Need help from Java-developers

Need help from Java-developers

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 2024 bytes --]

Hi,

as bootstrapping maven progresses (see my other post), the need for help
from Java-developers arises.

To finish all the work to be done after maven bootstrapping, some
experienced Java developers are needed. The following questions need to
be answered.

  * Designing the maven-build-system as follows:
      o How to maven commands map to our build phases? Should they map,
        should there be new ones?
      o Is there a need to clean up the pom-files prior to building,
        e.g. remove version numbers?
      o How to make the maven-build-system to never ever include other
        jar? Perhaps we need to post-process the generated jars.
      o How to handle pom-files (see below)

  * Which naming conventions should be used for packages? Maven has the
    notion of "group-id" and "atrtifact". Should this be kept? OTOH,
    there are very common packages like "commons-io" aka
    "apache-commons-io".

  * Where should the repo-files be kept in Guix? Debian seems to bot
    them into a dir-structure which I assume is leaned on some
    file-structure in the maven central repository. See
    <https://wiki.debian.org/Java/MavenRepoSpec>
    <https://wiki.debian.org/Java/MavenRepoHelper> and
    <https://packages.debian.org/jessie/maven-repo-helper>

  * Where to keep the pom-files? Are there other files we need (I've
    seen "effective pom", and "maven-fragments" in other distros)? Can
    or should we strip these files, like Debian seems to to with the
    maven-repo-helper? If so: What do we need? can this be done in guix,
    is there a maven-plugin, or …?

  * Help finding official sources, homepages, etc. for all packages. For
    many packages the data in the pom is outdated, since e.b.
    codehaus.org and code.google.com are gone.


Please comment!

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: 0xBF773B65.asc --]
[-- Type: application/pgp-keys, Size: 15087 bytes --]

Re: Need help from Java-developers

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 4334 bytes --]


Hi Hartmut,

I want to help, but I'm not sure how much help I can actually provide.
I'm familiar with Java, but I've only worked a little bit with Maven.
Please forgive me if I say anything that is inaccurate or naive!  You
probably know more about Maven than I do.

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Hi,
>
> as bootstrapping maven progresses (see my other post), the need for help
> from Java-developers arises.
>
> To finish all the work to be done after maven bootstrapping, some
> experienced Java developers are needed. The following questions need to
> be answered.
>
>   * Designing the maven-build-system as follows:
>       o How to maven commands map to our build phases? Should they map,
>         should there be new ones?

I'm not sure, but whatever seems simplest and works is probably a good
place to start.

>       o Is there a need to clean up the pom-files prior to building,
>         e.g. remove version numbers?

Why would that be necessary?  Will the presence of the version numbers
interfere with the build somehow?

>       o How to make the maven-build-system to never ever include other
>         jar? Perhaps we need to post-process the generated jars.

Speaking of JAR files, shouldn't we try to avoid them entirely?  My
understanding is that they are compressed files, which means the Guix
daemon won't be able to scan them for references.  I don't know if it's
easy to use Maven to build a project without putting the build output
into a JAR file, though.

>       o How to handle pom-files (see below)
>
>   * Which naming conventions should be used for packages? Maven has the
>     notion of "group-id" and "atrtifact". Should this be kept? OTOH,
>     there are very common packages like "commons-io" aka
>     "apache-commons-io".

Whatever is simplest to do is a good place to start.  I think the name
of the Guix package is what matters most, right?  So, maybe the Guix
package definition for apache-commons-io would be
"java-apache-commons-io", but internally within Maven it would use some
other name?  I think this is less important than getting it working in
the first place, so I have no strong opinions on this.

>
>   * Where should the repo-files be kept in Guix? Debian seems to bot
>     them into a dir-structure which I assume is leaned on some
>     file-structure in the maven central repository. See
>     <https://wiki.debian.org/Java/MavenRepoSpec>
>     <https://wiki.debian.org/Java/MavenRepoHelper> and
>     <https://packages.debian.org/jessie/maven-repo-helper>
>

Am I correct in understanding that the "repo-filess" are the files that
Maven uses for doing stuff like fulfilling the dependencies defined in
the POM file?  If so, then I guess we would need to pick an arbitrary
subdirectory to use for this purpose, and that is where all Maven-built
outputs will go.  Would that work?

Is it possible to create a "manifest of files" (e.g., an environment
variable that points to many different locations in the store) that we
can provide to the "maven-build-system"?  If we could do that, maybe we
could trick Maven into treating that collection of files (residing in
the store) as its repository.  It would be nice if we did not have to
propagate all of the dependencies into a single directory just to enable
Maven to discover them.

>
>   * Where to keep the pom-files? Are there other files we need (I've
>     seen "effective pom", and "maven-fragments" in other distros)? Can
>     or should we strip these files, like Debian seems to to with the
>     maven-repo-helper? If so: What do we need? can this be done in guix,
>     is there a maven-plugin, or …?
>

I don't know.  Does Maven expect the POM files for each project to
reside in a specific location?

>
>   * Help finding official sources, homepages, etc. for all
>packages. For
>     many packages the data in the pom is outdated, since e.b.
>     codehaus.org and code.google.com are gone.

For this, I'm sure it might be tough, but we will probably just have to
figure it out on a case-by-case basis.

I'm excited about the prospect of doing Java development with Guix.
Thank you for helping to figure this complicated stuff out.  I hope I
can help a little, too.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Need help from Java-developers

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]

Chris Marusich <cmmarusich@gmail.com> writes:

>>       o How to make the maven-build-system to never ever include other
>>         jar? Perhaps we need to post-process the generated jars.
>
> Speaking of JAR files, shouldn't we try to avoid them entirely?  My
> understanding is that they are compressed files, which means the Guix
> daemon won't be able to scan them for references.  I don't know if it's
> easy to use Maven to build a project without putting the build output
> into a JAR file, though.

For the record, I looked into this a little more.  I was mistaken: JAR
files are not necessarily compressed.  In fact, it seems [1][2] that it
should always be possible to make un-compressed JARs.  So, perhaps the
Guix daemon will not have trouble scanning JARs for references, after
all.  (Whether or not any references will actually be retained in the
JARs produced by Maven is another question; I don't know the answer.)

[1] "The special value 0 instructs the packer to copy through the
original JAR file directly, with no compression. The JSR 200 standard
requires any unpacker to understand this special case as a pass-through
of the entire archive."
https://docs.oracle.com/javase/8/docs/api/java/util/jar/Pack200.Packer.html#EFFORT

[2] "(Zero) Creates (c) or updates (u) the JAR file without using ZIP
compression.":
https://docs.oracle.com/javase/8/docs/technotes/tools/windows/jar.html

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Need help from Java-developers

[Ricardo Wurmus]

Chris Marusich <cmmarusich@gmail.com> writes:

>>       o How to make the maven-build-system to never ever include other
>>         jar? Perhaps we need to post-process the generated jars.
>
> Speaking of JAR files, shouldn't we try to avoid them entirely?  My
> understanding is that they are compressed files, which means the Guix
> daemon won't be able to scan them for references.  I don't know if it's
> easy to use Maven to build a project without putting the build output
> into a JAR file, though.

We are already using Jar files.  They are generated by the
ant-build-system.  I don’t think there’s any way around using them,
because that’s the format everybody uses.  You just add a jar to the
CLASSPATH to let java find it.

The format is like Zip, but with an additional manifest.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

JARs and reference scanning (was: Need help from Java-developers)

[Hartmut Goebel]

Am 24.04.2017 um 00:57 schrieb Chris Marusich:
> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> Speaking of JAR files, shouldn't we try to avoid them entirely?  My
>> understanding is that they are compressed files, which means the Guix
>> daemon won't be able to scan them for references.  I don't know if it's
>> easy to use Maven to build a project without putting the build output
>> into a JAR file, though.
> For the record, I looked into this a little more.  I was mistaken: JAR
> files are not necessarily compressed.  In fact, it seems [1][2] that it
> should always be possible to make un-compressed JARs.  So, perhaps the
> Guix daemon will not have trouble scanning JARs for references, after
> all.  (Whether or not any references will actually be retained in the
> JARs produced by Maven is another question; I don't know the answer.)

I have to admit that I do not know at all how the reference scanning and
dependency-tracking in the store works.

Regarding the jar-files ny understanding is:

- JAR-files are Zip-files with additional data (as Ricardo already stated)
- Zip-files can be used to store data without compressions (I assume
this is what you are refering to).
- My experience is that the contents of the JAR file can also be
*unpacked* into the file-system, so not archives would be needed. Some
Java guy might know better. I'm not sure it this is desired at all, though.
- My understanding is that Java normally does not have any reference
from one package (or jar-file) to another one. There is no such thing
like "rpath" but is more like Python or Perl where the garbage collector
AFAIK can not track references either.
- According to [3, 2] the MANIFEST.INF file *may* specify a Class-Path
containing the relative paths to other Jar-files. If this would help we
*could* add references here, but the entry-length is limited to 65353
bytes, so we might hit this limit with the long paths of the store.
- Fedora forbids to use this Class-Path entry in MANIFEST files [1].
- If it helps the garbage collector, we could add some ".dependencies"
file alongside each Java package. But we don't do this for Python or
Perl, either.


[1]
https://fedoraproject.org/wiki/Packaging:Java#No_class-path_in_MANIFEST.MF
[2] https://en.wikipedia.org/wiki/JAR_(file_format)#Dependencies
[3]
http://docs.oracle.com/javase/6/docs/technotes/guides/jar/jar.html#JAR%20Manifest


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: JARs and reference scanning

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3370 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 24.04.2017 um 00:57 schrieb Chris Marusich:
>> Chris Marusich <cmmarusich@gmail.com> writes:
>>
>>> Speaking of JAR files, shouldn't we try to avoid them entirely?  My
>>> understanding is that they are compressed files, which means the Guix
>>> daemon won't be able to scan them for references.  I don't know if it's
>>> easy to use Maven to build a project without putting the build output
>>> into a JAR file, though.
>> For the record, I looked into this a little more.  I was mistaken: JAR
>> files are not necessarily compressed.  In fact, it seems [1][2] that it
>> should always be possible to make un-compressed JARs.  So, perhaps the
>> Guix daemon will not have trouble scanning JARs for references, after
>> all.  (Whether or not any references will actually be retained in the
>> JARs produced by Maven is another question; I don't know the answer.)
>
> I have to admit that I do not know at all how the reference scanning and
> dependency-tracking in the store works.

As I understand it, the mechanism used by the Guix daemon (and the Nix
daemon) for scanning references doesn't work when the output of a
derivation is scrambled in some way (e.g. compressed).  Therefore, if we
use JAR files, they should not be compressed.

> Regarding the jar-files ny understanding is:
>
> - JAR-files are Zip-files with additional data (as Ricardo already stated)
> - Zip-files can be used to store data without compressions (I assume
> this is what you are refering to).
> - My experience is that the contents of the JAR file can also be
> *unpacked* into the file-system, so not archives would be needed. Some
> Java guy might know better. I'm not sure it this is desired at all, though.
> - My understanding is that Java normally does not have any reference
> from one package (or jar-file) to another one. There is no such thing
> like "rpath" but is more like Python or Perl where the garbage collector
> AFAIK can not track references either.

I don't think it's necessary to unpack them, as long as we can ensure
they are not compressed.

> - According to [3, 2] the MANIFEST.INF file *may* specify a Class-Path
> containing the relative paths to other Jar-files. If this would help
> we *could* add references here, but the entry-length is limited to
> 65353 bytes, so we might hit this limit with the long paths of the
> store.  - Fedora forbids to use this Class-Path entry in MANIFEST
> files [1].  - If it helps the garbage collector, we could add some
> ".dependencies" file alongside each Java package. But we don't do this
> for Python or Perl, either.

That's an interesting idea.  This would certainly make it clear which
Java packages depend on which other packages (e.g., for garbage
collection purposes), but this wouldn't by itself help Java (or Python,
or Perl, etc.) actually find the classes.  Some other stuff (propagation
of inputs like we do with Python libraries, a Guix-managed CLASSPATH, a
custom ClassLoader, etc.) would probably be needed to ensure that Java
would be able to find the classes.

Thank you for your work on this.  It's easy for me to just give
opinions, but you've actually done the real work.  If I can find more
time, I will try to look at what you've submitted to the list and help.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

store reference detection (was Re: JARs and reference scanning)

[Thomas Danckaert]

From: Chris Marusich <cmmarusich@gmail.com>
Subject: Re: JARs and reference scanning
Date: Tue, 25 Apr 2017 22:34:02 -0700

>> I have to admit that I do not know at all how the reference 
>> scanning and
>> dependency-tracking in the store works.
>
> As I understand it, the mechanism used by the Guix daemon (and the 
> Nix
> daemon) for scanning references doesn't work when the output of a
> derivation is scrambled in some way (e.g. compressed).  Therefore, 
> if we
> use JAR files, they should not be compressed.

The code scanning for reference is in nix/libstore/references.cc .  
It looks for base32 hashes encoded as character strings in the 
binaries.

Could/should this be generalized somehow?  Apart from compression, 
store filenames encoded with 16-bit character encodings also cause 
problems (can happen with Qt or WxWidgets).  And the are probably 
more cases where it fails.

Does it make sense to expand the reference detecting code (perhaps 
this would lead to too many different special cases?), or maybe 
provide a mechanism to force references when the daemon cannot detect 
them.  I suppose you can always add a text file with a list of store 
items to the output, but maybe there's a more elegant way?

Thomas

Re: store reference detection (was Re: JARs and reference scanning)

[Maxim Cournoyer]

On April 26, 2017 4:53:33 AM PDT, Thomas Danckaert <post@thomasdanckaert.be> wrote:
[...]
>The code scanning for reference is in nix/libstore/references.cc .  
>It looks for base32 hashes encoded as character strings in the 
>binaries.
>
>Could/should this be generalized somehow?  Apart from compression, 
>store filenames encoded with 16-bit character encodings also cause 
>problems (can happen with Qt or WxWidgets).  And the are probably 
>more cases where it fails.
>
>Does it make sense to expand the reference detecting code (perhaps 
>this would lead to too many different special cases?), or maybe 
>provide a mechanism to force references when the daemon cannot detect 
>them.  I suppose you can always add a text file with a list of store 
>items to the output, but maybe there's a more elegant way?
>
>Thomas

The text file idea with the dependencies that Hartmut proposed seems an elegant solution; it would make the reference searching process cheaper and available for any kind of format.

Re: store reference detection (was Re: JARs and reference scanning)

[Ludovic Courtès]

Hello,

Thomas Danckaert <post@thomasdanckaert.be> skribis:

> From: Chris Marusich <cmmarusich@gmail.com>
> Subject: Re: JARs and reference scanning
> Date: Tue, 25 Apr 2017 22:34:02 -0700
>
>>> I have to admit that I do not know at all how the reference
>>> scanning and
>>> dependency-tracking in the store works.
>>
>> As I understand it, the mechanism used by the Guix daemon (and the
>> Nix
>> daemon) for scanning references doesn't work when the output of a
>> derivation is scrambled in some way (e.g. compressed).  Therefore,
>> if we
>> use JAR files, they should not be compressed.
>
> The code scanning for reference is in nix/libstore/references.cc .  It
> looks for base32 hashes encoded as character strings in the binaries.
>
> Could/should this be generalized somehow?  Apart from compression,
> store filenames encoded with 16-bit character encodings also cause
> problems (can happen with Qt or WxWidgets).  And the are probably more
> cases where it fails.

Really? Qt/WxWidgets “hide” store references by default?

> Does it make sense to expand the reference detecting code (perhaps
> this would lead to too many different special cases?), or maybe
> provide a mechanism to force references when the daemon cannot detect
> them.  I suppose you can always add a text file with a list of store
> items to the output, but maybe there's a more elegant way?

‘propagated-inputs’ is one way to manually specify run-time references.
It works at the package level and not at the store level—that is, the
store item’s references are unaffected by what ‘propagated-inputs’
contains.  It’s usually enough for our purposes though.

In that sense, I think jars are comparable to Python/Perl libraries,
which do not carry reference information by themselves and thus need a
manually-provided ‘propagated-inputs’.

WDYT?

Ludo’.

Re: store reference detection

[Thomas Danckaert]

From: ludo@gnu.org (Ludovic Courtès)
Subject: Re: store reference detection (was Re: JARs and reference 
scanning)
Date: Thu, 27 Apr 2017 15:46:59 +0200

> Really? Qt/WxWidgets “hide” store references by default?

Not by default.  But there are cases were I've been bitten:

  - Qt has a “QStringLiteral”, which transforms your character literal
    to a UTF-16-encoded string at compile-time.  For example the
    kdbusaddons source used such a literal to encode the kdeinit5
    executable.  (In this case I patched the code, replacing it by a
    regular string.)

  - Similarly, when I packaged DVDStyler (a wxWidgets program), I
    noticed it also uses wchar_t strings to encode the name of
    executables.  (Here I used wrap-program to set the PATH, instead 
of
    embedding the full store location of the executables)

>> Does it make sense to expand the reference detecting code (perhaps
>> this would lead to too many different special cases?), or maybe
>> provide a mechanism to force references when the daemon cannot 
>> detect
>> them.  I suppose you can always add a text file with a list of 
>> store
>> items to the output, but maybe there's a more elegant way?
>
> ‘propagated-inputs’ is one way to manually specify run-time 
> references.
> It works at the package level and not at the store level—that is, 
> the
> store item’s references are unaffected by what ‘propagated-inputs’
> contains.  It’s usually enough for our purposes though.
>
> In that sense, I think jars are comparable to Python/Perl libraries,
> which do not carry reference information by themselves and thus 
> need a
> manually-provided ‘propagated-inputs’.

I think it's nice to avoid propagated inputs if possible, because 
they pollute the profile with things the user didn't strictly as for. 
 So Hartmuts proposed workaround (embed dependencies in the .jar's, 
if it's really possible, and use a .deps text file in the store to 
enable reference detection) sounds better.

Thomas

Re: store reference detection (was Re: JARs and reference scanning)

[Hartmut Goebel]

Am 27.04.2017 um 15:46 schrieb Ludovic Courtès:
> ‘propagated-inputs’ is one way to manually specify run-time references.
> It works at the package level and not at the store level—that is, the
> store item’s references are unaffected by what ‘propagated-inputs’
> contains.  It’s usually enough for our purposes though.

I'm not sure if 'propagated-inputs' are enough. For example
"python-passlib" as propagated-input python-py-bcrypt, but the later
does not show up as reference, requisite nor referrer:

$ ./pre-inst-env guix build --fallback python-passlib
[…]
/gnu/store/n9h6wr1ydgz2a2az04jpswmnl756x48r-python-passlib-1.7.1
$ ./pre-inst-env guix gc -R
/gnu/store/n9h6wr1ydgz2a2az04jpswmnl756x48r-python-passlib-1.7.1
/gnu/store/n9h6wr1ydgz2a2az04jpswmnl756x48r-python-passlib-1.7.1
$ ./pre-inst-env guix gc --references
/gnu/store/n9h6wr1ydgz2a2az04jpswmnl756x48r-python-passlib-1.7.1
/gnu/store/n9h6wr1ydgz2a2az04jpswmnl756x48r-python-passlib-1.7.1
$ ./pre-inst-env guix gc --referrers
/gnu/store/n9h6wr1ydgz2a2az04jpswmnl756x48r-python-passlib-1.7.1
/gnu/store/n9h6wr1ydgz2a2az04jpswmnl756x48r-python-passlib-1.7.1


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: store reference detection (was Re: JARs and reference scanning)

[Ludovic Courtès]

Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:

> Am 27.04.2017 um 15:46 schrieb Ludovic Courtès:
>> ‘propagated-inputs’ is one way to manually specify run-time references.
>> It works at the package level and not at the store level—that is, the
>> store item’s references are unaffected by what ‘propagated-inputs’
>> contains.  It’s usually enough for our purposes though.
>
> I'm not sure if 'propagated-inputs' are enough. For example
> "python-passlib" as propagated-input python-py-bcrypt, but the later
> does not show up as reference, requisite nor referrer:

Right, that’s what I meant by “not at the store level” above.

Ludo’.

Re: store reference detection (was Re: JARs and reference scanning)

[Hartmut Goebel]

Am 02.05.2017 um 14:43 schrieb Ludovic Courtès:
> Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:
>
>> Am 27.04.2017 um 15:46 schrieb Ludovic Courtès:
>>> ‘propagated-inputs’ is one way to manually specify run-time references.
>>> It works at the package level and not at the store level—that is, the
>>> store item’s references are unaffected by what ‘propagated-inputs’
>>> contains.  It’s usually enough for our purposes though.
>> I'm not sure if 'propagated-inputs' are enough. For example
>> "python-passlib" as propagated-input python-py-bcrypt, but the later
>> does not show up as reference, requisite nor referrer:
> Right, that’s what I meant by “not at the store level” above.
>
> Ludo’.
 So I propose to add a small text file ".guix-dependencies' to all
language's packages which do not add some kind of references themselves:
Python, Perl, Java, etc.

Question: How can the builder access the "propagated" inputs only?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: store reference detection (was Re: JARs and reference scanning)

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 3742 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 02.05.2017 um 14:43 schrieb Ludovic Courtès:
>> Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:
>>
>>> Am 27.04.2017 um 15:46 schrieb Ludovic Courtès:
>>>> ‘propagated-inputs’ is one way to manually specify run-time references.
>>>> It works at the package level and not at the store level—that is, the
>>>> store item’s references are unaffected by what ‘propagated-inputs’
>>>> contains.  It’s usually enough for our purposes though.
>>> I'm not sure if 'propagated-inputs' are enough. For example
>>> "python-passlib" as propagated-input python-py-bcrypt, but the later
>>> does not show up as reference, requisite nor referrer:
>> Right, that’s what I meant by “not at the store level” above.
>>
>> Ludo’.
>  So I propose to add a small text file ".guix-dependencies' to all
> language's packages which do not add some kind of references themselves:
> Python, Perl, Java, etc.

What is the goal of doing this?  If the goal is simply to make it so
that the daemon will know what references are live, I'm not sure it's
necessary, since the "propagated inputs" mechanism already accomplishes
that goal.

If the goal is to enable interpreters like Python, Perl, Java, etc. to
discover their program's dependencies (e.g., so Java can import
classes), then simply adding a .guix-dependencies file is not sufficient
by itself.  We would also need "something else" that (using the
.guix-dependencies file) arranges for the relevant interpreter to
discover those specified dependencies.  As far as I know, no design for
that "something else" has been proposed.

Currently, our options are limited because we are relying on the default
import mechanism for our interpreters.  For example, in the case of
Python, I believe we are using "propagated inputs" so that the default
import mechanism finds the modules.  (Please correct me if I'm
mistaken.)  If we were to modify the import mechanism, it might not be
necessary to use propagated inputs at all.

There are probably many ways to accomplish that.  For example, in Java,
perhaps we could implement a custom classloader.  Or maybe we could
patch the built-in class loading mechanism [1].  Similarly, in Python
there are also ways to customize the import mechanism [2].  Perhaps Perl
also has a similar mechanism?

This kind of low-level fiddling might sound drastic, but it isn't
without precedent.  Nix has already demonstrated that customizing the
dynamic linker [3] and the ELF executable files [4] is not a crazy idea.
In fact, Nix and Guix wouldn't work without such customizations.  So,
maybe it's time we thought about "fiddling" with the way these
interpreters (Java, Python, Perl, etc.) find their dependencies, too.

[1] These classes appear to be involved in the default loading process,
so they would be the place to start looking, I think:

  http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/misc/Launcher.java#l259
  http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/net/URLClassLoader.java#l356
  http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/misc/URLClassPath.java#l62

[2] https://docs.python.org/3/reference/import.html#importsystem

[3]
https://sandervanderburg.blogspot.co.uk/2015/10/deploying-prebuilt-binary-software-with.html

[4] https://github.com/NixOS/patchelf

>
> Question: How can the builder access the "propagated" inputs only?

Can you clarify what you mean?  I don't understand the question, and I
don't want to assume you meant one thing when you actually meant
something different.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: store reference detection (was Re: JARs and reference scanning)

[Ricardo Wurmus]

Chris Marusich <cmmarusich@gmail.com> writes:

> There are probably many ways to accomplish that.  For example, in Java,
> perhaps we could implement a custom classloader.  Or maybe we could
> patch the built-in class loading mechanism [1].  Similarly, in Python
> there are also ways to customize the import mechanism [2].  Perhaps Perl
> also has a similar mechanism?

Jar files can be told to import classes from another Jar by adding it to
the “Class-Path” field of the Jar’s manifest.

Here’s an example:
https://docs.oracle.com/javase/tutorial/deployment/jar/downman.html

I don’t know if this mechanism permits the use of absolute paths, but
it’s worth a try.  I should note that this doesn’t help with store
reference detection, because the manifest will be part of the compressed
Jar.  But if this works we could avoid propagation for Java packages.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: store reference detection (was Re: JARs and reference scanning)

[Ludovic Courtès]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> There are probably many ways to accomplish that.  For example, in Java,
>> perhaps we could implement a custom classloader.  Or maybe we could
>> patch the built-in class loading mechanism [1].  Similarly, in Python
>> there are also ways to customize the import mechanism [2].  Perhaps Perl
>> also has a similar mechanism?
>
> Jar files can be told to import classes from another Jar by adding it to
> the “Class-Path” field of the Jar’s manifest.

Likewise, Guile 2.2 uses ELF for its .go files.  It does not support
RUNPATH yet, but we could add support for it I suppose.

Ludo’.

Re: store reference detection (was Re: JARs and reference scanning)

[Chris Marusich]

[-- Attachment #1.1: Type: text/plain, Size: 3031 bytes --]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>> There are probably many ways to accomplish that.  For example, in Java,
>> perhaps we could implement a custom classloader.  Or maybe we could
>> patch the built-in class loading mechanism [1].  Similarly, in Python
>> there are also ways to customize the import mechanism [2].  Perhaps Perl
>> also has a similar mechanism?
>
> Jar files can be told to import classes from another Jar by adding it to
> the “Class-Path” field of the Jar’s manifest.
>
> Here’s an example:
> https://docs.oracle.com/javase/tutorial/deployment/jar/downman.html

I didn't know this!  That's awesome; it might be just what we need.

> I don’t know if this mechanism permits the use of absolute paths, but
> it’s worth a try.  I should note that this doesn’t help with store
> reference detection, because the manifest will be part of the compressed
> Jar.  But if this works we could avoid propagation for Java packages.

Preliminary testing shows that this does in fact work.  See the attached
tarball; when you run "make" on its extracted contents, you can verify
for yourself that it works (don't forget to do 'guix -i icedtea:jdk'):

--8<---------------cut here---------------start------------->8---
[0] marusich@garuda:~
$ java -cp /tmp/embed-absolute-classpath-in-jar/Main.jar Main
Hello world!
[0] marusich@garuda:~
$ jar -tvf /tmp/embed-absolute-classpath-in-jar/Main.jar
     0 Thu May 11 01:35:26 PDT 2017 META-INF/
   131 Thu May 11 01:35:26 PDT 2017 META-INF/MANIFEST.MF
   309 Thu May 11 01:35:26 PDT 2017 Main.class
[0] marusich@garuda:~
$ jar -tvf /tmp/embed-absolute-classpath-in-jar/Greeter.jar
     0 Thu May 11 01:35:28 PDT 2017 META-INF/
    69 Thu May 11 01:35:28 PDT 2017 META-INF/MANIFEST.MF
   396 Thu May 11 01:35:26 PDT 2017 Greeter.class
[0] marusich@garuda:~
$ grep /tmp/embed-absolute-classpath-in-jar /tmp/embed-absolute-classpath-in-jar/*
Binary file /tmp/embed-absolute-classpath-in-jar/Main.jar matches
/tmp/embed-absolute-classpath-in-jar/Main.txt:Class-Path: /tmp/embed-absolute-classpath-in-jar/Greeter.jar
[0] marusich@garuda:~
$ strings /tmp/embed-absolute-classpath-in-jar/Main.jar | grep /tmp/embed-absolute-classpath-in-jar
Class-Path: /tmp/embed-absolute-classpath-in-jar/Greeter.jar
[0] marusich@garuda:~
$ 
--8<---------------cut here---------------end--------------->8---

Based on this test, it looks like we can embed absolute paths in
uncompressed JAR files.  This is great!  It means that if we can package
Java libraries in this way, we won't need to propagate inputs for Java
libraries (because Java will find the classes via the embedded
Class-Path values), and the usual reference scanning mechanism will just
work (because the JAR files are not compressed).

I don't know what limits there are on the number of entries one can put
into the Class-Path line, but hopefully that won't be an issue.

-- 
Chris

[-- Attachment #1.2: embed-absolute-classpath-in-jar.tar.xz --]
[-- Type: application/octet-stream, Size: 1228 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: store reference detection (was Re: JARs and reference scanning)

[Ricardo Wurmus]

Chris Marusich <cmmarusich@gmail.com> writes:

>> Jar files can be told to import classes from another Jar by adding it to
>> the “Class-Path” field of the Jar’s manifest.
>>
>> Here’s an example:
>> https://docs.oracle.com/javase/tutorial/deployment/jar/downman.html
>
> I didn't know this!  That's awesome; it might be just what we need.
[…]

Thanks for testing this!

One limitation appears to be that this only works for applications, not
for libraries.  This could be a problem for us.  We don’t really need
this urgently for applications if we automatically generate shell
wrappers (as we do for Python executables).

It would be interesting to know if this could be used for libraries as
well, so that the application in the end does not need to know about all
transitive dependencies, but only its first-level dependencies.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: store reference detection (was Re: JARs and reference scanning)

[Mark H Weaver]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 02.05.2017 um 14:43 schrieb Ludovic Courtès:
>> Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:
>>
>>> Am 27.04.2017 um 15:46 schrieb Ludovic Courtès:
>>>> ‘propagated-inputs’ is one way to manually specify run-time references.
>>>> It works at the package level and not at the store level—that is, the
>>>> store item’s references are unaffected by what ‘propagated-inputs’
>>>> contains.  It’s usually enough for our purposes though.
>>> I'm not sure if 'propagated-inputs' are enough. For example
>>> "python-passlib" as propagated-input python-py-bcrypt, but the later
>>> does not show up as reference, requisite nor referrer:
>> Right, that’s what I meant by “not at the store level” above.
>>
>> Ludo’.
>  So I propose to add a small text file ".guix-dependencies' to all
> language's packages which do not add some kind of references themselves:
> Python, Perl, Java, etc.

I have thought of doing this in the past, but there's another more
difficult problem that would also need to be solved: how to make
grafting work for these non-plaintext references.  If grafting doesn't
work, there's a good chance that software with known security flaws will
continue to be executed.

       Mark

Re: store reference detection (was Re: JARs and reference scanning)

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2151 bytes --]

Ricardo Wurmus <rekado@elephly.net> writes:

> Chris Marusich <cmmarusich@gmail.com> writes:
>
>>> Jar files can be told to import classes from another Jar by adding it to
>>> the “Class-Path” field of the Jar’s manifest.
>>>
>>> Here’s an example:
>>> https://docs.oracle.com/javase/tutorial/deployment/jar/downman.html
>>
>> I didn't know this!  That's awesome; it might be just what we need.
> […]
>
> Thanks for testing this!
>
> One limitation appears to be that this only works for applications, not
> for libraries.

In what way does this not work for libraries?  I'm not criticizing you;
I'm genuinely curious.  To ask this question another way: how would a
solution that "works for libraries" behave, exactly?  I'm not sure what
the phrase "it works for libraries" might mean, since I suspect its
meaning varies depending on what one is trying to accomplish.

> This could be a problem for us.  We don’t really need this urgently
> for applications if we automatically generate shell wrappers (as we do
> for Python executables).
>

I agree.

> It would be interesting to know if this could be used for libraries as
> well, so that the application in the end does not need to know about all
> transitive dependencies, but only its first-level dependencies.

I think there are at least two concrete goals here.  I would phrase them
as follows:

1) I can run Java applications (built with Guix).

2) I can use Java libraries (built with Guix) in an IDE to develop Java
   applications.

We can accomplish (1) by using wrapper scripts.  I suppose we could also
maybe accomplish (1) by using JARs with embedded classpaths, but as long
as wrapper scripts are sufficient, it isn't really necessary to do this.
As for (2), I think it's probably trickier, since the exact way in which
an IDE might want to be informed about the dependencies of a project may
vary.  I'm still not sure how I would develop Java applications using
Java libraries built with Guix, without putting in a lot of manual
effort to tell my IDE where the dependencies live.  Surely there is a
way...

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: store reference detection (was Re: JARs and reference scanning)

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1986 bytes --]

Mark H Weaver <mhw@netris.org> writes:

> Hartmut Goebel <h.goebel@crazy-compilers.com> writes:
>
>> Am 02.05.2017 um 14:43 schrieb Ludovic Courtès:
>>> Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:
>>>
>>>> Am 27.04.2017 um 15:46 schrieb Ludovic Courtès:
>>>>> ‘propagated-inputs’ is one way to manually specify run-time references.
>>>>> It works at the package level and not at the store level—that is, the
>>>>> store item’s references are unaffected by what ‘propagated-inputs’
>>>>> contains.  It’s usually enough for our purposes though.
>>>> I'm not sure if 'propagated-inputs' are enough. For example
>>>> "python-passlib" as propagated-input python-py-bcrypt, but the later
>>>> does not show up as reference, requisite nor referrer:
>>> Right, that’s what I meant by “not at the store level” above.
>>>
>>> Ludo’.
>>  So I propose to add a small text file ".guix-dependencies' to all
>> language's packages which do not add some kind of references themselves:
>> Python, Perl, Java, etc.
>
> I have thought of doing this in the past, but there's another more
> difficult problem that would also need to be solved: how to make
> grafting work for these non-plaintext references.  If grafting doesn't
> work, there's a good chance that software with known security flaws will
> continue to be executed.

That's a good thing to keep in mind.  I think that the references we're
talking about putting into a ".guix-dependencies" file or into an
uncompressed JAR file are in fact "plaintext" in the sense that these
files are not using compression, encryption, esoteric encodings, or
other obfuscations which might defeat the reference scanning or grafting
mechanisms.  I'm not convinced we need these things (a list of
dependencies in ".guix-dependencies" or embedded classpaths in JAR
files), but if we used them, I don't think it would interfere with
reference scanning or grafting.  Would it?

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: store reference detection (was Re: JARs and reference scanning)

[Ricardo Wurmus]

Chris Marusich <cmmarusich@gmail.com> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> Chris Marusich <cmmarusich@gmail.com> writes:
>>
>>>> Jar files can be told to import classes from another Jar by adding it to
>>>> the “Class-Path” field of the Jar’s manifest.
>>>>
>>>> Here’s an example:
>>>> https://docs.oracle.com/javase/tutorial/deployment/jar/downman.html
>>>
>>> I didn't know this!  That's awesome; it might be just what we need.
>> […]
>>
>> Thanks for testing this!
>>
>> One limitation appears to be that this only works for applications, not
>> for libraries.
>
> In what way does this not work for libraries?  I'm not criticizing you;
> I'm genuinely curious.  To ask this question another way: how would a
> solution that "works for libraries" behave, exactly?  I'm not sure what
> the phrase "it works for libraries" might mean, since I suspect its
> meaning varies depending on what one is trying to accomplish.

What I mean is that it may not be transitive.  Let’s say “java-foo”
depends on “java-hamcrest-core”, so we record the absolute path to the
hamcrest core jar in the Class-Path field of the manifest.

Now let’s also say that the application “java-bar” depends on
“java-foo”, but not directly on “java-hamcrest-core”.  Now what I wonder
about is whether we would need to record the paths to the jars of both
“java-foo” *and* “java-hamcrest-core” in the Class-Path field of the
manifest for “java-bar”, or if it would be sufficient to record
“java-foo".

Would Java respect the Class-Path for all dependencies recursively
within their scope?  Is this ‘lexically scoped’ (i.e. the specified
class path will apply only for the jar on whose manifest it was
specified) or will Java traverse all Class-Path fields and create one
concatenated big class path that is used to resolve all classes?

In the latter case there could be conflicts when one node in the graph
demands *one* version of a jar and some other node demands the classes
from a *different* jar to be used.

Maybe none of this is implemented and all Class-Path does is tell the
current *application* where to get its dependencies; in that case the
Class-Path field would just be an alternative to using a shell wrapper
that sets the CLASSPATH environment variable.

Does this make it a little clearer?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: store reference detection (was Re: JARs and reference scanning)

[Hartmut Goebel]

Am 11.05.2017 um 10:41 schrieb Chris Marusich:
> Based on this test, it looks like we can embed absolute paths in
> uncompressed JAR files.

Only the MANIFEST within the JAR file needs to be uncompressed, the
remaining files can be compressed.

JARs are zip files, which include data compressed individually, thus the
above could be achieved.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: store reference detection

[Hartmut Goebel]

[-- Attachment #1.1: Type: text/plain, Size: 606 bytes --]

Am 12.05.2017 um 10:19 schrieb Chris Marusich:
> I'm not convinced we need these things (a list of
> dependencies in ".guix-dependencies" or embedded classpaths in JAR
> files),

I'm convinced that we need such a beast, since references are not kept
at all for quite some kind of packages. See
<http://lists.gnu.org/archive/html/guix-devel/2017-04/msg00639.html> for
an example with *not* working references.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: store reference detection (was Re: JARs and reference scanning)

[Mark H Weaver]

Chris Marusich <cmmarusich@gmail.com> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Hartmut Goebel <h.goebel@crazy-compilers.com> writes:
>>
>>> Am 02.05.2017 um 14:43 schrieb Ludovic Courtès:
>>>> Hartmut Goebel <h.goebel@crazy-compilers.com> skribis:
>>>>
>>>>> Am 27.04.2017 um 15:46 schrieb Ludovic Courtès:
>>>>>> ‘propagated-inputs’ is one way to manually specify run-time references.
>>>>>> It works at the package level and not at the store level—that is, the
>>>>>> store item’s references are unaffected by what ‘propagated-inputs’
>>>>>> contains.  It’s usually enough for our purposes though.
>>>>> I'm not sure if 'propagated-inputs' are enough. For example
>>>>> "python-passlib" as propagated-input python-py-bcrypt, but the later
>>>>> does not show up as reference, requisite nor referrer:
>>>> Right, that’s what I meant by “not at the store level” above.
>>>>
>>>> Ludo’.
>>>  So I propose to add a small text file ".guix-dependencies' to all
>>> language's packages which do not add some kind of references themselves:
>>> Python, Perl, Java, etc.
>>
>> I have thought of doing this in the past, but there's another more
>> difficult problem that would also need to be solved: how to make
>> grafting work for these non-plaintext references.  If grafting doesn't
>> work, there's a good chance that software with known security flaws will
>> continue to be executed.
>
> That's a good thing to keep in mind.  I think that the references we're
> talking about putting into a ".guix-dependencies" file or into an
> uncompressed JAR file are in fact "plaintext" in the sense that these
> files are not using compression, encryption, esoteric encodings, or
> other obfuscations which might defeat the reference scanning or grafting
> mechanisms.  I'm not convinced we need these things (a list of
> dependencies in ".guix-dependencies" or embedded classpaths in JAR
> files), but if we used them, I don't think it would interfere with
> reference scanning or grafting.  Would it?

It would not interfere, but it could have the effect of *hiding*
security problems due to a failure to graft properly.

If there are embedded references that Guix cannot see and therefore are
not transformed by grafting, we are most likely to become aware of this
problem when the referenced store item is reclaimed by the garbage
collector, thus breaking things and prompting us to investigate.  This
is in fact exactly how I discovered another problem of this kind:

  https://bugs.gnu.org/24703

If we create a redundant set of references in another file, then
problems like this could go undetected for a long time.

Adding more 'propagating-inputs' would have the same problem, and cause
other problems as well.

One thing we could do is add a phase to certain build systems, which
would try to use additional methods to detect these hidden references,
and deliberately cause the build to fail in that case, raising the alarm
that grafting would silently fail.

What do you think?

      Mark

Re: store reference detection (was Re: JARs and reference scanning)

[Mark H Weaver]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 11.05.2017 um 10:41 schrieb Chris Marusich:
>> Based on this test, it looks like we can embed absolute paths in
>> uncompressed JAR files.
>
> Only the MANIFEST within the JAR file needs to be uncompressed, the
> remaining files can be compressed.
>
> JARs are zip files, which include data compressed individually, thus the
> above could be achieved.

If we leave the remaining files compressed, what will this mean for
grafting?  Might the compressed portions contain store references that
will fail to be grafted?

       Mark

Re: store reference detection (was Re: JARs and reference scanning)

[Leo Famulari]

On Fri, May 12, 2017 at 01:39:18PM -0400, Mark H Weaver wrote:
> One thing we could do is add a phase to certain build systems, which
> would try to use additional methods to detect these hidden references,
> and deliberately cause the build to fail in that case, raising the alarm
> that grafting would silently fail.

It's a good idea, and worth doing. I do think we will always find new
types of obfuscated references, unfortunately, but we should try to
address the types we know about.

Re: store reference detection (was Re: JARs and reference scanning)

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

Am 12.05.2017 um 19:39 schrieb Mark H Weaver:
> It would not interfere, but it could have the effect of *hiding*
> security problems due to a failure to graft properly.
> [...]
> If we create a redundant set of references in another file, then
> problems like this could go undetected for a long time.

Reading you comments (and words like "hidden"), I assume you are
referring to some compressed or otherwise unreadable data.

Please don't confuse this: We are *not* talking about compressed files,
but about plain text (or stored uncomressed within e.g. a zip-file).

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 1410 bytes --]

Re: store reference detection (was Re: JARs and reference scanning)

[Hartmut Goebel]

Am 12.05.2017 um 20:22 schrieb Mark H Weaver:
> Might the compressed portions contain store references that
> will fail to be grafted?

Class files per se do not contain references to any JAR file AFAIK. For
all other files (resources, etc.) its the same as for other programming
languages

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: store reference detection (was Re: JARs and reference scanning)

[Mark H Weaver]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 12.05.2017 um 20:22 schrieb Mark H Weaver:
>> Might the compressed portions contain store references that
>> will fail to be grafted?
>
> Class files per se do not contain references to any JAR file AFAIK. For
> all other files (resources, etc.) its the same as for other programming
> languages

Is it possible that class files might contain string literals whose
values are determined at compile time?  I don't know how things are done
in the Java world, but in C and C++ and I guess many other languages, it
is common for store references to become embedded in the source code
before compilation.

     Mark

Re: store reference detection (was Re: JARs and reference scanning)

[Mark H Weaver]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 12.05.2017 um 19:39 schrieb Mark H Weaver:
>
>  It would not interfere, but it could have the effect of *hiding*
> security problems due to a failure to graft properly.
> [...]
> If we create a redundant set of references in another file, then
> problems like this could go undetected for a long time.
>
> Reading you comments (and words like "hidden"), I assume you are
> referring to some compressed or otherwise unreadable data.
>
> Please don't confuse this: We are *not* talking about compressed
> files, but about plain text (or stored uncomressed within e.g. a
> zip-file).

Apologies if I've misunderstood.  Earlier, you wrote:

>  So I propose to add a small text file ".guix-dependencies' to all
> language's packages which do not add some kind of references
> themselves: Python, Perl, Java, etc.

What's the motivation for this proposal, if not to allow the scanner to
see references that would otherwise be obfuscated?

      Mark

Re: store reference detection (was Re: JARs and reference scanning)

[Hartmut Goebel]

Am 12.05.2017 um 23:51 schrieb Mark H Weaver:
> What's the motivation for this proposal, if not to allow the scanner to
> see references that would otherwise be obfuscated?
The motivation is to have references at all. See
<http://lists.gnu.org/archive/html/guix-devel/2017-04/msg00639.html> for
an example of a package having propagated inputs which are not
recognized as references by the gc.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: store reference detection (was Re: JARs and reference scanning)

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1880 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 12.05.2017 um 23:51 schrieb Mark H Weaver:
>> What's the motivation for this proposal, if not to allow the scanner to
>> see references that would otherwise be obfuscated?
> The motivation is to have references at all. See
> <http://lists.gnu.org/archive/html/guix-devel/2017-04/msg00639.html> for
> an example of a package having propagated inputs which are not
> recognized as references by the gc.

Hartmut, the question of whether or not a "package" (actually, the store
paths created by building the package) is eligible for garbage
collection has little to do with whether or not the package is a
propagated input.  With that in mind, the example you provided does not
seem to be a problem.

If a package A happens to be a propagated input of another package B,
and if package B happens to be installed in some profile (e.g., via
'guix package -i B'), then package A is also installed in the same
profile.  Therefore, package A is not eligible for garbage collection
(i.e., it is "live") because every profile is a GC root.  However, if
one has just built package B with 'guix build B' without installing it
into any profile, then package A hasn't been installed, either.  In this
case, package A is eligible for garbage collection because no GC root
refers to it directly or transitively [1].

To summarize: When package A is a propagated input of package B, it is
possible for package A to be live, and it is possible for package A not
to be live.  By itself, the fact that package A is a propagated input
tells us nothing about whether or not package A should be eligible for
garbage collection.  And it is not the case that package A should be
live simply because package B has been built.

[1] This assumes, of course, that package A wasn't live to begin with.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]