bug#31719: icedtea-3 binaries contain references to icedtea-2

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Ricardo Wurmus]

Many binaries of the icedtea-3 package contain references to icedtea-2,
which was used to build it.  Because of these references users of
icedtea-3 have to *also* download icedtea-2.  And because icedtea-2
contains references to icedtea-1, they *also* need to download that
version.

It is possible that icedtea-1 contains references to the bootstrap JDK,
which will then also have to be downloaded.

We should figure out a way to remove all of these references.

-- 
Ricardo

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 301 bytes --]

Do you think we can do something like rebuild icedtea3 with icedtea3, and
then rewrite the references? I guess this way the dependency loop could be
broken, as we could detach the last icedtea3 build from the packages used
for bootstrapping.
We could even make on more rebuilding, if necessary. WDYT?

[-- Attachment #2: Type: text/html, Size: 348 bytes --]

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Ricardo Wurmus]

Gábor Boskovits <boskovits@gmail.com> writes:

> Do you think we can do something like rebuild icedtea3 with icedtea3, and
> then rewrite the references? I guess this way the dependency loop could be
> broken, as we could detach the last icedtea3 build from the packages used
> for bootstrapping.
> We could even make on more rebuilding, if necessary. WDYT?

Yes, that’s an option, but I’m not sure it is the best option.  Icedtea
builds already take a very long time.  Rebuilding the JDK after the
“build” phase would almost double the build time.

I would be happier if we could prevent references to the bootstrap JDK
from ending up in the binaries.  Before we can do that we need to
understand why they are there.  Are they necessary or are they just
accidental?

--
Ricardo

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

On Tue, Jun 05, 2018 at 01:49:12PM +0200, Gábor Boskovits wrote:
> Do you think we can do something like rebuild icedtea3 with icedtea3, and
> then rewrite the references? I guess this way the dependency loop could be
> broken, as we could detach the last icedtea3 build from the packages used
> for bootstrapping.

If the references are not important (that is, if they are not used), we
could also rewrite them to something that doesn't exist. Currently the
go-build-system does this so that Go executables don't refer to the Go
compiler, which is very large.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

2018-06-05 14:45 GMT+02:00 Leo Famulari <leo@famulari.name>:

> On Tue, Jun 05, 2018 at 01:49:12PM +0200, Gábor Boskovits wrote:
> > Do you think we can do something like rebuild icedtea3 with icedtea3, and
> > then rewrite the references? I guess this way the dependency loop could
> be
> > broken, as we could detach the last icedtea3 build from the packages used
> > for bootstrapping.
>
> If the references are not important (that is, if they are not used), we
> could also rewrite them to something that doesn't exist. Currently the
> go-build-system does this so that Go executables don't refer to the Go
> compiler, which is very large.
>

Ok, I will have a look at his, and try to isolate these references. This
looks like a
good first step to evaluate what can be done next.

[-- Attachment #2: Type: text/html, Size: 1238 bytes --]

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 127 bytes --]

It seems that this is caused by having the files in a directory that is not
stripped.
I will try to have another look at that.

[-- Attachment #2: Type: text/html, Size: 158 bytes --]

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Ricardo Wurmus]

This seems to affect the openjdk packages as well, so a user of OpenJDK
12 will have to download *all* JDKs.

Gábor, have you been able to identify locations that retain references
to the build JDK?

-- 
Ricardo

bug#31719: Chains of dependencies getting longer

[Andreas Enge]

Hello,

trying a "guix build openjdk", I see this:
2136.0 MB would be downloaded:
   /gnu/store/243algr6h60j46spn5dqhjc4mhkd0a0p-libelf-0.8.13
   /gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
   /gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
   /gnu/store/52bp22jrvwfqa5fmlv2y5vxcir74m8jb-openjdk-10.46-jdk
   /gnu/store/bp86qd2vgaqahqam3n89xl1wz02g0srs-openjdk-10.46
   /gnu/store/a8f5ksk3gb4i1wrn7m5fm5ijc0zwwg2w-openjdk-11.28-jdk
   /gnu/store/9ni61z4nhvkrxbzj6gwip8zrb7zbkx6l-openjdk-11.28
   /gnu/store/837pl2vmq9haiw3kmsnh72fkwvcljipk-openjdk-12.33-jdk
   /gnu/store/ll5dgf42x9smclrc1mx4gyk3dl683m77-openjdk-13.0-jdk
   /gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
   /gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
   /gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0

So the problem is getting worse and worse over time, which makes a solution
more important.

Andreas

bug#31719: Chains of dependencies getting longer

[Ludovic Courtès]

Hi,

Andreas Enge <andreas@enge.fr> skribis:

> trying a "guix build openjdk", I see this:
> 2136.0 MB would be downloaded:
>    /gnu/store/243algr6h60j46spn5dqhjc4mhkd0a0p-libelf-0.8.13
>    /gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
>    /gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
>    /gnu/store/52bp22jrvwfqa5fmlv2y5vxcir74m8jb-openjdk-10.46-jdk
>    /gnu/store/bp86qd2vgaqahqam3n89xl1wz02g0srs-openjdk-10.46
>    /gnu/store/a8f5ksk3gb4i1wrn7m5fm5ijc0zwwg2w-openjdk-11.28-jdk
>    /gnu/store/9ni61z4nhvkrxbzj6gwip8zrb7zbkx6l-openjdk-11.28
>    /gnu/store/837pl2vmq9haiw3kmsnh72fkwvcljipk-openjdk-12.33-jdk
>    /gnu/store/ll5dgf42x9smclrc1mx4gyk3dl683m77-openjdk-13.0-jdk
>    /gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
>    /gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
>    /gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0

Weird.  I see this:

--8<---------------cut here---------------start------------->8---
$ guix build openjdk -n
369.8 MB would be downloaded:
   /gnu/store/676kcvxnnjfcy960fy4xkr10fbn3ciay-openjdk-14.0-doc
   /gnu/store/dk8rkrfkdgsmy1a33n0sc39fhqggy1nh-openjdk-14.0-jdk
   /gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0
$ guix build openjdk@13 -n
358.0 MB would be downloaded:
   /gnu/store/hipy0g8zpnicqcwhx5ygx9fmmkbgbw6w-openjdk-13.0-doc
   /gnu/store/ff9nn7ipbpnrrvsg0djdrzm0a8v1jx5j-openjdk-13.0-jdk
   /gnu/store/nb6p8aqjjw923vwd7safbh0w5q3mr9fq-openjdk-13.0
$ guix size openjdk@14 | grep openjdk
/gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0           389.3   295.8  76.0%
$ guix describe
Generacio 175   Feb 04 2021 22:52:40    (nuna)
  guix 5ae09d7
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 5ae09d7979a0696d862b9555314eab199f7ce576
--8<---------------cut here---------------end--------------->8---

What this shows is that the compiled openjdk 14 does not depend on
previous versions; likewise for version 13.

So the long dependency chain is a built-time-only dependency chain.
It’s a problem when building from source but not when using substitutes.

Ludo’.

bug#31719: Chains of dependencies getting longer

[Andreas Enge]

Hello,

Am Mon, Mar 01, 2021 at 11:04:02AM +0100 schrieb Ludovic Courtès:
> Andreas Enge <andreas@enge.fr> skribis:
> > trying a "guix build openjdk", I see this:
> > 2136.0 MB would be downloaded
...
> >    /gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
> >    /gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
...
> >    /gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
> >    /gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
> >    /gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0
> 
> Weird.  I see this:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build openjdk -n
> 369.8 MB would be downloaded:
>    /gnu/store/676kcvxnnjfcy960fy4xkr10fbn3ciay-openjdk-14.0-doc
>    /gnu/store/dk8rkrfkdgsmy1a33n0sc39fhqggy1nh-openjdk-14.0-jdk
>    /gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0

It is strange we do not see the same thing! I just tried it again, and am
getting the same result as above. Notice that openjdk@14 is available as a
substitute, but still all previous versions are downloaded, while nothing
is built. The same with the most recent master commit (7ca43b0a1e).

Andreas

bug#31719: Chains of dependencies getting longer

[Ludovic Courtès]

Hi!

Andreas Enge <andreas@enge.fr> skribis:

> Am Mon, Mar 01, 2021 at 11:04:02AM +0100 schrieb Ludovic Courtès:
>> Andreas Enge <andreas@enge.fr> skribis:
>> > trying a "guix build openjdk", I see this:
>> > 2136.0 MB would be downloaded
> ...
>> >    /gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
>> >    /gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
> ...
>> >    /gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
>> >    /gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
>> >    /gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0
>> 
>> Weird.  I see this:
>> 
>> --8<---------------cut here---------------start------------->8---
>> $ guix build openjdk -n
>> 369.8 MB would be downloaded:
>>    /gnu/store/676kcvxnnjfcy960fy4xkr10fbn3ciay-openjdk-14.0-doc
>>    /gnu/store/dk8rkrfkdgsmy1a33n0sc39fhqggy1nh-openjdk-14.0-jdk
>>    /gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0
>
> It is strange we do not see the same thing! I just tried it again, and am
> getting the same result as above. Notice that openjdk@14 is available as a
> substitute, but still all previous versions are downloaded, while nothing
> is built.

Wait, with a newer commit, I get the whole chain as well:

--8<---------------cut here---------------start------------->8---
$ guix build openjdk -n
2,135.0 MB would be downloaded:
   /gnu/store/ymyvcc0871lwbnkv5acf5lldv0ahm5z1-openjdk-9.181-jdk
   /gnu/store/3hb80r2l48ix8pq2kvcsr1i9aj76d681-openjdk-9.181
   /gnu/store/52bp22jrvwfqa5fmlv2y5vxcir74m8jb-openjdk-10.46-jdk
   /gnu/store/bp86qd2vgaqahqam3n89xl1wz02g0srs-openjdk-10.46
   /gnu/store/a8f5ksk3gb4i1wrn7m5fm5ijc0zwwg2w-openjdk-11.28-jdk
   /gnu/store/9ni61z4nhvkrxbzj6gwip8zrb7zbkx6l-openjdk-11.28
   /gnu/store/837pl2vmq9haiw3kmsnh72fkwvcljipk-openjdk-12.33-jdk
   /gnu/store/ll5dgf42x9smclrc1mx4gyk3dl683m77-openjdk-13.0-jdk
   /gnu/store/bjvy5ab6f0gff52m808vzcn3rw55yxjn-openjdk-14.0-doc
   /gnu/store/r8kxk031fighc4dxr57nl0wirzhk8v08-openjdk-14.0-jdk
   /gnu/store/25kx1vna763xh4mpszp4p4sy4j6iifs3-openjdk-14.0
$ guix describe
Generacio 176   Mar 01 2021 11:39:49    (nuna)
  guix 7ca43b0
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 7ca43b0a1e2215abe0df0708f31decace8e68911
--8<---------------cut here---------------end--------------->8---

But again, from that Feb. 4th commit, everything looks fine:

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=5ae09d7979a0696d862b9555314eab199f7ce576 -- build openjdk -n
369.8 MB would be downloaded:
   /gnu/store/676kcvxnnjfcy960fy4xkr10fbn3ciay-openjdk-14.0-doc
   /gnu/store/dk8rkrfkdgsmy1a33n0sc39fhqggy1nh-openjdk-14.0-jdk
   /gnu/store/ylakfs5fwq5sqsk5zbkwwf2d1lp0jfxs-openjdk-14.0
--8<---------------cut here---------------end--------------->8---

It must be a recent regression.  I think this is an unintended
side-effect of 44425e1f2a96aee39a21dff634abb9b77b44261e and
84805ef205b7fa12bfaa7b23da06993cab64c40b (see
<https://issues.guix.gnu.org/41177>):

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=44425e1f2a96aee39a21dff634abb9b77b44261e -- build openjdk -n
2,135.9 MB would be downloaded:
   /gnu/store/wd9jyzy7i2dxbxahc0pa8mq22any08zx-openjdk-9.181-jdk
   /gnu/store/7zxhm2xmyqds8hvlhifi9d16p9ln9k4b-openjdk-9.181
   /gnu/store/j5sh6c7qhm6vis6fvjyxs2jch9dhz085-openjdk-10.46-jdk
   /gnu/store/7w3vd4y0jac7d74gw62qmfv7br11k64r-openjdk-10.46
   /gnu/store/0sb9z8ds8ly7pwmviaqixq9wq1rp13g5-openjdk-11.28-jdk
   /gnu/store/4nf35d4v363bxrfmp85sk671swzmbj5k-openjdk-11.28
   /gnu/store/iwi28gcfxrsq541367m4hnp44jnnrkdq-openjdk-12.33-jdk
   /gnu/store/2w5gg7v9sy54s1jfnymgvg6qwqnkzr8h-openjdk-13.0-jdk
   /gnu/store/sgqw343zlqqn4vpdq4v9m4jy6ar9i99f-openjdk-14.0-doc
   /gnu/store/8l4pb8rh9zbqjn14ipbps7nn8wd0jg4w-openjdk-14.0-jdk
   /gnu/store/1d5bzdzrir8k13bkclx3a9cq59gw5j4h-openjdk-14.0
--8<---------------cut here---------------end--------------->8---

Björn, WDYT?  I suppose we need to filter out some library names in
‘patch-jni-libs’?

While at it, we should add a #:disallowed-references keyword to prevent
that from popping up in the future.

Thanks,
Ludo’.

bug#31719: Chains of dependencies getting longer

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1819 bytes --]

On Mon, 01 Mar 2021 22:33:55 +0100
Ludovic Courtès <ludo@gnu.org> wrote:

> It must be a recent regression.  I think this is an unintended
> side-effect of 44425e1f2a96aee39a21dff634abb9b77b44261e and
> 84805ef205b7fa12bfaa7b23da06993cab64c40b (see
> <https://issues.guix.gnu.org/41177>):
> 
> --8<---------------cut here---------------start------------->8---
> $ guix time-machine --commit=44425e1f2a96aee39a21dff634abb9b77b44261e
> -- build openjdk -n 2,135.9 MB would be downloaded:
>    /gnu/store/wd9jyzy7i2dxbxahc0pa8mq22any08zx-openjdk-9.181-jdk
>    /gnu/store/7zxhm2xmyqds8hvlhifi9d16p9ln9k4b-openjdk-9.181
>    /gnu/store/j5sh6c7qhm6vis6fvjyxs2jch9dhz085-openjdk-10.46-jdk
>    /gnu/store/7w3vd4y0jac7d74gw62qmfv7br11k64r-openjdk-10.46
>    /gnu/store/0sb9z8ds8ly7pwmviaqixq9wq1rp13g5-openjdk-11.28-jdk
>    /gnu/store/4nf35d4v363bxrfmp85sk671swzmbj5k-openjdk-11.28
>    /gnu/store/iwi28gcfxrsq541367m4hnp44jnnrkdq-openjdk-12.33-jdk
>    /gnu/store/2w5gg7v9sy54s1jfnymgvg6qwqnkzr8h-openjdk-13.0-jdk
>    /gnu/store/sgqw343zlqqn4vpdq4v9m4jy6ar9i99f-openjdk-14.0-doc
>    /gnu/store/8l4pb8rh9zbqjn14ipbps7nn8wd0jg4w-openjdk-14.0-jdk
>    /gnu/store/1d5bzdzrir8k13bkclx3a9cq59gw5j4h-openjdk-14.0
> --8<---------------cut here---------------end--------------->8---
> 
> Björn, WDYT?  I suppose we need to filter out some library names in
> ‘patch-jni-libs’?

Ouch. While installing a JDK last week, I had the same feeling that the
dependency chain down was a bit too long, but then forgot to look at it
again. 

When I read today Andreas' email, it popped up again ... I will have a
look at it.

> While at it, we should add a #:disallowed-references keyword to
> prevent that from popping up in the future.

I wasn't aware of this and will also have a look.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#31719: Chains of dependencies getting longer

[Ricardo Wurmus]

Hi Björn,

> On Mon, 01 Mar 2021 22:33:55 +0100
> Ludovic Courtès <ludo@gnu.org> wrote:
>
>> It must be a recent regression.  I think this is an unintended
>> side-effect of 44425e1f2a96aee39a21dff634abb9b77b44261e and
>> 84805ef205b7fa12bfaa7b23da06993cab64c40b (see
>> <https://issues.guix.gnu.org/41177>):
>> 
>> --8<---------------cut 
>> here---------------start------------->8---
>> $ guix time-machine 
>> --commit=44425e1f2a96aee39a21dff634abb9b77b44261e
>> -- build openjdk -n 2,135.9 MB would be downloaded:
>>    /gnu/store/wd9jyzy7i2dxbxahc0pa8mq22any08zx-openjdk-9.181-jdk
>>    /gnu/store/7zxhm2xmyqds8hvlhifi9d16p9ln9k4b-openjdk-9.181
>>    /gnu/store/j5sh6c7qhm6vis6fvjyxs2jch9dhz085-openjdk-10.46-jdk
>>    /gnu/store/7w3vd4y0jac7d74gw62qmfv7br11k64r-openjdk-10.46
>>    /gnu/store/0sb9z8ds8ly7pwmviaqixq9wq1rp13g5-openjdk-11.28-jdk
>>    /gnu/store/4nf35d4v363bxrfmp85sk671swzmbj5k-openjdk-11.28
>>    /gnu/store/iwi28gcfxrsq541367m4hnp44jnnrkdq-openjdk-12.33-jdk
>>    /gnu/store/2w5gg7v9sy54s1jfnymgvg6qwqnkzr8h-openjdk-13.0-jdk
>>    /gnu/store/sgqw343zlqqn4vpdq4v9m4jy6ar9i99f-openjdk-14.0-doc
>>    /gnu/store/8l4pb8rh9zbqjn14ipbps7nn8wd0jg4w-openjdk-14.0-jdk
>>    /gnu/store/1d5bzdzrir8k13bkclx3a9cq59gw5j4h-openjdk-14.0
>> --8<---------------cut 
>> here---------------end--------------->8---
>> 
>> Björn, WDYT?  I suppose we need to filter out some library 
>> names in
>> ‘patch-jni-libs’?
>
> Ouch. While installing a JDK last week, I had the same feeling 
> that the
> dependency chain down was a bit too long, but then forgot to 
> look at it
> again. 
>
> When I read today Andreas' email, it popped up again ... I will 
> have a
> look at it.
>
>> While at it, we should add a #:disallowed-references keyword to
>> prevent that from popping up in the future.
>
> I wasn't aware of this and will also have a look.

Have you been able to identify what’s wrong with the JDK chain? 
I’d like us to drop these extra references soon so that they don’t 
appear in the upcoming release.

-- 
Ricardo

bug#31719: Chains of dependencies getting longer

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

Hi Ricardo,

On Wed, 14 Apr 2021 09:36:23 +0200
Ricardo Wurmus <rekado@elephly.net> wrote:

> Have you been able to identify what’s wrong with the JDK chain? 
> I’d like us to drop these extra references soon so that they don’t 
> appear in the upcoming release.

Sorry, I currently don't find the time to look at the problem. Would
someone else give it a try?

Björn


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#31719: Chains of dependencies getting longer

[Carlo Zancanaro]

On Sat, Apr 17 2021, Björn Höfling wrote:
> Sorry, I currently don't find the time to look at the problem. 
> Would someone else give it a try?

I just had a quick look, and I think we can resolve it by changing 
the patch-jni-libs to not rewrite references to "mlib_image" and 
"splashscreen". It looks like they're provided as part of the JVM 
built itself, but our build phase hard codes a reference to the 
previous JVM's build result instead of using the current JVM's 
build result. They're the only dependencies I've found so far, but 
I've only looked at openjdk10 and openjdk14.

I'll put together a patch later today, if I haven't been beaten to 
it by then.

Carlo

bug#31719: Chains of dependencies getting longer

[Carlo Zancanaro]

[-- Attachment #1: Type: text/plain, Size: 827 bytes --]

Here's a patch that should clean up these runtime dependencies. 
It's a bit specific to this particular case, but I think that 
might be fine for now.

I think it would make more sense for native inputs to not have 
their paths included in LIBRARY_PATH. Does it even make sense for 
them to be there? I thought LIBRARY_PATH was for compilers to find 
dependencies when compiling so they can link their output binaries 
against them. Having native inputs show up there seems wrong.

I'm in the process of rebuilding Java from icedtea-8 upwards to 
check, but I have already tested that modifying openjdk 9 and 10 
leads to "guix gc --references" show that openjdk 10 does not 
depend on openjdk 9. I have also tested that I can run some 
complex Java programs on my machine using the openjdk 10 built 
using this patch.

Carlo


[-- Attachment #2: 0001-gnu-Clean-up-runtime-dependencies-between-Java-versi.patch --]
[-- Type: text/x-diff, Size: 4380 bytes --]

From f98dc5ad5662cc62f198d8f50e7dd719cf941315 Mon Sep 17 00:00:00 2001
From: Carlo Zancanaro <carlo@zancanaro.id.au>
Date: Sat, 17 Apr 2021 16:33:06 +1000
Subject: [PATCH] gnu: Clean up runtime dependencies between Java versions.

* gnu/packages/java.scm (icedtea-8, openjdk9, openjdk11): Don't consider
icedtea/openjdk input paths when rewriting JNI libraries.
---
 gnu/packages/java.scm | 36 ++++++++++++++++++++++++++----------
 1 file changed, 26 insertions(+), 10 deletions(-)

diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 207f136513..3c4013ab6f 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -2,7 +2,7 @@
 ;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016 Leo Famulari <leo@famulari.name>
 ;;; Copyright © 2016, 2017 Roel Janssen <roel@gnu.org>
-;;; Copyright © 2017, 2019 Carlo Zancanaro <carlo@zancanaro.id.au>
+;;; Copyright © 2017, 2019, 2021 Carlo Zancanaro <carlo@zancanaro.id.au>
 ;;; Copyright © 2017-2020 Julien Lepiller <julien@lepiller.eu>
 ;;; Copyright © 2017 Thomas Danckaert <post@thomasdanckaert.be>
 ;;; Copyright © 2016, 2017, 2018 Alex Vong <alexvong1995@gmail.com>
@@ -1792,8 +1792,13 @@ new Date();"))
                  (add-after 'unpack 'patch-jni-libs
                    ;; Hardcode dynamically loaded libraries.
                    (lambda _
-                     (let* ((library-path (search-path-as-string->list
-                                           (getenv "LIBRARY_PATH")))
+                     (use-modules (srfi srfi-1))
+                     (define (icedtea-or-openjdk? path)
+                       (or (string-contains path "openjdk")
+                           (string-contains path "icedtea")))
+                     (let* ((library-path (remove icedtea-or-openjdk?
+                                                  (search-path-as-string->list
+                                                   (getenv "LIBRARY_PATH"))))
                             (find-library (lambda (name)
                                             (search-path
                                              library-path
@@ -1931,12 +1936,18 @@ new Date();"))
          (add-after 'unpack 'patch-jni-libs
            ;; Hardcode dynamically loaded libraries.
            (lambda _
-             (let* ((library-path (search-path-as-string->list
-                                   (getenv "LIBRARY_PATH")))
+             (use-modules (srfi srfi-1))
+             (define (icedtea-or-openjdk? path)
+               (or (string-contains path "openjdk")
+                   (string-contains path "icedtea")))
+             (let* ((library-path (remove icedtea-or-openjdk?
+                                          (search-path-as-string->list
+                                           (getenv "LIBRARY_PATH"))))
                     (find-library (lambda (name)
-                                    (search-path
-                                     library-path
-                                     (string-append "lib" name ".so")))))
+                                    (or (search-path
+                                         library-path
+                                         (string-append "lib" name ".so"))
+                                        (string-append "lib" name ".so")))))
                (for-each
                 (lambda (file)
                   (catch 'decoding-error
@@ -2139,8 +2150,13 @@ new Date();"))
          (add-after 'unpack 'patch-jni-libs
            ;; Hardcode dynamically loaded libraries.
            (lambda _
-             (let* ((library-path (search-path-as-string->list
-                                   (getenv "LIBRARY_PATH")))
+             (use-modules (srfi srfi-1))
+             (define (icedtea-or-openjdk? path)
+               (or (string-contains path "openjdk")
+                   (string-contains path "icedtea")))
+             (let* ((library-path (remove icedtea-or-openjdk?
+                                          (search-path-as-string->list
+                                           (getenv "LIBRARY_PATH"))))
                     (find-library (lambda (name)
                                     (search-path
                                      library-path
-- 
2.31.1

bug#31719: Chains of dependencies getting longer

[Carlo Zancanaro]

On Sat, Apr 17 2021, Carlo Zancanaro wrote:
> I'm in the process of rebuilding Java from icedtea-8 upwards to 
> check,

I've now built and checked the JRE/JDKs from 10 to 14, and none of 
them retain a reference to any other JRE/JDK according to "guix gc 
--references".

Carlo

bug#31719: Chains of dependencies getting longer

[Andreas Enge]

Hello Carlo,

Am Sat, Apr 17, 2021 at 07:38:11PM +1000 schrieb Carlo Zancanaro:
> On Sat, Apr 17 2021, Carlo Zancanaro wrote:
> > I'm in the process of rebuilding Java from icedtea-8 upwards to check,
> 
> I've now built and checked the JRE/JDKs from 10 to 14, and none of them
> retain a reference to any other JRE/JDK according to "guix gc --references".

thanks a lot for your patch! I am not very knowledgeable on this matter, but
after updating my system am right now once again bitten by the problem:
The following derivations would be built:
   /gnu/store/0bxpgqps79007jky37dwzgm08wlsgj02-openjdk-10.46.drv
   /gnu/store/2xwsm2plrbk7l4510hrqp9y7b5vv9v5j-module-import-compiled.drv
1683.5 MB would be downloaded:
   /gnu/store/0gzv5208m2prqbnsqdffcnz7mwfqa684-openjdk-10.46.tar.xz
   /gnu/store/5g6ng9a2ifgrv57mzdaysf2lq9rkixxk-make-4.2.1
   /gnu/store/243algr6h60j46spn5dqhjc4mhkd0a0p-libelf-0.8.13
   /gnu/store/crc4cgsdls10isy8prjdddvsp21wafx6-openjdk-9.181-jdk
   /gnu/store/6amc3php7qyqxagak5xnmv2k81wm7w26-openjdk-9.181
   /gnu/store/bms6jg5qwn927qmbh9xi30ifhsxxdazq-openjdk-11.28-jdk
   /gnu/store/lylv6ng5gwqpi930c6y7sglh2k8byjn6-openjdk-11.28
   /gnu/store/z27mhqpylrbmwkcgrb100p6jbflxhq5h-openjdk-12.33-jdk
   /gnu/store/xz500ry11dihwn9kij1kmzkci1lmnqjf-openjdk-13.0-jdk
   /gnu/store/a1s66hlj10nkkcxlk6s2dzq4iip4mh4k-openjdk-14.0-doc
   /gnu/store/cyssia0a5lh8mb5czd7155y7sl31aryl-openjdk-14.0-jdk
   /gnu/store/630lvja8vak1bkf9vsbbh29cp79j4pwp-openjdk-14.0
guix build: warning: at least 5922.4 MB needed but only 3539.4 MB available in /gnu/store

So I would have to download lots of openjdk and even build one in the middle
of the chain myself.

So unless someone else objects, I intend to apply the patch, build the
latest openjdk, and push it if everything goes well.

Notice that the original problem reported by Ricardo seems to have been
fixed independently - "guix build icedtea" only downloads icedtea@3, and
not @2 on current master.

Thanks,

Andreas

bug#31719: Chains of dependencies getting longer

[Ricardo Wurmus]

Andreas Enge <andreas@enge.fr> writes:

> So unless someone else objects, I intend to apply the patch, 
> build the
> latest openjdk, and push it if everything goes well.

Sounds good.

I just looked over the patch, and while I’m not sure it’s the best 
way to do things (matching “openjdk” or “icedtea” in the package 
name seems a little error prone in the presence of packages whose 
names might include these strings), but I think it’s a definite 
improvement.

Thank you Carlo!

-- 
Ricardo

bug#31719: Chains of dependencies getting longer

[Andreas Enge]

Hello,

Am Mon, Apr 19, 2021 at 08:18:03PM +0200 schrieb Ricardo Wurmus:
> I just looked over the patch, and while I’m not sure it’s the best way to do
> things (matching “openjdk” or “icedtea” in the package name seems a little
> error prone in the presence of packages whose names might include these
> strings), but I think it’s a definite improvement.

I just pushed the patch to master. Thanks a lot, Carlo! It has definitely
solved my problem: I can now compile an Android project after downloading
a single openjdk package.

It would be nice if someone else could close the bug if you feel the problem
is solved, or otherwise leave it open to discuss further possible improvements.

Andreas

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

Hi,

Andreas Enge <andreas@enge.fr> skribis:

> Am Mon, Apr 19, 2021 at 08:18:03PM +0200 schrieb Ricardo Wurmus:
>> I just looked over the patch, and while I’m not sure it’s the best way to do
>> things (matching “openjdk” or “icedtea” in the package name seems a little
>> error prone in the presence of packages whose names might include these
>> strings), but I think it’s a definite improvement.
>
> I just pushed the patch to master. Thanks a lot, Carlo! It has definitely
> solved my problem: I can now compile an Android project after downloading
> a single openjdk package.
>
> It would be nice if someone else could close the bug if you feel the problem
> is solved, or otherwise leave it open to discuss further possible improvements.

I think we can close it.  I have the attached improvements that I can
commit to ‘staging’ (or ‘core-updates’?) to avoid another rebuild.

Thanks!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: the patch --]
[-- Type: text/x-patch, Size: 4535 bytes --]

From 253d0485a9307c4e08afc058d7dafcd56025f9a6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Tue, 20 Apr 2021 00:26:01 +0200
Subject: [PATCH] java fixlet

---
 gnu/packages/java.scm | 35 +++++++++++++++++++++++++++--------
 1 file changed, 27 insertions(+), 8 deletions(-)

diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index 3c4013ab6f..b780f7a85f 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -1749,6 +1749,9 @@ IcedTea build harness.")
          ((guix build ant-build-system)
           (guix build syscalls)
           ,@%gnu-build-system-modules)
+
+         #:disallowed-references ((,icedtea-7 "jdk"))
+
          ,@(substitute-keyword-arguments (package-arguments icedtea-7)
              ((#:modules modules)
               `((guix build utils)
@@ -1792,10 +1795,13 @@ new Date();"))
                  (add-after 'unpack 'patch-jni-libs
                    ;; Hardcode dynamically loaded libraries.
                    (lambda _
-                     (use-modules (srfi srfi-1))
+                     (define remove
+                       (@ (srfi srfi-1) remove))
+
                      (define (icedtea-or-openjdk? path)
                        (or (string-contains path "openjdk")
                            (string-contains path "icedtea")))
+
                      (let* ((library-path (remove icedtea-or-openjdk?
                                                   (search-path-as-string->list
                                                    (getenv "LIBRARY_PATH"))))
@@ -1898,6 +1904,9 @@ new Date();"))
        #:imported-modules
        ((guix build syscalls)
         ,@%gnu-build-system-modules)
+
+       #:disallowed-references (,icedtea-8 (,icedtea-8 "jdk"))
+
        #:phases
        (modify-phases %standard-phases
          (add-after 'patch-source-shebangs 'fix-java-shebangs
@@ -1936,18 +1945,20 @@ new Date();"))
          (add-after 'unpack 'patch-jni-libs
            ;; Hardcode dynamically loaded libraries.
            (lambda _
-             (use-modules (srfi srfi-1))
+             (define remove
+               (@ (srfi srfi-1) remove))
+
              (define (icedtea-or-openjdk? path)
                (or (string-contains path "openjdk")
                    (string-contains path "icedtea")))
+
              (let* ((library-path (remove icedtea-or-openjdk?
                                           (search-path-as-string->list
                                            (getenv "LIBRARY_PATH"))))
                     (find-library (lambda (name)
-                                    (or (search-path
-                                         library-path
-                                         (string-append "lib" name ".so"))
-                                        (string-append "lib" name ".so")))))
+                                    (search-path
+                                     library-path
+                                     (string-append "lib" name ".so")))))
                (for-each
                 (lambda (file)
                   (catch 'decoding-error
@@ -2090,7 +2101,9 @@ new Date();"))
                        "--with-libjpeg=system"
                        "--with-native-debug-symbols=zipped"
                        (string-append "--prefix=" (assoc-ref outputs "out")))
-               #t))))))
+               #t))))
+       ((#:disallowed-references _ '())
+        `(,openjdk9 (,openjdk9 "jdk")))))
     (native-inputs
      `(("openjdk9" ,openjdk9)
        ("openjdk9:jdk" ,openjdk9 "jdk")
@@ -2120,6 +2133,9 @@ new Date();"))
     (arguments
      `(#:imported-modules ((guix build syscalls)
                            ,@%gnu-build-system-modules)
+
+       #:disallowed-references (,openjdk10 (,openjdk10 "jdk"))
+
        #:tests? #f; requires jtreg
        ;; TODO package jtreg
        #:configure-flags
@@ -2150,10 +2166,13 @@ new Date();"))
          (add-after 'unpack 'patch-jni-libs
            ;; Hardcode dynamically loaded libraries.
            (lambda _
-             (use-modules (srfi srfi-1))
+             (define remove
+               (@ (srfi srfi-1) remove))
+
              (define (icedtea-or-openjdk? path)
                (or (string-contains path "openjdk")
                    (string-contains path "icedtea")))
+
              (let* ((library-path (remove icedtea-or-openjdk?
                                           (search-path-as-string->list
                                            (getenv "LIBRARY_PATH"))))
-- 
2.31.1

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Andreas Enge]

Am Tue, Apr 20, 2021 at 11:24:59AM +0200 schrieb Ludovic Courtès:
> I think we can close it.  I have the attached improvements that I can
> commit to ‘staging’ (or ‘core-updates’?) to avoid another rebuild.

I think they can easily go to master; the time for rebuilds is not very
high, I just ran into trouble since I had little space left on the device
so some manual intervention was required to gc intermediate packages.

There also seem to be very few dependencies if "guix refresh -l openjdk"
is to be trusted.

Andreas

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Carlo Zancanaro]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

Hi Ludo!

On Tue, Apr 20 2021, Ludovic Courtès wrote:
>                      (find-library (lambda (name)
> -                                    (or (search-path
> -                                         library-path
> -                                         (string-append "lib" 
> name ".so"))
> -                                        (string-append "lib" 
> name ".so")))))
> +                                    (search-path
> +                                     library-path
> +                                     (string-append "lib" name 
> ".so")))))
>                 (for-each

As discussed on IRC, the "or" is actually important here to avoid 
substituting #f as the library name. I've attached a patch on top 
of yours that adds the "or" back (including the other two that I 
missed in my earlier patch), and also switches to "string-append" 
which is less sensitive to this problem.

I have built up to openjdk11 with this patch, and I see less #f's 
in the result. There are still some in the compiled libraries, but 
I haven't investigated thoroughly as to whether they're correct or 
not.

Carlo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Fix-openjdk-library-substitution-when-libraries-.patch --]
[-- Type: text/x-diff, Size: 5205 bytes --]

From 60101b27543b7cc41a052d5bec95234ea4977d35 Mon Sep 17 00:00:00 2001
From: Carlo Zancanaro <carlo@zancanaro.id.au>
Date: Tue, 20 Apr 2021 21:22:20 +1000
Subject: [PATCH] gnu: Fix openjdk library substitution when libraries aren't
 found

* gnu/packages/java.scm (icedtea-8, openjdk9, openjdk11): Fix JNI library
substitution to not substitute #f if the library can't be found.
---
 gnu/packages/java.scm | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)

diff --git a/gnu/packages/java.scm b/gnu/packages/java.scm
index b780f7a85f..8a1ba5f262 100644
--- a/gnu/packages/java.scm
+++ b/gnu/packages/java.scm
@@ -1806,9 +1806,10 @@ new Date();"))
                                                   (search-path-as-string->list
                                                    (getenv "LIBRARY_PATH"))))
                             (find-library (lambda (name)
-                                            (search-path
-                                             library-path
-                                             (string-append "lib" name ".so")))))
+                                            (or (search-path
+                                                 library-path
+                                                 (string-append "lib" name ".so"))
+                                                (string-append "lib" name ".so")))))
                        (for-each
                         (lambda (file)
                           (catch 'decoding-error
@@ -1816,9 +1817,9 @@ new Date();"))
                               (substitute* file
                                 (("VERSIONED_JNI_LIB_NAME\\(\"(.*)\", \"(.*)\"\\)"
                                   _ name version)
-                                 (format #f "\"~a\""  (find-library name)))
+                                 (string-append "\"" (find-library name) "\""))
                                 (("JNI_LIB_NAME\\(\"(.*)\"\\)" _ name)
-                                 (format #f "\"~a\"" (find-library name)))))
+                                 (string-append "\"" (find-library name) "\""))))
                             (lambda _
                               ;; Those are safe to skip.
                               (format (current-error-port)
@@ -1956,9 +1957,10 @@ new Date();"))
                                           (search-path-as-string->list
                                            (getenv "LIBRARY_PATH"))))
                     (find-library (lambda (name)
-                                    (search-path
-                                     library-path
-                                     (string-append "lib" name ".so")))))
+                                    (or (search-path
+                                         library-path
+                                         (string-append "lib" name ".so"))
+                                        (string-append "lib" name ".so")))))
                (for-each
                 (lambda (file)
                   (catch 'decoding-error
@@ -1966,9 +1968,9 @@ new Date();"))
                       (substitute* file
                         (("VERSIONED_JNI_LIB_NAME\\(\"(.*)\", \"(.*)\"\\)"
                           _ name version)
-                         (format #f "\"~a\""  (find-library name)))
+                         (string-append "\"" (find-library name) "\""))
                         (("JNI_LIB_NAME\\(\"(.*)\"\\)" _ name)
-                         (format #f "\"~a\"" (find-library name)))))
+                         (string-append "\"" (find-library name) "\""))))
                     (lambda _
                       ;; Those are safe to skip.
                       (format (current-error-port)
@@ -2177,9 +2179,10 @@ new Date();"))
                                           (search-path-as-string->list
                                            (getenv "LIBRARY_PATH"))))
                     (find-library (lambda (name)
-                                    (search-path
-                                     library-path
-                                     (string-append "lib" name ".so")))))
+                                    (or (search-path
+                                         library-path
+                                         (string-append "lib" name ".so"))
+                                        (string-append "lib" name ".so")))))
                (for-each
                 (lambda (file)
                   (catch 'decoding-error
@@ -2187,9 +2190,9 @@ new Date();"))
                       (substitute* file
                         (("VERSIONED_JNI_LIB_NAME\\(\"(.*)\", \"(.*)\"\\)"
                           _ name version)
-                         (format #f "\"~a\""  (find-library name)))
+                         (string-append "\"" (find-library name) "\""))
                         (("JNI_LIB_NAME\\(\"(.*)\"\\)" _ name)
-                         (format #f "\"~a\"" (find-library name)))))
+                         (string-append "\"" (find-library name) "\""))))
                     (lambda _
                       ;; Those are safe to skip.
                       (format (current-error-port)
-- 
2.31.1

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Ludovic Courtès]

Hi Carlo,

Carlo Zancanaro <carlo@zancanaro.id.au> skribis:

> From 60101b27543b7cc41a052d5bec95234ea4977d35 Mon Sep 17 00:00:00 2001
> From: Carlo Zancanaro <carlo@zancanaro.id.au>
> Date: Tue, 20 Apr 2021 21:22:20 +1000
> Subject: [PATCH] gnu: Fix openjdk library substitution when libraries aren't
>  found
>
> * gnu/packages/java.scm (icedtea-8, openjdk9, openjdk11): Fix JNI library
> substitution to not substitute #f if the library can't be found.

Thanks for the update patch!  Please mention ‘find-library’ in the
commit log, for clarity.

I think you can go and push it to master given that OpenJDK is probably
broken due to those #f right now.  However, please push an additional
commit at the same time that adds those #:disallowed-references I
proposed.  (Though you’ll have to rebuild openjdk@11 to be sure.)

Sounds good?

Ludo’.

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Carlo Zancanaro]

On 21 April 2021 2:38:56 am AEST, "Ludovic Courtès" <ludo@gnu.org> wrote:
>Sounds good?

Sounds good, except I don't have commit privileges. 😊 Can someone else push it for me?

bug#31719: icedtea-3 binaries contain references to icedtea-2

[Ludovic Courtès]

Hi,

Carlo Zancanaro <carlo@zancanaro.id.au> skribis:

> On 21 April 2021 2:38:56 am AEST, "Ludovic Courtès" <ludo@gnu.org> wrote:
>>Sounds good?
>
> Sounds good, except I don't have commit privileges. 😊 Can someone else push it for me?

Oops!  I adjusted the patch so it would apply to ‘master’, and followed
up with two commits:

  f8acd1aeef gnu: openjdk: Disallow references to the JDK used for build.
  e511a1d327 gnu: openjdk: Avoid non-top-level 'use-modules'.
  698c4365ba gnu: openjdk: Fix library substitution when libraries aren't found.

I rebuilt everything up to openjdk@11.

Thanks!

Ludo’.

bug#31719: Chains of dependencies getting longer

[Carlo Zancanaro]

On 20 April 2021 4:18:03 am AEST, Ricardo Wurmus <rekado@elephly.net> wrote:
>I just looked over the patch, and while I’m not sure it’s the best way to do things (matching “openjdk” or “icedtea” in the package name seems a little error prone in the presence of packages whose names might include these strings), but I think it’s a definite improvement.

Yeah, I'm agreed on that. I tried for a while to find a way to exclude native inputs, but as far as I could tell they all ended up as inputs within the build phases. I'd be happy to change it if somebody can give me a hint about how to pick out the native inputs.

Carlo