bug#62071: openjdk@9/10 sources not reproducible

bug#62071: openjdk@9/10 sources not reproducible

[Lars-Dominik Braun]

Hi,

it looks like the (auto-generated) tarballs for openjdk@9 and openjdk@10
changed their hash, causing a hash mismatch via

    guix build -S openjdk@9 --no-substitutes --no-grafts

I’m not sure why it uses these tarballs in the first place, since we
have a hg-download.

Lars

bug#62071: openjdk@9/10 sources not reproducible

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Thu, 9 Mar 2023 10:48:53 +0100
Lars-Dominik Braun <lars@6xq.net> wrote:

> Hi,
> 
> it looks like the (auto-generated) tarballs for openjdk@9 and
> openjdk@10 changed their hash, causing a hash mismatch via
> 
>     guix build -S openjdk@9 --no-substitutes --no-grafts
 

I can confirm this.

I found the old versions of openjdk 9 and 10 on a server of mine. I
will compare old/new tomorrow (oh, better say: later today).

> I’m not sure why it uses these tarballs in the first place, since we
> have a hg-download.

I don't know. Maybe there was no hg-download yet when we added OpenJDK9?

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#62071: openjdk@9/10 sources not reproducible

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

On Thu, 9 Mar 2023 10:48:53 +0100
Lars-Dominik Braun <lars@6xq.net> wrote:

> Hi,
> 
> it looks like the (auto-generated) tarballs for openjdk@9 and
> openjdk@10 changed their hash, causing a hash mismatch via
> 
>     guix build -S openjdk@9 --no-substitutes --no-grafts
> 
> I’m not sure why it uses these tarballs in the first place, since we
> have a hg-download.

I compared for JDK9 the two tarballs (old and new hash) and there is no
difference in the content (according to diffoscope). Also, if I
hg-clone the repository/tag (and add the .hg_archival.txt file), all
three directory trees have the same hash value according to guix hash
-rx

Thus, it seams like their artifacts are not stable, as we saw it
for autogenerated artifacts on github.

I will check the same for JDK10 and will prepare a patch within the
next two days.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#62071: openjdk@9/10 sources not reproducible

[Simon Tournier]

Hi,

On dim., 12 mars 2023 at 22:00, Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

> I compared for JDK9 the two tarballs (old and new hash) and there is no
> difference in the content (according to diffoscope). Also, if I
> hg-clone the repository/tag (and add the .hg_archival.txt file), all
> three directory trees have the same hash value according to guix hash
> -rx

So maybe it comes from some parameters of the compressor; maybe they
changed their level.  Well, I do not know how to check that.

Maybe the best is to switch from url-fetch to hg-fetch.  WDYT?

Cheers,
simon

bug#62071: openjdk@9/10 sources not reproducible

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

On Sun, 12 Mar 2023 22:00:21 +0100
Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

> On Thu, 9 Mar 2023 10:48:53 +0100
> Lars-Dominik Braun <lars@6xq.net> wrote:
> 
> > Hi,
> > 
> > it looks like the (auto-generated) tarballs for openjdk@9 and
> > openjdk@10 changed their hash, causing a hash mismatch via
> > 
> >     guix build -S openjdk@9 --no-substitutes --no-grafts
> > 
> > I’m not sure why it uses these tarballs in the first place, since we
> > have a hg-download.  

Changed to hg-download in commit(s):

7636c49b45adb9870cf416c64bde032ec858a820

Thanks for pointing this out.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#62071: openjdk@9/10 sources not reproducible

[Ludovic Courtès]

Hi Björn,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> I will check the same for JDK10 and will prepare a patch within the
> next two days.

Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
commit!

For the record, there are two remaining issues:

  1. Reproducibility of past revisions.  If we lose copies of the
     auto-generated tarballs, then OpenJDK in past revisions of Guix is
     irreparably lost.  We should check whether/how to get them in
     Disarchive + SWH.

  2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with Git
     (you can ask it for a specific Git commit ID), that’s not true for
     hg.  This is a more general problem, but as things are today,
     there’s no automatic SWH fallback if the upstream hg server
     vanishes.

Thanks,
Ludo’.

bug#62071: openjdk@9/10 sources not reproducible

[Jonathan Brielmaier]

I’m not sure why it uses these tarballs in the first place, since we
have a hg-download.

-> I guess a reason could be that downloading via hg is quite slow.
Thats at least my impression when fetching the "comm" repository for
Thunderbird with mecurial. Tarballs and git checkout tend to be way
faster...

bug#62071: openjdk@9/10 sources not reproducible

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

On Thu, 16 Mar 2023 12:48:19 +0100
Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> Hi Björn,
> 
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
> 
> > I will check the same for JDK10 and will prepare a patch within the
> > next two days.  
> 
> Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
> commit!
> 
> For the record, there are two remaining issues:
> 
>   1. Reproducibility of past revisions.  If we lose copies of the
>      auto-generated tarballs, then OpenJDK in past revisions of Guix
> is irreparably lost.  We should check whether/how to get them in
>      Disarchive + SWH.

How do we do that? Adding git repos to SWH is something I can achieve,
but adding tarballs to SWH was always opaque to me.

I find no reference in the manual about Disarchive. Ideally, is there a
linter for just adding a package to the disarchive database?

 
>   2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with
> Git (you can ask it for a specific Git commit ID), that’s not true for
>      hg.  This is a more general problem, but as things are today,
>      there’s no automatic SWH fallback if the upstream hg server
>      vanishes.

For git, I know and appreciate that the linter can directly add a
missing repo to SWH. During linting, I recogniced this is missing for
hg.

I was thinking a second time about it and found that not only the newer
development of OpenJDK is on GitHub, but also the older versions are
available. So I could add another patch like this: 

+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/openjdk/jdk9")

WDYT?

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#62071: openjdk@9/10 sources not reproducible

[Ludovic Courtès]

Hello,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> On Thu, 16 Mar 2023 12:48:19 +0100
> Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> Hi Björn,
>> 
>> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
>> 
>> > I will check the same for JDK10 and will prepare a patch within the
>> > next two days.  
>> 
>> Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
>> commit!
>> 
>> For the record, there are two remaining issues:
>> 
>>   1. Reproducibility of past revisions.  If we lose copies of the
>>      auto-generated tarballs, then OpenJDK in past revisions of Guix
>> is irreparably lost.  We should check whether/how to get them in
>>      Disarchive + SWH.
>
> How do we do that? Adding git repos to SWH is something I can achieve,
> but adding tarballs to SWH was always opaque to me.
>
> I find no reference in the manual about Disarchive. Ideally, is there a
> linter for just adding a package to the disarchive database?

SWH periodically ingests the contents of tarballs (not tarballs
themselves) that appear in <https://guix.gnu.org/sources.json>.  We’d
need to check whether it has the contents of those tarballs.

Then <https://disarchive.guix.gnu.org> is populated by the CI job at
<https://ci.guix.gnu.org/jobset/disarchive>.

Are the openjdk 9 and 10 tarballs archived?

Let’s look at their origins as of commit
1e6ddceb8318d413745ca1c9d91fde01b1e0364b.  We can construct their
Disarchive URL by first converting their SHA256 to hex:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix base32)
scheme@(guile-user)> ,use(guix base16)
scheme@(guile-user)> (bytevector->base16-string (nix-base32-string->bytevector "01ihmyf7k5z17wbr7xig7y40l9f01d5zjgkcmawn1102hw5kchpq"))
$5 = "f842360b87028460b9aa6c3ef94b0bc0250a883f2ff693173fe197799caf3006"
--8<---------------cut here---------------end--------------->8---

That gives us:

  https://disarchive.guix.gnu.org/sha256/f842360b87028460b9aa6c3ef94b0bc0250a883f2ff693173fe197799caf3006
  https://disarchive.guix.gnu.org/sha256/249fd462bdd32571c6d0a45f3cb698a305c9e4e66b275d25e990ac0184c0dc7f

Both are 404.  But like I wrote, this is expected: they are bzip2
tarballs and Disarchive doesn’t support bzip2 (yet).

>>   2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with
>> Git (you can ask it for a specific Git commit ID), that’s not true for
>>      hg.  This is a more general problem, but as things are today,
>>      there’s no automatic SWH fallback if the upstream hg server
>>      vanishes.
>
> For git, I know and appreciate that the linter can directly add a
> missing repo to SWH. During linting, I recogniced this is missing for
> hg.
>
> I was thinking a second time about it and found that not only the newer
> development of OpenJDK is on GitHub, but also the older versions are
> available. So I could add another patch like this: 
>
> +              (method git-fetch)
> +              (uri (git-reference
> +                    (url "https://github.com/openjdk/jdk9")
>
> WDYT?

That’s a good idea.  It shouldn’t change the SHA256 (if it does,
something’s wrong) so it looks like an no-brainer.

Thanks!

Ludo’.

bug#62071: openjdk@9/10 sources not reproducible

[Simon Tournier]

Hi,

On Mon, 20 Mar 2023 at 10:08, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>> I was thinking a second time about it and found that not only the newer
>> development of OpenJDK is on GitHub, but also the older versions are
>> available. So I could add another patch like this: 
>>
>> +              (method git-fetch)
>> +              (uri (git-reference
>> +                    (url "https://github.com/openjdk/jdk9")
>
> That’s a good idea.  It shouldn’t change the SHA256 (if it does,
> something’s wrong) so it looks like an no-brainer.

Well, by experience, it is rare that the released tarball contain the
exact same content as the tagged commit.  If it is the case (same SHA256
for both tarball and GitHub tagged release), indeed no-brainer. :-)

Else, some work still remain for the complete preservation of Guix. ;-)

Björn, what is the status of this SHA256?

Cheers,
simon