Making javadoc reproducible

Making javadoc reproducible

[Gábor Boskovits]

Hello guix,

I've tracked down the javadoc timestamp problem.
There is a command line flag for javadoc (notimestamp), that disables
generating the comment in the docs that contains the timestamp.
Currently I see two ways forward:
1. Track down the calls to javadoc, and add the flag to all calls.
2. Write a simple patch to make javadoc behave as if notimestamp was
specified, whenever
SOURCE_DATE_EPOCH is defined.
I do not think, that the patch produced by 2 is upstreamable, but it
seems much less work. WDYT?

Re: Making javadoc reproducible

[Gábor Boskovits]

Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. okt.
12., P, 19:00):
>
> Hello guix,
>
> I've tracked down the javadoc timestamp problem.
> There is a command line flag for javadoc (notimestamp), that disables
> generating the comment in the docs that contains the timestamp.
> Currently I see two ways forward:
> 1. Track down the calls to javadoc, and add the flag to all calls.
> 2. Write a simple patch to make javadoc behave as if notimestamp was
> specified, whenever
> SOURCE_DATE_EPOCH is defined.
> I do not think, that the patch produced by 2 is upstreamable, but it
> seems much less work. WDYT?

Also we can simply turn off the timestamp generation unconditionally...

Re: Making javadoc reproducible

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

On Fri, 12 Oct 2018 19:35:51 +0200
Gábor Boskovits <boskovits@gmail.com> wrote:

> Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. okt.
> 12., P, 19:00):
> >
> > Hello guix,
> >
> > I've tracked down the javadoc timestamp problem.
> > There is a command line flag for javadoc (notimestamp), that
> > disables generating the comment in the docs that contains the
> > timestamp. Currently I see two ways forward:
> > 1. Track down the calls to javadoc, and add the flag to all calls.
> > 2. Write a simple patch to make javadoc behave as if notimestamp was
> > specified, whenever
> > SOURCE_DATE_EPOCH is defined.
> > I do not think, that the patch produced by 2 is upstreamable, but it
> > seems much less work. WDYT?  
> 
> Also we can simply turn off the timestamp generation
> unconditionally...

Number 2 sounds good, and why not giving it a try to place it upstream?

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Making javadoc reproducible

[Gábor Boskovits]

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> ezt írta (időpont:
2018. okt. 12., P, 20:01):
>
> On Fri, 12 Oct 2018 19:35:51 +0200
> Gábor Boskovits <boskovits@gmail.com> wrote:
>
> > Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. okt.
> > 12., P, 19:00):
> > >
> > > Hello guix,
> > >
> > > I've tracked down the javadoc timestamp problem.
> > > There is a command line flag for javadoc (notimestamp), that
> > > disables generating the comment in the docs that contains the
> > > timestamp. Currently I see two ways forward:
> > > 1. Track down the calls to javadoc, and add the flag to all calls.
> > > 2. Write a simple patch to make javadoc behave as if notimestamp was
> > > specified, whenever
> > > SOURCE_DATE_EPOCH is defined.
> > > I do not think, that the patch produced by 2 is upstreamable, but it
> > > seems much less work. WDYT?
> >
> > Also we can simply turn off the timestamp generation
> > unconditionally...
>
> Number 2 sounds good, and why not giving it a try to place it upstream?

Ok, i will go for it, and try to get it upsreamed for jdk8 and jdk11.

>
> Björn

Re: Making javadoc reproducible

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]

On 2018-10-12, Björn Höfling wrote:
> On Fri, 12 Oct 2018 19:35:51 +0200
> Gábor Boskovits <boskovits@gmail.com> wrote:
>> Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. okt.
>> 12., P, 19:00):
>> > I've tracked down the javadoc timestamp problem.
>> > There is a command line flag for javadoc (notimestamp), that
>> > disables generating the comment in the docs that contains the
>> > timestamp. Currently I see two ways forward:
>> > 1. Track down the calls to javadoc, and add the flag to all calls.
>> > 2. Write a simple patch to make javadoc behave as if notimestamp was
>> > specified, whenever
>> > SOURCE_DATE_EPOCH is defined.
>> > I do not think, that the patch produced by 2 is upstreamable, but it
>> > seems much less work. WDYT?  
>> 
>> Also we can simply turn off the timestamp generation
>> unconditionally...
>
> Number 2 sounds good, and why not giving it a try to place it upstream?

There's been some discussion about this in Debian and in reproducible
builds:

  https://bugs.debian.org/783938

  https://wiki.debian.org/ReproducibleBuilds/TimestampsInDocumentationGeneratedByJavadoc

  https://tests.reproducible-builds.org/debian/issues/unstable/timestamps_in_documentation_generated_by_javadoc_issue.html

Hope it's useful!


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Making javadoc reproducible

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 1671 bytes --]

Hello,

Vagrant Cascadian <vagrant@debian.org> writes:

> On 2018-10-12, Björn Höfling wrote:
>> On Fri, 12 Oct 2018 19:35:51 +0200
>> Gábor Boskovits <boskovits@gmail.com> wrote:
>>> Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. okt.
>>> 12., P, 19:00):
>>> > I've tracked down the javadoc timestamp problem.
>>> > There is a command line flag for javadoc (notimestamp), that
>>> > disables generating the comment in the docs that contains the
>>> > timestamp. Currently I see two ways forward:
>>> > 1. Track down the calls to javadoc, and add the flag to all calls.
>>> > 2. Write a simple patch to make javadoc behave as if notimestamp was
>>> > specified, whenever
>>> > SOURCE_DATE_EPOCH is defined.
>>> > I do not think, that the patch produced by 2 is upstreamable, but it
>>> > seems much less work. WDYT?  
>>> 
>>> Also we can simply turn off the timestamp generation
>>> unconditionally...
>>
>> Number 2 sounds good, and why not giving it a try to place it upstream?
>
> There's been some discussion about this in Debian and in reproducible
> builds:
>
>   https://bugs.debian.org/783938
>
>   https://wiki.debian.org/ReproducibleBuilds/TimestampsInDocumentationGeneratedByJavadoc
>
>   https://tests.reproducible-builds.org/debian/issues/unstable/timestamps_in_documentation_generated_by_javadoc_issue.html
>
In the above, 2 solutions are mentioned:

1. Strip timestamp in files generated by javadoc
2. Patch javadoc to honor SOURCE_DATE_EPOCH

It seems 2 is easier but 1 is also possible since we have 'xml->sxml'
and friends in guile.

> Hope it's useful!
>
>
> live well,
>   vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Making javadoc reproducible

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2151 bytes --]

Hi Gábor and Vagrant,

Vagrant Cascadian <vagrant@debian.org> writes:

> There's been some discussion about this in Debian and in reproducible
> builds:
>
>   https://bugs.debian.org/783938
>
>   https://wiki.debian.org/ReproducibleBuilds/TimestampsInDocumentationGeneratedByJavadoc
>
>   https://tests.reproducible-builds.org/debian/issues/unstable/timestamps_in_documentation_generated_by_javadoc_issue.html
>
> Hope it's useful!

Thank you for the links!

Gábor Boskovits <boskovits@gmail.com> writes:

> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> ezt írta (időpont:
> 2018. okt. 12., P, 20:01):
>>
>> On Fri, 12 Oct 2018 19:35:51 +0200
>> Gábor Boskovits <boskovits@gmail.com> wrote:
>>
>> > Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. okt.
>> > 12., P, 19:00):
>> > >
>> > > Hello guix,
>> > >
>> > > I've tracked down the javadoc timestamp problem.
>> > > There is a command line flag for javadoc (notimestamp), that
>> > > disables generating the comment in the docs that contains the
>> > > timestamp. Currently I see two ways forward:
>> > > 1. Track down the calls to javadoc, and add the flag to all calls.
>> > > 2. Write a simple patch to make javadoc behave as if notimestamp was
>> > > specified, whenever
>> > > SOURCE_DATE_EPOCH is defined.
>> > > I do not think, that the patch produced by 2 is upstreamable, but it
>> > > seems much less work. WDYT?
>> >
>> > Also we can simply turn off the timestamp generation
>> > unconditionally...
>>
>> Number 2 sounds good, and why not giving it a try to place it upstream?
>
> Ok, i will go for it, and try to get it upsreamed for jdk8 and jdk11.

Be sure to check out the links Vagrant posted.  Specifically this one:

https://bugs.debian.org/783938

In that bug report, Samuel Thibault says: "Perhaps javadoc could be made
to use by default the SOURCE_DATE_EPOCH environment variable when it is
defined?"  There seems to be agreement that teaching javadoc to honor
the SOURCE_DATE_EPOCH environment variable would be preferable to
unconditionally disabling the timestamp behavior.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Making javadoc reproducible

[Gábor Boskovits]

Chris Marusich <cmmarusich@gmail.com> ezt írta (időpont: 2018. okt.
14., V, 5:35):
>
> Hi Gábor and Vagrant,
>
> Vagrant Cascadian <vagrant@debian.org> writes:
>
> > There's been some discussion about this in Debian and in reproducible
> > builds:
> >
> >   https://bugs.debian.org/783938
> >
> >   https://wiki.debian.org/ReproducibleBuilds/TimestampsInDocumentationGeneratedByJavadoc
> >
> >   https://tests.reproducible-builds.org/debian/issues/unstable/timestamps_in_documentation_generated_by_javadoc_issue.html
> >
> > Hope it's useful!
>
> Thank you for the links!
>

Yes, thank you!

> Gábor Boskovits <boskovits@gmail.com> writes:
>
> > Björn Höfling <bjoern.hoefling@bjoernhoefling.de> ezt írta (időpont:
> > 2018. okt. 12., P, 20:01):
> >>
> >> On Fri, 12 Oct 2018 19:35:51 +0200
> >> Gábor Boskovits <boskovits@gmail.com> wrote:
> >>
> >> > Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. okt.
> >> > 12., P, 19:00):
> >> > >
> >> > > Hello guix,
> >> > >
> >> > > I've tracked down the javadoc timestamp problem.
> >> > > There is a command line flag for javadoc (notimestamp), that
> >> > > disables generating the comment in the docs that contains the
> >> > > timestamp. Currently I see two ways forward:
> >> > > 1. Track down the calls to javadoc, and add the flag to all calls.
> >> > > 2. Write a simple patch to make javadoc behave as if notimestamp was
> >> > > specified, whenever
> >> > > SOURCE_DATE_EPOCH is defined.
> >> > > I do not think, that the patch produced by 2 is upstreamable, but it
> >> > > seems much less work. WDYT?
> >> >
> >> > Also we can simply turn off the timestamp generation
> >> > unconditionally...
> >>
> >> Number 2 sounds good, and why not giving it a try to place it upstream?
> >
> > Ok, i will go for it, and try to get it upsreamed for jdk8 and jdk11.
>
> Be sure to check out the links Vagrant posted.  Specifically this one:
>
> https://bugs.debian.org/783938
>
> In that bug report, Samuel Thibault says: "Perhaps javadoc could be made
> to use by default the SOURCE_DATE_EPOCH environment variable when it is
> defined?"  There seems to be agreement that teaching javadoc to honor
> the SOURCE_DATE_EPOCH environment variable would be preferable to
> unconditionally disabling the timestamp behavior.
>

Yes, I've also came to that conclusion reading the discussion, andI
have a came up with a patch.
I'm testing it right now, will report back if I have the results.

> --
> Chris

Re: Making javadoc reproducible

[Gábor Boskovits]

Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. okt.
14., V, 8:43):
>
> Chris Marusich <cmmarusich@gmail.com> ezt írta (időpont: 2018. okt.
> 14., V, 5:35):
> >
> > Hi Gábor and Vagrant,
> >
> > Vagrant Cascadian <vagrant@debian.org> writes:
> >
> > > There's been some discussion about this in Debian and in reproducible
> > > builds:
> > >
> > >   https://bugs.debian.org/783938
> > >
> > >   https://wiki.debian.org/ReproducibleBuilds/TimestampsInDocumentationGeneratedByJavadoc
> > >
> > >   https://tests.reproducible-builds.org/debian/issues/unstable/timestamps_in_documentation_generated_by_javadoc_issue.html
> > >
> > > Hope it's useful!
> >
> > Thank you for the links!
> >
>
> Yes, thank you!
>
> > Gábor Boskovits <boskovits@gmail.com> writes:
> >
> > > Björn Höfling <bjoern.hoefling@bjoernhoefling.de> ezt írta (időpont:
> > > 2018. okt. 12., P, 20:01):
> > >>
> > >> On Fri, 12 Oct 2018 19:35:51 +0200
> > >> Gábor Boskovits <boskovits@gmail.com> wrote:
> > >>
> > >> > Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. okt.
> > >> > 12., P, 19:00):
> > >> > >
> > >> > > Hello guix,
> > >> > >
> > >> > > I've tracked down the javadoc timestamp problem.
> > >> > > There is a command line flag for javadoc (notimestamp), that
> > >> > > disables generating the comment in the docs that contains the
> > >> > > timestamp. Currently I see two ways forward:
> > >> > > 1. Track down the calls to javadoc, and add the flag to all calls.
> > >> > > 2. Write a simple patch to make javadoc behave as if notimestamp was
> > >> > > specified, whenever
> > >> > > SOURCE_DATE_EPOCH is defined.
> > >> > > I do not think, that the patch produced by 2 is upstreamable, but it
> > >> > > seems much less work. WDYT?
> > >> >
> > >> > Also we can simply turn off the timestamp generation
> > >> > unconditionally...
> > >>
> > >> Number 2 sounds good, and why not giving it a try to place it upstream?
> > >
> > > Ok, i will go for it, and try to get it upsreamed for jdk8 and jdk11.
> >
> > Be sure to check out the links Vagrant posted.  Specifically this one:
> >
> > https://bugs.debian.org/783938
> >
> > In that bug report, Samuel Thibault says: "Perhaps javadoc could be made
> > to use by default the SOURCE_DATE_EPOCH environment variable when it is
> > defined?"  There seems to be agreement that teaching javadoc to honor
> > the SOURCE_DATE_EPOCH environment variable would be preferable to
> > unconditionally disabling the timestamp behavior.
> >
>
> Yes, I've also came to that conclusion reading the discussion, andI
> have a came up with a patch.
> I'm testing it right now, will report back if I have the results.

The results are good. You can see the patch at
http://issues.guix.info/issue/33041.
However a new bug surfaced. I am now testing a patch related to that.
The patch here is in the langtools component, and each javadoc generated file is
having a timestamp. Now icedtea6 javadoc respects SOURCE_DATE_EPOCH.
The patch attached is based on staging, and is not intended for inclusion as is.

The new bug is in the corba component, the IDL-to-Java compiler embeds
a timestamp into
the documentation. I am working on a similar patch. Anyone has any
information on that?
If this is something new, who should we inform?

>
> > --
> > Chris

Re: Making javadoc reproducible

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

Hi Gabor,

> There is a command line flag for javadoc (notimestamp), that disables
> generating the comment in the docs that contains the timestamp.
> Currently I see two ways forward:
> 1. Track down the calls to javadoc, and add the flag to all calls.
> 2. Write a simple patch to make javadoc behave as if notimestamp was
> specified, whenever
> SOURCE_DATE_EPOCH is defined.
> I do not think, that the patch produced by 2 is upstreamable, but it
> seems much less work. WDYT?

I'd like option 2 more.  Also, I think it should be upstreamable.

Thanks for working on tihs!

I've meant to tackle this for quite some time but I didn't find how icedtea
gets the jdk sources in the first place (in order to patch it).

Putting a patch into the "patches" subdir blindly should work, but I
didn't try.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Making javadoc reproducible

[Mykola Nikishov]

Gábor Boskovits <boskovits@gmail.com> writes:

>> > Also we can simply turn off the timestamp generation
>> > unconditionally...
>>
>> Number 2 sounds good, and why not giving it a try to place it upstream?
>
> Ok, i will go for it, and try to get it upsreamed for jdk8 and jdk11.

Gábor, do you have anything to share?

-- 
Mykola

Libre/Free Java Software Developer
https://manandbytes.gitlab.io/

Re: Making javadoc reproducible

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

Hello,

Mykola Nikishov <mn@mn.com.ua> ezt írta (időpont: 2019. máj. 19., V, 12:23):

> Gábor Boskovits <boskovits@gmail.com> writes:
>
> >> > Also we can simply turn off the timestamp generation
> >> > unconditionally...
> >>
> >> Number 2 sounds good, and why not giving it a try to place it upstream?
> >
> > Ok, i will go for it, and try to get it upsreamed for jdk8 and jdk11.
>
> Gábor, do you have anything to share?
>
>
It's not upstreamed, but our jdk11 javadoc is reproducilble.
Current status is that all outputs except the jdk could be easily made
reproducible.
Jdk is hard to solve, but the jre and javadoc are ok on jdk11. I plan to
package jdk12 soon,
then do a cleanup on the bootstrap toolchain.

For hints on the problem around jdk see:
https://hg.openjdk.java.net/jdk/jdk12/file/06222165c35f/make/scripts/compare.sh
from line 160.

That is the last reproducibility issue we have in jdk :)

-- 
> Mykola
>
> Libre/Free Java Software Developer
> https://manandbytes.gitlab.io/
>
>
> Best regards,
g_bor

[-- Attachment #2: Type: text/html, Size: 1950 bytes --]