Timestamps in tarballs created by 'patch-and-repack'

Timestamps in tarballs created by 'patch-and-repack'

[Mark H Weaver]

Hello Guix,

One of the MIPS build slaves has a malfunctioning clock, and sometimes I
discover that the clock is several days behind or ahead.  In the past
I've been able to simply reset the clock and restart builds, but now I
have a more serious problem:

Patched tarballs created by 'patch-and-repack' were built on a build
slave with the clock set about 8 days ahead.  Those tarballs are now in
Hydra's store, and include timestamps in the tar file that are set so
far ahead that attempts to build them are failing.  I'm currently
attempting to find and delete the bad tarballs from hydra's store.

However, this raises a deeper problem: all of the outputs of
'patch-and-repack' contain non-deterministic timestamps, and these
timestamps can cause problems with future builds.

Would it be sufficient to simply zero out all of the timestamps before
repacking?

     Mark

Re: Timestamps in tarballs created by 'patch-and-repack'

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Hello Guix,
>
> One of the MIPS build slaves has a malfunctioning clock, and sometimes I
> discover that the clock is several days behind or ahead.  In the past
> I've been able to simply reset the clock and restart builds, but now I
> have a more serious problem:
>
> Patched tarballs created by 'patch-and-repack' were built on a build
> slave with the clock set about 8 days ahead.  Those tarballs are now in
> Hydra's store, and include timestamps in the tar file that are set so
> far ahead that attempts to build them are failing.  I'm currently
> attempting to find and delete the bad tarballs from hydra's store.

I ran into difficulties doing this, and anyway I'm not sure how to
achieve confidence that I've found all of the bad archives, which might
result is subtle breakage.  So now I'm more inclined to just fix the
problem in 'patch-and-repack'.

> However, this raises a deeper problem: all of the outputs of
> 'patch-and-repack' contain non-deterministic timestamps, and these
> timestamps can cause problems with future builds.
>
> Would it be sufficient to simply zero out all of the timestamps before
> repacking?

I looked into this a bit more.  In addition to timestamps, the other
impure bits getting into tar files are the names and numeric ids of the
owner and groups.  Section 4.3.1 (Overriding File Metadata) of the GNU
tar manual describes how to override these when creating an archive.

My guess is that the following options would be sufficient to make the
generated tar archives deterministic:

  --mtime=@0 --owner=root:0 --group=root:0

I'm going to try this on my local machine and see how it goes...

      Mark

Re: Timestamps in tarballs created by 'patch-and-repack'

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> However, this raises a deeper problem: all of the outputs of
>> 'patch-and-repack' contain non-deterministic timestamps, and these
>> timestamps can cause problems with future builds.
>>
>> Would it be sufficient to simply zero out all of the timestamps before
>> repacking?
>
> I looked into this a bit more.  In addition to timestamps, the other
> impure bits getting into tar files are the names and numeric ids of the
> owner and groups.  Section 4.3.1 (Overriding File Metadata) of the GNU
> tar manual describes how to override these when creating an archive.
>
> My guess is that the following options would be sufficient to make the
> generated tar archives deterministic:
>
>   --mtime=@0 --owner=root:0 --group=root:0

This seems to work well, so I've added these flags when creating
tarballs in commits 2e9511dfbdb5ddd78c2f205c4ca7fc23738d79f8 and
c09e6a5f5e2a77beff89d68069f3037c1b6310e5 in core-updates.

      Mark

Re: Timestamps in tarballs created by 'patch-and-repack'

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> However, this raises a deeper problem: all of the outputs of
>>> 'patch-and-repack' contain non-deterministic timestamps, and these
>>> timestamps can cause problems with future builds.
>>>
>>> Would it be sufficient to simply zero out all of the timestamps before
>>> repacking?
>>
>> I looked into this a bit more.  In addition to timestamps, the other
>> impure bits getting into tar files are the names and numeric ids of the
>> owner and groups.  Section 4.3.1 (Overriding File Metadata) of the GNU
>> tar manual describes how to override these when creating an archive.
>>
>> My guess is that the following options would be sufficient to make the
>> generated tar archives deterministic:
>>
>>   --mtime=@0 --owner=root:0 --group=root:0
>
> This seems to work well, so I've added these flags when creating
> tarballs in commits 2e9511dfbdb5ddd78c2f205c4ca7fc23738d79f8 and
> c09e6a5f5e2a77beff89d68069f3037c1b6310e5 in core-updates.

That sounds like the right thing (it’s also what is used in
gnu/system/install.scm for the binary tarball.)

Thanks for noticing and fixing it!

Ludo’.