[bug#70169] [PATCH 0/7] Reproducible `make dist' tarball in defiance of Autotools and Gettext

[Janneke Nieuwenhuizen <janneke@gnu.org>]

Ludovic Courtès writes:

Hello,

> Janneke Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> The recent XZ-utils <https://www.openwall.com/lists/oss-security/2024/03/29/4>
>> debacle inspired me to resurrect and finish my patch set for creating a
>> reproducible source tarball for Guix, i.e., finally have `make dist' be
>> reproducible (when run from Git).  I've been using a version of these patches
>> in simpler projects for some years now and stole one from Timothy Samplet's
>> Gash project.
>
> Yay, kudos to you and Timothy!

\o/

>> Autotools and Gettext still make it harder than necessary to do reproducible
>> (responsible?) computing, which is especially sad given the fact that the
>> Reproducible Builds project recently had their 10th birthday
>> <https://reproducible-builds.org/_lfs/presentations/2023-05-27-R-B-the-first-10-years/#/>.
>>
>> Gettext tooling embeds timestamps found in the file-system, fails to respect
>> SOURCE_DATE_EPOCH, and lacks options like `--pot-creation-date' so that we
>> have to resort to SED to fixup.  The caching of all sorts of information, in
>> separate build stages, also doesn't help.
>
> Sadness indeed.  Hopefully things will improve in the coming weeks, now
> that there’s an impetus.

Yes, that would be nice.  With more people joining the effort, it could
be fixed brilliantly, in no time :)

>> To create a reproducible source tarball, having a reproducible build
>> environment is a prerequitite, so this would have to be recorded too.
>> Using this patch set, I created a tarball doing something like
>>
>> guix pull --commit=1dbe492b993a7629df3b35146ce0272b52913776
>> guix shell
>> bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc && make dist
>> guix hash guix-1.3.0.57425-80a228.tar.gz
>> 0mk59ay5k2dxmjni9fx4i8qyfhvlgxbhqzsjpg2pbw381nskkxbj
>
> I applied the whole series on top of
> df64d48e6f9f648044aa5279c045b8d6f7bee604 (the ‘base-commit’ at the
> bottom of your message).  Thus I got the same content as you but with a
> different commit ID.

Yeah..., that's why I pushed `wip-tarball'.  We even look at the
committer's timestamp (not author, as that could be quite old).

> “make dist” gave me guix-1.3.0.57425-9f4a4a.tar.gz.  The name indeed
> corresponds to the tip of my tree:

[..]

> But as a result, I get a different hash, and since the directory in the
> tarball has a different name, ‘diffoscope’ isn’t very helpful.
>
> There’s at least one relevant difference in the gzip metadata:
>
> --- guix-1.3.0.57425-9f4a4a.tar.gz
> +++ /tmp/guix-1.3.0.57425-80a228.tar.gz
> ├── filetype from file(1)
> │ @@ -1 +1 @@
> │ -gzip compressed data, from Unix, original size modulo 2^32 208138240 gzip compressed data, rese
> rved method, ASCII, extra field, encrypted, from FAT filesystem (MS-DOS, OS/2, NT)
> │ +gzip compressed data, from Unix, original size modulo 2^32 222504960 gzip compressed data, rese
> rved method, ASCII, has CRC, was "", has comment, encrypted, from FAT filesystem (MS-DOS, OS/2, NT
>
>
> (Your tarball has a CRC and comment, mine doesn’t.)

I believe this is a red herring.  I saw this all day whenever one file
had a difference...

> Maybe we’ll have to iterate once you’ve pushed a first version, so we
> can truly build the same thing.  Or we should push the branch somewhere

I boldly pushed `origin/wip-tarball', you may try that :)

Be sure to git diff between your tip and origin/wip-tarball to ascertain
I didn't place easter eggs (just kidding).

> (or use the one from <https://data.qa.guix.gnu.org/> once it’s been
> created).

Thanks a lot for trying!

Greetings,
Janneke