emacs reproducible builds part1 of 2 : eln

emacs reproducible builds part1 of 2 : eln

[Bernhard M. Wiedemann]

[-- Attachment #1.1: Type: text/plain, Size: 1364 bytes --]

Dear emacs devs,

I'm working on reproducible builds for openSUSE
and in that process found that our emacs-29.1
package varies across builds from 2 different issues

The diff is visible at
https://rb.zq1.de/compare.factory-20231231/diffs/emacs-compare.out

Here is a simple reproducer that works in our build env:

cd ~/rpmbuild/BUILD/emacs-29.1/native-lisp &&
  for i in $(seq 10) ; do
   ../src/emacs -batch \
   --eval "(batch-native-compile t)" ../lisp/term/x-win.el &&
   md5sum 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
  done | sort | uniq -c


That prints variations of
       6 0c749bdbef8d9fa7b1afc2f042caf45b 
29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
       2 3022ea8c675b69ac79005281fd824179 
29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
       2 57bc581456c38395e9ac6409bbcb0d6e 
29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln


while the same with setarch -R ../src/emacs
produces deterministic
      10 0c749bdbef8d9fa7b1afc2f042caf45b 
29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln

So somewhere in the batch-native-compile code is a place that adds 
ASLR-related non-determinism into the .eln file. From the diff, it 
probably is some ordering issue.

I'd appreciate pointers to where that non-determinism gets added and how 
to patch it out.

And once that is solved, there is still part2: pdmp

Ciao
Bernhard M.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: emacs reproducible builds part1 of 2 : eln

[Eli Zaretskii]

> Date: Thu, 11 Jan 2024 22:17:24 +0100
> From: "Bernhard M. Wiedemann" <bernhardout@lsmod.de>
> 
> I'm working on reproducible builds for openSUSE
> and in that process found that our emacs-29.1
> package varies across builds from 2 different issues
> 
> The diff is visible at
> https://rb.zq1.de/compare.factory-20231231/diffs/emacs-compare.out
> 
> Here is a simple reproducer that works in our build env:
> 
> cd ~/rpmbuild/BUILD/emacs-29.1/native-lisp &&
>   for i in $(seq 10) ; do
>    ../src/emacs -batch \
>    --eval "(batch-native-compile t)" ../lisp/term/x-win.el &&
>    md5sum 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
>   done | sort | uniq -c
> 
> 
> That prints variations of
>        6 0c749bdbef8d9fa7b1afc2f042caf45b 
> 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
>        2 3022ea8c675b69ac79005281fd824179 
> 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
>        2 57bc581456c38395e9ac6409bbcb0d6e 
> 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
> 
> 
> while the same with setarch -R ../src/emacs
> produces deterministic
>       10 0c749bdbef8d9fa7b1afc2f042caf45b 
> 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
> 
> So somewhere in the batch-native-compile code is a place that adds 
> ASLR-related non-determinism into the .eln file. From the diff, it 
> probably is some ordering issue.
> 
> I'd appreciate pointers to where that non-determinism gets added and how 
> to patch it out.

Andrea, can you perhaps help Bernhard understand where the differences
come from?

Re: emacs reproducible builds part1 of 2 : eln

[Simon Tournier]

Hi,

On Thu, 11 Jan 2024 at 22:17, "Bernhard M. Wiedemann" <bernhardout@lsmod.de> wrote:

> I'm working on reproducible builds for openSUSE
> and in that process found that our emacs-29.1
> package varies across builds from 2 different issues

Similar results with GNU Guix.  Let first build a first time, then let
build a second time (--check).  And let compare the two builds:

--8<---------------cut here---------------start------------->8---
$ guix build emacs --no-grafts
$ guix build emacs --no-grafts --check -K
guix build: error: derivation `/gnu/store/p73n8hvadw1q8m0k47xgvj0qpqlws09s-emacs-29.1.drv' may not be deterministic: output `/gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1' differs from `/gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1-check'

$ diff -qr --no-dereference /gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1 /gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1-check | wc -l
56

$ diff -qr --no-dereference /gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1 /gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1-check | grep eln | wc -l
53

$ diff -qr --no-dereference /gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1 /gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1-check | grep elc | wc -l
2

$ diff -qr --no-dereference /gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1 /gnu/store/ab5l8h98a56fb7r1r40iqjkxzv4smv1i-emacs-29.1-check | grep pdmp | wc -l
1
--8<---------------cut here---------------end--------------->8---

In addition to .eln files, two .elc files are also different:

    share/emacs/29.1/lisp/emacs-lisp/package.elc
    share/emacs/29.1/lisp/gnus/nnspool.elc

Well, any pointers for helping in removing this non-determinism would be
appreciated.


Cheers,
simon

Re: emacs reproducible builds part1 of 2 : eln

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 11 Jan 2024 22:17:24 +0100
>> From: "Bernhard M. Wiedemann" <bernhardout@lsmod.de>
>> 
>> I'm working on reproducible builds for openSUSE
>> and in that process found that our emacs-29.1
>> package varies across builds from 2 different issues
>> 
>> The diff is visible at
>> https://rb.zq1.de/compare.factory-20231231/diffs/emacs-compare.out
>> 
>> Here is a simple reproducer that works in our build env:
>> 
>> cd ~/rpmbuild/BUILD/emacs-29.1/native-lisp &&
>>   for i in $(seq 10) ; do
>>    ../src/emacs -batch \
>>    --eval "(batch-native-compile t)" ../lisp/term/x-win.el &&
>>    md5sum 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
>>   done | sort | uniq -c
>> 
>> 
>> That prints variations of
>>        6 0c749bdbef8d9fa7b1afc2f042caf45b 
>> 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
>>        2 3022ea8c675b69ac79005281fd824179 
>> 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
>>        2 57bc581456c38395e9ac6409bbcb0d6e 
>> 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
>> 
>> 
>> while the same with setarch -R ../src/emacs
>> produces deterministic
>>       10 0c749bdbef8d9fa7b1afc2f042caf45b 
>> 29.1-70b784e9/x-win-3c49581f-6ddb7e08.eln
>> 
>> So somewhere in the batch-native-compile code is a place that adds 
>> ASLR-related non-determinism into the .eln file. From the diff, it 
>> probably is some ordering issue.
>> 
>> I'd appreciate pointers to where that non-determinism gets added and how 
>> to patch it out.
>
> Andrea, can you perhaps help Bernhard understand where the differences
> come from?

Mmmh, from what I can see from the diff looks like to me in x-win we
have some circular object.  For some reason when we serialize (print) it
in order to store it into the eln the result is not reproducible with
ASLR (!?).

That's bizarre.

  Andrea

Re: emacs reproducible builds part1 of 2 : eln

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: "Bernhard M. Wiedemann" <bernhardout@lsmod.de>,  emacs-devel@gnu.org
> Date: Fri, 12 Jan 2024 11:19:58 -0500
> 
> Mmmh, from what I can see from the diff looks like to me in x-win we
> have some circular object.  For some reason when we serialize (print) it
> in order to store it into the eln the result is not reproducible with
> ASLR (!?).

I guess the way to find that circular object is to bisect x-win.el?
Or are there better ways?

Re: emacs reproducible builds part1 of 2 : eln

[Eli Zaretskii]

> From: Simon Tournier <zimon.toutoune@gmail.com>
> Date: Fri, 12 Jan 2024 17:14:57 +0100
> 
> In addition to .eln files, two .elc files are also different:
> 
>     share/emacs/29.1/lisp/emacs-lisp/package.elc
>     share/emacs/29.1/lisp/gnus/nnspool.elc
> 
> Well, any pointers for helping in removing this non-determinism would be
> appreciated.

Would you please help us help you by showing the differences?  Bonus
points for analyzing the byte codes that are different and telling us
why.

Thanks.

Re: emacs reproducible builds part1 of 2 : eln

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: "Bernhard M. Wiedemann" <bernhardout@lsmod.de>,  emacs-devel@gnu.org
>> Date: Fri, 12 Jan 2024 11:19:58 -0500
>> 
>> Mmmh, from what I can see from the diff looks like to me in x-win we
>> have some circular object.  For some reason when we serialize (print) it
>> in order to store it into the eln the result is not reproducible with
>> ASLR (!?).
>
> I guess the way to find that circular object is to bisect x-win.el?
> Or are there better ways?

I think an initial approach would be to set `native-comp-debug' to 2
during compilation and compare the outputted pseudo C for x-win.el during
various compilations.

It should be rather easy and should definitely give more information
(unfortunately I can't try it myself before next week).

  Andrea

Re: emacs reproducible builds part1 of 2 : eln

[Andrea Corallo]

Okay did some more investigation on this and I think I've a decent
understanding of the problem, this is not related to the printer but is
entirely on the native compiler.

When we compile 'vendor-specific-keysyms' the native compiler is doing
its value type propagation collecting all possible return values.

This function is returning various possible different lists.

We collect all possible return values iterating over the basic blocks of
the functions, these are stored in hash table so we walk order is I
believe *not* deterministic.  That's the source of the trouble because
these lists gets collected (and then stored into the function type)
inside the eln.

With other type of objects we normalize the list of possible returned
objects (ex symbols) sorting them, but how to do it with list?

Option 1 would be to ignore completly objects behind conses and just
propagating the type (cons).  IOW going from:

(subr-type (native-compile '(lambda (x)
                              (if x
                                  '(1 2)
                                '(2 3)))))

=> (function (t) (member (1 2) (2 3)))

to

=> (function (t) cons)

Option 2 is to sort those lists to keep the computed function type
stable over different compilation runs.

BTW IIUC CL goes for option 1 [1] but at the time I thought it was easy
(and nice as well) to support the full object behind the cons.  Was it a
bad idea?

Thanks

  Andrea

[1] <http://clhs.lisp.se/Body/t_member.htm>

Re: emacs reproducible builds part1 of 2 : eln

[Stefan Monnier]

> We collect all possible return values iterating over the basic blocks of
> the functions, these are stored in hash table so we walk order is I
> believe *not* deterministic.

The walk order is officially undefined, but in practice with the current
implementation of hash-tables it is fully determined by the order in
which elements have been added/removed (and unaffected by the hash
values).

So, I'm not sure it explains the phenomenon you're seeing (I haven't
seen the rest of this thread yet).


        Stefan

Re: emacs reproducible builds part1 of 2 : eln

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> We collect all possible return values iterating over the basic blocks of
>> the functions, these are stored in hash table so we walk order is I
>> believe *not* deterministic.
>
> The walk order is officially undefined, but in practice with the current
> implementation of hash-tables it is fully determined by the order in
> which elements have been added/removed (and unaffected by the hash
> values).
>
> So, I'm not sure it explains the phenomenon you're seeing (I haven't
> seen the rest of this thread yet).

Interesting, I imagined the walk order is defined but my question is
what about two hash table with the same content but created in two
different sessions?

Thanks

  Andrea

Re: emacs reproducible builds part1 of 2 : eln

[Stefan Monnier]

>> So, I'm not sure it explains the phenomenon you're seeing (I haven't
>> seen the rest of this thread yet).
>
> Interesting, I imagined the walk order is defined but my question is
> what about two hash table with the same content but created in two
> different sessions?

Depends how they were filled: if they were filles by the same sequence
of operations, then they should have the same walk-order.
If not, then all bets are off.


        Stefan

Re: emacs reproducible builds part1 of 2 : eln

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> So, I'm not sure it explains the phenomenon you're seeing (I haven't
>>> seen the rest of this thread yet).
>>
>> Interesting, I imagined the walk order is defined but my question is
>> what about two hash table with the same content but created in two
>> different sessions?
>
> Depends how they were filled: if they were filles by the same sequence
> of operations, then they should have the same walk-order.
> If not, then all bets are off.

Okay that's good news, I'll investigate more tomorrow, the case is
pretty well defined now so should be possible to understand exactly what
is going on.

Thanks

  Andrea

Re: emacs reproducible builds part1 of 2 : eln

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> So, I'm not sure it explains the phenomenon you're seeing (I haven't
>>>> seen the rest of this thread yet).
>>>
>>> Interesting, I imagined the walk order is defined but my question is
>>> what about two hash table with the same content but created in two
>>> different sessions?
>>
>> Depends how they were filled: if they were filles by the same sequence
>> of operations, then they should have the same walk-order.
>> If not, then all bets are off.
>
> Okay that's good news, I'll investigate more tomorrow, the case is
> pretty well defined now so should be possible to understand exactly what
> is going on.

Okay cool I pushed a fix in emacs29, with that installed we don't try to
sort anymore conses based on their 'sxhash-equal' value because this is
not stable over different Emacs sessions.  With the change installed the
build looks finally reproducible here.

Bernhard please let us know if this solves the problem on your side as
well.

Thanks

  Andrea

Re: emacs reproducible builds part1 of 2 : eln

[Stefan Monnier]

> Okay cool I pushed a fix in emacs29, with that installed we don't try to
> sort anymore conses based on their 'sxhash-equal' value because this is
> not stable over different Emacs sessions.  With the change installed the
> build looks finally reproducible here.

Maybe we should introduce an `sxhash-*` variant which does not
depend on object addresses.


        Stefan

Re: emacs reproducible builds part1 of 2 : eln

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Okay cool I pushed a fix in emacs29, with that installed we don't try to
>> sort anymore conses based on their 'sxhash-equal' value because this is
>> not stable over different Emacs sessions.  With the change installed the
>> build looks finally reproducible here.
>
> Maybe we should introduce an `sxhash-*` variant which does not
> depend on object addresses.

Yes, BTW ATM I think we could leave without 'sxhash-equal' interely in
the native compiler, also IIRC there are not many other users of this
function in the codebase.

Bests

  Andrea

Re: emacs reproducible builds part1 of 2 : eln

[Björn Bidar]

Hey,

I tried exactly the a few months ago last year.
The old thread about the topic might be helpful:
https://mail.gnu.org/archive/html/emacs-devel/2023-10/msg00127.html

Br,

Björn Bidar

Re: emacs reproducible builds part1 of 2 : eln

[Andrea Corallo]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Hey,
>
> I tried exactly the a few months ago last year.
> The old thread about the topic might be helpful:
> https://mail.gnu.org/archive/html/emacs-devel/2023-10/msg00127.html
>
> Br,
>
> Björn Bidar

Hi Björn,

it would be helpful if you could very that now the build is stable for
you as well.

Thanks

  Andrea

Re: emacs reproducible builds part1 of 2 : eln

[Bernhard M. Wiedemann]

On 11/02/2024 11.24, Andrea Corallo wrote:
> Andrea Corallo <acorallo@gnu.org> writes:
> 
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>>> So, I'm not sure it explains the phenomenon you're seeing (I haven't
>>>>> seen the rest of this thread yet).
>>>>
>>>> Interesting, I imagined the walk order is defined but my question is
>>>> what about two hash table with the same content but created in two
>>>> different sessions?
>>>
>>> Depends how they were filled: if they were filles by the same sequence
>>> of operations, then they should have the same walk-order.
>>> If not, then all bets are off.
>>
>> Okay that's good news, I'll investigate more tomorrow, the case is
>> pretty well defined now so should be possible to understand exactly what
>> is going on.
> 
> Okay cool I pushed a fix in emacs29, with that installed we don't try to
> sort anymore conses based on their 'sxhash-equal' value because this is
> not stable over different Emacs sessions.  With the change installed the
> build looks finally reproducible here.
> 
> Bernhard please let us know if this solves the problem on your side as
> well.

Thanks for the fix.
I had missed the email and only now got to retest.
I found your commit
614b244a7fa03fcb27d76757e14ef0fa895d6f23
is part of emacs-29.3 and the x-win.el compilation is now deterministic.

The bad news is that there are still 4 more non-deterministic .eln files.
I made a reproducer for one of them:

cd ~/rpmbuild/BUILD/emacs-29.3/native-lisp &&
  for i in $(seq 10) ; do
   ../src/emacs -batch \
   --eval "(batch-native-compile t)" ../lisp/international/utf7.el &&
   md5sum 29.*/utf7.eln
  done | sort | uniq -c

again with ASLR as factor in the result.

The varying files are
emacs/29.3/native-lisp/29.3-f8a6a23e/el-72f9fa70-1a554ff6.eln
emacs/29.3/native-lisp/29.3-f8a6a23e/ox-9aa46d10-040d281c.eln
emacs/29.3/native-lisp/29.3-f8a6a23e/utf7-8aab9346-0cfd2c82.eln
emacs/29.3/native-lisp/29.3-fc431f4a/utf7-8aab9346-0cfd2c82.eln

Unfortunately, the el.el and ox.el don't reproduce non-determinism the 
same way.


And finally there is still part2 left:
.pdmp files have variations from some unknown source.


Ciao
Bernhard M.

Re: emacs reproducible builds part1 of 2 : eln

[Eli Zaretskii]

> Date: Wed, 22 May 2024 08:06:05 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>  Mattias Engdegård <mattias.engdegard@gmail.com>
> From: "Bernhard M. Wiedemann" <bernhardout@lsmod.de>
> 
> The varying files are
> emacs/29.3/native-lisp/29.3-f8a6a23e/el-72f9fa70-1a554ff6.eln

Please help us by telling which el.el gave birth to this one.  There
are 2 of them in the tree.

> emacs/29.3/native-lisp/29.3-f8a6a23e/utf7-8aab9346-0cfd2c82.eln
> emacs/29.3/native-lisp/29.3-fc431f4a/utf7-8aab9346-0cfd2c82.eln

This is the same file, just built for 2 different builds of Emacs.  So
there are only 3 problematic files, not 4.