Emacs is not reproducible

Emacs is not reproducible

[Bone Baboon]

I am using the Guix operating system.  It provides tools to verify if a software package is reproducible. <https://guix.gnu.org/en/manual/en/html_node/On-Trusting-Binaries.html#On-Trusting-Binaries> 

For details on how Emacs 27.2 is not reproducible see <https://issues.guix.gnu.org/issue/35085#7>.  Previous versions of Emacs are also not reproducible as reported in that thread.

For information on why reproducible software is important see <https://reproducible-builds.org/>.

For information on how to make software reproducible see <https://reproducible-builds.org/docs/>.

Is there currently any work underway to make Emacs reproducible?

Re: Emacs is not reproducible

[Alan Mackenzie]

Hello, Bone.

On Mon, May 17, 2021 at 11:28:18 -0400, Bone Baboon wrote:
> I am using the Guix operating system.  It provides tools to verify if
> a software package is reproducible.

What does it even mean to say a software package is "reproducible"?

> <https://guix.gnu.org/en/manual/en/html_node/On-Trusting-Binaries.html#On-Trusting-Binaries> 

> For details on how Emacs 27.2 is not reproducible see
> <https://issues.guix.gnu.org/issue/35085#7>.  Previous versions of
> Emacs are also not reproducible as reported in that thread.

> For information on why reproducible software is important see
> <https://reproducible-builds.org/>.

> For information on how to make software reproducible see
> <https://reproducible-builds.org/docs/>.

> Is there currently any work underway to make Emacs reproducible?

Without knowing what it means, I can't really say, but I suspect not.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Emacs is not reproducible

[Eli Zaretskii]

> From: Bone Baboon <bone.baboon@disroot.org>
> Date: Mon, 17 May 2021 11:28:18 -0400
> 
> I am using the Guix operating system.  It provides tools to verify if a software package is reproducible. <https://guix.gnu.org/en/manual/en/html_node/On-Trusting-Binaries.html#On-Trusting-Binaries> 
> 
> For details on how Emacs 27.2 is not reproducible see <https://issues.guix.gnu.org/issue/35085#7>.  Previous versions of Emacs are also not reproducible as reported in that thread.
> 
> For information on why reproducible software is important see <https://reproducible-builds.org/>.
> 
> For information on how to make software reproducible see <https://reproducible-builds.org/docs/>.
> 
> Is there currently any work underway to make Emacs reproducible?

We are aware of the issues, and solve them as it becomes possible,
yes.

Re: Emacs is not reproducible

[Achim Gratz]

Alan Mackenzie writes:
> On Mon, May 17, 2021 at 11:28:18 -0400, Bone Baboon wrote:
>> I am using the Guix operating system.  It provides tools to verify if
>> a software package is reproducible.
>
> What does it even mean to say a software package is "reproducible"?

It means this: you can build it multiple times on the same or even on
different systems (which need to enable the same configuration of
course) and you get exactly identical build artefacts out of each such
build.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: Emacs is not reproducible

[Stefan Monnier]

> Is there currently any work underway to make Emacs reproducible?

There has been changes made in the past to fix some problems.

I have a pending patch in bug#46502 (see below) which aims to fix some
more of those problems, but still haven't heard confirmation that it
helps.


        Stefan


diff --git a/lisp/emacs-lisp/byte-opt.el b/lisp/emacs-lisp/byte-opt.el
index e0feb95a46..06c925b7bf 100644
--- a/lisp/emacs-lisp/byte-opt.el
+++ b/lisp/emacs-lisp/byte-opt.el
@@ -268,32 +269,16 @@ byte-compile-inline-expand
        ;; The byte-code will be really inlined in byte-compile-unfold-bcf.
        `(,fn ,@(cdr form)))
       ((or `(lambda . ,_) `(closure . ,_))
-       (if (not (or (eq fn localfn)     ;From the same file => same mode.
-                    (eq (car fn)        ;Same mode.
-                        (if lexical-binding 'closure 'lambda))))
-           ;; While byte-compile-unfold-bcf can inline dynbind byte-code into
-           ;; letbind byte-code (or any other combination for that matter), we
-           ;; can only inline dynbind source into dynbind source or letbind
-           ;; source into letbind source.
-           (progn
-             ;; We can of course byte-compile the inlined function
-             ;; first, and then inline its byte-code.
-             (byte-compile name)
-             `(,(symbol-function name) ,@(cdr form)))
-         (let ((newfn (if (eq fn localfn)
-                          ;; If `fn' is from the same file, it has already
-                          ;; been preprocessed!
-                          `(function ,fn)
-                        ;; Try and process it "in its original environment".
-                        (let ((byte-compile-bound-variables nil))
-                          (byte-compile-preprocess
-                           (byte-compile--reify-function fn))))))
-           (if (eq (car-safe newfn) 'function)
-               (macroexp--unfold-lambda `(,(cadr newfn) ,@(cdr form)))
-             ;; This can happen because of macroexp-warn-and-return &co.
-             (byte-compile-warn
-              "Inlining closure %S failed" name)
-             form))))
+       (if (eq fn localfn)     ;From the same file => same mode.
+           (macroexp--unfold-lambda `(,fn ,@(cdr form)))
+         ;; While byte-compile-unfold-bcf can inline dynbind byte-code into
+         ;; letbind byte-code (or any other combination for that matter), we
+         ;; can only inline dynbind source into dynbind source or letbind
+         ;; source into letbind source.
+         ;; We can of course byte-compile the inlined function
+         ;; first, and then inline its byte-code.
+         (byte-compile name)
+         `(,(symbol-function name) ,@(cdr form))))
 
       (_ ;; Give up on inlining.
        form))))

Re: Emacs is not reproducible

[Stefan Monnier]

>> Is there currently any work underway to make Emacs reproducible?
> There has been changes made in the past to fix some problems.
> I have a pending patch in bug#46502 (see below) which aims to fix some
> more of those problems, but still haven't heard confirmation that it
> helps.

This is quite odd: both in bug#46502 and here again I don't hear
anything back from my proposal (not even some complaint that it doesn't
apply or it breaks something), even tho all I'm asking if for someone to
try and confirm whether it makes the build more (or less?) reproducible,
which I thought should be reasonably easy to do.

What am I missing?


        Stefan

Re: Emacs is not reproducible

[Glenn Morris]

Stefan Monnier wrote:

>> There has been changes made in the past to fix some problems.
>> I have a pending patch in bug#46502 (see below) which aims to fix some
>> more of those problems, but still haven't heard confirmation that it
>> helps.

I did 10 builds with -j8; 5 without your patch and 5 with.
The builds without your patch had changes in 12, 10, 9, 11 elc files.
The builds with your patch had changes in 4, 4, 7, 4 elc files.

Mostly tramp.elc, ox-odt.elc, ox.elc, cc-mode.elc; sometimes in
eieio files. At least some of these seem to be https://debbugs.gnu.org/34322

So it seems better in terms of less elc variation.
It causes all tests in test/lisp/cedet/semantic-utest-ia.el to fail.

> What am I missing?

You forgot to make a grammar mistake, or propose changing the default
value of a user option. That's how you get comments on Emacs patches.

Re: Emacs is not reproducible

[Stefan Monnier]

Glenn Morris [2021-05-23 19:52:08] wrote:
> Stefan Monnier wrote:
>>> There has been changes made in the past to fix some problems.
>>> I have a pending patch in bug#46502 (see below) which aims to fix some
>>> more of those problems, but still haven't heard confirmation that it
>>> helps.
> I did 10 builds with -j8; 5 without your patch and 5 with.
> The builds without your patch had changes in 12, 10, 9, 11 elc files.
> The builds with your patch had changes in 4, 4, 7, 4 elc files.

Thanks for confirming what I was seeing.  I pushed the patch to `master`.

> Mostly tramp.elc, ox-odt.elc, ox.elc, cc-mode.elc; sometimes in
> eieio files. At least some of these seem to be https://debbugs.gnu.org/34322

The mystery for those remains,

> So it seems better in terms of less elc variation.
> It causes all tests in test/lisp/cedet/semantic-utest-ia.el to fail.

I'm indeed looking at it,


        Stefan

Re: Emacs is not reproducible

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1579 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:


This may or may not be related --- but I reported a few months ago that
compiling emacspeak with make -j 4 resulted in unpredictable behavior --
I got rid of the parallel build and problems went away.

1. There were no warnings during the parallel build.

2. Things were "weird" in that some advice definitions were not active,
   reloading files fixed it.

I decided to keep life simple at that point and got rid of the parallel
build.
> Glenn Morris [2021-05-23 19:52:08] wrote:
>> Stefan Monnier wrote:
>>>> There has been changes made in the past to fix some problems.
>>>> I have a pending patch in bug#46502 (see below) which aims to fix some
>>>> more of those problems, but still haven't heard confirmation that it
>>>> helps.
>> I did 10 builds with -j8; 5 without your patch and 5 with.
>> The builds without your patch had changes in 12, 10, 9, 11 elc files.
>> The builds with your patch had changes in 4, 4, 7, 4 elc files.
>
> Thanks for confirming what I was seeing.  I pushed the patch to `master`.
>
>> Mostly tramp.elc, ox-odt.elc, ox.elc, cc-mode.elc; sometimes in
>> eieio files. At least some of these seem to be https://debbugs.gnu.org/34322
>
> The mystery for those remains,
>
>> So it seems better in terms of less elc variation.
>> It causes all tests in test/lisp/cedet/semantic-utest-ia.el to fail.
>
> I'm indeed looking at it,
>
>
>         Stefan
>
>

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Emacs is not reproducible

[Glenn Morris]

>> It causes all tests in test/lisp/cedet/semantic-utest-ia.el to fail.

FTR, I should have said "four", not "all".

Re: Emacs is not reproducible

[Basil L. Contovounesios]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Glenn Morris [2021-05-23 19:52:08] wrote:
>> Stefan Monnier wrote:
>>>> There has been changes made in the past to fix some problems.
>>>> I have a pending patch in bug#46502 (see below) which aims to fix some
>>>> more of those problems, but still haven't heard confirmation that it
>>>> helps.
>> I did 10 builds with -j8; 5 without your patch and 5 with.
>> The builds without your patch had changes in 12, 10, 9, 11 elc files.
>> The builds with your patch had changes in 4, 4, 7, 4 elc files.
>
> Thanks for confirming what I was seeing.  I pushed the patch to `master`.
>
>> So it seems better in terms of less elc variation.
>> It causes all tests in test/lisp/cedet/semantic-utest-ia.el to fail.
>
> I'm indeed looking at it,

FYI if those test failures prove too boring there are some new and
exciting bootstrap warnings to play with too :).

In end of data:
calc/calc.el:3126:27: Warning: the function ‘math-compare’ is not known to be
    defined.
calc/calc.el:2805:27: Warning: the function ‘math-zerop’ is not known to be
    defined.
In tramp-handle-access-file:
net/tramp.el:3238:43: Warning: attempt to inline ‘tramp-error’ before it was
    defined
net/tramp.el:3238:43: Warning: attempt to inline ‘tramp-error’ before it was
    defined

Thanks,

-- 
Basil

Re: Emacs is not reproducible

[Stefan Monnier]

Glenn Morris [2021-05-25 14:03:52] wrote:
>>> It causes all tests in test/lisp/cedet/semantic-utest-ia.el to fail.
> FTR, I should have said "four", not "all".

OK, I think I found the culprit!

When `byte-compile-inline-expand` calls `byte-compile`, the nested
compilation may end up using the outer value of
`byte-optimize--lexvars` resulting in plain-old broken code.

In the case at hand, during compilation of
`semantic-analyze-find-tag-sequence-default` the call to
`semantic-tag-of-class-p` gets miscompiled because
`byte-optimize--lexvars` says that `tag` is known to be nil and that
info gets propagated to the body of `semantic-tag-of-class-p` (where
`tag` refers to another variable).

Matthias, could you try and see where byte-opt.el should let-bind
`byte-optimize--lexvars` to nil to avoid this problem?
I guess we could add it to `byte-compile-top-level`, but it would be
better if we could do it somewhere in `byte-opt.el`.

See sample session below for "the evidence".


        Stefan


% rm lisp/cedet/semantic/analyze.elc lisp/cedet/semantic/tag.elc; make -C lisp cedet/semantic/analyze.elc
make : on entre dans le répertoire « /home/monnier/src/emacs/trunk/lisp »
  ELC      cedet/semantic/analyze.elc
Byte-compiling for inlinling: semantic-tag-p
Byte-compiling for inlinling: semantic-tag-class
Source = (closure (t) (tag) "Return the class of TAG.
This is a symbol like `variable', `function', or `type'.
There is no limit to the symbols that may represent the class of a tag.
Each parser generates tags with classes defined by it.

For functional languages, typical tag classes are:

@table @code
@item type
Data types, named map for a memory block.
@item function
A function or method, or named execution location.
@item variable
A variable, or named storage for data.
@item include
Statement that represents a file from which more tags can be found.
@item package
Statement that declares this file's package name.
@item code
Code that has not name or binding to any other symbol, such as in a script.
@end table
" (nth 1 tag))
Byte-code = #[257 "\211A@\207" [] 2 "Return the class of TAG.
This is a symbol like `variable', `function', or `type'.
There is no limit to the symbols that may represent the class of a tag.
Each parser generates tags with classes defined by it.

For functional languages, typical tag classes are:

@table @code
@item type
Data types, named map for a memory block.
@item function
A function or method, or named execution location.
@item variable
A variable, or named storage for data.
@item include
Statement that represents a file from which more tags can be found.
@item package
Statement that declares this file's package name.
@item code
Code that has not name or binding to any other symbol, such as in a script.
@end table

(fn TAG)"]
lexvars = nil

Disass:
byte code:
  doc:  Return the class of TAG. ...
  args: (arg1)
0       dup       
1       cdr       
2       car       
3       return    

Byte-compiling for inlinling: semantic-tag-type
Byte-compiling for inlinling: semantic-tag-get-attribute
Byte-compiling for inlinling: semantic-tag-attributes
Byte-compiling for inlinling: semantic-tag-of-class-p
Source = (closure (t) (tag class) "Return non-nil if class of TAG is CLASS." (eq (semantic-tag-class tag) class))
Byte-code = #[514 "\211A@\262=\207" [] 4 "Return non-nil if class of TAG is CLASS.

(fn TAG CLASS)"]
lexvars = ((tagclass nil) (miniscope nil) (fname nil nil) (tagtype nil nil) (tag nil nil) (tmp t) (s nil))

Disass:
byte code:
  doc:  Return non-nil if class of TAG is CLASS. ...
  args: (arg1 arg2)
0       stack-ref 1
1       dup       
2       cdr       
3       car       
4       stack-set 1
6       stack-ref 1
7       eq        
8       return    

Byte-compiling for inlinling: semantic-tag-name
Byte-compiling for inlinling: semantic-tag
Byte-compiling for inlinling: semantic-tag-type-members
Byte-compiling for inlinling: semantic-tag-function-arguments

Re: Emacs is not reproducible

[Mattias Engdegård]

26 maj 2021 kl. 22.58 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> Matthias, could you try and see where byte-opt.el should let-bind
> `byte-optimize--lexvars` to nil to avoid this problem?
> I guess we could add it to `byte-compile-top-level`, but it would be
> better if we could do it somewhere in `byte-opt.el`.

Oh dear, I can't really find any good place to bind the variable either, but ended up doing it in `byte-compile-inline-expand` on the grounds that it's the only place we re-enter the compiler from the optimiser.

It would probably be better if we had a single entry for `byte-optimize-form` that wasn't used recursively; then it could be done there.

Anyway, feel free to improve. Writing the test was what took most of the time anyway!

Re: Emacs is not reproducible

[Stefan Monnier]

> FYI if those test failures prove too boring there are some new and
> exciting bootstrap warnings to play with too :).
>
> In end of data:
> calc/calc.el:3126:27: Warning: the function ‘math-compare’ is not known to be
>     defined.
> calc/calc.el:2805:27: Warning: the function ‘math-zerop’ is not known to be
>     defined.
> In tramp-handle-access-file:
> net/tramp.el:3238:43: Warning: attempt to inline ‘tramp-error’ before it was
>     defined
> net/tramp.el:3238:43: Warning: attempt to inline ‘tramp-error’ before it was
>     defined

I believe these should be fixed now.


        Stefan

Re: Emacs is not reproducible

[Stefan Monnier]

>> Matthias, could you try and see where byte-opt.el should let-bind
>> `byte-optimize--lexvars` to nil to avoid this problem?
>> I guess we could add it to `byte-compile-top-level`, but it would be
>> better if we could do it somewhere in `byte-opt.el`.
> Oh dear, I can't really find any good place to bind the variable either, but
> ended up doing it in `byte-compile-inline-expand` on the grounds that it's
> the only place we re-enter the compiler from the optimiser.

Fair enough.

> It would probably be better if we had a single entry for
> `byte-optimize-form` that wasn't used recursively; then it could be
> done there.

Indeed, that was also my conclusion.

> Anyway, feel free to improve. Writing the test was what took most of the time anyway!

Thanks for the test,


        Stefan

Re: Emacs is not reproducible

[Basil L. Contovounesios]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> FYI if those test failures prove too boring there are some new and
>> exciting bootstrap warnings to play with too :).
>>
>> In end of data:
>> calc/calc.el:3126:27: Warning: the function ‘math-compare’ is not known to be
>>     defined.
>> calc/calc.el:2805:27: Warning: the function ‘math-zerop’ is not known to be
>>     defined.
>> In tramp-handle-access-file:
>> net/tramp.el:3238:43: Warning: attempt to inline ‘tramp-error’ before it was
>>     defined
>> net/tramp.el:3238:43: Warning: attempt to inline ‘tramp-error’ before it was
>>     defined
>
> I believe these should be fixed now.

All but the tramp-error one (at least with sufficiently parallel
bootstrap)!

Thanks,

-- 
Basil

Warning in tramp.el (was: Emacs is not reproducible)

[Stefan Monnier]

Basil L. Contovounesios [2021-05-28 10:00:39] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> FYI if those test failures prove too boring there are some new and
>>> exciting bootstrap warnings to play with too :).
>>>
>>> In end of data:
>>> calc/calc.el:3126:27: Warning: the function ‘math-compare’ is not known to be
>>>     defined.
>>> calc/calc.el:2805:27: Warning: the function ‘math-zerop’ is not known to be
>>>     defined.
>>> In tramp-handle-access-file:
>>> net/tramp.el:3238:43: Warning: attempt to inline ‘tramp-error’ before it was
>>>     defined
>>> net/tramp.el:3238:43: Warning: attempt to inline ‘tramp-error’ before it was
>>>     defined
>>
>> I believe these should be fixed now.
>
> All but the tramp-error one (at least with sufficiently parallel
> bootstrap)!

This one is a bit more tricky:

`tramp-handle-access-file` (in `tramp.el`) calls the inlinable
`tramp-compat-file-missing` (from `tramp-compat.el`) which calls the
inlinable `tramp-error` (from `tramp.el`).

If `tramp-compat.el` is compiled first, we don't get any warning, but
`tramp-error` is not inlined because `tramp-compat.el` doesn't require
`tramp` (instead it `declare-function` on `tramp-error`).
This case should arguably also signal a warning, because we fail to
inline the function, and the only good fix for that would be to
change the dependencies such that `tramp-compat.el` requires the file in
which `tramp-error` is defined (which may require moving `tramp-error`
to some other file to avoid circular dependencies).

OTOH if `tramp-compat.el` is not yet compiled when we compile
`tramp.el`, we get the above warning.  In that case, we *could* actually
do the right thing and inline `tramp-error` (and the old inlining code
got that right), but the info about `tramp-error` is kept inside
`byte-compile-function-environment` at that point, and we don't pass
that down to the recursive `byte-compile` (and to some extent for good
reason: the inlined function should be compiled in "its" environment
rather than in the environment of the caller).

So we could try and fix this warning in the byte-compiler, but it
could require non-trivial (or ugly) changes to the compiler.

I think the better fix is to change the Tramp code to avoid this
circular dependency.  This will not only avoid the warning but it will
also make the inlining work in both cases (and avoid the need for
`declare-function` here).


        Stefan

Re: Warning in tramp.el

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

> `tramp-handle-access-file` (in `tramp.el`) calls the inlinable
> `tramp-compat-file-missing` (from `tramp-compat.el`) which calls the
> inlinable `tramp-error` (from `tramp.el`).
>
> If `tramp-compat.el` is compiled first, we don't get any warning, but
> `tramp-error` is not inlined because `tramp-compat.el` doesn't require
> `tramp` (instead it `declare-function` on `tramp-error`).
> This case should arguably also signal a warning, because we fail to
> inline the function, and the only good fix for that would be to
> change the dependencies such that `tramp-compat.el` requires the file in
> which `tramp-error` is defined (which may require moving `tramp-error`
> to some other file to avoid circular dependencies).
>
> OTOH if `tramp-compat.el` is not yet compiled when we compile
> `tramp.el`, we get the above warning.  In that case, we *could* actually
> do the right thing and inline `tramp-error` (and the old inlining code
> got that right), but the info about `tramp-error` is kept inside
> `byte-compile-function-environment` at that point, and we don't pass
> that down to the recursive `byte-compile` (and to some extent for good
> reason: the inlined function should be compiled in "its" environment
> rather than in the environment of the caller).
>
> So we could try and fix this warning in the byte-compiler, but it
> could require non-trivial (or ugly) changes to the compiler.
>
> I think the better fix is to change the Tramp code to avoid this
> circular dependency.  This will not only avoid the warning but it will
> also make the inlining work in both cases (and avoid the need for
> `declare-function` here).

I've converted tramp-error from defsubst to defun. Unlike other Tramp
message functions, there's no reason for it to be a defsubst.

>         Stefan

Best regards, Michael.

Re: Warning in tramp.el

[Stefan Monnier]

> I've converted tramp-error from defsubst to defun. Unlike other Tramp
> message functions, there's no reason for it to be a defsubst.

Ah, indeed, that's an even better way to solve this, thanks,


        Stefan