bug#34322: reproducibility: absolute file name in tramp.elc

bug#34322: reproducibility: absolute file name in tramp.elc

[Glenn Morris]

Package: emacs
Version: 26.1.91
Severity: minor

Perhaps the same issue as https://debbugs.gnu.org/34321 .
The definition of tramp-lookup-syntax in tramp.elc contains the absolute
file name of the build directory, making the file non-reproducible.
Eg in the 26.1.91 pretest tarfile, it contains
"/home/nico/work/emacs-26/lisp/net/tramp-compat.elc"

bug#34322: reproducibility: absolute file name in tramp.elc

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> Perhaps the same issue as https://debbugs.gnu.org/34321 .
> The definition of tramp-lookup-syntax in tramp.elc contains the absolute
> file name of the build directory, making the file non-reproducible.
> Eg in the 26.1.91 pretest tarfile, it contains
> "/home/nico/work/emacs-26/lisp/net/tramp-compat.elc"

Don't know how to fix it. For the records, in the emacs-26 branch,
tramp.elc contains

--8<---------------cut here---------------start------------->8---
(defalias 'tramp-lookup-syntax #[257 "\301 \236A\206\f\302\303\b\"\207" [tramp-syntax #[0 "\b\301\267\202\n\302\207\303\207\b\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 8)) default separate] 2 ("/usr/local/src/emacs-26/lisp/net/tramp-compat.elc" . 7408)] error "Wrong `tramp-syntax' %s"] 4 (#$ . 25891)])
--8<---------------cut here---------------end--------------->8---

OTOH, in the Tramp git repository, branch tramp-2-3-stable, tramp.elc contains

--8<---------------cut here---------------start------------->8---
(defalias 'tramp-lookup-syntax #[257 "\b\301\267\202

bug#34322: reproducibility: absolute file name in tramp.elc

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Tue, 05 Feb 2019 10:00:15 +0100
> Cc: 34322@debbugs.gnu.org
> 
> Don't know how to fix it. For the records, in the emacs-26 branch,
> tramp.elc contains
> 
> --8<---------------cut here---------------start------------->8---
> (defalias 'tramp-lookup-syntax #[257 "\301 \236A\206\f\302\303\b\"\207" [tramp-syntax #[0 "\b\301\267\202\n\302\207\303\207\b\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 8)) default separate] 2 ("/usr/local/src/emacs-26/lisp/net/tramp-compat.elc" . 7408)] error "Wrong `tramp-syntax' %s"] 4 (#$ . 25891)])
> --8<---------------cut here---------------end--------------->8---
> 
> OTOH, in the Tramp git repository, branch tramp-2-3-stable, tramp.elc contains
> 
> --8<---------------cut here---------------start------------->8---
> (defalias 'tramp-lookup-syntax #[257 "\b\301\267\202 \302\202 \303\202 \b\236A\206 \304\305\b\"\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10)) default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
> --8<---------------cut here---------------end--------------->8---
> 
> I have no idea where the absolute file name in the former comes
> from.

I tried to explain what it means in another message.

> Both repositories use the same Emacs 26.1.91 binary for the build.

I think whether there is or isn't such a reference depends on whether
tramp-compat.elc exists when you byte-compile tramp.c.  Try removing
the former or renaming it, and you get the contents you have in the
second excerpt.  So perhaps whether the reference exists or not
depends on the order of the compilation, and maybe also on how many
compilation jobs in parallel are run when boostrapping Emacs.

bug#34322: reproducibility: absolute file name in tramp.elc

[Stefan Monnier]

>> --8<---------------cut here---------------start------------->8---
>> (defalias 'tramp-lookup-syntax #[257 "\301 \236A\206\f\302\303\b\"\207"
>> [tramp-syntax #[0 "\b\301\267\202\n\302\207\303\207\b\207" [tramp-syntax
>> #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125
>> purecopy t data (ftp 6 sep 8)) default separate]
>> 2 ("/usr/local/src/emacs-26/lisp/net/tramp-compat.elc" . 7408)] error
>> "Wrong `tramp-syntax' %s"] 4 (#$ . 25891)])
>> --8<---------------cut here---------------end--------------->8---

Hmm... looks like a bug in the implementation of defsubst: it works by
replacing references to tramp-compat-tramp-syntax with a copy of the
function's byte-code (this is the #[0 ...] object above, which includes
the source file name info), and later on those copied bytecode objects
are normally spliced into the surrounding bytecode (at which point the
extra source info is dropped), but here it seems like it hasn't been
spliced as it should.

>> --8<---------------cut here---------------start------------->8---
>> (defalias 'tramp-lookup-syntax #[257 "\b\301\267\202 \302\202 \303\202
>> \b\236A\206 \304\305\b\"\207" [tramp-syntax #s(hash-table size 2 test eq
>> rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10))
>> default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
>> --8<---------------cut here---------------end--------------->8---

Here you can see that there is no nested bytecode object (i.e. no
#[...] with the main #[...]) so the call has been correctly inlined.

> I think whether there is or isn't such a reference depends on whether
> tramp-compat.elc exists when you byte-compile tramp.c.  Try removing

I don't think we've changed anything significant in the implementation
of defsubst so indeed it's likely triggered by something like the order
of compilation.


        Stefan

bug#34322: reproducibility: absolute file name in tramp.elc

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> --8<---------------cut here---------------start------------->8---
>>> (defalias 'tramp-lookup-syntax #[257 "\b\301\267\202 \302\202 \303\202
>>> \b\236A\206 \304\305\b\"\207" [tramp-syntax #s(hash-table size 2 test eq
>>> rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10))
>>> default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
>>> --8<---------------cut here---------------end--------------->8---
>
> Here you can see that there is no nested bytecode object (i.e. no
> #[...] with the main #[...]) so the call has been correctly inlined.

Given, that the absolute file name is not needed in the bytecode, I'm
wondering why we insert it there. Couldn't we live without?

>         Stefan

Best regards, Michael.

bug#34322: reproducibility: absolute file name in tramp.elc

[Stefan Monnier]

>> Here you can see that there is no nested bytecode object (i.e. no
>> #[...] with the main #[...]) so the call has been correctly inlined.
> Given, that the absolute file name is not needed in the bytecode,

It's present in the in-memory bytecode object so that `C-h f` can jump
to the source upon demand, so it is needed.

> I'm wondering why we insert it there.

It's because we just insert the bytecode object wholesale (not a copy of
the object), so it comes with all its fields.

> Couldn't we live without?

We'd have to make a copy of the bytecode object, skipping the
source-code reference.

But really, this is actually a side-problem: the inlined bytecode is not
spliced the way it should, so the inlining optimization is basically
missing.  Such a half-assed inlining gives you all the downsides of
inlining without its upsides.  Once we fix that, the reproducibility
problem will also be fixed.


        Stefan

bug#34322: bug#34321: reproducibility: absolute file name in newst-treeview.elc

[Lars Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> But really, this is actually a side-problem: the inlined bytecode is not
> spliced the way it should, so the inlining optimization is basically
> missing.  Such a half-assed inlining gives you all the downsides of
> inlining without its upsides.  Once we fix that, the reproducibility
> problem will also be fixed.

It's two years later, so I checked whether this problem is still present
on the trunk, and indeed:

(defalias 'tramp-lookup-syntax #[257 "..." [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 8)) default separate] 2 ("/home/larsi/src/emacs/trunk/lisp/net/tramp-compat.elc" . 7627)] error "Wrong `tramp-syntax' %s"] 4 (#$ . 29248)])

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34322: reproducibility: absolute file name in tramp.elc

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Mon, 04 Feb 2019 17:58:50 -0500
> 
> Package: emacs
> Version: 26.1.91
> Severity: minor
> 
> Perhaps the same issue as https://debbugs.gnu.org/34321 .
> The definition of tramp-lookup-syntax in tramp.elc contains the absolute
> file name of the build directory, making the file non-reproducible.
> Eg in the 26.1.91 pretest tarfile, it contains
> "/home/nico/work/emacs-26/lisp/net/tramp-compat.elc"

It seems to be a reference to the place in tramp-compat.elc where
tramp-compat-tramp-syntax is defined.  Maybe some byte-compiler guru
could explain or guess when does the byte compiler use such
references.  Stefan, any ideas?

bug#34321: reproducibility: absolute file name in newst-treeview.elc

[Glenn Morris]

Package: emacs
Version: 26.1.91
Severity: minor

For some reason, the definition of newsticker--group-shift in the
compiled file newst-treeview.elc ends up containing a string that points
to the build directory. Eg in the 26.1.91 pretest tarfile, it is:

"/home/nico/work/emacs-26/lisp/emacs-lisp/cl-extra.elc"

This means that newst-treeview.elc is non-reproducible (contents change
as build directory changes).

(Issued spotted in
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/emacs.html
)

bug#34322: reproducibility: absolute file name in tramp.elc

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> For some reason, the definition of newsticker--group-shift in the
> compiled file newst-treeview.elc ends up containing a string that points
> to the build directory. Eg in the 26.1.91 pretest tarfile, it is:
>
> "/home/nico/work/emacs-26/lisp/emacs-lisp/cl-extra.elc"
>
> This means that newst-treeview.elc is non-reproducible (contents change
> as build directory changes).

I grepped through the news*.elc files for /home, but couldn't find any.
So has this problem fixed itself since it was reported?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34321: bug#34322: reproducibility: absolute file name in tramp.elc

[Glenn Morris]

Lars Ingebrigtsen wrote:

> So has this problem fixed itself since it was reported?

No it has not.

find . -name '*.elc' | xargs grep /scratch
Binary file ./lisp/org/ox-odt.elc matches
Binary file ./lisp/org/ox.elc matches
Binary file ./lisp/progmodes/cc-mode.elc matches

Even if that didn't return any matches, the onus would still be there to
verify that it was due to a genuine bytecode fix rather than
coincidental changes in the source files or build ordering.

bug#34322: reproducibility: absolute file name in tramp.elc

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

>> So has this problem fixed itself since it was reported?
>
> No it has not.
>
> find . -name '*.elc' | xargs grep /scratch
> Binary file ./lisp/org/ox-odt.elc matches
> Binary file ./lisp/org/ox.elc matches
> Binary file ./lisp/progmodes/cc-mode.elc matches

Ah, right, I also get my complete path in those .elc files.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no