bug#34321: reproducibility: absolute file name in newst-treeview.elc

bug#34321: reproducibility: absolute file name in newst-treeview.elc

[Glenn Morris]

Package: emacs
Version: 26.1.91
Severity: minor

For some reason, the definition of newsticker--group-shift in the
compiled file newst-treeview.elc ends up containing a string that points
to the build directory. Eg in the 26.1.91 pretest tarfile, it is:

"/home/nico/work/emacs-26/lisp/emacs-lisp/cl-extra.elc"

This means that newst-treeview.elc is non-reproducible (contents change
as build directory changes).

(Issued spotted in
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/emacs.html
)

bug#34322: reproducibility: absolute file name in tramp.elc

[Glenn Morris]

Package: emacs
Version: 26.1.91
Severity: minor

Perhaps the same issue as https://debbugs.gnu.org/34321 .
The definition of tramp-lookup-syntax in tramp.elc contains the absolute
file name of the build directory, making the file non-reproducible.
Eg in the 26.1.91 pretest tarfile, it contains
"/home/nico/work/emacs-26/lisp/net/tramp-compat.elc"

bug#34322: reproducibility: absolute file name in tramp.elc

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> Perhaps the same issue as https://debbugs.gnu.org/34321 .
> The definition of tramp-lookup-syntax in tramp.elc contains the absolute
> file name of the build directory, making the file non-reproducible.
> Eg in the 26.1.91 pretest tarfile, it contains
> "/home/nico/work/emacs-26/lisp/net/tramp-compat.elc"

Don't know how to fix it. For the records, in the emacs-26 branch,
tramp.elc contains

--8<---------------cut here---------------start------------->8---
(defalias 'tramp-lookup-syntax #[257 "\301 \236A\206\f\302\303\b\"\207" [tramp-syntax #[0 "\b\301\267\202\n\302\207\303\207\b\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 8)) default separate] 2 ("/usr/local/src/emacs-26/lisp/net/tramp-compat.elc" . 7408)] error "Wrong `tramp-syntax' %s"] 4 (#$ . 25891)])
--8<---------------cut here---------------end--------------->8---

OTOH, in the Tramp git repository, branch tramp-2-3-stable, tramp.elc contains

--8<---------------cut here---------------start------------->8---
(defalias 'tramp-lookup-syntax #[257 "\b\301\267\202

bug#34322: reproducibility: absolute file name in tramp.elc

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Mon, 04 Feb 2019 17:58:50 -0500
> 
> Package: emacs
> Version: 26.1.91
> Severity: minor
> 
> Perhaps the same issue as https://debbugs.gnu.org/34321 .
> The definition of tramp-lookup-syntax in tramp.elc contains the absolute
> file name of the build directory, making the file non-reproducible.
> Eg in the 26.1.91 pretest tarfile, it contains
> "/home/nico/work/emacs-26/lisp/net/tramp-compat.elc"

It seems to be a reference to the place in tramp-compat.elc where
tramp-compat-tramp-syntax is defined.  Maybe some byte-compiler guru
could explain or guess when does the byte compiler use such
references.  Stefan, any ideas?

bug#34321: reproducibility: absolute file name in newst-treeview.elc

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Mon, 04 Feb 2019 17:55:09 -0500
> 
> For some reason, the definition of newsticker--group-shift in the
> compiled file newst-treeview.elc ends up containing a string that points
> to the build directory. Eg in the 26.1.91 pretest tarfile, it is:
> 
> "/home/nico/work/emacs-26/lisp/emacs-lisp/cl-extra.elc"

Looks like a pointer to a place where cl-concatenate is defined.

bug#34322: reproducibility: absolute file name in tramp.elc

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Tue, 05 Feb 2019 10:00:15 +0100
> Cc: 34322@debbugs.gnu.org
> 
> Don't know how to fix it. For the records, in the emacs-26 branch,
> tramp.elc contains
> 
> --8<---------------cut here---------------start------------->8---
> (defalias 'tramp-lookup-syntax #[257 "\301 \236A\206\f\302\303\b\"\207" [tramp-syntax #[0 "\b\301\267\202\n\302\207\303\207\b\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 8)) default separate] 2 ("/usr/local/src/emacs-26/lisp/net/tramp-compat.elc" . 7408)] error "Wrong `tramp-syntax' %s"] 4 (#$ . 25891)])
> --8<---------------cut here---------------end--------------->8---
> 
> OTOH, in the Tramp git repository, branch tramp-2-3-stable, tramp.elc contains
> 
> --8<---------------cut here---------------start------------->8---
> (defalias 'tramp-lookup-syntax #[257 "\b\301\267\202 \302\202 \303\202 \b\236A\206 \304\305\b\"\207" [tramp-syntax #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10)) default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
> --8<---------------cut here---------------end--------------->8---
> 
> I have no idea where the absolute file name in the former comes
> from.

I tried to explain what it means in another message.

> Both repositories use the same Emacs 26.1.91 binary for the build.

I think whether there is or isn't such a reference depends on whether
tramp-compat.elc exists when you byte-compile tramp.c.  Try removing
the former or renaming it, and you get the contents you have in the
second excerpt.  So perhaps whether the reference exists or not
depends on the order of the compilation, and maybe also on how many
compilation jobs in parallel are run when boostrapping Emacs.

bug#34322: reproducibility: absolute file name in tramp.elc

[Stefan Monnier]

>> --8<---------------cut here---------------start------------->8---
>> (defalias 'tramp-lookup-syntax #[257 "\301 \236A\206\f\302\303\b\"\207"
>> [tramp-syntax #[0 "\b\301\267\202\n\302\207\303\207\b\207" [tramp-syntax
>> #s(hash-table size 2 test eq rehash-size 1.5 rehash-threshold 0.8125
>> purecopy t data (ftp 6 sep 8)) default separate]
>> 2 ("/usr/local/src/emacs-26/lisp/net/tramp-compat.elc" . 7408)] error
>> "Wrong `tramp-syntax' %s"] 4 (#$ . 25891)])
>> --8<---------------cut here---------------end--------------->8---

Hmm... looks like a bug in the implementation of defsubst: it works by
replacing references to tramp-compat-tramp-syntax with a copy of the
function's byte-code (this is the #[0 ...] object above, which includes
the source file name info), and later on those copied bytecode objects
are normally spliced into the surrounding bytecode (at which point the
extra source info is dropped), but here it seems like it hasn't been
spliced as it should.

>> --8<---------------cut here---------------start------------->8---
>> (defalias 'tramp-lookup-syntax #[257 "\b\301\267\202 \302\202 \303\202
>> \b\236A\206 \304\305\b\"\207" [tramp-syntax #s(hash-table size 2 test eq
>> rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10))
>> default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
>> --8<---------------cut here---------------end--------------->8---

Here you can see that there is no nested bytecode object (i.e. no
#[...] with the main #[...]) so the call has been correctly inlined.

> I think whether there is or isn't such a reference depends on whether
> tramp-compat.elc exists when you byte-compile tramp.c.  Try removing

I don't think we've changed anything significant in the implementation
of defsubst so indeed it's likely triggered by something like the order
of compilation.


        Stefan

bug#34322: reproducibility: absolute file name in tramp.elc

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> --8<---------------cut here---------------start------------->8---
>>> (defalias 'tramp-lookup-syntax #[257 "\b\301\267\202 \302\202 \303\202
>>> \b\236A\206 \304\305\b\"\207" [tramp-syntax #s(hash-table size 2 test eq
>>> rehash-size 1.5 rehash-threshold 0.8125 purecopy t data (ftp 6 sep 10))
>>> default separate error "Wrong `tramp-syntax' %s"] 4 (#$ . 25800)])
>>> --8<---------------cut here---------------end--------------->8---
>
> Here you can see that there is no nested bytecode object (i.e. no
> #[...] with the main #[...]) so the call has been correctly inlined.

Given, that the absolute file name is not needed in the bytecode, I'm
wondering why we insert it there. Couldn't we live without?

>         Stefan

Best regards, Michael.

bug#34321: reproducibility: absolute file name in newst-treeview.elc

[Glenn Morris]

Eli Zaretskii wrote:

>> to the build directory. Eg in the 26.1.91 pretest tarfile, it is:
>> 
>> "/home/nico/work/emacs-26/lisp/emacs-lisp/cl-extra.elc"
>
> Looks like a pointer to a place where cl-concatenate is defined.

I was going to say that this is a defun, not a defsubst, but then I see
that it is "proclaimed inline" in cl-macs.el, so it is a (confusing)
defsubst, so this is probably the same root cause as #34322.

bug#34322: reproducibility: absolute file name in tramp.elc

[Stefan Monnier]

>> Here you can see that there is no nested bytecode object (i.e. no
>> #[...] with the main #[...]) so the call has been correctly inlined.
> Given, that the absolute file name is not needed in the bytecode,

It's present in the in-memory bytecode object so that `C-h f` can jump
to the source upon demand, so it is needed.

> I'm wondering why we insert it there.

It's because we just insert the bytecode object wholesale (not a copy of
the object), so it comes with all its fields.

> Couldn't we live without?

We'd have to make a copy of the bytecode object, skipping the
source-code reference.

But really, this is actually a side-problem: the inlined bytecode is not
spliced the way it should, so the inlining optimization is basically
missing.  Such a half-assed inlining gives you all the downsides of
inlining without its upsides.  Once we fix that, the reproducibility
problem will also be fixed.


        Stefan

bug#34322: reproducibility: absolute file name in tramp.elc

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> For some reason, the definition of newsticker--group-shift in the
> compiled file newst-treeview.elc ends up containing a string that points
> to the build directory. Eg in the 26.1.91 pretest tarfile, it is:
>
> "/home/nico/work/emacs-26/lisp/emacs-lisp/cl-extra.elc"
>
> This means that newst-treeview.elc is non-reproducible (contents change
> as build directory changes).

I grepped through the news*.elc files for /home, but couldn't find any.
So has this problem fixed itself since it was reported?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34321: bug#34322: reproducibility: absolute file name in tramp.elc

[Glenn Morris]

Lars Ingebrigtsen wrote:

> So has this problem fixed itself since it was reported?

No it has not.

find . -name '*.elc' | xargs grep /scratch
Binary file ./lisp/org/ox-odt.elc matches
Binary file ./lisp/org/ox.elc matches
Binary file ./lisp/progmodes/cc-mode.elc matches

Even if that didn't return any matches, the onus would still be there to
verify that it was due to a genuine bytecode fix rather than
coincidental changes in the source files or build ordering.

bug#34322: reproducibility: absolute file name in tramp.elc

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

>> So has this problem fixed itself since it was reported?
>
> No it has not.
>
> find . -name '*.elc' | xargs grep /scratch
> Binary file ./lisp/org/ox-odt.elc matches
> Binary file ./lisp/org/ox.elc matches
> Binary file ./lisp/progmodes/cc-mode.elc matches

Ah, right, I also get my complete path in those .elc files.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34321: bug#34322: reproducibility: absolute file name in tramp.elc

[Andreas Schwab]

On Jan 30 2022, Lars Ingebrigtsen wrote:

> Glenn Morris <rgm@gnu.org> writes:
>
>>> So has this problem fixed itself since it was reported?
>>
>> No it has not.
>>
>> find . -name '*.elc' | xargs grep /scratch
>> Binary file ./lisp/org/ox-odt.elc matches
>> Binary file ./lisp/org/ox.elc matches
>> Binary file ./lisp/progmodes/cc-mode.elc matches
>
> Ah, right, I also get my complete path in those .elc files.

That appears to be a bug in the bytecode inliner.  In ox.elc the
function org-element-class (a defsubst) is supposed to be inlined into
org-export-data and org-export-expand, but instead the original bytecode
of org-element-class is put into the bytecode, including its doc string
reference (pointing to org-element.elc, which would normally be written
as $# if it appeared there instead of in ex.elc).

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#34321: bug#34322: reproducibility: absolute file name in tramp.elc

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> That appears to be a bug in the bytecode inliner.  In ox.elc the
> function org-element-class (a defsubst) is supposed to be inlined into
> org-export-data and org-export-expand, but instead the original bytecode
> of org-element-class is put into the bytecode, including its doc string
> reference (pointing to org-element.elc, which would normally be written
> as $# if it appeared there instead of in ex.elc).

Hm, yes.

(As an aside, org-element-class should probably not be a defsubst,
because three other org files declare-function it because they don't
load it at compile time.  But ox.el does load it when compiling.)

Perhaps Stefan or Mattias have some input on what's going wrong with
inlining here; added to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no