Building a release tarball generates trampoline files in eln-cache

Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

Andrea,

When Emacs 28 is built from a release tarball, and the preloaded *.el
files are native-compiled with batch-native-compile, the compilation
produces trampoline files in the user's eln-cache directory.  Here's
the list of such files I have in my eln-cache, after building a
tarball on MS-Windows:

  subr--trampoline-782d6c6973742d666f6e7473_x_list_fonts_0.eln
  subr--trampoline-782d646973706c61792d706978656c2d686569676874_x_display_pixel_height_0.eln
  subr--trampoline-782d646973706c61792d706978656c2d7769647468_x_display_pixel_width_0.eln
  subr--trampoline-782d646973706c61792d706c616e6573_x_display_planes_0.eln
  subr--trampoline-782d646973706c61792d636f6c6f722d63656c6c73_x_display_color_cells_0.eln
  subr--trampoline-782d7365727665722d6d61782d726571756573742d73697a65_x_server_max_request_size_0.eln
  subr--trampoline-782d7365727665722d76656e646f72_x_server_vendor_0.eln
  subr--trampoline-782d646973706c61792d73637265656e73_x_display_screens_0.eln
  subr--trampoline-782d7365727665722d76657273696f6e_x_server_version_0.eln
  subr--trampoline-782d646973706c61792d6261636b696e672d73746f7265_x_display_backing_store_0.eln
  subr--trampoline-782d646973706c61792d6d6d2d686569676874_x_display_mm_height_0.eln
  subr--trampoline-782d646973706c61792d6d6d2d7769647468_x_display_mm_width_0.eln
  subr--trampoline-782d646973706c61792d76697375616c2d636c617373_x_display_visual_class_0.eln

Is this expected, and if so, why aren't these trampoline files put
under native-lisp/ directory in the build tree?  I presume that these
trampoline files need to be installed together with the rest of *.eln,
or else the installed Emacs will not work, right?

Re: Building a release tarball generates trampoline files in eln-cache

[Ken Brown]

On 10/9/2021 4:19 AM, Eli Zaretskii wrote:
> Andrea,
> 
> When Emacs 28 is built from a release tarball, and the preloaded *.el
> files are native-compiled with batch-native-compile, the compilation
> produces trampoline files in the user's eln-cache directory.  Here's
> the list of such files I have in my eln-cache, after building a
> tarball on MS-Windows:
> 
>    subr--trampoline-782d6c6973742d666f6e7473_x_list_fonts_0.eln
>    subr--trampoline-782d646973706c61792d706978656c2d686569676874_x_display_pixel_height_0.eln
>    subr--trampoline-782d646973706c61792d706978656c2d7769647468_x_display_pixel_width_0.eln
>    subr--trampoline-782d646973706c61792d706c616e6573_x_display_planes_0.eln
>    subr--trampoline-782d646973706c61792d636f6c6f722d63656c6c73_x_display_color_cells_0.eln
>    subr--trampoline-782d7365727665722d6d61782d726571756573742d73697a65_x_server_max_request_size_0.eln
>    subr--trampoline-782d7365727665722d76656e646f72_x_server_vendor_0.eln
>    subr--trampoline-782d646973706c61792d73637265656e73_x_display_screens_0.eln
>    subr--trampoline-782d7365727665722d76657273696f6e_x_server_version_0.eln
>    subr--trampoline-782d646973706c61792d6261636b696e672d73746f7265_x_display_backing_store_0.eln
>    subr--trampoline-782d646973706c61792d6d6d2d686569676874_x_display_mm_height_0.eln
>    subr--trampoline-782d646973706c61792d6d6d2d7769647468_x_display_mm_width_0.eln
>    subr--trampoline-782d646973706c61792d76697375616c2d636c617373_x_display_visual_class_0.eln
> 
> Is this expected, and if so, why aren't these trampoline files put
> under native-lisp/ directory in the build tree?  I presume that these
> trampoline files need to be installed together with the rest of *.eln,
> or else the installed Emacs will not work, right?

Is this a 32-bit build?  I saw the same issue when I built on 32-bit Cygwin, but 
not when I built on 64-bit Cygwin.  I don't remember which system I used when I 
created the tarball (i.e., when I byte-compiled the elisp files).  It shouldn't 
matter, but Paul's report in Bug#51104 makes me wonder.

Ken

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Sat, 9 Oct 2021 10:37:38 -0400
> 
> >    subr--trampoline-782d6c6973742d666f6e7473_x_list_fonts_0.eln
> >    subr--trampoline-782d646973706c61792d706978656c2d686569676874_x_display_pixel_height_0.eln
> >    subr--trampoline-782d646973706c61792d706978656c2d7769647468_x_display_pixel_width_0.eln
> >    subr--trampoline-782d646973706c61792d706c616e6573_x_display_planes_0.eln
> >    subr--trampoline-782d646973706c61792d636f6c6f722d63656c6c73_x_display_color_cells_0.eln
> >    subr--trampoline-782d7365727665722d6d61782d726571756573742d73697a65_x_server_max_request_size_0.eln
> >    subr--trampoline-782d7365727665722d76656e646f72_x_server_vendor_0.eln
> >    subr--trampoline-782d646973706c61792d73637265656e73_x_display_screens_0.eln
> >    subr--trampoline-782d7365727665722d76657273696f6e_x_server_version_0.eln
> >    subr--trampoline-782d646973706c61792d6261636b696e672d73746f7265_x_display_backing_store_0.eln
> >    subr--trampoline-782d646973706c61792d6d6d2d686569676874_x_display_mm_height_0.eln
> >    subr--trampoline-782d646973706c61792d6d6d2d7769647468_x_display_mm_width_0.eln
> >    subr--trampoline-782d646973706c61792d76697375616c2d636c617373_x_display_visual_class_0.eln
> > 
> > Is this expected, and if so, why aren't these trampoline files put
> > under native-lisp/ directory in the build tree?  I presume that these
> > trampoline files need to be installed together with the rest of *.eln,
> > or else the installed Emacs will not work, right?
> 
> Is this a 32-bit build?

Yes.  But I don't think I understand why this should matter.  Do you?

> I saw the same issue when I built on 32-bit Cygwin, but 
> not when I built on 64-bit Cygwin.  I don't remember which system I used when I 
> created the tarball (i.e., when I byte-compiled the elisp files).  It shouldn't 
> matter, but Paul's report in Bug#51104 makes me wonder.

You are saying that this is because the tarball (and thus the *.elc
files) were built on a 64-bit system?  But I see the same if I build
the tarball on a 64-bit GNU/Linux machine.

Re: Building a release tarball generates trampoline files in eln-cache

[Ken Brown]

On 10/9/2021 11:05 AM, Eli Zaretskii wrote:
>> From: Ken Brown <kbrown@cornell.edu>
>> Is this a 32-bit build?
> 
> Yes.  But I don't think I understand why this should matter.  Do you?

No, I'm just reporting my observations in case it helps Andrea.

>> I saw the same issue when I built on 32-bit Cygwin, but
>> not when I built on 64-bit Cygwin.  I don't remember which system I used when I
>> created the tarball (i.e., when I byte-compiled the elisp files).  It shouldn't
>> matter, but Paul's report in Bug#51104 makes me wonder.
> 
> You are saying that this is because the tarball (and thus the *.elc
> files) were built on a 64-bit system?  But I see the same if I build
> the tarball on a 64-bit GNU/Linux machine.

Let me restart from a clean checkout and do some more careful experiments.  I'll 
report back later.

Ken

Re: Building a release tarball generates trampoline files in eln-cache

[Ken Brown]

On 10/9/2021 11:39 AM, Ken Brown wrote:
> Let me restart from a clean checkout and do some more careful experiments.  I'll 
> report back later.

I apologize.  There's no 32-bit/64-bit difference.  My earlier remarks were 
based on recollections that were obviously faulty.  I just rebuilt, and I got 
exactly the results you reported, on both 32-bit and 64-bit Cygwin.

Ken

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Ken Brown <kbrown@cornell.edu>
> Cc: akrl@sdf.org, eggert@cs.ucla.edu, emacs-devel@gnu.org
> Date: Sat, 9 Oct 2021 13:33:10 -0400
> 
> On 10/9/2021 11:39 AM, Ken Brown wrote:
> > Let me restart from a clean checkout and do some more careful experiments.  I'll 
> > report back later.
> 
> I apologize.  There's no 32-bit/64-bit difference.  My earlier remarks were 
> based on recollections that were obviously faulty.  I just rebuilt, and I got 
> exactly the results you reported, on both 32-bit and 64-bit Cygwin.

OK, thanks.  No need to apologize: I've seen this earlier, but also
mis-interpreted the results.

I hope Andrea will help us out here.

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ken Brown <kbrown@cornell.edu>
>> Cc: akrl@sdf.org, eggert@cs.ucla.edu, emacs-devel@gnu.org
>> Date: Sat, 9 Oct 2021 13:33:10 -0400
>> 
>> On 10/9/2021 11:39 AM, Ken Brown wrote:
>> > Let me restart from a clean checkout and do some more careful experiments.  I'll 
>> > report back later.
>> 
>> I apologize.  There's no 32-bit/64-bit difference.  My earlier remarks were 
>> based on recollections that were obviously faulty.  I just rebuilt, and I got 
>> exactly the results you reported, on both 32-bit and 64-bit Cygwin.
>
> OK, thanks.  No need to apologize: I've seen this earlier, but also
> mis-interpreted the results.
>
> I hope Andrea will help us out here.

Hi Eli,

I certainly will very soon.  I'm digesting all the backlog of mails.
Should have time to look into this tomorrow evening or Friday at the
latest.

BR

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Ken Brown <kbrown@cornell.edu>, eggert@cs.ucla.edu, emacs-devel@gnu.org
> Date: Wed, 13 Oct 2021 19:37:49 +0000
> 
> > I hope Andrea will help us out here.
> 
> Hi Eli,
> 
> I certainly will very soon.  I'm digesting all the backlog of mails.
> Should have time to look into this tomorrow evening or Friday at the
> latest.

Thanks.

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Ken Brown <kbrown@cornell.edu>, eggert@cs.ucla.edu, emacs-devel@gnu.org
>> Date: Wed, 13 Oct 2021 19:37:49 +0000
>> 
>> > I hope Andrea will help us out here.
>> 
>> Hi Eli,
>> 
>> I certainly will very soon.  I'm digesting all the backlog of mails.
>> Should have time to look into this tomorrow evening or Friday at the
>> latest.
>
> Thanks.

Hi Eli,

ced72b6e4c should fix this, please have a look if it works for you when
you've the time.

Best Regards

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Ken Brown]

On 10/15/2021 3:35 AM, Andrea Corallo wrote:
> ced72b6e4c should fix this, please have a look if it works for you when
> you've the time.

This fixes it for me on Cygwin.

Thanks.

Ken

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: eggert@cs.ucla.edu, kbrown@cornell.edu, emacs-devel@gnu.org
> Date: Fri, 15 Oct 2021 07:35:49 +0000
> 
> ced72b6e4c should fix this, please have a look if it works for you when
> you've the time.

Thanks.  Now those subr--trampoline*.eln files are no longer under
eln-cache, but they are directly under native-lisp/, not under its
28.0.60-xxxx/ subdirectory.  That doesn't sound right, does it?  And I
think it's because of that the built Emacs won't start normally,
showing these errors at startup:

  lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
  lisp/term/xterm.elc: Error: Symbol’s function definition is void: t

Ken, don't you see this problem in your Cygwin build of the pretest
tarball?

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: eggert@cs.ucla.edu, kbrown@cornell.edu, emacs-devel@gnu.org
>> Date: Fri, 15 Oct 2021 07:35:49 +0000
>> 
>> ced72b6e4c should fix this, please have a look if it works for you when
>> you've the time.
>
> Thanks.  Now those subr--trampoline*.eln files are no longer under
> eln-cache, but they are directly under native-lisp/, not under its
> 28.0.60-xxxx/ subdirectory.  That doesn't sound right, does it?

Correct, it does not.  2971a6890f hopefully should fix this.

Sorry I don't know how to test the build over the release tarball so
haven't tested it, unless I've made some other stupid error this should
work.

> And I
> think it's because of that the built Emacs won't start normally,
> showing these errors at startup:
>
>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>
> Ken, don't you see this problem in your Cygwin build of the pretest
> tarball?

Perhaps he still had the necessary trampolines previously compiled
reachable by Emacs?

BR

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> Date: Sat, 16 Oct 2021 09:04:28 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andrea Corallo <akrl@sdf.org>
> >> Cc: eggert@cs.ucla.edu, kbrown@cornell.edu, emacs-devel@gnu.org
> >> Date: Fri, 15 Oct 2021 07:35:49 +0000
> >> 
> >> ced72b6e4c should fix this, please have a look if it works for you when
> >> you've the time.
> >
> > Thanks.  Now those subr--trampoline*.eln files are no longer under
> > eln-cache, but they are directly under native-lisp/, not under its
> > 28.0.60-xxxx/ subdirectory.  That doesn't sound right, does it?
> 
> Correct, it does not.  2971a6890f hopefully should fix this.

Will test shortly.

> Sorry I don't know how to test the build over the release tarball so
> haven't tested it, unless I've made some other stupid error this should
> work.

Testing this is simple: after you build Emacs normally, do this:

  $ ./make-dist --snapshot --no-compress --no-check
  $ tar -xf emacs-28.0.60.tar
  $ cd emacs-28.0.60
  $ ./configure ... && make -jN

> >   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
> >   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
> >
> > Ken, don't you see this problem in your Cygwin build of the pretest
> > tarball?
> 
> Perhaps he still had the necessary trampolines previously compiled
> reachable by Emacs?

Maybe.

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> Date: Sat, 16 Oct 2021 12:30:02 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> 
> > >> ced72b6e4c should fix this, please have a look if it works for you when
> > >> you've the time.
> > >
> > > Thanks.  Now those subr--trampoline*.eln files are no longer under
> > > eln-cache, but they are directly under native-lisp/, not under its
> > > 28.0.60-xxxx/ subdirectory.  That doesn't sound right, does it?
> > 
> > Correct, it does not.  2971a6890f hopefully should fix this.
> 
> Will test shortly.

Now the subr--trampoline*.eln files are in the right place, but this
problem:

>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t

still exists.  So something else is amiss here.

Btw, why are these subr--trampoline*.eln files get generated when
building the tarball, but not when building from Git "normally", where
the *.eln files are produced by batch-byte+native-compile?  Because I
don't see any such files that correspond to the build from Git
anywhere on my system.

Re: Building a release tarball generates trampoline files in eln-cache

[Ken Brown]

On 10/16/2021 5:55 AM, Eli Zaretskii wrote:
>> Date: Sat, 16 Oct 2021 12:30:02 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>>
>>>>> ced72b6e4c should fix this, please have a look if it works for you when
>>>>> you've the time.
>>>>
>>>> Thanks.  Now those subr--trampoline*.eln files are no longer under
>>>> eln-cache, but they are directly under native-lisp/, not under its
>>>> 28.0.60-xxxx/ subdirectory.  That doesn't sound right, does it?
>>>
>>> Correct, it does not.  2971a6890f hopefully should fix this.
>>
>> Will test shortly.
> 
> Now the subr--trampoline*.eln files are in the right place, but this
> problem:
> 
>>    lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>>    lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
> 
> still exists.  So something else is amiss here.

I'm seeing the errors too.  It's possible that when I tested yesterday I only 
checked the build, to make sure the trampoline files weren't being created in 
eln-cache, but didn't actually run the built emacs.

Ken

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 16 Oct 2021 12:30:02 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> 
>> > >> ced72b6e4c should fix this, please have a look if it works for you when
>> > >> you've the time.
>> > >
>> > > Thanks.  Now those subr--trampoline*.eln files are no longer under
>> > > eln-cache, but they are directly under native-lisp/, not under its
>> > > 28.0.60-xxxx/ subdirectory.  That doesn't sound right, does it?
>> > 
>> > Correct, it does not.  2971a6890f hopefully should fix this.
>> 
>> Will test shortly.
>
> Now the subr--trampoline*.eln files are in the right place, but this
> problem:
>
>>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>
> still exists.  So something else is amiss here.

Okay, I'll have a look into this I guess Monday.

> Btw, why are these subr--trampoline*.eln files get generated when
> building the tarball, but not when building from Git "normally", where
> the *.eln files are produced by batch-byte+native-compile?  Because I
> don't see any such files that correspond to the build from Git
> anywhere on my system.

The obvious answer is that something is either redefining or advising
those primitives.  If it's important to know what and where we can
investigate it, it should not be extremely difficult.

BR

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> Date: Sat, 16 Oct 2021 20:15:09 +0000
> 
> > Now the subr--trampoline*.eln files are in the right place, but this
> > problem:
> >
> >>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
> >>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
> >
> > still exists.  So something else is amiss here.
> 
> Okay, I'll have a look into this I guess Monday.

Thanks.

> > Btw, why are these subr--trampoline*.eln files get generated when
> > building the tarball, but not when building from Git "normally", where
> > the *.eln files are produced by batch-byte+native-compile?  Because I
> > don't see any such files that correspond to the build from Git
> > anywhere on my system.
> 
> The obvious answer is that something is either redefining or advising
> those primitives.  If it's important to know what and where we can
> investigate it, it should not be extremely difficult.

I'm curious why 2 methods of native compilation that are supposed to
produce identical results, in reality produce slightly different
results: one produces these trampoline files, the other doesn't.  It
could be that there's some hidden issue here, perhaps even related to
those errors at startup described above.

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> Date: Sat, 16 Oct 2021 20:15:09 +0000
>> 
>> > Now the subr--trampoline*.eln files are in the right place, but this
>> > problem:
>> >
>> >>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>> >>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>> >
>> > still exists.  So something else is amiss here.
>> 
>> Okay, I'll have a look into this I guess Monday.
>
> Thanks.

I haven't found the reason for the issue so far, but at least I wanted
to mention I can reproduce the error on my X86_64 GNU/Linux machine as
well.

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> Date: Sun, 17 Oct 2021 08:56:20 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> 
> > From: Andrea Corallo <akrl@sdf.org>
> > Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> > Date: Sat, 16 Oct 2021 20:15:09 +0000
> > 
> > > Now the subr--trampoline*.eln files are in the right place, but this
> > > problem:
> > >
> > >>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
> > >>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
> > >
> > > still exists.  So something else is amiss here.
> > 
> > Okay, I'll have a look into this I guess Monday.
> 
> Thanks.
> 
> > > Btw, why are these subr--trampoline*.eln files get generated when
> > > building the tarball, but not when building from Git "normally", where
> > > the *.eln files are produced by batch-byte+native-compile?  Because I
> > > don't see any such files that correspond to the build from Git
> > > anywhere on my system.
> > 
> > The obvious answer is that something is either redefining or advising
> > those primitives.  If it's important to know what and where we can
> > investigate it, it should not be extremely difficult.
> 
> I'm curious why 2 methods of native compilation that are supposed to
> produce identical results, in reality produce slightly different
> results: one produces these trampoline files, the other doesn't.  It
> could be that there's some hidden issue here, perhaps even related to
> those errors at startup described above.

Andrea, any progress in investigating this strange problem?  It is
currently the only hard blocking issue that prevents us from making a
pretest tarball.

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> Date: Wed, 27 Oct 2021 19:32:30 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> 
> > Date: Sun, 17 Oct 2021 08:56:20 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> > 
> > > From: Andrea Corallo <akrl@sdf.org>
> > > Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> > > Date: Sat, 16 Oct 2021 20:15:09 +0000
> > > 
> > > > Now the subr--trampoline*.eln files are in the right place, but this
> > > > problem:
> > > >
> > > >>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
> > > >>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
> > > >
> > > > still exists.  So something else is amiss here.
> > > 
> > > Okay, I'll have a look into this I guess Monday.
> > 
> > Thanks.
> > 
> > > > Btw, why are these subr--trampoline*.eln files get generated when
> > > > building the tarball, but not when building from Git "normally", where
> > > > the *.eln files are produced by batch-byte+native-compile?  Because I
> > > > don't see any such files that correspond to the build from Git
> > > > anywhere on my system.
> > > 
> > > The obvious answer is that something is either redefining or advising
> > > those primitives.  If it's important to know what and where we can
> > > investigate it, it should not be extremely difficult.
> > 
> > I'm curious why 2 methods of native compilation that are supposed to
> > produce identical results, in reality produce slightly different
> > results: one produces these trampoline files, the other doesn't.  It
> > could be that there's some hidden issue here, perhaps even related to
> > those errors at startup described above.
> 
> Andrea, any progress in investigating this strange problem?  It is
> currently the only hard blocking issue that prevents us from making a
> pretest tarball.

I now have the same problem in the "normal" build with native-comp of
the release branch, the one that builds from Git clone and uses
ELC+ELN to compile all the preloaded files (as opposed to building a
release tarball).  The problem shows at startup:

  $ emacs -Q -nw

This shows errors in the *Compile Log* buffer(??)

  lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
  lisp/term/xterm.elc: Error: Symbol’s function definition is void: t

If I delete seq.elc, the problem goes away.

How do I investigate this?

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 27 Oct 2021 19:32:30 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> 
>> > Date: Sun, 17 Oct 2021 08:56:20 +0300
>> > From: Eli Zaretskii <eliz@gnu.org>
>> > Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> > 
>> > > From: Andrea Corallo <akrl@sdf.org>
>> > > Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> > > Date: Sat, 16 Oct 2021 20:15:09 +0000
>> > > 
>> > > > Now the subr--trampoline*.eln files are in the right place, but this
>> > > > problem:
>> > > >
>> > > >>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>> > > >>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>> > > >
>> > > > still exists.  So something else is amiss here.
>> > > 
>> > > Okay, I'll have a look into this I guess Monday.
>> > 
>> > Thanks.
>> > 
>> > > > Btw, why are these subr--trampoline*.eln files get generated when
>> > > > building the tarball, but not when building from Git "normally", where
>> > > > the *.eln files are produced by batch-byte+native-compile?  Because I
>> > > > don't see any such files that correspond to the build from Git
>> > > > anywhere on my system.
>> > > 
>> > > The obvious answer is that something is either redefining or advising
>> > > those primitives.  If it's important to know what and where we can
>> > > investigate it, it should not be extremely difficult.
>> > 
>> > I'm curious why 2 methods of native compilation that are supposed to
>> > produce identical results, in reality produce slightly different
>> > results: one produces these trampoline files, the other doesn't.  It
>> > could be that there's some hidden issue here, perhaps even related to
>> > those errors at startup described above.
>> 
>> Andrea, any progress in investigating this strange problem?  It is
>> currently the only hard blocking issue that prevents us from making a
>> pretest tarball.

Hi Eli,

I must apologize, I've also completely missed your previous mail.

I'm behind schedule as I'm in a late with some GCC work where I'm trying
to meet the stage1 deadline.  As a result haven't progressed much on
this investigation.

> I now have the same problem in the "normal" build with native-comp of
> the release branch, the one that builds from Git clone and uses
> ELC+ELN to compile all the preloaded files (as opposed to building a
> release tarball).  The problem shows at startup:
>
>   $ emacs -Q -nw
>
> This shows errors in the *Compile Log* buffer(??)
>
>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>
> If I delete seq.elc, the problem goes away.
>
> How do I investigate this?

I guess a start is to run in gdb to see where the function definition of
gv-setter was last time changed (if ever).

I guess `gv-setter' was compiled and dumped, therefore its definition
it's expected to be revived by 'load_comp_unit' called from
pdumper.c:5355 while Emacs is starting-up.

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Stefan Monnier]

> I guess `gv-setter' was compiled and dumped, therefore its definition
> it's expected to be revived by 'load_comp_unit' called from
> pdumper.c:5355 while Emacs is starting-up.

Hmm... gv-setter is usually not dumped, tho it is probably indeed dumped
into the `bootstrap-emacs.pdmp` since `gv.el` is probably loaded during
the first dump where the .el files haven't been (byte)compiled yet.


        Stefan

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 02 Nov 2021 19:22:56 +0000
> 
> > This shows errors in the *Compile Log* buffer(??)
> >
> >   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
> >   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
> >
> > If I delete seq.elc, the problem goes away.
> >
> > How do I investigate this?
> 
> I guess a start is to run in gdb to see where the function definition of
> gv-setter was last time changed (if ever).
> 
> I guess `gv-setter' was compiled and dumped, therefore its definition
> it's expected to be revived by 'load_comp_unit' called from
> pdumper.c:5355 while Emacs is starting-up.

Thanks, will try.

Any idea why the errors show in *Compile Log* buffer?  Where did that
buffer come from?

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

[...]

> Hi Eli,
>
> I must apologize, I've also completely missed your previous mail.
>
> I'm behind schedule as I'm in a late with some GCC work where I'm trying
> to meet the stage1 deadline.  As a result haven't progressed much on
> this investigation.
>
>> I now have the same problem in the "normal" build with native-comp of
>> the release branch, the one that builds from Git clone and uses
>> ELC+ELN to compile all the preloaded files (as opposed to building a
>> release tarball).  The problem shows at startup:
>>
>>   $ emacs -Q -nw
>>
>> This shows errors in the *Compile Log* buffer(??)
>>
>>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>>
>> If I delete seq.elc, the problem goes away.
>>
>> How do I investigate this?
>
> I guess a start is to run in gdb to see where the function definition of
> gv-setter was last time changed (if ever).
>
> I guess `gv-setter' was compiled and dumped, therefore its definition
> it's expected to be revived by 'load_comp_unit' called from
> pdumper.c:5355 while Emacs is starting-up.
>
>   Andrea

I tried a fresh bootstrap of emacs-28 now and it is working for me.  I
guess what we are observing here is something related to some state of
the build?  Is it possible that your Emacs dumped two times?  See my
following notes:

Reading the source code I might have done some progress on the
investigation.  I guess this is just related to bug#45103 (read native
Emacs not supporting so far redumps).

When we execute loadup.el we prepare the native compilation units for
dump fixing up their filenames.

Comment at loadup.el:463

  ;; Fix the compilation unit filename to have it working when                                                                                                        
  ;; installed or if the source directory got moved.  This is set to be                                                                                               
  ;; a pair in the form of:                                                                                                                                           
  ;;     (rel-filename-from-install-bin . rel-filename-from-local-bin).

Once dump is done reviving it at pdumper.c:5301 we expect to find a cons
as compilation unit filename.  If before dumping we do not prepare that
this will not happen so I don't think it can work.

I'm preparing a patch to add a check (I think is very healthy anyway) and
testing that my theory make sense, will follow up very soon.

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 02 Nov 2021 19:22:56 +0000
>> 
>> > This shows errors in the *Compile Log* buffer(??)
>> >
>> >   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>> >   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>> >
>> > If I delete seq.elc, the problem goes away.
>> >
>> > How do I investigate this?
>> 
>> I guess a start is to run in gdb to see where the function definition of
>> gv-setter was last time changed (if ever).
>> 
>> I guess `gv-setter' was compiled and dumped, therefore its definition
>> it's expected to be revived by 'load_comp_unit' called from
>> pdumper.c:5355 while Emacs is starting-up.
>
> Thanks, will try.
>
> Any idea why the errors show in *Compile Log* buffer?  Where did that
> buffer come from?

Nope, but that might be a side effect of something very basic going
wrong earlier.

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
> [...]
>
>> Hi Eli,
>>
>> I must apologize, I've also completely missed your previous mail.
>>
>> I'm behind schedule as I'm in a late with some GCC work where I'm trying
>> to meet the stage1 deadline.  As a result haven't progressed much on
>> this investigation.
>>
>>> I now have the same problem in the "normal" build with native-comp of
>>> the release branch, the one that builds from Git clone and uses
>>> ELC+ELN to compile all the preloaded files (as opposed to building a
>>> release tarball).  The problem shows at startup:
>>>
>>>   $ emacs -Q -nw
>>>
>>> This shows errors in the *Compile Log* buffer(??)
>>>
>>>   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>>>   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>>>
>>> If I delete seq.elc, the problem goes away.
>>>
>>> How do I investigate this?
>>
>> I guess a start is to run in gdb to see where the function definition of
>> gv-setter was last time changed (if ever).
>>
>> I guess `gv-setter' was compiled and dumped, therefore its definition
>> it's expected to be revived by 'load_comp_unit' called from
>> pdumper.c:5355 while Emacs is starting-up.
>>
>>   Andrea
>
> I tried a fresh bootstrap of emacs-28 now and it is working for me.  I
> guess what we are observing here is something related to some state of
> the build?  Is it possible that your Emacs dumped two times?  See my
> following notes:
>
> Reading the source code I might have done some progress on the
> investigation.  I guess this is just related to bug#45103 (read native
> Emacs not supporting so far redumps).
>
> When we execute loadup.el we prepare the native compilation units for
> dump fixing up their filenames.
>
> Comment at loadup.el:463
>
>   ;; Fix the compilation unit filename to have it working when                                                                                                        
>   ;; installed or if the source directory got moved.  This is set to be                                                                                               
>   ;; a pair in the form of:                                                                                                                                           
>   ;;     (rel-filename-from-install-bin . rel-filename-from-local-bin).
>
> Once dump is done reviving it at pdumper.c:5301 we expect to find a cons
> as compilation unit filename.  If before dumping we do not prepare that
> this will not happen so I don't think it can work.
>
> I'm preparing a patch to add a check (I think is very healthy anyway) and
> testing that my theory make sense, will follow up very soon.

Okay that's not the issue as I guess while preparing the release binary
we don't redump native code.  I've pushed 9d6162053e to add the sanity
check anyway as I think it's still good to check everything is coherent
there.

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Stefan Monnier]

Eli Zaretskii [2021-11-02 21:47:42] wrote:
>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 02 Nov 2021 19:22:56 +0000
>> > This shows errors in the *Compile Log* buffer(??)
>> >
>> >   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>> >   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>> >
>> > If I delete seq.elc, the problem goes away.
>> >
>> > How do I investigate this?
>> 
>> I guess a start is to run in gdb to see where the function definition of
>> gv-setter was last time changed (if ever).
>> 
>> I guess `gv-setter' was compiled and dumped, therefore its definition
>> it's expected to be revived by 'load_comp_unit' called from
>> pdumper.c:5355 while Emacs is starting-up.
>
> Thanks, will try.
>
> Any idea why the errors show in *Compile Log* buffer?  Where did that
> buffer come from?

I'd guess that it's when `cl-generic.el` calls `byte-compile` (both
seq.el and xterm.el use `cl-defmethod`).
`cl-generic.el` is also one of the rare users of `gv-setter`.
OTOH, `gv-setter` normally doesn't appear within the code that
`cl-generic.el` passes to `byte-compile`, instead `gv-setter` is called
directly by `cl-generic.el`, so my guess might be off.


        Stefan

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Andrea Corallo <akrl@sdf.org>,  emacs-devel@gnu.org
> Date: Tue, 02 Nov 2021 16:26:22 -0400
> 
> Eli Zaretskii [2021-11-02 21:47:42] wrote:
> >> From: Andrea Corallo <akrl@sdf.org>
> >> Cc: emacs-devel@gnu.org
> >> Date: Tue, 02 Nov 2021 19:22:56 +0000
> >> > This shows errors in the *Compile Log* buffer(??)
> >> >
> >> >   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
> >> >   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
> >> >
> >> > If I delete seq.elc, the problem goes away.
> >> >
> >> > How do I investigate this?
> >> 
> >> I guess a start is to run in gdb to see where the function definition of
> >> gv-setter was last time changed (if ever).
> >> 
> >> I guess `gv-setter' was compiled and dumped, therefore its definition
> >> it's expected to be revived by 'load_comp_unit' called from
> >> pdumper.c:5355 while Emacs is starting-up.
> >
> > Thanks, will try.
> >
> > Any idea why the errors show in *Compile Log* buffer?  Where did that
> > buffer come from?
> 
> I'd guess that it's when `cl-generic.el` calls `byte-compile` (both
> seq.el and xterm.el use `cl-defmethod`).
> `cl-generic.el` is also one of the rare users of `gv-setter`.
> OTOH, `gv-setter` normally doesn't appear within the code that
> `cl-generic.el` passes to `byte-compile`, instead `gv-setter` is called
> directly by `cl-generic.el`, so my guess might be off.

Maybe, maybe not.

The crucial detail here is that I invoke "emacs -nw" (are you using
"-nw" in your attempts to reproduce?).  What happens then is that we
load the terminal-specific support file, in this case xterm.elc,
during startup.  And since xterm.el is not preloaded, it is not
natively-compiled by the build process.  So once we load it, Emacs
tries to native-compile it, and all of its prerequisites, and that's
where those errors and that compile log buffer come from, I guess.

I tried to mark xterm.el and seq.el as not for native-compile, but
that didn't help.  However, if I manually compile gv.el natively into
a *.eln file, the problem goes away.

So why would Emacs want to native-compile gv.el in this case, and why
trying that causes this problem?  Maybe we should add gv.el to the
list of files that are natively-compiled as part of the build?

Interestingly, if gv-XXX.eln is available, Emacs still loads bytecomp
at startup, but I don't see any evidence that it compiles xterm.el
natively: no subprocesses and no xterm-XXX.eln files are created.  Why
does it load bytecomp, and what prevents it from natively compiling
xterm.el in this case?

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Andrea Corallo <akrl@sdf.org>,  emacs-devel@gnu.org
>> Date: Tue, 02 Nov 2021 16:26:22 -0400
>> 
>> Eli Zaretskii [2021-11-02 21:47:42] wrote:
>> >> From: Andrea Corallo <akrl@sdf.org>
>> >> Cc: emacs-devel@gnu.org
>> >> Date: Tue, 02 Nov 2021 19:22:56 +0000
>> >> > This shows errors in the *Compile Log* buffer(??)
>> >> >
>> >> >   lisp/emacs-lisp/seq.elc: Error: Symbol’s function definition is void: gv-setter
>> >> >   lisp/term/xterm.elc: Error: Symbol’s function definition is void: t
>> >> >
>> >> > If I delete seq.elc, the problem goes away.
>> >> >
>> >> > How do I investigate this?
>> >> 
>> >> I guess a start is to run in gdb to see where the function definition of
>> >> gv-setter was last time changed (if ever).
>> >> 
>> >> I guess `gv-setter' was compiled and dumped, therefore its definition
>> >> it's expected to be revived by 'load_comp_unit' called from
>> >> pdumper.c:5355 while Emacs is starting-up.
>> >
>> > Thanks, will try.
>> >
>> > Any idea why the errors show in *Compile Log* buffer?  Where did that
>> > buffer come from?
>> 
>> I'd guess that it's when `cl-generic.el` calls `byte-compile` (both
>> seq.el and xterm.el use `cl-defmethod`).
>> `cl-generic.el` is also one of the rare users of `gv-setter`.
>> OTOH, `gv-setter` normally doesn't appear within the code that
>> `cl-generic.el` passes to `byte-compile`, instead `gv-setter` is called
>> directly by `cl-generic.el`, so my guess might be off.
>
> Maybe, maybe not.
>
> The crucial detail here is that I invoke "emacs -nw" (are you using
> "-nw" in your attempts to reproduce?).  What happens then is that we
> load the terminal-specific support file, in this case xterm.elc,
> during startup.  And since xterm.el is not preloaded, it is not
> natively-compiled by the build process.  So once we load it, Emacs
> tries to native-compile it, and all of its prerequisites, and that's
> where those errors and that compile log buffer come from, I guess.
>
> I tried to mark xterm.el and seq.el as not for native-compile, but
> that didn't help.  However, if I manually compile gv.el natively into
> a *.eln file, the problem goes away.
>
> So why would Emacs want to native-compile gv.el in this case, 

Apparently emacs (-nw -Q) wants to compile gv.el because once started
`gv-get' is used (still in byte compiled form).

This is the backtrace from where we take the decision:

===========================

#0  maybe_defer_native_compilation (function_name=XIL(0x7ffff3c4f3e8), definition=XIL(0xb9f5ed)) at comp.c:4848
#1  0x000000000055b053 in Fdefalias (symbol=XIL(0x7ffff3c4f3e8), definition=XIL(0xb9f5ed), docstring=XIL(0)) at data.c:830
#2  0x00000000005766af in eval_sub (form=<optimized out>) at lisp.h:2108
#3  0x00000000005a51b6 in readevalloop (readcharfun=XIL(0x68a0), infile0=0x7fffffffb860, sourcename=XIL(0xc4e604), printflag=false, unibyte=<optimized out>, 
    readfun=XIL(0), start=XIL(0), end=<optimized out>) at lread.c:2320
#4  0x00000000005a5fc7 in Fload (file=<optimized out>, noerror=<optimized out>, nomessage=XIL(0x30), nosuffix=<optimized out>, must_suffix=<optimized out>)
    at lisp.h:1008
#5  0x00000000005a65aa in save_match_data_load (file=XIL(0x7ffff47519cc), noerror=noerror@entry=XIL(0), nomessage=nomessage@entry=XIL(0x30), 
    nosuffix=nosuffix@entry=XIL(0), must_suffix=must_suffix@entry=XIL(0x30)) at lread.c:1623
#6  0x00000000005736a5 in Fautoload_do_load (fundef=XIL(0x7ffff475199b), funname=XIL(0x7ffff3c4f3e8), macro_only=XIL(0)) at lisp.h:1008
#7  0x0000000000573966 in Ffuncall (nargs=3, args=args@entry=0x7fffffffba28) at lisp.h:1008
#8  0x00000000005b75c0 in exec_byte_code (bytestr=<optimized out>, vector=<optimized out>, maxdepth=<optimized out>, args_template=<optimized out>, 
    nargs=<optimized out>, args=<optimized out>) at bytecode.c:632
#9  0x00000000005738b9 in Ffuncall (nargs=3, args=0x7fffffffbd10) at eval.c:3042
#10 0x0000000000573d08 in Fapply (nargs=2, args=0x7fffffffbda0) at eval.c:2656
#11 0x0000000000577ed7 in apply1 (arg=<optimized out>, fn=<optimized out>) at lisp.h:1376
#12 Fmacroexpand (form=XIL(0xd49813), environment=XIL(0xcbe143)) at eval.c:1139
#13 0x0000000000573a63 in Ffuncall (nargs=3, args=0x7fffffffbe30) at lisp.h:2108
#14 0x00007ffff3f55c92 in F6d6163726f6578702d6d6163726f657870616e64_macroexp_macroexpand_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#15 0x0000000000573a63 in Ffuncall (nargs=3, args=0x7fffffffbf70) at lisp.h:2108
#16 0x00007ffff3f56f72 in F6d6163726f6578702d2d657870616e642d616c6c_macroexp__expand_all_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#17 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffc0f0) at lisp.h:2108
#18 0x00007ffff3f54f69 in F6d6163726f6578702d2d616c6c2d666f726d73_macroexp__all_forms_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#19 0x0000000000573a63 in Ffuncall (nargs=3, args=0x7fffffffc1e0) at lisp.h:2108
#20 0x00007ffff3f56cfb in F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_5 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#21 0x00007ffff3f5789f in F6d6163726f6578702d2d657870616e642d616c6c_macroexp__expand_all_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#22 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffc420) at lisp.h:2108
#23 0x00007ffff3f54f69 in F6d6163726f6578702d2d616c6c2d666f726d73_macroexp__all_forms_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#24 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffc4f0) at lisp.h:2108
#25 0x00007ffff3f575e0 in F6d6163726f6578702d2d657870616e642d616c6c_macroexp__expand_all_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#26 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffc6a0) at lisp.h:2108
#27 0x00007ffff3f54f69 in F6d6163726f6578702d2d616c6c2d666f726d73_macroexp__all_forms_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#28 0x0000000000573a63 in Ffuncall (nargs=3, args=0x7fffffffc7e0) at lisp.h:2108
#29 0x00007ffff3f574c6 in F6d6163726f6578702d2d657870616e642d616c6c_macroexp__expand_all_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#30 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffc920) at lisp.h:2108
#31 0x00007ffff3f54f69 in F6d6163726f6578702d2d616c6c2d666f726d73_macroexp__all_forms_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#32 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffc9f0) at lisp.h:2108
#33 0x00007ffff3f575e0 in F6d6163726f6578702d2d657870616e642d616c6c_macroexp__expand_all_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#34 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffcba0) at lisp.h:2108
#35 0x00007ffff3f54f69 in F6d6163726f6578702d2d616c6c2d666f726d73_macroexp__all_forms_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#36 0x0000000000573a63 in Ffuncall (nargs=3, args=0x7fffffffcce0) at lisp.h:2108
#37 0x00007ffff3f574c6 in F6d6163726f6578702d2d657870616e642d616c6c_macroexp__expand_all_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#38 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffcdf0) at lisp.h:2108
#39 0x00007ffff3f57974 in F6d6163726f657870616e642d616c6c_macroexpand_all_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/macroexp-2c3e1495-8e8af7d5.eln
#40 0x0000000000573a63 in Ffuncall (nargs=3, args=0x7fffffffce90) at lisp.h:2108
#41 0x00007ffff2c43b7f in F627974652d636f6d70696c652d70726570726f63657373_byte_compile_preprocess_0 ()
   from .../emacs-28.0.60/native-lisp/28.0.60-e950671c/bytecomp-12882072-29d9ad62.eln
#42 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffcf60) at lisp.h:2108
#43 0x00007ffff2c469c9 in F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_50 ()
   from .../emacs-28.0.60/native-lisp/28.0.60-e950671c/bytecomp-12882072-29d9ad62.eln
#44 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffd068) at lisp.h:2108
#45 0x00007ffff2c46f30 in F627974652d636f6d70696c65_byte_compile_0 ()
   from .../emacs-28.0.60/native-lisp/28.0.60-e950671c/bytecomp-12882072-29d9ad62.eln
#46 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffd1b0) at lisp.h:2108
#47 0x00007ffff4275860 in F636c2d2d67656e657269632d6765742d64697370617463686572_cl__generic_get_dispatcher_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/cl-generic-be68ad15-12e9508b.eln
#48 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffd2d0) at lisp.h:2108
#49 0x00007ffff4275b4c in F636c2d2d67656e657269632d6d616b652d6e6578742d66756e6374696f6e_cl__generic_make_next_function_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/cl-generic-be68ad15-12e9508b.eln
#50 0x0000000000573a63 in Ffuncall (nargs=4, args=0x7fffffffd3a0) at lisp.h:2108
#51 0x00007ffff4275a1f in F636c2d2d67656e657269632d6d616b652d66756e6374696f6e_cl__generic_make_function_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/cl-generic-be68ad15-12e9508b.eln
#52 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffd4e0) at lisp.h:2108
#53 0x00007ffff4274a3b in F636c2d67656e657269632d646566696e652d6d6574686f64_cl_generic_define_method_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/cl-generic-be68ad15-12e9508b.eln
#54 0x0000000000573a63 in Ffuncall (nargs=6, args=args@entry=0x7fffffffd648) at lisp.h:2108
#55 0x00000000005b75c0 in exec_byte_code (bytestr=<optimized out>, vector=<optimized out>, maxdepth=<optimized out>, args_template=<optimized out>, 
    nargs=<optimized out>, args=<optimized out>) at bytecode.c:632
#56 0x00000000005766af in eval_sub (form=<optimized out>) at lisp.h:2108
#57 0x00000000005a51b6 in readevalloop (readcharfun=XIL(0x68a0), infile0=0x7fffffffdb00, sourcename=XIL(0xd09114), printflag=false, unibyte=<optimized out>, 
    readfun=XIL(0), start=XIL(0), end=<optimized out>) at lread.c:2320
#58 0x00000000005a5fc7 in Fload (file=<optimized out>, noerror=<optimized out>, nomessage=XIL(0x30), nosuffix=<optimized out>, must_suffix=<optimized out>)
    at lisp.h:1008
#59 0x00007ffff4208cb5 in F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_48 ()
#54 0x0000000000573a63 in Ffuncall (nargs=6, args=args@entry=0x7fffffffd648) at lisp.h:2108
#55 0x00000000005b75c0 in exec_byte_code (bytestr=<optimized out>, vector=<optimized out>, maxdepth=<optimized out>, args_template=<optimized out>, 
    nargs=<optimized out>, args=<optimized out>) at bytecode.c:632
#56 0x00000000005766af in eval_sub (form=<optimized out>) at lisp.h:2108
#57 0x00000000005a51b6 in readevalloop (readcharfun=XIL(0x68a0), infile0=0x7fffffffdb00, sourcename=XIL(0xd09114), printflag=false, unibyte=<optimized out>, 
    readfun=XIL(0), start=XIL(0), end=<optimized out>) at lread.c:2320
#58 0x00000000005a5fc7 in Fload (file=<optimized out>, noerror=<optimized out>, nomessage=XIL(0x30), nosuffix=<optimized out>, must_suffix=<optimized out>)
    at lisp.h:1008
#59 0x00007ffff4208cb5 in F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_48 ()
--Type <RET> for more, q to quit, c to continue without paging--
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/faces-b9447c93-7f74610c.eln
#60 0x0000000000573a63 in Ffuncall (nargs=2, args=0x7fffffffdc90) at lisp.h:2108
#61 0x00007ffff4208abc in F7474792d66696e642d74797065_tty_find_type_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/faces-b9447c93-7f74610c.eln
#62 0x0000000000573a63 in Ffuncall (nargs=3, args=0x7fffffffddb0) at lisp.h:2108
#63 0x00007ffff4208ec7 in F7474792d72756e2d7465726d696e616c2d696e697469616c697a6174696f6e_tty_run_terminal_initialization_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/faces-b9447c93-7f74610c.eln
#64 0x0000000000573a63 in Ffuncall (nargs=4, args=0x7fffffffdeb0) at lisp.h:2108
#65 0x00007ffff3d881e2 in F636f6d6d616e642d6c696e65_command_line_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/startup-bbc6ea72-8e089fac.eln
#66 0x0000000000573a63 in Ffuncall (nargs=1, args=0x7fffffffdfb8) at lisp.h:2108
#67 0x00007ffff3d82ca9 in F6e6f726d616c2d746f702d6c6576656c_normal_top_level_0 ()
   from .../emacs-28.0.60/src/../native-lisp/28.0.60-e950671c/preloaded/startup-bbc6ea72-8e089fac.eln
#68 0x00000000005769b9 in eval_sub (form=<optimized out>) at lisp.h:2108
#69 0x0000000000578697 in Feval (form=XIL(0x7ffff479b3d3), lexical=<optimized out>) at eval.c:2330
#70 0x00000000005728f7 in internal_condition_case (bfun=bfun@entry=0x4e0580 <top_level_2>, handlers=handlers@entry=XIL(0x90), hfun=hfun@entry=0x4e6ed0 <cmd_error>)
    at eval.c:1453
#71 0x00000000004e1374 in top_level_1 (ignore=ignore@entry=XIL(0)) at lisp.h:1008
#72 0x0000000000572831 in internal_catch (tag=<optimized out>, func=func@entry=0x4e1330 <top_level_1>, arg=arg@entry=XIL(0)) at eval.c:1184
#73 0x00000000004e0d59 in command_loop () at lisp.h:1008
#74 0x00000000004e6a0d in recursive_edit_1 () at keyboard.c:720
#75 0x00000000004e6daa in Frecursive_edit () at keyboard.c:803
#76 0x000000000042a6f1 in main (argc=<optimized out>, argv=<optimized out>) at emacs.c:2310

Lisp Backtrace:

eval.c:122: Emacs fatal error: assertion failed: pdl->kind == SPECPDL_BACKTRACE

Breakpoint 1, terminate_due_to_signal (sig=sig@entry=6, backtrace_limit=backtrace_limit@entry=2147483647) at emacs.c:400
400	  signal (sig, SIG_DFL);

===========================

> and why trying that causes this problem?

Still unclear.

After Emacs decideds gv.el should be compiled we require comp.el.
Probably as a consequence of that we queue for native compilation also
(in order): cl-lib.el, seq.el, warnings.el.  Then the error.

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

[...]
>> and why trying that causes this problem?
> Still unclear.

To summarize, apparently:

- `cl--generic-get-dispatcher' wants to bytecompile
- This requires the usage of `gv-get'
- We require comp.el to compile gv.el
- Something goes wrong

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> Date: Fri, 05 Nov 2021 15:49:03 +0000
> 
> Andrea Corallo <akrl@sdf.org> writes:
> 
> [...]
> >> and why trying that causes this problem?
> > Still unclear.
> 
> To summarize, apparently:
> 
> - `cl--generic-get-dispatcher' wants to bytecompile
> - This requires the usage of `gv-get'
> - We require comp.el to compile gv.el
> - Something goes wrong

So what's the road forward?  Do we add gv.el to the list of *.el files
that need to be natively-compiled as part of the build?  Or are you
still investigating the issue to figure out what "goes wrong" there?

Thanks.

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
>> Date: Fri, 05 Nov 2021 15:49:03 +0000
>> 
>> Andrea Corallo <akrl@sdf.org> writes:
>> 
>> [...]
>> >> and why trying that causes this problem?
>> > Still unclear.
>> 
>> To summarize, apparently:
>> 
>> - `cl--generic-get-dispatcher' wants to bytecompile
>> - This requires the usage of `gv-get'
>> - We require comp.el to compile gv.el
>> - Something goes wrong
>
> So what's the road forward?  Do we add gv.el to the list of *.el files
> that need to be natively-compiled as part of the build?  Or are you
> still investigating the issue to figure out what "goes wrong" there?

Hi Eli,

sorry for taking that long.

This is what I reached after further investigation:

The function slot of `gv-setter' was *never* set when we signal the
error.

Here the lisp backtrace when we signal the error "Symbol’s function
definition is void: gv-setter":

Lisp Backtrace:
"gv-setter" (0xffff5248)
"gv-get" (0xffff57b0)
0xf4a1bb58 PVEC_COMPILED
"macroexpand" (0xffff5ea8)
"macroexp-macroexpand" (0xffff6088)
"macroexp--expand-all" (0xffff62a8)
"macroexp--all-forms" (0xffff6438)
"macroexp--expand-all" (0xffff6718)
"macroexp--all-forms" (0xffff6888)
"macroexp--expand-all" (0xffff6ad8)
"macroexp--all-forms" (0xffff6cb8)
"macroexp--expand-all" (0xffff6e98)
"macroexp--all-forms" (0xffff7008)
"macroexp--expand-all" (0xffff7258)
"macroexp--all-forms" (0xffff7438)
"macroexp--expand-all" (0xffff75e8)
"macroexpand-all" (0xffff7728)
"byte-compile-preprocess" (0xffff7898)
0xbda8c0 PVEC_SUBR
"byte-compile" (0xffff7c28)
"cl--generic-get-dispatcher" (0xffff7de8)
"cl--generic-make-next-function" (0xffff7f58)
"cl--generic-make-function" (0xffff8138)
"cl-generic-define-method" (0xffff8340)
"byte-code" (0xffff8750)
"require" (0xffff8c50)
"byte-code" (0xffff9010)
"require" (0xffff9510)
"byte-code" (0xffff9a70)
"defalias" (0xffffa100)
"gv-get" (0xffffa590)
0xf4a1bb58 PVEC_COMPILED
"macroexpand" (0xffffac88)
"macroexp-macroexpand" (0xffffae68)
"macroexp--expand-all" (0xffffb088)
"macroexp--all-forms" (0xffffb218)
"macroexp--expand-all" (0xffffb4f8)
"macroexp--all-forms" (0xffffb668)
"macroexp--expand-all" (0xffffb8b8)
"macroexp--all-forms" (0xffffba98)
"macroexp--expand-all" (0xffffbc78)
"macroexp--all-forms" (0xffffbde8)
"macroexp--expand-all" (0xffffc038)
"macroexp--all-forms" (0xffffc218)
"macroexp--expand-all" (0xffffc3c8)
"macroexpand-all" (0xffffc508)
"byte-compile-preprocess" (0xffffc678)
0xbda8c0 PVEC_SUBR
"byte-compile" (0xffffca08)
"cl--generic-get-dispatcher" (0xffffcbc8)
"cl--generic-make-next-function" (0xffffcd38)
"cl--generic-make-function" (0xffffcf18)
"cl-generic-define-method" (0xffffd120)
"byte-code" (0xffffd500)
0xf4bc8898 PVEC_SUBR
"tty-find-type" (0xffffdb78)
"tty-run-terminal-initialization" (0xffffdd18)
"command-line" (0xffffdec0)
"normal-top-level" (0xffffdf50)

Note that in the stack we have two `gv-get'.  The first one is the one
(as mentioned in my prev mail) that is causing comp.el to be required.
This is really a dependency issue.

I guess option 1 is to native compile gv.el AOT as you've suggested.

Option 2 would be to postpone the native compiler load (if needed) to a
point were we are sure it can be loaded.

If we know gv.el is the only file causing this I'd say the least
impacting solution would be option 1.  But I'm happy also to implement
option 2 if we prefer it and we define where to perform the check and
load of comp.el if needed.

Best Regards

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Tue, 09 Nov 2021 21:55:03 +0000
> 
> Option 2 would be to postpone the native compiler load (if needed) to a
> point were we are sure it can be loaded.

You mean, bind no-native-compile to a non-nil value until after some
point in startup.el?  That could work (I actually tried that, but
under the assumption that xterm.el was the problem, so it didn't
help).  But why do you think this problem happens only early during
startup?  Suppose we cross that bridge, and then Emacs needs to load
some Lisp file that wasn't natively-compiled -- won't we have the same
problem and for the same reason?  The first gv-get in the backtrace is
called as part of compiling, so it sounds like every compilation is in
the danger of triggering this problem, because compiling needs
gv-setter again.  Or what am I missing?

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> Date: Tue, 09 Nov 2021 21:55:03 +0000
>> 
>> Option 2 would be to postpone the native compiler load (if needed) to a
>> point were we are sure it can be loaded.
>
> You mean, bind no-native-compile to a non-nil value until after some
> point in startup.el?

I was thinking to something a little different like recording what we
wanted to compile when it was not possible cause too early in the
startup.

When startup is finished if this list of pending files is not empty we
should load comp.el (and its dependencies) and then consume the list of
pending .el files to be compiled (possibly including comp.el as well).

> That could work (I actually tried that, but
> under the assumption that xterm.el was the problem, so it didn't
> help).  But why do you think this problem happens only early during
> startup?  Suppose we cross that bridge, and then Emacs needs to load
> some Lisp file that wasn't natively-compiled -- won't we have the same
> problem and for the same reason?  The first gv-get in the backtrace is
> called as part of compiling, so it sounds like every compilation is in
> the danger of triggering this problem, because compiling needs
> gv-setter again.  Or what am I missing?

The first `gv-get' is part of a byte compilation.  With the proposed
mechanism it should go through postponing the comp.el load to say when
we return to `normal-top-level'.

That's not a trivial problem, I might be missing something I don't see
ATM.

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Thu, 11 Nov 2021 15:08:16 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > You mean, bind no-native-compile to a non-nil value until after some
> > point in startup.el?
> 
> I was thinking to something a little different like recording what we
> wanted to compile when it was not possible cause too early in the
> startup.
> 
> When startup is finished if this list of pending files is not empty we
> should load comp.el (and its dependencies) and then consume the list of
> pending .el files to be compiled (possibly including comp.el as well).
> 
> > That could work (I actually tried that, but
> > under the assumption that xterm.el was the problem, so it didn't
> > help).  But why do you think this problem happens only early during
> > startup?  Suppose we cross that bridge, and then Emacs needs to load
> > some Lisp file that wasn't natively-compiled -- won't we have the same
> > problem and for the same reason?  The first gv-get in the backtrace is
> > called as part of compiling, so it sounds like every compilation is in
> > the danger of triggering this problem, because compiling needs
> > gv-setter again.  Or what am I missing?
> 
> The first `gv-get' is part of a byte compilation.  With the proposed
> mechanism it should go through postponing the comp.el load to say when
> we return to `normal-top-level'.
> 
> That's not a trivial problem, I might be missing something I don't see
> ATM.

Would the code to implement this be complicated?  If not, please show
the code you had in mind, so I could make sure we are on the same
page.  Otherwise, I guess I'll go with compiling gv.el ATM, at least
for now.

And while I have your attention, another question: suppose that Emacs
with all the preloaded files compiled to *.eln is installed on a
system that has libgccjit (which AFAIU is necessary to run the
native-compiled code), but doesn't have Binutils -- what will happen
when Emacs loads some .el file and tries to compile it?  Will the
response to the compilation failure be graceful?  (This situation is
likely to happen when users install a binary distro, but don't have
Binutils available.)

Thanks.

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

[-- Attachment #1: Type: text/plain, Size: 1839 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> Date: Thu, 11 Nov 2021 15:08:16 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > You mean, bind no-native-compile to a non-nil value until after some
>> > point in startup.el?
>> 
>> I was thinking to something a little different like recording what we
>> wanted to compile when it was not possible cause too early in the
>> startup.
>> 
>> When startup is finished if this list of pending files is not empty we
>> should load comp.el (and its dependencies) and then consume the list of
>> pending .el files to be compiled (possibly including comp.el as well).
>> 
>> > That could work (I actually tried that, but
>> > under the assumption that xterm.el was the problem, so it didn't
>> > help).  But why do you think this problem happens only early during
>> > startup?  Suppose we cross that bridge, and then Emacs needs to load
>> > some Lisp file that wasn't natively-compiled -- won't we have the same
>> > problem and for the same reason?  The first gv-get in the backtrace is
>> > called as part of compiling, so it sounds like every compilation is in
>> > the danger of triggering this problem, because compiling needs
>> > gv-setter again.  Or what am I missing?
>> 
>> The first `gv-get' is part of a byte compilation.  With the proposed
>> mechanism it should go through postponing the comp.el load to say when
>> we return to `normal-top-level'.
>> 
>> That's not a trivial problem, I might be missing something I don't see
>> ATM.
>
> Would the code to implement this be complicated?  If not, please show
> the code you had in mind, so I could make sure we are on the same
> page.

I'm attaching the following patch.  It is very much untested but should
clarify what I had in mind.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: comp.patch --]
[-- Type: text/x-diff, Size: 3631 bytes --]

diff --git a/lisp/startup.el b/lisp/startup.el
index 505d7b83f4..fe51c1de64 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -519,6 +519,18 @@ startup--xdg-or-homedot
       xdg-dir)
      (t emacs-d-dir))))
 
+(defvar comp--delayed-sources)
+(defvar comp--loadable)
+(declare-function native--compile-async "comp.el"
+                  (files &optional recursively load selector))
+(defun honor-delayed-native-compilations ()
+  (when (native-comp-available-p)
+    (setq comp--loadable t)
+    (when comp--delayed-sources
+      (require 'comp)
+      (native--compile-async comp--delayed-sources nil 'late)
+      (setq comp--delayed-sources nil))))
+
 (defvar native-comp-eln-load-path)
 (defun normal-top-level ()
   "Emacs calls this function when it first starts up.
@@ -785,7 +797,8 @@ normal-top-level
           (if (string-match "\\`DISPLAY=" varval)
               (setq display varval))))
       (when display
-        (delete display process-environment)))))
+        (delete display process-environment))))
+  (honor-delayed-native-compilations))
 
 ;; Precompute the keyboard equivalents in the menu bar items.
 ;; Command-line options supported by tty's:
diff --git a/src/comp.c b/src/comp.c
index bc1adcf4e2..900e2ee82a 100644
--- a/src/comp.c
+++ b/src/comp.c
@@ -4786,10 +4786,6 @@ register_native_comp_unit (Lisp_Object comp_u)
 /* Deferred compilation mechanism. */
 /***********************************/
 
-/* List of sources we'll compile and load after having conventionally
-   loaded the compiler and its dependencies.  */
-static Lisp_Object delayed_sources;
-
 /* Queue an asynchronous compilation for the source file defining
    FUNCTION_NAME and perform a late load.
 
@@ -4846,30 +4842,16 @@ maybe_defer_native_compilation (Lisp_Object function_name,
 
   /* This is so deferred compilation is able to compile comp
      dependencies breaking circularity.  */
-  if (!NILP (Ffeaturep (Qcomp, Qnil)))
+  if (comp__loadable)
     {
-      /* Comp already loaded.  */
-      if (!NILP (delayed_sources))
-	{
-	  CALLN (Ffuncall, intern_c_string ("native--compile-async"),
-		 delayed_sources, Qnil, Qlate);
-	  delayed_sources = Qnil;
-	}
+      /* Startup is done, comp is usable.  */
+      Frequire (Qcomp, Qnil, Qnil);
       Fputhash (function_name, definition, Vcomp_deferred_pending_h);
       CALLN (Ffuncall, intern_c_string ("native--compile-async"),
 	     src, Qnil, Qlate);
     }
   else
-    {
-      delayed_sources = Fcons (src, delayed_sources);
-      /* Require comp only once.  */
-      static bool comp_required = false;
-      if (!comp_required)
-	{
-	  comp_required = true;
-	  Frequire (Qcomp, Qnil, Qnil);
-	}
-    }
+    Vcomp__delayed_sources = Fcons (src, Vcomp__delayed_sources);
 }
 
 \f
@@ -5327,6 +5309,13 @@ DEFUN ("native-comp-available-p", Fnative_comp_available_p,
 syms_of_comp (void)
 {
 #ifdef HAVE_NATIVE_COMP
+  DEFVAR_LISP ("comp--delayed-sources", Vcomp__delayed_sources,
+	       doc: /* List of sources to be native compiled when
+		       startup is finished.  For internal use.  */);
+  DEFVAR_BOOL ("comp--loadable",
+	       comp__loadable,
+	       doc: /* Non-nil when comp.el can be loaded.  For
+		       internal use. */);
   /* Compiler control customizes.  */
   DEFVAR_BOOL ("native-comp-deferred-compilation",
 	       native_comp_deferred_compilation,
@@ -5467,8 +5456,6 @@ syms_of_comp (void)
   staticpro (&comp.func_blocks_h);
   staticpro (&comp.emitter_dispatcher);
   comp.emitter_dispatcher = Qnil;
-  staticpro (&delayed_sources);
-  delayed_sources = Qnil;
   staticpro (&loadsearch_re_list);
   loadsearch_re_list = Qnil;
 

[-- Attachment #3: Type: text/plain, Size: 667 bytes --]


> And while I have your attention, another question: suppose that Emacs
> with all the preloaded files compiled to *.eln is installed on a
> system that has libgccjit (which AFAIU is necessary to run the
> native-compiled code), but doesn't have Binutils -- what will happen
> when Emacs loads some .el file and tries to compile it?  Will the
> response to the compilation failure be graceful?  (This situation is
> likely to happen when users install a binary distro, but don't have
> Binutils available.)

I believe binutils is a dependecy of libgccjit, so I guess having
libgccjit without binutils should be classified as missconfiguration no?

Regards

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Thu, 11 Nov 2021 21:01:37 +0000
> 
> > Would the code to implement this be complicated?  If not, please show
> > the code you had in mind, so I could make sure we are on the same
> > page.
> 
> I'm attaching the following patch.  It is very much untested but should
> clarify what I had in mind.

Thanks.  I think we should do this on master, but on the release
branch compile gv.el AOT, as that will be safer.  WDYT?

> > And while I have your attention, another question: suppose that Emacs
> > with all the preloaded files compiled to *.eln is installed on a
> > system that has libgccjit (which AFAIU is necessary to run the
> > native-compiled code), but doesn't have Binutils -- what will happen
> > when Emacs loads some .el file and tries to compile it?  Will the
> > response to the compilation failure be graceful?  (This situation is
> > likely to happen when users install a binary distro, but don't have
> > Binutils available.)
> 
> I believe binutils is a dependecy of libgccjit, so I guess having
> libgccjit without binutils should be classified as missconfiguration no?

Dependency in what way?  maybe if one installs from a Linux distro,
Binutils will be installed as a dependency.  But if someone downloads
a precompiled libgccjit DLL on Windows, for example, they will not
have Binutils unless they also install it.  Because libgccjit is part
of GCC, and one can AFAIK install MinGW GCC without installing
Binutils.

So I think my question still stands.

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> Date: Thu, 11 Nov 2021 21:01:37 +0000
>> 
>> > Would the code to implement this be complicated?  If not, please show
>> > the code you had in mind, so I could make sure we are on the same
>> > page.
>> 
>> I'm attaching the following patch.  It is very much untested but should
>> clarify what I had in mind.
>
> Thanks.  I think we should do this on master, but on the release
> branch compile gv.el AOT, as that will be safer.  WDYT?

I think is sensible.  I'll prepare and test this for master then.

>> > And while I have your attention, another question: suppose that Emacs
>> > with all the preloaded files compiled to *.eln is installed on a
>> > system that has libgccjit (which AFAIU is necessary to run the
>> > native-compiled code), but doesn't have Binutils -- what will happen
>> > when Emacs loads some .el file and tries to compile it?  Will the
>> > response to the compilation failure be graceful?  (This situation is
>> > likely to happen when users install a binary distro, but don't have
>> > Binutils available.)
>> 
>> I believe binutils is a dependecy of libgccjit, so I guess having
>> libgccjit without binutils should be classified as missconfiguration no?
>
> Dependency in what way?  maybe if one installs from a Linux distro,
> Binutils will be installed as a dependency.

Yes this was the scenarion I was thinking of.

> But if someone downloads
> a precompiled libgccjit DLL on Windows, for example, they will not
> have Binutils unless they also install it.  Because libgccjit is part
> of GCC, and one can AFAIK install MinGW GCC without installing
> Binutils.

Interesting, I didn't know it was a realistic scenarion.

Anyway this was never tested by me but... what should happen is that we
signal and error in `comp--compile-ctxt-to-file'.  For async compilation
this should be eventually reported in the async log buffer.

Best Regards

  Andrea

Re: Building a release tarball generates trampoline files in eln-cache

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Sat, 13 Nov 2021 20:33:39 +0000
> 
> > Thanks.  I think we should do this on master, but on the release
> > branch compile gv.el AOT, as that will be safer.  WDYT?
> 
> I think is sensible.

OK.  I ended up adding a few more files, all the dependencies of
comp.el and comp-cstr.el, just to be on the safe side.  The result is
satisfactory, I think: both "emacs -Q" and "emacs -Q -nw" come up
without a problem.

So I think we are approaching a point where we can finally make the
first pretest, thanks for your help on the way.  I just want to see
what's going on in bug#51689, which sounds like some similar problem,
perhaps, and do some more local testing.

And another question (I think you already answered that at some point,
sorry for my faulty memory): Emacs with native-compilation is not
supposed to try to compile ~/.emacs when it loads it at startup,
right?  But it does compile packages loaded by ~/.emacs, right?  And
the same with explicit *.el files loaded from the command-line, as in
"emacs -Q -l foo.el" -- the foo.el file will not be compiled, only any
packages it loads.  Correct?

Re: Building a release tarball generates trampoline files in eln-cache

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> Date: Sat, 13 Nov 2021 20:33:39 +0000
>> 
>> > Thanks.  I think we should do this on master, but on the release
>> > branch compile gv.el AOT, as that will be safer.  WDYT?
>> 
>> I think is sensible.
>
> OK.  I ended up adding a few more files, all the dependencies of
> comp.el and comp-cstr.el, just to be on the safe side.  The result is
> satisfactory, I think: both "emacs -Q" and "emacs -Q -nw" come up
> without a problem.

Very nice.

> So I think we are approaching a point where we can finally make the
> first pretest, thanks for your help on the way.  I just want to see
> what's going on in bug#51689, which sounds like some similar problem,
> perhaps, and do some more local testing.
>
> And another question (I think you already answered that at some point,
> sorry for my faulty memory): Emacs with native-compilation is not
> supposed to try to compile ~/.emacs when it loads it at startup,
> right?  But it does compile packages loaded by ~/.emacs, right?  And
> the same with explicit *.el files loaded from the command-line, as in
> "emacs -Q -l foo.el" -- the foo.el file will not be compiled, only any
> packages it loads.  Correct?

Correct, Emacs triggers automatically native compilation only when byte
compiled code is loaded.

~/.emacs is tipically not byte compiled and so is typically not native
compiled automatically, on the contrary packages tipically are.

Similarly explicit load of plain .el files will not cause native
compilation.


Best Regards

  Andrea