bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

[Pip Cet <pipcet@gmail.com>]

On Sat, Mar 6, 2021 at 1:48 AM Andy Moreton <andrewjmoreton@gmail.com> wrote:
> On Fri 05 Mar 2021, Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

[Does anyone know where this via "name" comes from? I believe it is
Google's joke which somehow makes it through into the gmail
interface...]

> Is the problem that dlopen resolves to use an unlinked file kept alive
> by having open handles, rather than a new file with the filename used
> by the old file before it was unlinked ?

I believe so, and that's what I think we can work around.

IIUC, we don't actually call dlclose() until we GC (and might not do
so even then, since GC is conservative).

> >> Merely verifying that the ABI is correct could be done at runtime, so
> >> that's no reason to keep a hash in the filename.
> >>
> >> So my vague idea is this:
> >>
> >> 1. implement fixed_dlopen(), which keeps track of filenames that have
> >> been opened and, if necessary, creates a temporary file and loads that
> >> instead of its argument.
> >> 2. compile lisp/emacs-lisp/bytecomp.el to lisp/emacs-lisp/bytecomp.elc
> >> and native-lisp/emacs-lisp/bytecomp.eln
> >
> > So it was at the beginning, I think we moved away from that before the
> > odd dlopen behavior.
>
> As above, this odd dlopen behaviour needs to be fully explained to
> ensure that design choices are not driven by possible misunderstandings.

I'm unsure what Andrea is saying here; is the dlopen thing relevant to
the decision to use hashes in names, or isn't it?

> >> 3. add extra code in the top level function of each .eln to check that
> >> the ABI is correct.
> >>
> >> This would allow us to use standard make rules. It would also make
> >> .eln filenames predictable. It might even draw someone's attention to
> >> the fact that dlopen() is broken and make them fix it.
> >>
> >> I'm probably missing other good reasons for the hashed filename scheme.
> >
> > Yep, this was discussed in length on emacs-devel, IIRC mainly on a long
> > standing thread called "native compilation the bird-eye view" (or
> > something close).
>
> Thread "Native compilation: the bird-eye view" starts here:
> https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg02186.html

Thanks for the link. I do think it would be good to summarize the
reasons for the hash-based naming scheme somewhere, because I've read
the thread and all I've taken away is that the dlopen oddity requires
a workaround (but, really, a different one). I don't think I've read
all of the followup threads, though.

> I agree with Pip that using standard make rules eases several development
> pains and should be used if at all possible.

What I think should be discussed, or should have been discussed, is
whether we really need hashes in the names of files in the Emacs build
tree. Whether we need them for installed files, or for files in the
eln cache, is a separate issue.

A second question is whether it's really worth it to build the elc and
eln files at the same time. Make would be a lot happier not having two
targets in the rule, and so would I.

But if this has been discussed and resolved, we merely need to
document the decision and the reasons for it rather than reopening
discussion because I missed it the first time around. If someone can
provide a link to the relevant messages, I'd be glad to try.

Pip