bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

[Andy Moreton <andrewjmoreton@gmail.com>]

On Fri 05 Mar 2021, Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of text editors" wrote:

> Pip Cet <pipcet@gmail.com> writes:
>
>> On Fri, Mar 5, 2021 at 9:33 AM Andrea Corallo via Bug reports for GNU
>> Emacs, the Swiss army knife of text editors <bug-gnu-emacs@gnu.org>
>> wrote:
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> > So currently the only way to fill up a newly created subdirectory of
>>> > native-lisp/ is to manually delete the *.elc files of all the files in
>>> > lisp.mk's $shortlisp list, is that sufficient?
>>>
>>> Yes I think so.
>>>
>>> The trouble of using make for building such a system is that make is not
>>> aware of the .eln filename, so it should be necessary to ask the Emacs
>>> binary about that to create dynamically the precise (multiple target)
>>> rule.  Not very practical IMO...
>>
>> I do wonder whether the whole filename scheme is really the best option.
>>
>> IIUC, and that's a big if in this case, the main motivation for using
>> hashes in the .eln filenames is that dlopen() is broken and may return
>> the same handle for subsequent dlopen()s of the same name, even if the
>> underlying file changed in between.
>
> Unfortunately this was only an unfortunate discover along the road...
> this design predates that.

Can you explain what the problem is with dlopen ? I have not found a
complete and precise description of the problem in earlier messages as a
reproducer.

Is the problem that dlopen resolves to use an unlinked file kept alive
by having open handles, rather than a new file with the filename used
by the old file before it was unlinked ?

>> Merely verifying that the ABI is correct could be done at runtime, so
>> that's no reason to keep a hash in the filename.
>>
>> So my vague idea is this:
>>
>> 1. implement fixed_dlopen(), which keeps track of filenames that have
>> been opened and, if necessary, creates a temporary file and loads that
>> instead of its argument.
>> 2. compile lisp/emacs-lisp/bytecomp.el to lisp/emacs-lisp/bytecomp.elc
>> and native-lisp/emacs-lisp/bytecomp.eln
>
> So it was at the beginning, I think we moved away from that before the
> odd dlopen behavior.

As above, this odd dlopen behaviour needs to be fully explained to
ensure that design choices are not driven by possible misunderstandings.

>> 3. add extra code in the top level function of each .eln to check that
>> the ABI is correct.
>>
>> This would allow us to use standard make rules. It would also make
>> .eln filenames predictable. It might even draw someone's attention to
>> the fact that dlopen() is broken and make them fix it.
>>
>> I'm probably missing other good reasons for the hashed filename scheme.
>
> Yep, this was discussed in length on emacs-devel, IIRC mainly on a long
> standing thread called "native compilation the bird-eye view" (or
> something close).

Thread "Native compilation: the bird-eye view" starts here:
https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg02186.html

I agree with Pip that using standard make rules eases several development
pains and should be used if at all possible.

    AndyM