bug#46256: [feature/native-comp] AOT eln files ignored if run from build tree

[Eli Zaretskii <eliz@gnu.org>]

> From: Pip Cet <pipcet@gmail.com>
> Date: Mon, 8 Mar 2021 14:50:50 +0000
> Cc: Andrea Corallo <akrl@sdf.org>, 46256@debbugs.gnu.org, andrewjmoreton@gmail.com
> 
> > Well, how about explaining the details in terms that are simple enough
> > that I could understand and do the testing?
> 
> Excellent idea. I'll try!

Thanks.  Somewhat clearer now, but I'm still not out of the woods
yet.  Bear with me.

> Native-comp uses a pseudo-vector type representing a dlopen()ed handle.

You mean, the Lisp_Native_Comp_Unit structure?  If so, it doesn't
really represent a handle, AFAIU, it represents a .eln file we
loaded.  Right?

> In addition to the handle being stored in the pseudo-vector, a pointer
> to the pseudo-vector is stored in the data space belonging to the
> handle. I'll refer to that as the "reverse pointer" because I can't
> think of a better term right now.

Why do we need this reverse pointer, and how do we use it?

> When we cleanup the pseudo-vector, we don't reset the reverse pointer
> to NULL, or Qnil.

What do you mean by "cleanup" here?  Under what circumstances does it
happen?

And no, Qnil won't do, because a Lisp_Object can be wider than a
pointer (it is in the 32-bit build --with-wide-int).  NULL is fine.

> That is because we assume that the dlclose() we perform on cleanup
> will unmap the data space belonging to the handle, anyway.

But the call to dlclose doesn't happen immediately, it only happens in
GC.  Right?

(A nit: please don't use "foo()" to refer to a function, because that
looks like a call to 'foo' with no arguments, which is not what you
mean.)

> That assumption is wrong in certain specific circumstances.
> 
> In those circumstances, the reverse pointer is dereferenced after the
> vector has been deallocated. It points to a random different vector
> now.

I need to understand the circumstances under which this could happen.
If the vector has been deallocated, it means it was GC'ed, right?  And
if it was GC'ed, how come the .eln was not unloaded?

> One of the circumstances in which the assumption (that the reverse
> pointer won't be used) becomes invalid is when two pseudo-vectors
> share a handle (and, thus, a reverse pointer).

How can that happen? can you describe a series of events that make
this possible?

> My patch, thus, resets the reverse pointer to Qnil when cleanup is
> performed.

You can't use Qnil, for the reasons described above.

P.S. The stuff described in this sub-thread should eventually find its
way into comments in comp.c.