Re: Dynamic loading progress

[Eli Zaretskii <eliz@gnu.org>]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 14 Feb 2015 19:02:48 -0600
> 
> I took a stab at creating emacs_module_api.h; attached.
> 
> To do this, I replaced "#include <lisp.h>"
> in modules/curl/curl.c with "#include <emacs_modules_api.h>", and copied
> stuff from lisp.h and other headers to emacs_module_api.h until it
> compiled. Note the FIXMEs; I could not figure out what was going on in
> some places, so I just kludged it for now.

Thanks.

I started to write comments to that file, but gave up in despair.  My
overall conclusion from reviewing it is that, if we want the
tightly-coupled version of modules, we might as well compile them as
any other C source file in the Emacs tree, i.e. we could simply
include lisp.h in its entirety, and all the other headers that it
pulls in.  I see no other workable alternative that would let modules
use Lisp objects transparently (as opposed to using them as opaque
objects), call Lisp primitives directly like we do in Emacs sources,
etc.

> >    This already presents at least 2 problems.  The first is 32- vs
> >    64-bit issues.  We need some way of checking that a 32-bit module
> >    isn't loaded by a 64-bit Emacs, and vice versa.

> Good point. That's a special case of "don't load a module compiled for
> processor X in an Emacs compiled for processor Y". gnu binutils has
> facilities for that.

Not sure how Binutils could help here.  AFAIK, dlopen and dlsym (and
their equivalents) don't invoke Binutils on any platform.

> >    The other problem is more serious: Emacs can be built with or
> >    without --check-lisp-object-type, and with or without --wide-int;
> >    how will the module know which implementation to use?
> 
> Similar check when the module is loaded. Which means we need some
> metainfo for each module, and a standard function to retrieve it.

That "metainfo", whatever it will be, will also have to support
compilation of modules: we need to know at compile time what values of
USE_LSB_TAG, ENABLE_CHECKING, WIDE_EMACS_INT, optimization flags,
etc. were used when Emacs was built, because those affect the layout
of objects inside Emacs and also the availability of certain functions
in the Emacs binary.

> >  . Functions to access values of Lisp objects.  We shouldn't rely on C
> >    macros like XINT and XWINDOW for that, because macros track too
> >    closely the internals of each object.  So I propose to use
> >    functions that will be exposed via the API.
> 
> If we follow Stefan's suggestion, then either this function approach is
> not viable, or we need another #ifdef for module vs emacs.

If we don't use the function approach, we can only have modules that
are tightly coupled with the version of Emacs for which they were
written and compiled.  We have no hope of binary compatibility with
other versions of Emacs, except by sheer luck, and even source-level
compatibility will be a challenge.

> > They are currently written as if the code were an integral part of
> > Emacs on one of the Emacs C files. Going that way means that modules
> > will have to be recompiled from sources for each Emacs version, and
> > practically will have to live in the Emacs tree. Maybe this is what we
> > want, I don't know.
> 
> This is the model I had in mind. Since I need max speed with mixed
> Ada/lisp code, I need tight integration with the core. The people who
> need that speed for Ada mode will simply have to recompile the module
> for each Emacs release; they are coders, so that's not a problem.

Being an Ada programmer (or even a C programmer) doesn't necessarily
mean you know how to fix compile- and link-time problems with Emacs.
You've just bumped into that yourself.

In practice, I think this means maintainers of modules will have to
adapt their modules to each Emacs version, and perhaps also keep
several branches, one each for every version they want to support.

> Rather than splitting out emacs_module_api.h from lisp.h, we could add
> __declspec(dllexport) to a subset of the functions in lisp.h; I suspect
> that will be a small number.

What for?  If we are going to expose most or all of lisp.h to modules,
we are actually saying that all of Emacs internals should be exposed
as well, so all of the lisp.h stuff should be externally visible, as
it is used all over the place in Emacs sources, and modules will want
to be able to do the same, under the "tightly-coupled" concept.

> Is there an equivalent on Linux (I don't recall ever seeing one)?

You want '__attribute__((visibility ("default")))', and you want to
use -fvisibility=hidden on the command line.  See

  https://gcc.gnu.org/wiki/Visibility

But this looks futile wrt lisp.h, see above.  Perhaps we could use
that for other public functions in other headers, but I expect that
paradigm to collapse very quickly, since modules will want to call
Lisp primitives directly, not through funcall.  Look at the modules
already on the branch: they include buffer.h and character.h, and call
stuff like Fgethash, Fputhash, and Fmake_hash_table.  How much time
will pass before we see modules call Finsert_file_contents,
Fmake_frame, and Fredisplay?  How can we explain to module developers
that they need to go through funcall?  And then "more internal"
internals will quickly follow: scan_words, for example, or
recenter_overlay_lists.  It's a slippery slope, but I don't see how we
could prevent the slip if we are willing to go with "tightly-coupled"
modules.