Re: Dynamic loading progress

[Eli Zaretskii <eliz@gnu.org>]

> Date: Mon, 16 Feb 2015 12:01:40 -0800
> From: Daniel Colascione <dancol@dancol.org>
> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> 
> >>> This approach requires us to change the size each time we change the
> >>> layout, even if the change itself leaves the size intact.  Using a
> >>> version field doesn't have this disadvantage.
> >>
> >> Emacs is allowed to pass a new version of the runtime and context
> >> structures to modules expecting old versions. That can only work if the
> >> structures are append-only, so `size' is adequate here.
> > 
> > I understand that, but appending members does not necessarily increase
> > the size of the struct, due to alignment, bit fields, etc.
> 
> If it's just a table of function pointers and everything has natural
> alignment, adding members will increase the size.

Having to include only function pointers is also a kind of limitation
inherent to this method.

I guess I just don't understand why you didn't go with a version
field.  What are its disadvantages?  The value can be generated
automatically, so it isn't more error-prone than the size.

> Say do_frobricate has address 0xA3B41234. Implementation-wise, the value
> `frobricate` that `make_function' returned is a Lisp closure that
> essentially looks like this:
> 
>   (lambda (&rest args) (%do-module-call frobricate-module 0xA3B41234 2 2
> args))
> 
> %do-module-call then clears the pending-error information, packs its
> arguments into a C array, and calls 0xA3B41234 like this:
> 
>   emacs_subr module_function = (module_function) 0xA3B41234;
>   result = module_function(some_env, args, nargs);

It's the arguments that I was talking about.  AFAIU, you are saying
that %do-module-call takes arguments, which are normal Lisp_Object
values, generates the indirection table for them, which stores
_pointers_ to the actual arguments, then puts those pointers into a C
array, and calls 0xA3B41234.  Is that correct?

If so, I guess I don't see why we would need all that complexity and
extra processing.  Why not put the original Lisp_Object values into
the args[] array to begin with?

> >>> What about the doc string?
> >>
> No: that's silly. I'm saying that creating a globally-named function in
> Emacs has two parts: 1) create the actual _function object_. 2) bind
> that function object to a name by setting some symbol's function slot to
> that function object. We can do step #2 entirely in Lisp code that the
> module calls, and we can set a documentation property at this time. It's
> only step #1 that requires a magical incantation.
> 
> I've only described the magical, ABI-relevant bits of the interface. In
> order to make a module actually usable, we'll need glue, but since we
> can implement this glue in Lisp and call it from module code using the
> narrow interface I've described here, I'm not worried about it.

Sorry, I'm still not out of the woods.  Providing a doc string has
nothing to do with calling the function.  You need to have the doc
string available even if you never call the function.  The trigger for
accessing the doc string is _outside_ of the module, by one of the
help commands.  So how does all this work in your scenario above,
where we called make_function, which returned 'frobnicate', which is a
value in the function slot of some Lisp object?  Say now Emacs calls
'documentation-property' on that Lisp object or it calls
'documentation' on 'frobnicate' -- how will that produce a doc string?

> >>> Do we need 'unintern' as well?
> >>
> >> Modules can call unintern through Lisp.
> > 
> > Then why provide 'intern'?  It, too, can be called from Lisp.
> 
> Modules have no way other than `intern' of getting an emacs_value that
> corresponds to a known symbol. They can call `unintern' by first
> interning the symbol "unintern", then calling it.

You mean, modules cannot use funcall to call Lisp functions, without
some complicated dance with 'intern' before that?  Why is that needed?
why not let modules call Lisp functions by their string name, and do
the 'intern' stuff on the Emacs side of the interface?

> >>>>     emacs_value (*type_of)(
> >>>>       emacs_env* env,
> >>>>       emacs_value value);
> >>>>
> >>>> Like Lisp type-of: returns a symbol.
> >>>
> >>> What is a "symbol", from the module's C code POV?
> >>
> >> It's an emacs_value. You can compare symbols (by calling `eq'), call
> >> them as functions, or use them as function arguments. That's sufficient.
> >>
> >> Come to think of it, though, we do need a C-level `eq'.
> > 
> > Why not simply return a C enumeration type?
> 
> Why make the set of elisp primitive types part of the ABI?

An enum is just a list of named integer values.  Additions to that
list make more values valid, that's all.  How is that "part of the
ABI" that could introduce incompatibility?

OTOH, making the life of module writers harder on every turn is not a
good way of encouraging modules.

> >>>  . not sure how will a module "provide" its feature
> >>
> >> It doesn't. You don't require a module: you load it. Most modules will
> >> come with a small Lisp wrapper, just like Python modules.
> > 
> > So do we _require_ a Lisp wrapper?  Otherwise, how would Emacs know
> > whether the module is loaded or not?
> 
> By path, I imagine.

What path is that?  Where and how will Lisp code find it?

> Why don't you have this problem with (load "myfile.el")?

Because myfile.el normally does a (provide 'myfile), and then I can
use featurep.

> I'm not opposed to hooking modules into the provide-and-require
> machinery, but I don't see a particular need either.

The need is to have an easy way of checking that a module is
available, without risking loading it twice, or having to jump through
hoops.

> >>> One thing that's inconvenient is the need to drag the environment
> >>> pointer through all the calls.  Why exactly is that needed?
> >>
> >> Modules have no other way of accessing Emacs internals. Dragging one
> >> parameter through the system isn't that bad and lets us sanely track
> >> which module is doing what.
> > 
> > The environment must be known to the module, that's for sure.  But why
> > does it have to be plugged into every call to every interface
> > function?
> 
> How else are the interface functions supposed to know the module for
> which they're being called?

Why would the interface functions care?  Which parts of the
environment are needed for the interface to know about the module in
order to do its job?