dynamic reload of dynamic module not dynamic?

dynamic reload of dynamic module not dynamic?

[Emanuel Berg via Users list for the GNU Emacs text editor]

It seems recompiling the C for an Emacs dynamic module and
then loading the new SO file from the same Emacs instance that
loaded the old doesn't get you the new stuff?

So not so dynamic after all, ey?

See this example project with screenshots,

  https://dataswamp.org/~incal/emacs-init/random-urandom/

-- 
underground experts united
https://dataswamp.org/~incal

Re: dynamic reload of dynamic module not dynamic?

[Eduardo Ochs]

On Sun, 17 Apr 2022 at 14:20, Emanuel Berg via Users list for the GNU
Emacs text editor <help-gnu-emacs@gnu.org> wrote:
>
> It seems recompiling the C for an Emacs dynamic module and
> then loading the new SO file from the same Emacs instance that
> loaded the old doesn't get you the new stuff?
>
> So not so dynamic after all, ey?
>
> See this example project with screenshots,
>
>   https://dataswamp.org/~incal/emacs-init/random-urandom/
>
> --
> underground experts united
> https://dataswamp.org/~incal


Hi Emanuel,

Are you using `load' or `require'?

If I remember correctly, when I used

            (load "/tmp/emlua.so")

my module would be reloaded, but when I used

  (require 'emlua "/tmp/emlua.so")

it would not...

The full example is here:

  http://angg.twu.net/emlua/emlua.cpp.html
  http://angg.twu.net/emlua/emlua.cpp.html#tests-in-tmp
  https://github.com/edrx/emlua#introduction

Cheers,
  E. ...

Re: dynamic reload of dynamic module not dynamic?

[Philipp Stephani]

> Am 17.04.2022 um 06:42 schrieb Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>:
> 
> It seems recompiling the C for an Emacs dynamic module and
> then loading the new SO file from the same Emacs instance that
> loaded the old doesn't get you the new stuff?

Yeah, we never call dlclose on the returned shared object handle, so reloading a module will only increment the handle's reference count. It might make sense to support reloading in some way (by having unload-feature eventually call dlclose), but it's not trivial: we need to be careful to invalidate all references to affected module functions before calling dlclose.

Re: dynamic reload of dynamic module not dynamic?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Philipp Stephani wrote:

>> It seems recompiling the C for an Emacs dynamic module and
>> then loading the new SO file from the same Emacs instance
>> that loaded the old doesn't get you the new stuff?
>
> Yeah, we never call dlclose on the returned shared object
> handle, so reloading a module will only increment the
> handle's reference count. It might make sense to support
> reloading in some way (by having unload-feature eventually
> call dlclose), but it's not trivial: we need to be careful
> to invalidate all references to affected module functions
> before calling dlclose.

Okay, where are they stored then so one can apply invalidate
to all members and then call dlclose and `load' again?

-- 
underground experts united
https://dataswamp.org/~incal

Re: dynamic reload of dynamic module not dynamic?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Eduardo Ochs wrote:

> Are you using `load' or `require'?

`load', see the source and full/complete project [1]

> If I remember correctly, when I used
>
>   (load "/tmp/emlua.so")
>
> my module would be reloaded

Okay, did you edit the source to do something different after
the first load, recompile, load again, and eval the altered
function to observe the new behavior? Because I get the old
behavior ...

[1] https://dataswamp.org/~incal/emacs-init/random-urandom/

-- 
underground experts united
https://dataswamp.org/~incal

Re: dynamic reload of dynamic module not dynamic?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2106 bytes --]

On Sun, Apr 17, 2022 at 10:12:55PM +0200, Emanuel Berg via Users list for the GNU Emacs text editor wrote:
> Philipp Stephani wrote:
> 
> >> It seems recompiling the C for an Emacs dynamic module and
> >> then loading the new SO file from the same Emacs instance
> >> that loaded the old doesn't get you the new stuff?
> >
> > Yeah, we never call dlclose on the returned shared object
> > handle, so reloading a module will only increment the
> > handle's reference count. It might make sense to support
> > reloading in some way (by having unload-feature eventually
> > call dlclose), but it's not trivial: we need to be careful
> > to invalidate all references to affected module functions
> > before calling dlclose.
> 
> Okay, where are they stored then so one can apply invalidate
> to all members and then call dlclose and `load' again?

This is not completely trivial. Parts of the application may
have become dependent on the loaded library. That's why dlclose()
is described as "advisory" [1]. It may do, but then, it might
not.

Just imagine your application storing a pointer to some function
in that DL. You remove it, the address space becomes free and
is replaced by something totally different. The above code calls
into this pointer and... kaboom. You literally pulled the rug
from under your poor application :-)

Now you would say you wouldn't ever hoard pointers to random
library functions, but that's exactly what the dynamic loader
is doing for you.

Not saying it's impossible, but that it's trickier than it seems.
Most of the time you'll have to live with several incarnations
of "some functions" because your "old function" refuses to die.
Most applications choose to restart.

Just search for "dlclose problem" in the GNU libc mailing list
to see what I mean. Here's [2] one.

Cheers

[1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/dlclose.html
[2] https://sourceware.org/pipermail/libc-help/2010-November/thread.html#1644

-- 
t


> 
> -- 
> underground experts united
> https://dataswamp.org/~incal
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: dynamic reload of dynamic module not dynamic?

[Emanuel Berg via Users list for the GNU Emacs text editor]

tomas wrote:

> This is not completely trivial. Parts of the application may
> have become dependent on the loaded library. [...]

I don't see how that is any different from eval'ing a Lisp
function, if something else uses it and the new version don't
do what the old did it sure won't happen (or fail, if you
change the interface which you should always change with care
anyway). Or introduce another bug. But how would it else be,
dynamic update of code which only works when the updates
makes sense?

> Just imagine your application storing a pointer to some
> function in that DL. You remove it, the address space
> becomes free and is replaced by something totally different.
> The above code calls into this pointer and... kaboom.
> You literally pulled the rug from under your poor
> application :-)

What we are imagining is rather a change that made sense so
the poor application can have its own rags to riches story ...

> Now you would say you wouldn't ever hoard pointers to random
> library functions, but that's exactly what the dynamic
> loader is doing for you.

You mean like this?

  https://dataswamp.org/~incal/emacs-init/random-urandom/random-urandom.c

> Not saying it's impossible, but that it's trickier than it
> seems. Most of the time you'll have to live with several
> incarnations of "some functions" because your "old function"
> refuses to die. Most applications choose to restart.

Tricky but maybe it can be made simple, look

  1. a uses b, what happens is b
  2. b is edited (let's call it b' just to denote the change)
  3. a uses b, what happens is b'

?

-- 
underground experts united
https://dataswamp.org/~incal

Re: dynamic reload of dynamic module not dynamic?

[Philipp Stephani]

Am Mo., 18. Apr. 2022 um 23:07 Uhr schrieb Emanuel Berg via Users list
for the GNU Emacs text editor <help-gnu-emacs@gnu.org>:
>
> Philipp Stephani wrote:
>
> >> It seems recompiling the C for an Emacs dynamic module and
> >> then loading the new SO file from the same Emacs instance
> >> that loaded the old doesn't get you the new stuff?
> >
> > Yeah, we never call dlclose on the returned shared object
> > handle, so reloading a module will only increment the
> > handle's reference count. It might make sense to support
> > reloading in some way (by having unload-feature eventually
> > call dlclose), but it's not trivial: we need to be careful
> > to invalidate all references to affected module functions
> > before calling dlclose.
>
> Okay, where are they stored then so one can apply invalidate
> to all members and then call dlclose and `load' again?

I don't think we have a single exhaustive location, but at the very
least we'd need to invalidate all corresponding module functions
before unloading a module, probably repointing them to some function
that always signals an error. There is no mapping from module handles
to corresponding module functions yet, but it shouldn't be terribly
hard to add one (but I think first we should discuss whether the FR is
important enough for that). Another category would be user pointers,
since they are likely to point nowhere once a module gets unloaded.

Re: dynamic reload of dynamic module not dynamic?

[Emanuel Berg via Users list for the GNU Emacs text editor]

Philipp Stephani wrote:

>>> Yeah, we never call dlclose on the returned shared object
>>> handle, so reloading a module will only increment the
>>> handle's reference count. It might make sense to support
>>> reloading in some way (by having unload-feature eventually
>>> call dlclose), but it's not trivial: we need to be careful
>>> to invalidate all references to affected module functions
>>> before calling dlclose.
>>
>> Okay, where are they stored then so one can apply
>> invalidate to all members and then call dlclose and
>> `load' again?
>
> I don't think we have a single exhaustive location, but at
> the very least we'd need to invalidate all corresponding
> module functions before unloading a module, probably
> repointing them to some function that always signals an
> error. There is no mapping from module handles to
> corresponding module functions yet, but it shouldn't be
> terribly hard to add one (but I think first we should
> discuss whether the FR is important enough for that).
> Another category would be user pointers, since they are
> likely to point nowhere once a module gets unloaded.

I don't understand why anything has to be discontinued
necessarily, why can't it just be made available again and
what's new (or changed) is new? The way it already is
in Elisp?

That property with Emacs and Elisp is why we are all here
BTW :) If one had had to restart Emacs for every change no one
would ever get hooked and use Emacs for everything always.
Indeed it wouldn't be possible to do even ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: dynamic reload of dynamic module not dynamic?

[Eli Zaretskii]

> Date: Tue, 19 Apr 2022 13:08:21 +0200
> From:  Emanuel Berg via Users list for the GNU Emacs text editor <help-gnu-emacs@gnu.org>
> 
> tomas wrote:
> 
> > This is not completely trivial. Parts of the application may
> > have become dependent on the loaded library. [...]
> 
> I don't see how that is any different from eval'ing a Lisp
> function

Because C is a compiled language, not an interpreted one, and the
rules and machinery involved in loading shared libraries are (a) much
more complex than the rules involved in loading Lisp, and (b) we don't
control how this stuff works because this is done by the OS, not by
Emacs.

Re: dynamic reload of dynamic module not dynamic?

[Psionic K]

> but I think first we should discuss whether the FR is important enough for that

Full reload?  Absolutely indispensable and the right decision.

However, I think we're looking at the challenges wrong, worrying about
problems that could exist if dynamic modules haphazardly return
pointers to owned data rather than focusing on communicating the right
contract to dynamic module authors so that such problems don't occur.
The former conversation presumes too much responsibility on the part
of Emacs for things it can't be responsible for.

Instead of saying, "what if the dynamic module does this?" where
"this" would be hard for Emacs to manage in the next five or ten
years, tell dynamic module authors to conform to a contract that is
simple for Emacs already.  That way we're not all blocking each other.

Just looking at the Rust module code, I see a mechanism to give the
Emacs GC ownership of a value.  If that can be done, why would I hold
in Elisp any pointers into memory managed by the dynamic module?  When
we do want to name and point to things in the dynamic module, we
should do so by identifier, not a raw pointer.  That way the dynamic
module can look before leaping, using whatever identifier mapping the
module author sees fit.

Any loss in speed of not handing pointers over to Emacs is silly to
worry about because the presumption of the entire scheme is that Elisp
is delegating human speed command input to a fast and capable
implementation.  Lookups for identifiers are faster than human speed,
so the cost of looking up identifiers at the human to native interface
is negligible.  There was a keystroke.  We shouldn't be counting
microseconds.  The presumption is that we saved ten seconds of Elisp
or did something Elisp can't do well.

From the rest of the thread and in general, I got the impression that
Emacs devs are, out of experience with users making new problems for
themselves, being too protective in thinking.  Dynamic module authors
should be held to a higher standard and not treated like a potential
source of annoying bug reports.  Just set some ground rules that will
make upstream behavior within Emacs easier to plan on rather than
presuming too much responsibility.  The user is writing a dynamic
module.  They have opted into knowing what they are doing.  There
should minimally only be ways made available to write a good module.
The contract should be that Emacs can reload the module and that it's
the module's responsibility not to violate that contract by handing
back data that won't work through reload.  Emacs should treat them
like oracles that are responsible for not screwing up.

I think this is the right conversation to have.  It's a C interface.
The user is responsible.  Honestly it is more of a headache to
consider how to tell Emacs to invalidate raw pointers than to manage
the data correctly within the module.  There's infinite other ways to
crash programs that dynamic module authors are also responsible for.
It's nothing new.  Delegate this responsibility to the maximum extent
possible and dynamic module authors will figure it out with libraries.

The alternative is to develop by throwing away Emacs instances or load
lots of renamed and annoying duplicate modules?  A lot of Elisp state
might be built up as the user is hacking away with a module and
associated Elisp concurrently.  When forced to reload Emacs, it will
just break the workflow.  This is not a 1.0 version of the dynamic
module feature because it negates the core strength of interactive
dynamic development.

In the end, the worst case of badly behaved dynamic modules had nearly
identical impact to the user: reload Emacs (if it didn't politely
crash already).

Thanks