Callbacks from modules

Callbacks from modules

[Ivan Andrus]

I’ve started writing an Emacs module to access NSSpeechSynthesizer on OS X (and maybe GNUStep though I haven’t tested it there).  I’ve been using the mod-test module as an example and I have it working reasonably well except that I don’t have any idea how to run a callback.

When the synthesizer is done speaking it sends an Objective-C message speechSynthesizer:didFinishSpeaking.  From there I would like to run an elisp hook, say `ns-speech-finished-speaking-hook`.  How can I do this?  Do I have to squirrel away a pointer to an emacs_env somehow?  I tried naively storing env from a previous call but, not surprisingly, the pointer is invalid when I try to use it.

Thanks,
Ivan

Re: Callbacks from modules

[Eli Zaretskii]

> From: Ivan Andrus <darthandrus@gmail.com>
> Date: Wed, 25 Nov 2015 23:08:48 -0700
> 
> I’ve started writing an Emacs module to access NSSpeechSynthesizer on OS X (and 
> maybe GNUStep though I haven’t tested it there).  I’ve been using the mod-test 
> module as an example and I have it working reasonably well except that I don’t 
> have any idea how to run a callback.
> 
> When the synthesizer is done speaking it sends an Objective-C message 
> speechSynthesizer:didFinishSpeaking.  From there I would like to run an elisp 
> hook, say `ns-speech-finished-speaking-hook`.  How can I do this?  Do I have to 
> squirrel away a pointer to an emacs_env somehow?  I tried naively storing env 
> from a previous call but, not surprisingly, the pointer is invalid when I try 
> to use it.

How would you do that in Emacs's core code?  A module is just a
(limited) extension of the Emacs core, so when you ask such questions,
you should first think how Emacs core does that.

Anyway, I assume you have a C callback in your module that is
triggered by the speechSynthesizer:didFinishSpeaking message, is that
right?  Then one way would be to have that callback set a flag,
provide a Lisp-callable function that returns the flag, and then start
a timer that will test the flag and call your Lisp callback when the
flag is set.

Another, perhaps simpler, possibility would be to have the module
provide a Lisp-callable function that will register a Lisp callback.
Then your C callback will simply call that Lisp function.

Would any of these do the job?

Re: Callbacks from modules

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ivan Andrus <darthandrus@gmail.com>
>> Date: Wed, 25 Nov 2015 23:08:48 -0700
>> 
>> I’ve started writing an Emacs module to access NSSpeechSynthesizer on OS X (and 
>> maybe GNUStep though I haven’t tested it there).  I’ve been using the mod-test 
>> module as an example and I have it working reasonably well except that I don’t 
>> have any idea how to run a callback.
>> 
>> When the synthesizer is done speaking it sends an Objective-C message 
>> speechSynthesizer:didFinishSpeaking.  From there I would like to run an elisp 
>> hook, say `ns-speech-finished-speaking-hook`.  How can I do this?  Do I have to 
>> squirrel away a pointer to an emacs_env somehow?  I tried naively storing env 
>> from a previous call but, not surprisingly, the pointer is invalid when I try 
>> to use it.
>
> How would you do that in Emacs's core code?  A module is just a
> (limited) extension of the Emacs core, so when you ask such questions,
> you should first think how Emacs core does that.
>
> Anyway, I assume you have a C callback in your module that is
> triggered by the speechSynthesizer:didFinishSpeaking message, is that
> right?  Then one way would be to have that callback set a flag,
> provide a Lisp-callable function that returns the flag, and then start
> a timer that will test the flag and call your Lisp callback when the
> flag is set.
>
> Another, perhaps simpler, possibility would be to have the module
> provide a Lisp-callable function that will register a Lisp callback.
> Then your C callback will simply call that Lisp function.
>
> Would any of these do the job?
>
>

For the xwidget code I used events. Wouldn't that work here as well?

-- 
Joakim Verona

Re: Callbacks from modules

[Eli Zaretskii]

> From: joakim@verona.se
> Cc: Ivan Andrus <darthandrus@gmail.com>,  emacs-devel@gnu.org
> Date: Thu, 26 Nov 2015 17:17:40 +0100
> 
> For the xwidget code I used events. Wouldn't that work here as well?

The module API doesn't provide any means to inject events into the
Emacs event queue, AFAICT.

Re: Callbacks from modules

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 519 bytes --]

+Tom Tromey <tom@tromey.com>

This is a essentially the same issue as
https://github.com/aaptel/emacs-dynamic-module/issues/41.

Eli Zaretskii <eliz@gnu.org> schrieb am Do., 26. Nov. 2015 um 17:34 Uhr:

> > From: joakim@verona.se
> > Cc: Ivan Andrus <darthandrus@gmail.com>,  emacs-devel@gnu.org
> > Date: Thu, 26 Nov 2015 17:17:40 +0100
> >
> > For the xwidget code I used events. Wouldn't that work here as well?
>
> The module API doesn't provide any means to inject events into the
> Emacs event queue, AFAICT.
>
>

[-- Attachment #2: Type: text/html, Size: 1148 bytes --]

Re: Callbacks from modules

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 26 Nov 2015 17:01:48 +0000
> Cc: darthandrus@gmail.com, emacs-devel@gnu.org
> 
> This is a essentially the same issue as
> https://github.com/aaptel/emacs-dynamic-module/issues/41.

I didn't understand the issue well enough, but if you say so...


Once again, I think registering a Lisp callback will allow it to be
called by the C callback, when the event arrives.  Of course, if the
arriving event calls the C callback on another thread, then Emacs will
abort due to the thread check...

So I guess the timer solution is the way to go.

Re: Callbacks from modules

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2190 bytes --]

Ivan Andrus <darthandrus@gmail.com> schrieb am Do., 26. Nov. 2015 um
07:09 Uhr:

> I’ve started writing an Emacs module to access NSSpeechSynthesizer on OS X
> (and maybe GNUStep though I haven’t tested it there).  I’ve been using the
> mod-test module as an example and I have it working reasonably well except
> that I don’t have any idea how to run a callback.
>
> When the synthesizer is done speaking it sends an Objective-C message
> speechSynthesizer:didFinishSpeaking.  From there I would like to run an
> elisp hook, say `ns-speech-finished-speaking-hook`.  How can I do this?  Do
> I have to squirrel away a pointer to an emacs_env somehow?  I tried naively
> storing env from a previous call but, not surprisingly, the pointer is
> invalid when I try to use it.
>
>
Yes, that is currently working as designed. Environments are alive only
while Emacs calls some module functions.

There are two major issues with allowing callbacks at arbitrary times,
which need to be discussed and resolved first. The first is that such a
facility would change the execution model of Emacs in a quite fundamental
way. Right now all user-defined code inside the Emacs process gets run
through a path under the control of Emacs, either as a response to an input
event, or a timer, or a process filter etc. Adding a facility to run module
code from arbitrary callbacks would remove this invariant, which can have
global effects on Emacs. With the current design modules are a pretty
localized feature: almost no other part of Emacs has to know about their
existence. If we allow callbacks at arbitrary times all parts of Emacs
would suddenly have to be prepared for this, and after decades of
development there might well be hidden assumptions that such things cannot
happen. Therefore I'd be very cautious before introducing such a facility.

The second issue is how such a facility should be made available. Currently
all communication between modules and Emacs happens through modules, and
there is no facility how modules can request creation of an environment.
However, it makes little sense to discuss this issue until the first one is
resolved.

[-- Attachment #2: Type: text/html, Size: 2478 bytes --]

Re: Callbacks from modules

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Thu, 26 Nov 2015 18:16:13 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, Daniel Colascione <dancol@dancol.org>
> 
> There are two major issues with allowing callbacks at arbitrary times, which
> need to be discussed and resolved first. The first is that such a facility
> would change the execution model of Emacs in a quite fundamental way. Right now
> all user-defined code inside the Emacs process gets run through a path under
> the control of Emacs, either as a response to an input event, or a timer, or a
> process filter etc. Adding a facility to run module code from arbitrary
> callbacks would remove this invariant, which can have global effects on Emacs.

I don't even understand what is meant by "arbitrary callbacks".  If
they are called in the context of a separate thread, we will not allow
them to run.

IOW, such arbitrary callbacks need to run in the C land, and
communicate with Lisp via timers.  In the future, we might add
facilities that would allow a module register a file descriptor
through which events could flow into the Emacs event queue.  But that
is not yet available, and I'm not sure we'd want to allow it.

Re: Callbacks from modules

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Do., 26. Nov. 2015 um 19:41 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Thu, 26 Nov 2015 18:16:13 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, Daniel Colascione <dancol@dancol.org>
> >
> > There are two major issues with allowing callbacks at arbitrary times,
> which
> > need to be discussed and resolved first. The first is that such a
> facility
> > would change the execution model of Emacs in a quite fundamental way.
> Right now
> > all user-defined code inside the Emacs process gets run through a path
> under
> > the control of Emacs, either as a response to an input event, or a
> timer, or a
> > process filter etc. Adding a facility to run module code from arbitrary
> > callbacks would remove this invariant, which can have global effects on
> Emacs.
>
> I don't even understand what is meant by "arbitrary callbacks".  If
> they are called in the context of a separate thread, we will not allow
> them to run.
>

Yes, but there are other possibilities. For example, modules can create
additional windows in the main thread and install windows procedures
pointing to them. Such window procedures currently cannot communicate with
Emacs.


>
> IOW, such arbitrary callbacks need to run in the C land, and
> communicate with Lisp via timers.  In the future, we might add
> facilities that would allow a module register a file descriptor
> through which events could flow into the Emacs event queue.  But that
> is not yet available, and I'm not sure we'd want to allow it.
>
>
I agree.

[-- Attachment #2: Type: text/html, Size: 2268 bytes --]

Re: Callbacks from modules

[Ivan Andrus]

On Nov 26, 2015, at 8:43 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ivan Andrus <darthandrus@gmail.com>
>> Date: Wed, 25 Nov 2015 23:08:48 -0700
>> 
>> I’ve started writing an Emacs module to access NSSpeechSynthesizer on OS X (and 
>> maybe GNUStep though I haven’t tested it there).  I’ve been using the mod-test 
>> module as an example and I have it working reasonably well except that I don’t 
>> have any idea how to run a callback.
>> 
>> When the synthesizer is done speaking it sends an Objective-C message 
>> speechSynthesizer:didFinishSpeaking.  From there I would like to run an elisp 
>> hook, say `ns-speech-finished-speaking-hook`.  How can I do this?  Do I have to 
>> squirrel away a pointer to an emacs_env somehow?  I tried naively storing env 
>> from a previous call but, not surprisingly, the pointer is invalid when I try 
>> to use it.
> 
> How would you do that in Emacs's core code?  A module is just a
> (limited) extension of the Emacs core, so when you ask such questions,
> you should first think how Emacs core does that.

I did a version of this (in core) some time ago by adding a new event type.  I wasn’t ever happy with it and for various reasons I think it makes more sense as a module.

> Anyway, I assume you have a C callback in your module that is
> triggered by the speechSynthesizer:didFinishSpeaking message, is that
> right?  

Yes.

> Then one way would be to have that callback set a flag,
> provide a Lisp-callable function that returns the flag, and then start
> a timer that will test the flag and call your Lisp callback when the
> flag is set.

I was hoping to avoid polling, since the use case I have would like to start reading again as soon as the previous version finishes.

> Another, perhaps simpler, possibility would be to have the module
> provide a Lisp-callable function that will register a Lisp callback.
> Then your C callback will simply call that Lisp function.

I don’t understand how I would implement this option, though it sounds like what I was hoping for.  In order to call a lisp function, don’t I need an emacs_env?

> Would any of these do the job?

I hope so.  :-)  I’ll probably try the timer option.

Thanks,
Ivan

Re: Callbacks from modules

[Ivan Andrus]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

On Nov 26, 2015, at 11:44 AM, Philipp Stephani <p.stephani2@gmail.com> wrote:
> 
> IOW, such arbitrary callbacks need to run in the C land, and
> communicate with Lisp via timers.  In the future, we might add
> facilities that would allow a module register a file descriptor
> through which events could flow into the Emacs event queue.  But that
> is not yet available, and I'm not sure we'd want to allow it.
> 
> I agree. 

It seems desirable to me to have a function 

	void send_event( char const * type );

which would just queue a special type of event that could be dispatched on the elisp side through the normal even handling.  Though I admit I don’t really understand how it all works.

-Ivan

[-- Attachment #2: Type: text/html, Size: 2532 bytes --]

Re: Callbacks from modules

[Tom Tromey]

>>>>> "Philipp" == Philipp Stephani <p.stephani2@gmail.com> writes:

Philipp> The first is that such a facility would change the execution
Philipp> model of Emacs in a quite fundamental way.

I think instead that there is no fundamental change here.  The callback
scenario is no different from anything else in Emacs.

The things to disallow are calling into Emacs from a different thread,
or callbacks from signal handlers.  This is easily done -- the former is
done already and the latter can be done simply by fiat: "don't do that".

Philipp> Right now all user-defined code inside the Emacs process gets
Philipp> run through a path under the control of Emacs, either as a
Philipp> response to an input event, or a timer, or a process filter
Philipp> etc.

In my proposal in the relevant issue, this would not change.

Now, it should change a little.  It would be good to add fd-watching
capabilities to the module API; and it would be good to add some kind of
"do this later" API as well.  But even these aren't very different.

Philipp> The second issue is how such a facility should be made
Philipp> available. Currently all communication between modules and
Philipp> Emacs happens through modules, and there is no facility how
Philipp> modules can request creation of an environment. However, it
Philipp> makes little sense to discuss this issue until the first one is
Philipp> resolved.

I already proposed a solution for this as well: when a module requests a
runtime, keep it around.  Then let a module function request an
environment from the runtime at any time, and provide a way to tear down
such an environment.  Then a callback could get a new environment when
needed.

Tom

Re: Callbacks from modules

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Eli> In the future, we might add
Eli> facilities that would allow a module register a file descriptor
Eli> through which events could flow into the Emacs event queue.  But that
Eli> is not yet available, and I'm not sure we'd want to allow it.

Why not?

Tom

Re: Callbacks from modules

[Eli Zaretskii]

> From: Ivan Andrus <darthandrus@gmail.com>
> Date: Thu, 26 Nov 2015 20:23:48 -0700
> Cc: emacs-devel@gnu.org
> 
> > Then one way would be to have that callback set a flag,
> > provide a Lisp-callable function that returns the flag, and then start
> > a timer that will test the flag and call your Lisp callback when the
> > flag is set.
> 
> I was hoping to avoid polling, since the use case I have would like to start reading again as soon as the previous version finishes.

You could do that on the C level, no?  You don't need anything from
Lisp to start reading again.

> > Another, perhaps simpler, possibility would be to have the module
> > provide a Lisp-callable function that will register a Lisp callback.
> > Then your C callback will simply call that Lisp function.
> 
> I don’t understand how I would implement this option, though it sounds like what I was hoping for.  In order to call a lisp function, don’t I need an emacs_env?

Forget it, it probably won't work anyway, because your C callback runs
in another thread, right?  Module code cannot run in any thread but
the main thread.

Re: Callbacks from modules

[Eli Zaretskii]

> From: Ivan Andrus <darthandrus@gmail.com>
> Date: Thu, 26 Nov 2015 20:35:06 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  dancol@dancol.org,
>  emacs-devel@gnu.org
> 
> It seems desirable to me to have a function 
> 
> void send_event( char const * type );
> 
> which would just queue a special type of event that could be dispatched on the
> elisp side through the normal even handling.

You can't do that from another thread, because the Emacs event queue
is not ready to be accessed from several threads.

Re: Callbacks from modules

[Eli Zaretskii]

> From: Tom Tromey <tom@tromey.com>
> Cc: Philipp Stephani <p.stephani2@gmail.com>,  darthandrus@gmail.com,  dancol@dancol.org,  emacs-devel@gnu.org
> Date: Thu, 26 Nov 2015 21:22:07 -0700
> 
> >>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> Eli> In the future, we might add
> Eli> facilities that would allow a module register a file descriptor
> Eli> through which events could flow into the Emacs event queue.  But that
> Eli> is not yet available, and I'm not sure we'd want to allow it.
> 
> Why not?

Because we didn't discuss it? ;-)

Seriously, though: it might be dangerous.  A module that floods Emacs
with events could bring a session to its knees.  And that's just based
on 5-sec thinking; I'm sure there are other "interesting"
possibilities.

I think we should have some specific design presented, and then
discuss it.  Perhaps the dangers are not real, or could be kept in
check.

Re: Callbacks from modules

[Ivan Andrus]

On Nov 27, 2015, at 1:29 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ivan Andrus <darthandrus@gmail.com>
>> Date: Thu, 26 Nov 2015 20:23:48 -0700
>> Cc: emacs-devel@gnu.org
>> 
>>> Then one way would be to have that callback set a flag,
>>> provide a Lisp-callable function that returns the flag, and then start
>>> a timer that will test the flag and call your Lisp callback when the
>>> flag is set.
>> 
>> I was hoping to avoid polling, since the use case I have would like to start reading again as soon as the previous version finishes.
> 
> You could do that on the C level, no?  You don't need anything from
> Lisp to start reading again.

I should have said I want the option of continuing to read.  I wanted that logic (along with what to read next) in lisp.

>>> Another, perhaps simpler, possibility would be to have the module
>>> provide a Lisp-callable function that will register a Lisp callback.
>>> Then your C callback will simply call that Lisp function.
>> 
>> I don’t understand how I would implement this option, though it sounds like what I was hoping for.  In order to call a lisp function, don’t I need an emacs_env?
> 
> Forget it, it probably won't work anyway, because your C callback runs
> in another thread, right?  Module code cannot run in any thread but
> the main thread.

Okay, I was wondering about that.  Based on the back trace it did look like it was running in the same thread, but it’s probably best to not rely on that.

Thanks,
Ivan

Re: Callbacks from modules

[Ivan Andrus]

On Nov 27, 2015, at 1:30 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ivan Andrus <darthandrus@gmail.com>
>> Date: Thu, 26 Nov 2015 20:35:06 -0700
>> Cc: Eli Zaretskii <eliz@gnu.org>,
>> dancol@dancol.org,
>> emacs-devel@gnu.org
>> 
>> It seems desirable to me to have a function 
>> 
>> void send_event( char const * type );
>> 
>> which would just queue a special type of event that could be dispatched on the
>> elisp side through the normal even handling.
> 
> You can't do that from another thread, because the Emacs event queue
> is not ready to be accessed from several threads.

Ah.  That makes sense.

-Ivan

Re: Callbacks from modules

[Ted Zlatanov]

On Fri, 27 Nov 2015 10:35:40 +0200 Eli Zaretskii <eliz@gnu.org> wrote: 

EZ> A module that floods Emacs with events could bring a session to its
EZ> knees. And that's just based on 5-sec thinking; I'm sure there are
EZ> other "interesting" possibilities.

I agree. Callbacks as a whole are not going to play well with the ELisp
ecosystem so I would definitely make them events that can be ignored, as
Ivan also suggested.

EZ> I think we should have some specific design presented, and then
EZ> discuss it.  Perhaps the dangers are not real, or could be kept in
EZ> check.

I would delay this discussion until the current module system is solid
and we've made a release with it. It just doesn't seem essential to the
current release.

Ted

Re: Callbacks from modules

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Mon, 30 Nov 2015 09:03:24 -0500
> 
> EZ> I think we should have some specific design presented, and then
> EZ> discuss it.  Perhaps the dangers are not real, or could be kept in
> EZ> check.
> 
> I would delay this discussion until the current module system is solid
> and we've made a release with it. It just doesn't seem essential to the
> current release.

I fully agree.

Re: Callbacks from modules

[John Wiegley]

>>>>> Ted Zlatanov <tzz@lifelogs.com> writes:

EZ> I think we should have some specific design presented, and then discuss
EZ> it. Perhaps the dangers are not real, or could be kept in check.

> I would delay this discussion until the current module system is solid and
> we've made a release with it. It just doesn't seem essential to the current
> release.

Agreed.

John