bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

[Ivan Yonchovski]

This is needed for the cases when the module is listening for extenal
events and wants to call back emacs to process them. ATM this is kind of
possible by using signals on linux and using WM_INPUTLANGCHANGE as
described in https://nullprogram.com/blog/2017/02/14/ but this looks
more a hack for a missing feature.

bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

[Eli Zaretskii]

> From: Ivan Yonchovski <yyoncho@gmail.com>
> Date: Wed, 24 Jun 2020 10:14:44 +0300
> 
> This is needed for the cases when the module is listening for extenal
> events and wants to call back emacs to process them. ATM this is kind of
> possible by using signals on linux and using WM_INPUTLANGCHANGE as
> described in https://nullprogram.com/blog/2017/02/14/ but this looks
> more a hack for a missing feature.

It is almost trivial to let modules insert events into the event
queue.  The problem is that I expect a module that inserts such events
to want to be called to process those events as well.  Is that
expectation correct?  If it is, then we need to think about extending
the mechanism that calls event handlers, not just about exposing the
likes of kbd_buffer_store_event to modules.  We should also consider
how a module could define its own events.

IOW, this calls for a slightly more detailed specification, before
someone could sit down and code the stuff.  I suggest to take a look
at how existing events are handled, and propose such a detailed
specification.

Thanks.

bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

[yyoncho]

[-- Attachment #1: Type: text/plain, Size: 322 bytes --]

> The problem is that I expect a module that inserts such events
> to want to be called to process those events as well.  Is that
> expectation correct?
>

Yes. As a side note, this feature will be useful even for the core C part
for anything
that wants to perform async processing not including LispObject.

Thanks,
Ivan

[-- Attachment #2: Type: text/html, Size: 615 bytes --]

bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

[Eli Zaretskii]

> From: yyoncho <yyoncho@gmail.com>
> Date: Wed, 24 Jun 2020 19:17:30 +0300
> Cc: 42028@debbugs.gnu.org
> 
> As a side note, this feature will be useful even for the core C part for anything
> that wants to perform async processing not including LispObject. 

How will it help that?  The event queue is processed synchronously.

bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

[yyoncho]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

Here it is an example: If we oversimplify json parsing consist of 2 parts.

1. JSON parsing to Jansson data structures
2. Converting Jasson data structures to lisp data structures.

If we assume that 1 is taking most of the time then we could write the
following code:

start(string, callback) -> convert string to char* -> post it to worker
thread -> worker thread does the parsing
-> worker thread post to main loop -> main loop converts Jansson ds to lisp
ds -> main loop
calls the callback passed on start

The same strategy probably could be applied to Font locking with
TreeSitter.

Thanks,
Ivan

On Wed, Jun 24, 2020 at 7:29 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: yyoncho <yyoncho@gmail.com>
> > Date: Wed, 24 Jun 2020 19:17:30 +0300
> > Cc: 42028@debbugs.gnu.org
> >
> > As a side note, this feature will be useful even for the core C part for
> anything
> > that wants to perform async processing not including LispObject.
>
> How will it help that?  The event queue is processed synchronously.
>

[-- Attachment #2: Type: text/html, Size: 1633 bytes --]

bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

[Eli Zaretskii]

> From: yyoncho <yyoncho@gmail.com>
> Date: Wed, 24 Jun 2020 19:46:56 +0300
> Cc: 42028@debbugs.gnu.org
> 
> 1. JSON parsing to Jansson data structures
> 2. Converting Jasson data structures to lisp data structures.
> 
> If we assume that 1 is taking most of the time

Does it?  Did someone time these separately?

> start(string, callback) -> convert string to char* -> post it to worker thread -> worker thread does the parsing
> -> worker thread post to main loop -> main loop converts Jansson ds to lisp ds -> main loop
> calls the callback passed on start

You cannot easily post to the queue from a worker thread, because the
queue cannot be posted to asynchronously.

And I lack the larger picture: how would this work within a framework
of some Emacs feature?  E.g., how will this processed be triggered,
and by what kind of trigger?

> The same strategy probably could be applied to Font locking with TreeSitter. 

Again, I don't see the wider picture.  Are you familiar with how the
current JIT font-lock works?  If so, can you explain which parts of
that will be replaced/modified, and how?

bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

[Ivan Yonchovski]

Eli Zaretskii writes:

> Does it?  Did someone time these separately?

IDK, this was just an example.

>> start(string, callback) -> convert string to char* -> post it to worker thread -> worker thread does the parsing
>> -> worker thread post to main loop -> main loop converts Jansson ds to lisp ds -> main loop
>> calls the callback passed on start
>
> You cannot easily post to the queue from a worker thread, because the
> queue cannot be posted to asynchronously.

AFAIK this is not possible at all - thus this feature request.

>
> And I lack the larger picture: how would this work within a framework
> of some Emacs feature?  E.g., how will this processed be triggered,
> and by what kind of trigger?

The goal users to be able to handle more usecases. ATM the only way to
achieve that is hackish as described here
https://nullprogram.com/blog/2017/02/14/ + several practical usecases.


> Again, I don't see the wider picture.  Are you familiar with how the
> current JIT font-lock works?  If so, can you explain which parts of
> that will be replaced/modified, and how?

AFAIK the initial tree-sitter parsing (which does not involve anything
emacs related) may take 200-300ms. In other editors (e.g. vscode) this
part is not happening on the UI thread but in some extenal background
thread which then passes the AST to the main UI thread to do the actual
fontlock. IMO this model might be replicated in emacs in the event of TS
integration.

Thanks,
Ivan

bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

[Eli Zaretskii]

> From: Ivan Yonchovski <yyoncho@gmail.com>
> Cc: 42028@debbugs.gnu.org
> Date: Wed, 24 Jun 2020 20:33:29 +0300
> 
> > You cannot easily post to the queue from a worker thread, because the
> > queue cannot be posted to asynchronously.
> 
> AFAIK this is not possible at all - thus this feature request.

Your feature just asked for a way to queue events, it didn't say
anything about doing that asynchronously, not from a function invoked
by the main thread.  If you mean the latter, then it AFAIU would need
a serious redesign of the Emacs event queue, to make it accessible
from several threads running in parallel at once.

> > Again, I don't see the wider picture.  Are you familiar with how the
> > current JIT font-lock works?  If so, can you explain which parts of
> > that will be replaced/modified, and how?
> 
> AFAIK the initial tree-sitter parsing (which does not involve anything
> emacs related) may take 200-300ms. In other editors (e.g. vscode) this
> part is not happening on the UI thread but in some extenal background
> thread which then passes the AST to the main UI thread to do the actual
> fontlock. IMO this model might be replicated in emacs in the event of TS
> integration.

What you describe is very different from how JIT font-lock works now.
Which is why I asked the question: I know (more or less) how it works
in other editors, I just don't yet understand well enough how
something like that would fit into the existing fontification
framework.  I hope there is a way of fitting it, because otherwise it
would mean a serious surgery of the display engine as well, which will
make the job significantly larger and harder.

bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop

[Ivan Yonchovski]

Eli Zaretskii writes:

>
> Your feature just asked for a way to queue events, it didn't say
> anything about doing that asynchronously, not from a function invoked
> by the main thread.  If you mean the latter, then it AFAIU would need
> a serious redesign of the Emacs event queue, to make it accessible
> from several threads running in parallel at once.

Yes, sorry about that. I didn't mention it explicitly in the bug report
it was described in more details in the linked article.

> What you describe is very different from how JIT font-lock works now.
> Which is why I asked the question: I know (more or less) how it works
> in other editors, I just don't yet understand well enough how
> something like that would fit into the existing fontification
> framework.  I hope there is a way of fitting it, because otherwise it
> would mean a serious surgery of the display engine as well, which will
> make the job significantly larger and harder.

IMO we are good here. At least on lsp-mode side we were able to
implement delayed semantic fontification as a result of async source.
But in lsp-mode case we are reseiving the messages from the server on
the UI thread.

Thanks,
Ivan