Re: Running process filters in another thread

[yyoncho <yyoncho@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3703 bytes --]

Hi Stefan,

> So, IIUC the scenario is that the LSP sends us large JSON data
> structures, but we only ever use a fairly small portion of it, so
> presumably a significant part of the total processing time is spent
> gobbling data and parsing it into an internal data structure.

AFAIK this is not the case, LSP is pretty minimal. This could be true for
some
limited cases but in general we need all of the data.

> Beside limiting the total amount of time spent doing this (e.g. by
> processing the data more efficiently (e.g. in C rather in Elisp) or in
> parallel), the other issue is the fact that it shouldn't cause pauses in
> the user interaction.  This can be obtained via true concurrency, or by
> slicing the processing so it can be interleaved finely with user
> interaction without any noticeable pauses.

> Maybe an "easy" way to solve this particular problem is to define the
> json parser as a "yield point".  This would even allow the
> implementation of the json parser to spawn a separate thread and do the
> parsing in a truly concurrent thread (tho the GC would need to be
> extended to be able to deal with such truly concurrent non-Elisp
> threads allocating Elisp data structures, maybe this can be done by
> making those json threads use their own Elisp heap which gets joined
> into the main heap at the end).

Yeah, that is what I was looking for. I believe it will be a big
improvement from
UX standpoint of view. Also, it seems like if we have that kind of
infrastructure inplace this will open many other options.

> IIUC eglot does use the native json parser.  BTW, I'd be happy to hear
> about the use of jsonrpc.el in lsp-mode (either failures or successes)
> and more generally consolidation of effort between eglot and lsp-mode.

I would like to see consolidation too. I will file a task for investigating
jsonrpc.el.

Thanks,
Ivan


On Sat, Sep 29, 2018 at 11:30 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > Some of the responses/notifications might be big e. g.
> > 1. Diagnostics
> > 2. Semantic highlight data(proportional to file size)
>
> So, IIUC the scenario is that the LSP sends us large JSON data
> structures, but we only ever use a fairly small portion of it, so
> presumably a significant part of the total processing time is spent
> gobbling data and parsing it into an internal data structure.
>
> Beside limiting the total amount of time spent doing this (e.g. by
> processing the data more efficiently (e.g. in C rather in Elisp) or in
> parallel), the other issue is the fact that it shouldn't cause pauses in
> the user interaction.  This can be obtained via true concurrency, or by
> slicing the processing so it can be interleaved finely with user
> interaction without any noticeable pauses.
>
> Maybe an "easy" way to solve this particular problem is to define the
> json parser as a "yield point".  This would even allow the
> implementation of the json parser to spawn a separate thread and do the
> parsing in a truly concurrent thread (tho the GC would need to be
> extended to be able to deal with such truly concurrent non-Elisp
> threads allocating Elisp data structures, maybe this can be done by
> making those json threads use their own Elisp heap which gets joined
> into the main heap at the end).
>
> > a filter to that pipeline won't be a problem. One we (lsp-mode
> > contributors) move it to native json I will resume the discussion if
> > with some concrete testing data.
>
> IIUC eglot does use the native json parser.  BTW, I'd be happy to hear
> about the use of jsonrpc.el in lsp-mode (either failures or successes)
> and more generally consolidation of effort between eglot and lsp-mode.
>
>
>         Stefan
>

[-- Attachment #2: Type: text/html, Size: 4736 bytes --]