unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Re: Using incremental parsing in Emacs (via: emacs rendering comparisson between emacs23 and emacs26.3)
@ 2020-03-29 18:46 Stefan Monnier
  2020-03-29 19:05 ` Andrea Corallo
  0 siblings, 1 reply; 139+ messages in thread
From: Stefan Monnier @ 2020-03-29 18:46 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: emacs-devel

>> tree-sitter, like LSP, is something Emacs should embrace.
>   https://lists.gnu.org/archive/html/emacs-devel/2020-01/msg00059.html

Ah, thanks Eli: I guess I skipped over that while catching up.

> Would someone like to try to figure out how we could use the
> incremental parsing technology in Emacs for making our
> programming-language support more accurate and efficient?  One package
> that implements this technology is tree-sitter:
>
>   https://tree-sitter.github.io/tree-sitter/

Yes, adding support for this would be great.  

> AFAIU, these capabilities could be used as an alternative to
> regexp- and syntax-pps-based font-lock, better code folding,
> completion, refactoring, and other similar features; in general, any
> feature which would benefit from having a parse tree for the source
> code in a buffer.

Some of those features could be provided by LSP as well, but IIUC the
way LSP is designed and usually used makes it somewhat inadequate for
synchronous use, when you want an immediate answer.

tree-sitter is designed exactly for that: it can parse "immediately",
in the same sense as `syntax-ppss`, so LSP seems inapplicable (in the
near future at least) for things like font-lock and navigation, and
indentation, whereas tree-sitter should work great for that.

[ W.r.t disucssions around LSP's use of JSON: AFAICT, parsing and
  emitting json can be done as efficiently as any other format, AFAICT,
  so I don't see the use of JSON as a problem in the protocol.  ]

> To be able to use such libraries, we need to figure out how to
> integrate them into the core, what kind of interfaces would be needed
> for that, and what kind of infrastructure we would need for basing
> Lisp features on those libraries.

The existing third party packages should be good starting points to come
up with a design.  But I think an important issue is to figure out how
to make tree-sitter usable for the end users: AFAICT the main issue
being how to let end users download and install new grammars.
IIUC grammars are written in Javascript (or some subset thereof?) and
then somehow compiled to C code.  Having them as C code implies either
the end-user need to have a C compiler or distributing pre-compiled
binaries with all the trouble this entails (with all the variations of
OSes, and architectures, and ABIs, ..., plus issues related to
licensing, security, ...).

Maybe those grammars could be compiled to some other representation (I
don't know if it is made mostly of data-tables or actual code or what)?


        Stefan




^ permalink raw reply	[flat|nested] 139+ messages in thread
* Re: Reliable after-change-functions (via: Using incremental parsing in Emacs)
@ 2020-03-31 17:07 Tuấn Anh Nguyễn
  2020-03-31 17:50 ` Eli Zaretskii
  0 siblings, 1 reply; 139+ messages in thread
From: Tuấn Anh Nguyễn @ 2020-03-31 17:07 UTC (permalink / raw)
  To: emacs-devel

> In any case, I hope that passing the buffer to tree-sitter doesn't
> involve marshalling the entire buffer text via a function call as a
> huge string, or some such.  We should instead request that tree-sitter
> exposes an API through which we could give it direct access to buffer
> text as 2 parts, before and after the gap, like we do with regex
> code.  Otherwise this will be a bottleneck in the long run, not unlike
> the problem we have with LSP.

It does support parsing through direct access. Which is why I wanted
dynamic modules to have direct access to buffer text.

>> How large is "very large" here?
>
> xdisp.c comes to mind, obviously.

On my machine, a 3.39 GHz Intel Core i7:

    (0.150791 0 0.0) ; 1 full parse
    (2.142236 5 0.6105190000000107) ; 10 full parses
    (0.015423 0 0.0) ; incremental parsing, after typing 1 character

--
Tuấn-Anh Nguyễn
Software Engineer



^ permalink raw reply	[flat|nested] 139+ messages in thread

end of thread, other threads:[~2020-04-06 19:55 UTC | newest]

Thread overview: 139+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-03-29 18:46 Using incremental parsing in Emacs (via: emacs rendering comparisson between emacs23 and emacs26.3) Stefan Monnier
2020-03-29 19:05 ` Andrea Corallo
2020-03-29 19:18   ` Eli Zaretskii
2020-03-29 19:29     ` Reliable after-change-functions (via: Using incremental parsing in Emacs) Yuan Fu
2020-03-30 14:04       ` Eli Zaretskii
2020-03-30 15:06       ` Stefan Monnier
2020-03-30 17:14         ` Yuan Fu
2020-03-30 17:54           ` Stefan Monnier
2020-03-30 18:43             ` Štěpán Němec
2020-03-30 18:46               ` Stefan Monnier
2020-03-30 19:02                 ` Yuan Fu
2020-03-30 19:10                   ` Eli Zaretskii
2020-03-30 19:21                     ` Yuan Fu
2020-03-31  3:56                       ` Štěpán Němec
2020-03-31 13:16                         ` Eli Zaretskii
2020-03-31 13:36                           ` Štěpán Němec
2020-03-31 14:34                             ` Eli Zaretskii
2020-03-31 15:37                               ` Štěpán Němec
2020-03-31 15:58                                 ` Eli Zaretskii
2020-03-31 16:18                                   ` Štěpán Němec
2020-03-31 17:38                                     ` Eli Zaretskii
2020-04-01  0:57                     ` Stephen Leake
2020-03-30 19:42                   ` Stefan Monnier
2020-03-30 19:27                 ` Štěpán Němec
2020-03-31  2:24           ` Eli Zaretskii
2020-03-31  3:10             ` Stefan Monnier
2020-03-31 13:14               ` Eli Zaretskii
2020-03-31 14:31                 ` Dmitry Gutov
2020-03-31 15:36                   ` Eli Zaretskii
2020-03-31 15:45                     ` Dmitry Gutov
2020-03-31 17:16                     ` Stefan Monnier
2020-03-31 17:48                       ` Eli Zaretskii
2020-03-31 19:35                         ` Stefan Monnier
2020-04-01  2:23                           ` Eli Zaretskii
2020-03-31 15:11                 ` Stefan Monnier
2020-03-31 15:44                   ` Eli Zaretskii
2020-03-31 17:10                     ` Stefan Monnier
2020-03-31 17:19                       ` Jorge Javier Araya Navarro
2020-03-31 17:46                       ` Eli Zaretskii
2020-03-31 18:42                         ` 조성빈
2020-03-31 19:29                           ` Eli Zaretskii
2020-03-31 18:47                         ` Dmitry Gutov
2020-03-31 18:48                           ` Noam Postavsky
2020-03-31 19:02                             ` Dmitry Gutov
2020-03-31 19:26                           ` Eli Zaretskii
2020-03-31 19:50                             ` Dmitry Gutov
2020-04-01  2:28                               ` Eli Zaretskii
2020-04-01  3:49                                 ` Dmitry Gutov
2020-04-01  4:14                                   ` Eli Zaretskii
2020-04-01 13:47                                     ` Dmitry Gutov
2020-04-01 14:04                                       ` Eli Zaretskii
2020-04-01 14:55                                         ` Eli Zaretskii
2020-04-01 15:16                                         ` Dmitry Gutov
2020-04-01 15:59                                           ` Eli Zaretskii
2020-04-01 21:48                                             ` Dmitry Gutov
2020-04-01 22:29                                               ` Stefan Monnier
2020-04-02 14:23                                               ` Eli Zaretskii
2020-04-02 16:17                                                 ` Dmitry Gutov
2020-04-02 18:25                                                   ` Eli Zaretskii
2020-04-03 14:40                                                   ` Tuấn-Anh Nguyễn
2020-04-03 16:10                                                     ` Dmitry Gutov
2020-04-01 13:52                                     ` Alan Mackenzie
2020-04-01 14:10                                       ` Eli Zaretskii
2020-04-01 15:27                                         ` Dmitry Gutov
2020-04-01 15:44                                           ` Jorge Javier Araya Navarro
2020-04-01 16:03                                           ` Eli Zaretskii
2020-04-01 21:21                                             ` Dmitry Gutov
2020-04-02 14:09                                               ` Eli Zaretskii
2020-04-02 18:03                                                 ` 조성빈 via "Emacs development discussions.
2020-04-02 18:27                                                   ` Yuan Fu
2020-04-02 19:39                                                     ` Stefan Monnier
2020-04-01 15:22                                       ` Dmitry Gutov
2020-04-04 11:06                                         ` Alan Mackenzie
2020-04-04 11:26                                           ` Eli Zaretskii
2020-04-04 14:14                                             ` Andrea Corallo
2020-04-04 14:41                                               ` Eli Zaretskii
2020-04-04 15:04                                                 ` Andrea Corallo
2020-04-04 15:38                                                   ` Richard Copley
2020-04-04 11:27                                           ` Eli Zaretskii
2020-04-04 12:01                                           ` Dmitry Gutov
2020-04-04 12:36                                             ` Alan Mackenzie
2020-04-04 12:40                                               ` Dmitry Gutov
2020-04-04 13:02                                               ` Eli Zaretskii
2020-04-04 16:09                                                 ` Dmitry Gutov
2020-04-04 16:38                                                   ` Eli Zaretskii
2020-04-04 16:45                                                     ` Eli Zaretskii
2020-04-04 17:22                                                       ` Richard Copley
2020-04-04 17:50                                                         ` Eli Zaretskii
2020-04-04 18:29                                                         ` Andrea Corallo
2020-04-04 18:56                                                           ` Richard Copley
2020-04-04 20:36                                                             ` Andrea Corallo
2020-04-04 17:36                                                       ` Dmitry Gutov
2020-04-04 17:47                                                         ` Eli Zaretskii
2020-04-04 18:02                                                           ` Dmitry Gutov
2020-04-04 23:01                                                             ` Stefan Monnier
2020-04-06 14:25                                                               ` Yuan Fu
2020-04-06 19:55                                                                 ` Jorge Javier Araya Navarro
2020-04-04 17:29                                                     ` Dmitry Gutov
2020-04-04 17:38                                                       ` Eli Zaretskii
2020-04-04 17:57                                                         ` Dmitry Gutov
2020-03-31 16:13                 ` Alan Third
2020-03-31 17:55                   ` Eli Zaretskii
2020-03-30  3:35     ` Using incremental parsing in Emacs (via: emacs rendering comparisson between emacs23 and emacs26.3) Stefan Monnier
2020-03-30  6:02       ` Eli Zaretskii
2020-03-30 13:33         ` Stefan Monnier
2020-03-30 14:09           ` Eli Zaretskii
2020-03-30 15:03             ` Stefan Monnier
2020-04-01  0:39               ` Stephen Leake
  -- strict thread matches above, loose matches on Subject: below --
2020-03-31 17:07 Reliable after-change-functions (via: Using incremental parsing in Emacs) Tuấn Anh Nguyễn
2020-03-31 17:50 ` Eli Zaretskii
2020-04-01  6:17   ` Tuấn Anh Nguyễn
2020-04-01 13:26     ` Eli Zaretskii
2020-04-01 15:47       ` Jorge Javier Araya Navarro
2020-04-01 16:07         ` Eli Zaretskii
2020-04-01 17:55       ` Tuấn-Anh Nguyễn
2020-04-01 19:33         ` Eli Zaretskii
2020-04-01 23:38           ` Stephen Leake
2020-04-02  0:25             ` Stephen Leake
2020-04-02  2:46             ` Stefan Monnier
2020-04-02  4:36               ` Tuấn-Anh Nguyễn
2020-04-02 14:44               ` Eli Zaretskii
2020-04-02 15:19                 ` Stefan Monnier
2020-04-02  5:21             ` Tuấn-Anh Nguyễn
2020-04-02 14:36             ` Eli Zaretskii
2020-04-03  2:27               ` Stephen Leake
2020-04-03  7:43                 ` Eli Zaretskii
2020-04-03 17:45                   ` Stephen Leake
2020-04-03 18:31                     ` Eli Zaretskii
2020-04-04  0:04                       ` Stephen Leake
2020-04-04  7:13                         ` Eli Zaretskii
2020-04-02  4:21           ` Tuấn-Anh Nguyễn
2020-04-02  5:19             ` Jorge Javier Araya Navarro
2020-04-02  9:29               ` Stephen Leake
2020-04-02 10:37             ` Andrea Corallo
2020-04-02 11:14               ` Tuấn-Anh Nguyễn
2020-04-02 13:02             ` Stefan Monnier
2020-04-02 15:06               ` Eli Zaretskii
2020-04-02 15:02             ` Eli Zaretskii
2020-04-03 14:34               ` Tuấn-Anh Nguyễn

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).