unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Tree-sitter integration on feature/tree-sitter
@ 2022-05-07  8:29 Yuan Fu
  2022-05-07  8:44 ` Yuan Fu
                   ` (3 more replies)
  0 siblings, 4 replies; 150+ messages in thread
From: Yuan Fu @ 2022-05-07  8:29 UTC (permalink / raw)
  To: Emacs Devel

Guys,

I’ve pushed the tree-sitter integration to feature/tree-sitter. If anyone want to give it a try:
- get tree-sitter from GitHub (or your package manager, make sure the version is at least 0.20.2)
https://github.com/tree-sitter/tree-sitter
- pull and build the branch
- read (elisp)Parsing Program Source
- grab language definitions from https://github.com/casouri/tree-sitter-module
You can either build with my script, or download the prebuilt ones
- play with it

Also apologize in advance for that my response might be slow until like June :-)

Yuan


^ permalink raw reply	[flat|nested] 150+ messages in thread
* Re: Tree-sitter integration on feature/tree-sitter
@ 2022-05-09 17:50 Yoav Marco
  2022-05-09 20:51 ` Yuan Fu
  0 siblings, 1 reply; 150+ messages in thread
From: Yoav Marco @ 2022-05-09 17:50 UTC (permalink / raw)
  To: casouri; +Cc: emacs-devel

Looking at the code, isn't the while loop in treesit-query-capture
O(n²)? It essentially amounts to

  result = nil
  while (next capture group is avaliable) {
        captures = nil
        for (capture in capture group) {
          captures = cons(capture, captures)
        }
        captures = nreverse(captures)
        if (captures pass all predicates in their query)
           result = nconc(result, captures) // <----- THE OFFENDER
  }

A better way to do this would be to call nconc(captures, result) and
nreverse it all at the end instead of at the end of the for loop.

An even faster way would be to add unconditionally to result and roll it
back in case predicates fail. This doesn't use nconc at all:

  result = nil
  while (next capture group is avaliable) {
        prev_result = result;
        for (capture in capture group) {
          result = cons(capture, result)
        }
        if (captures *fail* at a predicate)
           result = prev_result
  }
  result = nreverse(result)


Context: I'm still working on profiling query compilation, and from what
I understand of gprof's output (not much) nconc indeed is very slow
here. Seeing nconc in the report is what made me look for nconc usage in
treesit-query-capture.

index % time    self  children    called     name
[1]     98.4    1.76    0.13     169+9031282 <cycle 1 as a whole> [1]
                1.59    0.00   41385             Fnconc <cycle 1> [2]
                0.06    0.05 2432616+3714        process_mark_stack <cycle 1> [24]
                0.08    0.00  147643             re_match_2_internal <cycle 1> [25]
                0.02    0.00    1707+56214       mark_char_table <cycle 1> [32]
                0.00    0.02      18             garbage_collect <cycle 1> [33]
                ...

The thing profiled is calling treesit-font-lock-fontify-region with c
queries accidentally on a go file with 8k lines. Still shouldn't take
the 2.2 seconds that it did, though.


> BTW, I would appreciate for someone to look at the manual and maybe
> touch up a bit, as I’m not a native speaker and might write something
> not very idiomatic/fluent.

One thing I've noticed - the manual node starts by introducing
treesit-available-p, a function that doesn't seem to exist anymore?



^ permalink raw reply	[flat|nested] 150+ messages in thread
* Re: Tree-sitter integration on feature/tree-sitter
@ 2022-05-19  1:35 Kiong-Ge Liau
  2022-05-20  2:01 ` Yuan Fu
  0 siblings, 1 reply; 150+ messages in thread
From: Kiong-Ge Liau @ 2022-05-19  1:35 UTC (permalink / raw)
  To: casouri, emacs-devel

Can you please share the mentioned "treesit-demo.el" file? I cannot see
it attached to any messages on emacs-devel mailing list. 

Thanks.






^ permalink raw reply	[flat|nested] 150+ messages in thread
* Re: Tree-sitter integration on feature/tree-sitter
@ 2022-05-19  1:35 Kiong-Ge Liau
  0 siblings, 0 replies; 150+ messages in thread
From: Kiong-Ge Liau @ 2022-05-19  1:35 UTC (permalink / raw)
  To: casouri, emacs-devel

Can you please share the mentioned "treesit-demo.el" file? I cannot it
attached to any message on emacs-devel maliing list.

Thanks.







^ permalink raw reply	[flat|nested] 150+ messages in thread
* Re: Tree-sitter integration on feature/tree-sitter
@ 2022-06-28 16:08 Yoav Marco
  2022-06-28 19:35 ` Yoav Marco
  0 siblings, 1 reply; 150+ messages in thread
From: Yoav Marco @ 2022-06-28 16:08 UTC (permalink / raw)
  To: Yuan Fu
  Cc: Eli Zaretskii, Theodor Thornhill, Stefan Monnier,
	Daniel Colascione, emacs-devel

References: <5bada349-2f43-4325-b696-70918584cd3d@email.android.com> <83mtfsuluo.fsf@gnu.org> <87sfpjhm33.fsf@thornhill.no> <83a6brufe5.fsf@gnu.org> <87pmkmhp8i.fsf@thornhill.no> <83v8ueuc7i.fsf@gnu.org> <73DE25BA-5EEF-4497-8F98-8C5F20853A61@gmail.com> <87v8uewfuq.fsf@thornhill.no> <B227CEAC-B3E9-45B2-B859-0C411CCECF3F@gmail.com> <87mtfkbt9n.fsf@thornhill.no> <1179E1EC-90EF-4989-BE1D-115498F77F60@gmail.com> <87k0ajygry.fsf@thornhill.no> <6EF70929-5759-4F1A-B878-0C1660FB6831@gmail.com> <87leuy5z46.fsf@thornhill.no> <2E68780C-9923-411E-A5DF-B1A54E2EC38B@gmail.com> <83k09fq0am.fsf@gnu.org> <38C272F6-828C-4478-9D90-326AF14D0C94@gmail.com> <87h74j2lin.fsf@gmail.com> <4C257709-0E3E-46A7-8CEB-569001EEE31D@gmail.com> <87edzm2vk8.fsf@gmail.com> <1CEC92F7-3CB9-48DA-887C-43CD4C58E406@gmail.com>
User-agent: mu4e 1.6.3; emacs 29.0.50
In-reply-to: <1CEC92F7-3CB9-48DA-887C-43CD4C58E406@gmail.com>

https://github.com/meain/evil-textobj-tree-sitter for example pulls
Yuan Fu <casouri@gmail.com> writes:
>>> Also I think it makes more sense if you just fork it rather than making a PR.
>>
>> Yeah, I don't really expect it to be merged.
>>
>> My reason for the PR is for it to be more of a talking point about
>> collaboration between treesit and tree-sitter-langs. I used
>> elisp-tree-sitter before trying the feature/tree-sitter branch, and I
>> really like the richness of its highlighting (which comes from the
>> highlights.scm files).
>
> Do you already have the highlighting working for treesit?

I only checked a few, but yeah, and the syntax highlighting is very pretty.

> If so, maybe you can packages it in a separate package and publish it,
> it would be a nice demonstration of treesit features.

I'm not comfortable with doing that, the query patterns update
frequently upstream (though ubolonton has been busy lately) and all I'm
doing is post-process them a little.

I could try to make a self-updating repo or something with CI -
highlights.scm changes from nvim's repo weeky.

Is that worth it? I thought people could just as easily use my PR
branch of tree-sitter-langs. That also takes care of fetching grammars.


  Sorry for the late response,
  Yoav



^ permalink raw reply	[flat|nested] 150+ messages in thread
* Re: Tree-sitter integration on feature/tree-sitter
@ 2022-06-29 16:51 Abin Simon
  2022-06-29 17:43 ` Yoav Marco
  0 siblings, 1 reply; 150+ messages in thread
From: Abin Simon @ 2022-06-29 16:51 UTC (permalink / raw)
  To: yoavm448; +Cc: casouri, eliz, theo, monnier, dancol, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

Yoav Marco <yoavm448@gmail.com> writes:

> I could try to make a self-updating repo or something with CI -
> https://github.com/meain/evil-textobj-tree-sitter for example pulls
> highlights.scm changes from nvim's repo weeky.

Just wanted to give a heads up that neovim has non standard items in
their queries. For example you will find things like `vim-match` and
`lua-match` in the queries.

https://github.com/nvim-treesitter/nvim-treesitter/blob/989c75046c46d2ed96bb65c5badd6b8f785e7f09/queries/go/highlights.scm#L19

I ran into similar issues in the meain/evil-textobj-tree-sitter and had
to write scritps to convert them to something that works in emacs.

ref: https://github.com/meain/evil-textobj-tree-sitter/issues/33

[-- Attachment #2: Type: text/html, Size: 1140 bytes --]

^ permalink raw reply	[flat|nested] 150+ messages in thread

end of thread, other threads:[~2022-06-30 14:37 UTC | newest]

Thread overview: 150+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-07  8:29 Tree-sitter integration on feature/tree-sitter Yuan Fu
2022-05-07  8:44 ` Yuan Fu
2022-05-07  8:47 ` Theodor Thornhill
2022-05-07 17:59   ` Yuan Fu
2022-05-07 18:16     ` Theodor Thornhill
2022-05-07  9:04 ` Eli Zaretskii
2022-05-07  9:34   ` Theodor Thornhill
2022-05-07 18:33     ` Yuan Fu
2022-05-07 19:02       ` Theodor Thornhill
2022-05-07 18:27   ` Yuan Fu
2022-05-07 18:48     ` Eli Zaretskii
2022-05-07 19:00       ` Theodor Thornhill
2022-05-07 19:21         ` Eli Zaretskii
2022-05-07 19:11       ` Yuan Fu
2022-05-07 19:25         ` Eli Zaretskii
2022-05-07 20:00           ` Yuan Fu
2022-05-07 20:12             ` Theodor Thornhill
2022-05-07 21:24               ` Stefan Monnier
2022-05-07 22:02                 ` Theodor Thornhill
2022-05-08  6:18                 ` Eli Zaretskii
2022-05-08 12:05                   ` Dmitry Gutov
2022-05-08 12:16                     ` Stefan Monnier
2022-05-08 13:23                       ` Eli Zaretskii
2022-05-08 20:57                         ` Dmitry Gutov
2022-05-08 13:21                     ` Eli Zaretskii
2022-05-08 20:42                       ` Dmitry Gutov
2022-05-09 11:18                         ` Eli Zaretskii
2022-05-08  6:16               ` Eli Zaretskii
2022-05-08  6:49                 ` Theodor Thornhill
2022-05-08  6:58                   ` Eli Zaretskii
2022-05-08  9:02                     ` Theodor Thornhill
2022-05-08  9:09                       ` Theodor Thornhill
2022-05-08  9:10                       ` Eli Zaretskii
2022-05-08  9:19                         ` Theodor Thornhill
2022-05-08 10:33                           ` Eli Zaretskii
2022-05-08 13:47                             ` Theodor Thornhill
2022-05-08 13:58                               ` Eli Zaretskii
2022-05-08 14:01                               ` Stefan Monnier
2022-05-08 14:25                                 ` Theodor Thornhill
2022-05-08 14:42                                   ` Eli Zaretskii
2022-05-08 19:16                                     ` Theodor Thornhill
2022-05-08 21:14                                       ` Yuan Fu
2022-05-09 11:14                                       ` Eli Zaretskii
2022-05-09 12:20                                         ` Theodor Thornhill
2022-05-09 12:23                                           ` Eli Zaretskii
2022-05-09 21:10                                             ` Yuan Fu
2022-05-09 21:33                                               ` Theodor Thornhill
2022-05-14  0:03                                                 ` Yuan Fu
2022-05-14  5:03                                                   ` Theodor Thornhill
2022-05-14  5:13                                                     ` Yuan Fu
2022-05-17 21:45                                                       ` Theodor Thornhill
2022-05-18 20:52                                                         ` Yuan Fu
2022-05-18 21:07                                                           ` Theodor Thornhill
2022-06-16 19:09                                                             ` Yuan Fu
2022-06-17  6:19                                                               ` Eli Zaretskii
2022-06-17  7:32                                                                 ` Yuan Fu
2022-06-17 10:42                                                                   ` Eli Zaretskii
2022-06-18  0:20                                                                     ` Yuan Fu
2022-06-18  6:23                                                                       ` Eli Zaretskii
2022-06-20 14:20                                                                       ` Daniel Martín
2022-06-20 20:03                                                                         ` Yuan Fu
2022-06-17 18:12                                                                   ` Yoav Marco
2022-06-18  0:35                                                                     ` Yuan Fu
2022-06-18  8:15                                                                       ` Yoav Marco
2022-06-18 20:11                                                                         ` Yuan Fu
2022-05-08 22:42                             ` Stephen Leake
2022-05-14 15:09 ` Daniel Martín
2022-05-14 15:55   ` Yuan Fu
2022-05-14 18:50     ` Daniel Martín
2022-05-14 19:09       ` Eli Zaretskii
2022-06-16 19:10       ` Yuan Fu
  -- strict thread matches above, loose matches on Subject: below --
2022-05-09 17:50 Yoav Marco
2022-05-09 20:51 ` Yuan Fu
     [not found]   ` <87lev9wyll.fsf@gmail.com>
2022-05-10 15:20     ` Yoav Marco
2022-05-10 15:43   ` Yoav Marco
2022-05-10 17:54     ` Yuan Fu
2022-05-10 18:18       ` Yoav Marco
2022-05-10 19:58         ` Stefan Monnier
2022-05-10 23:11           ` Yuan Fu
2022-05-10 23:53             ` Yuan Fu
2022-05-11 11:10         ` Eli Zaretskii
2022-05-11 11:16           ` Yoav Marco
2022-05-11 14:20             ` Eli Zaretskii
2022-05-11 15:40               ` Yoav Marco
2022-05-11 16:27                 ` Eli Zaretskii
2022-05-11 20:14                   ` Yuan Fu
2022-05-11 20:25                     ` Yuan Fu
2022-05-12  5:19                       ` Eli Zaretskii
2022-05-12  6:10                         ` Yuan Fu
2022-05-12  7:12                           ` Eli Zaretskii
2022-05-12 15:18                         ` Stefan Monnier
2022-05-12 15:53                           ` Eli Zaretskii
2022-05-12  5:17                     ` Eli Zaretskii
2022-05-12  6:07                       ` Yuan Fu
2022-05-12 14:16                       ` Yoav Marco
2022-05-12 16:04                         ` Eli Zaretskii
2022-05-12 16:26                           ` Yoav Marco
2022-05-12 17:18                             ` Eli Zaretskii
2022-05-12 17:22                               ` Yoav Marco
2022-05-13  6:34                                 ` Eli Zaretskii
2022-05-13  8:04                                   ` Theodor Thornhill
2022-05-13  8:36                                     ` Yoav Marco
2022-05-13  9:46                                       ` Theodor Thornhill
2022-05-13 10:37                                     ` Eli Zaretskii
2022-05-13 10:52                                       ` Theodor Thornhill
2022-05-13  8:42                                   ` Yoav Marco
2022-05-13 10:41                                     ` Eli Zaretskii
2022-05-14  0:04                                       ` Yuan Fu
2022-06-16 19:16                                         ` Yuan Fu
2022-06-16 21:57                                           ` yoavm448
2022-06-17  1:10                                             ` Yuan Fu
2022-05-12 15:15                       ` Stefan Monnier
2022-05-15 19:20       ` chad
2022-05-15 19:26         ` Eli Zaretskii
2022-05-19  1:35 Kiong-Ge Liau
2022-05-20  2:01 ` Yuan Fu
2022-06-16 19:03   ` Yuan Fu
2022-06-17  1:24     ` Po Lu
2022-06-18  0:09       ` Yuan Fu
2022-06-17  2:00     ` Ihor Radchenko
2022-06-17  5:23       ` Eli Zaretskii
2022-06-17 10:40         ` Ihor Radchenko
2022-06-17  6:15     ` Eli Zaretskii
2022-06-17  7:17       ` Yuan Fu
2022-06-17 10:37         ` Eli Zaretskii
2022-06-18  0:14           ` Yuan Fu
2022-06-18  6:22             ` Eli Zaretskii
2022-06-18  8:25               ` Yuan Fu
2022-06-18  8:50                 ` Eli Zaretskii
2022-06-18 20:07                   ` Yuan Fu
2022-06-19  5:39                     ` Eli Zaretskii
2022-06-20  3:00                       ` Yuan Fu
2022-06-20 11:44                         ` Eli Zaretskii
2022-06-20 20:01                           ` Yuan Fu
2022-06-21  2:26                             ` Eli Zaretskii
2022-06-21  4:39                               ` Yuan Fu
2022-06-21 10:18                                 ` Eli Zaretskii
2022-06-22  0:34                                   ` Yuan Fu
2022-06-17 11:06     ` Jostein Kjønigsen
2022-06-18  0:28       ` Yuan Fu
2022-06-18 20:57         ` Jostein Kjønigsen
2022-05-19  1:35 Kiong-Ge Liau
2022-06-28 16:08 Yoav Marco
2022-06-28 19:35 ` Yoav Marco
2022-06-29 15:35   ` Yuan Fu
2022-06-29 16:51 Abin Simon
2022-06-29 17:43 ` Yoav Marco
2022-06-30 11:21   ` Yoav Marco
2022-06-30 14:29     ` Abin Simon
2022-06-30 14:37       ` Yoav Marco

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).