bug#58711: Treesit hangs when calling treesit-search-forward

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 4741 bytes --]

I am attempting to implement treesit for the current elixir-mode using the
branch feature/trees-sitter. For implementing beginning-of-defun-function
using treesit I am running into an issue where the Emacs will hang
indefinitely. Using (setq-local treesit-defun-type-regexp (rx (or "call")))
yields the same result.

For the following node-at-point output

(do_block (call target: (identifier)))

with Elixir code

# foo.ex
defmodule Foo do
  <<point/cursor here>>def bar(), do: "bar"
end

When I call `(treesit-search-forward-goto (rx (or "call")) 'start nil t)`
the function `treesit-search-forward` seems to get stuck.

Elixir does not strictly have a begin function, but can be determined as
one of the following:
; * modules and protocols
(call
  target: (identifier) @ignore
  (arguments (alias) @name)
  (#match? @ignore "^(defmodule|defprotocol)$")) @definition.module

; * functions/macros
(call
  target: (identifier) @ignore
  (arguments
    [
      ; zero-arity functions with no parentheses
      (identifier) @name
      ; regular function clause
      (call target: (identifier) @name)
      ; function clause with a guard clause
      (binary_operator
        left: (call target: (identifier) @name)
        operator: "when")
    ])
  (#match? @ignore
"^(def|defp|defdelegate|defguard|defguardp|defmacro|defmacrop|defn|defnp)$"))
@definition.function

The elixir tree sitter implementation is here:
https://github.com/elixir-lang/tree-sitter-elixir

I lack the knowledge to further debug this or find a clean workaround, but
I'm almost sure that treesit-search-forward should never hang.

In GNU Emacs 29.0.50 (build 3, x86_64-pc-linux-gnu, GTK+ Version
 3.24.34, cairo version 1.17.6) of 2022-10-22 built on melissa.local
Repository revision: 7c750343be6309a78d3fd289959bca241d9daf5d
Repository branch: feature/tree-sitter
Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
System Description: Arch Linux

Configured using:
 'configure --with-tree-sitter --with-native-compilation'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON TREE-SITTER LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_ZA.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa epg rfc6068
epg-config gnus-util text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils time-date comp
comp-cstr warnings icons subr-x cl-extra help-mode bytecomp byte-compile
cconv elixir-mode derived rx pcase cl-macs gv treesit cl-seq cl-loaddefs
cl-lib rmc iso-transl tooltip eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit xinput2 x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 81171 6071)
 (symbols 48 7449 0)
 (strings 32 20132 1351)
 (string-bytes 1 613689)
 (vectors 16 15685)
 (vector-slots 8 324392 11268)
 (floats 8 27 29)
 (intervals 56 211 0)
 (buffers 1008 11))

[-- Attachment #2: Type: text/html, Size: 5569 bytes --]

bug#58711: The full tree-sitter parse output

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

This is the full tree-sitter parse output for the use case mentioned:

(source [0, 0] - [4, 3]
  (call [0, 0] - [4, 3]
    target: (identifier [0, 0] - [0, 9])
    (arguments [0, 10] - [0, 13]
      (alias [0, 10] - [0, 13]))
    (do_block [0, 14] - [4, 3]
      (call [1, 2] - [3, 5]
        target: (identifier [1, 2] - [1, 5])
        (arguments [1, 6] - [1, 11]
          (call [1, 6] - [1, 11]
            target: (identifier [1, 6] - [1, 9])
            (arguments [1, 9] - [1, 11])))
        (do_block [1, 12] - [3, 5]
          (string [2, 4] - [2, 9]
            (quoted_content [2, 5] - [2, 8])))))))

[-- Attachment #2: Type: text/html, Size: 734 bytes --]

bug#58711: Treesit hangs when calling treesit-search-forward

[Yuan Fu]

> On Oct 22, 2022, at 2:53 AM, Wilhelm Kirschbaum <wkirschbaum@gmail.com> wrote:
> 
> I am attempting to implement treesit for the current elixir-mode using the branch feature/trees-sitter. For implementing beginning-of-defun-function using treesit I am running into an issue where the Emacs will hang indefinitely. Using (setq-local treesit-defun-type-regexp (rx (or "call"))) yields the same result.
> 
> For the following node-at-point output
> 
> (do_block (call target: (identifier)))
> 
> with Elixir code
> 
> # foo.ex
> defmodule Foo do
>   <<point/cursor here>>def bar(), do: "bar"
> end
> 
> When I call `(treesit-search-forward-goto (rx (or "call")) 'start nil t)` the function `treesit-search-forward` seems to get stuck. 
> 
> Elixir does not strictly have a begin function, but can be determined as one of the following:
> ; * modules and protocols
> (call
>   target: (identifier) @ignore
>   (arguments (alias) @name)
>   (#match? @ignore "^(defmodule|defprotocol)$")) @definition.module
> 
> ; * functions/macros
> (call
>   target: (identifier) @ignore
>   (arguments
>     [
>       ; zero-arity functions with no parentheses
>       (identifier) @name
>       ; regular function clause
>       (call target: (identifier) @name)
>       ; function clause with a guard clause
>       (binary_operator
>         left: (call target: (identifier) @name)
>         operator: "when")
>     ])
>   (#match? @ignore "^(def|defp|defdelegate|defguard|defguardp|defmacro|defmacrop|defn|defnp)$")) @definition.function
> 
> The elixir tree sitter implementation is here: https://github.com/elixir-lang/tree-sitter-elixir
> 
> I lack the knowledge to further debug this or find a clean workaround, but I'm almost sure that treesit-search-forward should never hang. 

Thanks for reporting this. The way treesit-search-forward-goto works makes it very to have infinite loops. I revised the way it traverses the tree and now it should be impossible to fall into infinite loops.

Yuan

bug#58711: Treesit hangs when calling treesit-search-forward

[Yuan Fu]

> On Oct 22, 2022, at 11:42 PM, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> 
>> On Oct 22, 2022, at 2:53 AM, Wilhelm Kirschbaum <wkirschbaum@gmail.com> wrote:
>> 
>> I am attempting to implement treesit for the current elixir-mode using the branch feature/trees-sitter. For implementing beginning-of-defun-function using treesit I am running into an issue where the Emacs will hang indefinitely. Using (setq-local treesit-defun-type-regexp (rx (or "call"))) yields the same result.
>> 
>> For the following node-at-point output
>> 
>> (do_block (call target: (identifier)))
>> 
>> with Elixir code
>> 
>> # foo.ex
>> defmodule Foo do
>>  <<point/cursor here>>def bar(), do: "bar"
>> end
>> 
>> When I call `(treesit-search-forward-goto (rx (or "call")) 'start nil t)` the function `treesit-search-forward` seems to get stuck. 
>> 
>> Elixir does not strictly have a begin function, but can be determined as one of the following:
>> ; * modules and protocols
>> (call
>>  target: (identifier) @ignore
>>  (arguments (alias) @name)
>>  (#match? @ignore "^(defmodule|defprotocol)$")) @definition.module
>> 
>> ; * functions/macros
>> (call
>>  target: (identifier) @ignore
>>  (arguments
>>    [
>>      ; zero-arity functions with no parentheses
>>      (identifier) @name
>>      ; regular function clause
>>      (call target: (identifier) @name)
>>      ; function clause with a guard clause
>>      (binary_operator
>>        left: (call target: (identifier) @name)
>>        operator: "when")
>>    ])
>>  (#match? @ignore "^(def|defp|defdelegate|defguard|defguardp|defmacro|defmacrop|defn|defnp)$")) @definition.function
>> 
>> The elixir tree sitter implementation is here: https://github.com/elixir-lang/tree-sitter-elixir
>> 
>> I lack the knowledge to further debug this or find a clean workaround, but I'm almost sure that treesit-search-forward should never hang. 
> 
> Thanks for reporting this. The way treesit-search-forward-goto works makes it very to have infinite loops. I revised the way it traverses the tree and now it should be impossible to fall into infinite loops.
> 
> Yuan

Just a heads up: I changed the function signature of treesit-search-forward-goto in the latest commit.

Yuan

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Hugo Kirschbaum]

On Monday, 24 October 2022 01:20:01 SAST Yuan Fu wrote:
> > On Oct 22, 2022, at 11:42 PM, Yuan Fu <casouri@gmail.com> wrote:
> >> On Oct 22, 2022, at 2:53 AM, Wilhelm Kirschbaum <wkirschbaum@gmail.com>
> >> wrote:
> >> 
> >> I am attempting to implement treesit for the current elixir-mode using
> >> the branch feature/trees-sitter. For implementing
> >> beginning-of-defun-function using treesit I am running into an issue
> >> where the Emacs will hang indefinitely. Using (setq-local
> >> treesit-defun-type-regexp (rx (or "call"))) yields the same result.
> >> 
> >> For the following node-at-point output
> >> 
> >> (do_block (call target: (identifier)))
> >> 
> >> with Elixir code
> >> 
> >> # foo.ex
> >> defmodule Foo do
> >> 
> >>  <<point/cursor here>>def bar(), do: "bar"
> >> 
> >> end
> >> 
> >> When I call `(treesit-search-forward-goto (rx (or "call")) 'start nil t)`
> >> the function `treesit-search-forward` seems to get stuck.
> >> 
> >> Elixir does not strictly have a begin function, but can be determined as
> >> one of the following: ; * modules and protocols
> >> (call
> >> 
> >>  target: (identifier) @ignore
> >>  (arguments (alias) @name)
> >>  (#match? @ignore "^(defmodule|defprotocol)$")) @definition.module
> >> 
> >> ; * functions/macros
> >> (call
> >> 
> >>  target: (identifier) @ignore
> >>  (arguments
> >>  
> >>    [
> >>    
> >>      ; zero-arity functions with no parentheses
> >>      (identifier) @name
> >>      ; regular function clause
> >>      (call target: (identifier) @name)
> >>      ; function clause with a guard clause
> >>      (binary_operator
> >>      
> >>        left: (call target: (identifier) @name)
> >>        operator: "when")
> >>    
> >>    ])
> >>  
> >>  (#match? @ignore
> >>  "^(def|defp|defdelegate|defguard|defguardp|defmacro|defmacrop|defn|defn
> >>  p)$")) @definition.function>> 
> >> The elixir tree sitter implementation is here:
> >> https://github.com/elixir-lang/tree-sitter-elixir
> >> 
> >> I lack the knowledge to further debug this or find a clean workaround,
> >> but I'm almost sure that treesit-search-forward should never hang.> 
> > Thanks for reporting this. The way treesit-search-forward-goto works makes
> > it very to have infinite loops. I revised the way it traverses the tree
> > and now it should be impossible to fall into infinite loops.
> > 
> > Yuan
> 
> Just a heads up: I changed the function signature of
> treesit-search-forward-goto in the latest commit.
> 
> Yuan

Thanks, I would need some time to have a proper look. The reporting issue 
seems to have been fixed. There are some complications to identify the 
appropriate node for defun in elixir as a "call" node might, or might not be 
defun, depending on its child identifier ( i currently have to do a query on 
cycling up the parent ), but will try the respond in more detail in another 
thread.  

bug#58711: Treesit hangs when calling treesit-search-forward

[Yuan Fu]

> 
> Thanks, I would need some time to have a proper look. The reporting issue 
> seems to have been fixed. There are some complications to identify the 
> appropriate node for defun in elixir as a "call" node might, or might not be 
> defun, depending on its child identifier ( i currently have to do a query on 
> cycling up the parent ), but will try the respond in more detail in another 
> thread.  

While fixing the hanging issue, I also improved treesit-beginning-of-defun, which now first goes backward to find a node that matches tresit-defun-type-regexp, then goes upward and finds the top-most parent matching treesit-defun-type-regexp, IOW, it now only stops at top-level defun matches. Does that help with your case?

Yuan

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 2042 bytes --]

I finally had some time to have a look. I don't see any more issues, thank
you for the fantastic work on this. The defun-type-regexp is not enough to
identify a defun in elixir this is the query I am using currently:

(defvar elixir--treesit-query-defun
  (let ((query `((call
     target: (identifier) @type
     (arguments
      [
       (alias) @name
       (identifier) @name
       (call target: (identifier)) @name
       (binary_operator
        left: (call target: (identifier)) @name
        operator: "when")
       ])
     (:match ,elixir--definition-keywords-re @type)
     ))))
    (treesit-query-compile 'elixir query)))

Regex will work in most cases I guess, but does not really deal with more
complex queries for more complex cases like in elixir as there is not one
type which is always the defun. elixir relies heavily on macros and
different defun macros can be defined on the fly. Maybe if there is an
option for using either a regex or a function?

I am also not sure how forward-sexp can work with the current
treesit-search-forward-goto function as it does not take into consideration
the level. Is there perhaps a way to move forward/backward, but do not jump
to parents or children?


On Mon, 24 Oct 2022 at 22:19, Yuan Fu <casouri@gmail.com> wrote:

> >
> > Thanks, I would need some time to have a proper look. The reporting
> issue
> > seems to have been fixed. There are some complications to identify the
> > appropriate node for defun in elixir as a "call" node might, or might
> not be
> > defun, depending on its child identifier ( i currently have to do a
> query on
> > cycling up the parent ), but will try the respond in more detail in
> another
> > thread.
>
> While fixing the hanging issue, I also improved
> treesit-beginning-of-defun, which now first goes backward to find a node
> that matches tresit-defun-type-regexp, then goes upward and finds the
> top-most parent matching treesit-defun-type-regexp, IOW, it now only stops
> at top-level defun matches. Does that help with your case?
>
> Yuan

[-- Attachment #2: Type: text/html, Size: 2652 bytes --]

bug#58711: Treesit hangs when calling treesit-search-forward

[Yuan Fu]

> On Nov 9, 2022, at 10:44 PM, Wilhelm Kirschbaum <wilhelm@floatpays.co.za> wrote:
> 
> I finally had some time to have a look. I don't see any more issues, thank you for the fantastic work on this. The defun-type-regexp is not enough to identify a defun in elixir this is the query I am using currently:
> 
> (defvar elixir--treesit-query-defun
>   (let ((query `((call
>      target: (identifier) @type
>      (arguments
>       [
>        (alias) @name
>        (identifier) @name
>        (call target: (identifier)) @name
>        (binary_operator
>         left: (call target: (identifier)) @name
>         operator: "when")
>        ])
>      (:match ,elixir--definition-keywords-re @type)
>      ))))
>     (treesit-query-compile 'elixir query)))
> 
> Regex will work in most cases I guess, but does not really deal with more complex queries for more complex cases like in elixir as there is not one type which is always the defun. elixir relies heavily on macros and different defun macros can be defined on the fly.

You can try the following procedure: use a regex to find the next/previous call, then perform some check on whether it’s indeed a defun, if not, keep searching for the next/previous call.


> Maybe if there is an option for using either a regex or a function? 

Yes, instead of a regex you can pass a predicate function.

> 
> I am also not sure how forward-sexp can work with the current treesit-search-forward-goto function as it does not take into consideration the level. Is there perhaps a way to move forward/backward, but do not jump to parents or children? 

If you want to move in the same level, perhaps you can use treesit-next/prev/sibling?

Yuan

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 3161 bytes --]

forward-sexp works well with the below code, but when calling
treesit-node-prev-sibling it will traverse up the list, which is then
breaking backward-up-list when defining forward-sexp in the major mode.

(defun heex--treesit-largest-node-at-point ()
  "Find the largest node at point."
  (save-excursion
    (forward-comment (point-max))
    (let ((node-list
           (cl-loop for node = (treesit-node-at (point))
                    then (treesit-node-parent node)
                    while node
                    if (eq (treesit-node-start node)
                           (point))
                    collect node)))
      (car (last node-list)))))

(defun heex--treesit-backward-sexp ()
  "Forward sexp for Heex using treesit."
  (let* ((largest-node (heex--treesit-largest-node-at-point))
         (sibling (treesit-node-prev-sibling largest-node)))
    (when sibling
      (goto-char (treesit-node-start sibling)))))

(defun heex--treesit-forward-sexp ()
  "Forward sexp for Heex using treesit."
  (let* ((largest-node (heex--treesit-largest-node-at-point))
         (sibling (treesit-node-next-sibling largest-node)))
    (when sibling
      (goto-char (treesit-node-start sibling))
      (forward-comment (- (point-max))))))

On Thu, 10 Nov 2022 at 10:14, Yuan Fu <casouri@gmail.com> wrote:

>
>
> > On Nov 9, 2022, at 10:44 PM, Wilhelm Kirschbaum <wilhelm@floatpays.co.za>
> wrote:
> >
> > I finally had some time to have a look. I don't see any more issues,
> thank you for the fantastic work on this. The defun-type-regexp is not
> enough to identify a defun in elixir this is the query I am using currently:
> >
> > (defvar elixir--treesit-query-defun
> >   (let ((query `((call
> >      target: (identifier) @type
> >      (arguments
> >       [
> >        (alias) @name
> >        (identifier) @name
> >        (call target: (identifier)) @name
> >        (binary_operator
> >         left: (call target: (identifier)) @name
> >         operator: "when")
> >        ])
> >      (:match ,elixir--definition-keywords-re @type)
> >      ))))
> >     (treesit-query-compile 'elixir query)))
> >
> > Regex will work in most cases I guess, but does not really deal with
> more complex queries for more complex cases like in elixir as there is not
> one type which is always the defun. elixir relies heavily on macros and
> different defun macros can be defined on the fly.
>
> You can try the following procedure: use a regex to find the next/previous
> call, then perform some check on whether it’s indeed a defun, if not, keep
> searching for the next/previous call.
>
>
> > Maybe if there is an option for using either a regex or a function?
>
> Yes, instead of a regex you can pass a predicate function.
>
> >
> > I am also not sure how forward-sexp can work with the current
> treesit-search-forward-goto function as it does not take into consideration
> the level. Is there perhaps a way to move forward/backward, but do not jump
> to parents or children?
>
> If you want to move in the same level, perhaps you can use
> treesit-next/prev/sibling?
>
> Yuan

[-- Attachment #2: Type: text/html, Size: 3903 bytes --]

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 3462 bytes --]

Full implementation here:
https://github.com/wkirschbaum/elixir-mode/blob/main/heex-mode.el

On Thu, 10 Nov 2022 at 21:03, Wilhelm Kirschbaum <wilhelm@floatpays.co.za>
wrote:

> forward-sexp works well with the below code, but when calling
> treesit-node-prev-sibling it will traverse up the list, which is then
> breaking backward-up-list when defining forward-sexp in the major mode.
>
> (defun heex--treesit-largest-node-at-point ()
>   "Find the largest node at point."
>   (save-excursion
>     (forward-comment (point-max))
>     (let ((node-list
>            (cl-loop for node = (treesit-node-at (point))
>                     then (treesit-node-parent node)
>                     while node
>                     if (eq (treesit-node-start node)
>                            (point))
>                     collect node)))
>       (car (last node-list)))))
>
> (defun heex--treesit-backward-sexp ()
>   "Forward sexp for Heex using treesit."
>   (let* ((largest-node (heex--treesit-largest-node-at-point))
>          (sibling (treesit-node-prev-sibling largest-node)))
>     (when sibling
>       (goto-char (treesit-node-start sibling)))))
>
> (defun heex--treesit-forward-sexp ()
>   "Forward sexp for Heex using treesit."
>   (let* ((largest-node (heex--treesit-largest-node-at-point))
>          (sibling (treesit-node-next-sibling largest-node)))
>     (when sibling
>       (goto-char (treesit-node-start sibling))
>       (forward-comment (- (point-max))))))
>
> On Thu, 10 Nov 2022 at 10:14, Yuan Fu <casouri@gmail.com> wrote:
>
>>
>>
>> > On Nov 9, 2022, at 10:44 PM, Wilhelm Kirschbaum <
>> wilhelm@floatpays.co.za> wrote:
>> >
>> > I finally had some time to have a look. I don't see any more issues,
>> thank you for the fantastic work on this. The defun-type-regexp is not
>> enough to identify a defun in elixir this is the query I am using currently:
>> >
>> > (defvar elixir--treesit-query-defun
>> >   (let ((query `((call
>> >      target: (identifier) @type
>> >      (arguments
>> >       [
>> >        (alias) @name
>> >        (identifier) @name
>> >        (call target: (identifier)) @name
>> >        (binary_operator
>> >         left: (call target: (identifier)) @name
>> >         operator: "when")
>> >        ])
>> >      (:match ,elixir--definition-keywords-re @type)
>> >      ))))
>> >     (treesit-query-compile 'elixir query)))
>> >
>> > Regex will work in most cases I guess, but does not really deal with
>> more complex queries for more complex cases like in elixir as there is not
>> one type which is always the defun. elixir relies heavily on macros and
>> different defun macros can be defined on the fly.
>>
>> You can try the following procedure: use a regex to find the
>> next/previous call, then perform some check on whether it’s indeed a defun,
>> if not, keep searching for the next/previous call.
>>
>>
>> > Maybe if there is an option for using either a regex or a function?
>>
>> Yes, instead of a regex you can pass a predicate function.
>>
>> >
>> > I am also not sure how forward-sexp can work with the current
>> treesit-search-forward-goto function as it does not take into consideration
>> the level. Is there perhaps a way to move forward/backward, but do not jump
>> to parents or children?
>>
>> If you want to move in the same level, perhaps you can use
>> treesit-next/prev/sibling?
>>
>> Yuan
>
>

[-- Attachment #2: Type: text/html, Size: 4458 bytes --]

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 4201 bytes --]

Sorry, I see something like this makes sense and is not the prev-sibling
issue.

(defun heex--treesit-backward-sexp ()
  "Forward sexp for Heex using treesit."
  (let* ((node (treesit-search-forward
                (treesit-node-at (point))
                (rx (or "end_tag" "end_component" "end_slot"))
                t))
         (sibling (treesit-node-prev-sibling node)))
    (when sibling
      (goto-char (treesit-node-start sibling)))))

Just need to handle the outermost (fragment peace and should work then.

Thanks for the help.

On Thu, 10 Nov 2022 at 21:05, Wilhelm Kirschbaum <wilhelm@floatpays.co.za>
wrote:

> Full implementation here:
> https://github.com/wkirschbaum/elixir-mode/blob/main/heex-mode.el
>
> On Thu, 10 Nov 2022 at 21:03, Wilhelm Kirschbaum <wilhelm@floatpays.co.za>
> wrote:
>
>> forward-sexp works well with the below code, but when calling
>> treesit-node-prev-sibling it will traverse up the list, which is then
>> breaking backward-up-list when defining forward-sexp in the major mode.
>>
>> (defun heex--treesit-largest-node-at-point ()
>>   "Find the largest node at point."
>>   (save-excursion
>>     (forward-comment (point-max))
>>     (let ((node-list
>>            (cl-loop for node = (treesit-node-at (point))
>>                     then (treesit-node-parent node)
>>                     while node
>>                     if (eq (treesit-node-start node)
>>                            (point))
>>                     collect node)))
>>       (car (last node-list)))))
>>
>> (defun heex--treesit-backward-sexp ()
>>   "Forward sexp for Heex using treesit."
>>   (let* ((largest-node (heex--treesit-largest-node-at-point))
>>          (sibling (treesit-node-prev-sibling largest-node)))
>>     (when sibling
>>       (goto-char (treesit-node-start sibling)))))
>>
>> (defun heex--treesit-forward-sexp ()
>>   "Forward sexp for Heex using treesit."
>>   (let* ((largest-node (heex--treesit-largest-node-at-point))
>>          (sibling (treesit-node-next-sibling largest-node)))
>>     (when sibling
>>       (goto-char (treesit-node-start sibling))
>>       (forward-comment (- (point-max))))))
>>
>> On Thu, 10 Nov 2022 at 10:14, Yuan Fu <casouri@gmail.com> wrote:
>>
>>>
>>>
>>> > On Nov 9, 2022, at 10:44 PM, Wilhelm Kirschbaum <
>>> wilhelm@floatpays.co.za> wrote:
>>> >
>>> > I finally had some time to have a look. I don't see any more issues,
>>> thank you for the fantastic work on this. The defun-type-regexp is not
>>> enough to identify a defun in elixir this is the query I am using currently:
>>> >
>>> > (defvar elixir--treesit-query-defun
>>> >   (let ((query `((call
>>> >      target: (identifier) @type
>>> >      (arguments
>>> >       [
>>> >        (alias) @name
>>> >        (identifier) @name
>>> >        (call target: (identifier)) @name
>>> >        (binary_operator
>>> >         left: (call target: (identifier)) @name
>>> >         operator: "when")
>>> >        ])
>>> >      (:match ,elixir--definition-keywords-re @type)
>>> >      ))))
>>> >     (treesit-query-compile 'elixir query)))
>>> >
>>> > Regex will work in most cases I guess, but does not really deal with
>>> more complex queries for more complex cases like in elixir as there is not
>>> one type which is always the defun. elixir relies heavily on macros and
>>> different defun macros can be defined on the fly.
>>>
>>> You can try the following procedure: use a regex to find the
>>> next/previous call, then perform some check on whether it’s indeed a defun,
>>> if not, keep searching for the next/previous call.
>>>
>>>
>>> > Maybe if there is an option for using either a regex or a function?
>>>
>>> Yes, instead of a regex you can pass a predicate function.
>>>
>>> >
>>> > I am also not sure how forward-sexp can work with the current
>>> treesit-search-forward-goto function as it does not take into consideration
>>> the level. Is there perhaps a way to move forward/backward, but do not jump
>>> to parents or children?
>>>
>>> If you want to move in the same level, perhaps you can use
>>> treesit-next/prev/sibling?
>>>
>>> Yuan
>>
>>

[-- Attachment #2: Type: text/html, Size: 5595 bytes --]

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 5586 bytes --]

Given the following code

 <.foo>  ;; component, start_component
 <bar>  ;; tag, start_tag
  </bar>;; [cursor here 1)] end_tag
</.foo> ;; [cursor here 2] end_component

and ast

(fragment [0, 0] - [5, 0]
  (component [0, 0] - [3, 7]
    (start_component [0, 0] - [0, 6]
      (component_name [0, 1] - [0, 5]
        (function [0, 2] - [0, 5])))
    (tag [1, 2] - [2, 8]
      (start_tag [1, 2] - [1, 7]
        (tag_name [1, 3] - [1, 6]))
      (end_tag [2, 2] - [2, 8]
        (tag_name [2, 4] - [2, 7])))
    (end_component [3, 0] - [3, 7]
      (component_name [3, 2] - [3, 6]
        (function [3, 3] - [3, 6])))))

I do not know how to reliably move from cursor position 1 to start of <bar>
and from cursor position 2 to start of <.foo>

as (treesit-search-forward (treesit-node-at (point)) (rx (or "end_tag"
"end_component" "end_slot")) t) on position 1 will not look backwards, but
find the <./foo> end_component.

if i am at the point after the node, how do I find the node before the
point? once we have the node before the point we can find the correct
parent and then find the prev sibling to goto.

Hope this makes sense.


On Thu, 10 Nov 2022 at 21:32, Wilhelm Kirschbaum <wilhelm@floatpays.co.za>
wrote:

> Sorry, I see something like this makes sense and is not the prev-sibling
> issue.
>
> (defun heex--treesit-backward-sexp ()
>   "Forward sexp for Heex using treesit."
>   (let* ((node (treesit-search-forward
>                 (treesit-node-at (point))
>                 (rx (or "end_tag" "end_component" "end_slot"))
>                 t))
>          (sibling (treesit-node-prev-sibling node)))
>     (when sibling
>       (goto-char (treesit-node-start sibling)))))
>
> Just need to handle the outermost (fragment peace and should work then.
>
> Thanks for the help.
>
> On Thu, 10 Nov 2022 at 21:05, Wilhelm Kirschbaum <wilhelm@floatpays.co.za>
> wrote:
>
>> Full implementation here:
>> https://github.com/wkirschbaum/elixir-mode/blob/main/heex-mode.el
>>
>> On Thu, 10 Nov 2022 at 21:03, Wilhelm Kirschbaum <wilhelm@floatpays.co.za>
>> wrote:
>>
>>> forward-sexp works well with the below code, but when calling
>>> treesit-node-prev-sibling it will traverse up the list, which is then
>>> breaking backward-up-list when defining forward-sexp in the major mode.
>>>
>>> (defun heex--treesit-largest-node-at-point ()
>>>   "Find the largest node at point."
>>>   (save-excursion
>>>     (forward-comment (point-max))
>>>     (let ((node-list
>>>            (cl-loop for node = (treesit-node-at (point))
>>>                     then (treesit-node-parent node)
>>>                     while node
>>>                     if (eq (treesit-node-start node)
>>>                            (point))
>>>                     collect node)))
>>>       (car (last node-list)))))
>>>
>>> (defun heex--treesit-backward-sexp ()
>>>   "Forward sexp for Heex using treesit."
>>>   (let* ((largest-node (heex--treesit-largest-node-at-point))
>>>          (sibling (treesit-node-prev-sibling largest-node)))
>>>     (when sibling
>>>       (goto-char (treesit-node-start sibling)))))
>>>
>>> (defun heex--treesit-forward-sexp ()
>>>   "Forward sexp for Heex using treesit."
>>>   (let* ((largest-node (heex--treesit-largest-node-at-point))
>>>          (sibling (treesit-node-next-sibling largest-node)))
>>>     (when sibling
>>>       (goto-char (treesit-node-start sibling))
>>>       (forward-comment (- (point-max))))))
>>>
>>> On Thu, 10 Nov 2022 at 10:14, Yuan Fu <casouri@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> > On Nov 9, 2022, at 10:44 PM, Wilhelm Kirschbaum <
>>>> wilhelm@floatpays.co.za> wrote:
>>>> >
>>>> > I finally had some time to have a look. I don't see any more issues,
>>>> thank you for the fantastic work on this. The defun-type-regexp is not
>>>> enough to identify a defun in elixir this is the query I am using currently:
>>>> >
>>>> > (defvar elixir--treesit-query-defun
>>>> >   (let ((query `((call
>>>> >      target: (identifier) @type
>>>> >      (arguments
>>>> >       [
>>>> >        (alias) @name
>>>> >        (identifier) @name
>>>> >        (call target: (identifier)) @name
>>>> >        (binary_operator
>>>> >         left: (call target: (identifier)) @name
>>>> >         operator: "when")
>>>> >        ])
>>>> >      (:match ,elixir--definition-keywords-re @type)
>>>> >      ))))
>>>> >     (treesit-query-compile 'elixir query)))
>>>> >
>>>> > Regex will work in most cases I guess, but does not really deal with
>>>> more complex queries for more complex cases like in elixir as there is not
>>>> one type which is always the defun. elixir relies heavily on macros and
>>>> different defun macros can be defined on the fly.
>>>>
>>>> You can try the following procedure: use a regex to find the
>>>> next/previous call, then perform some check on whether it’s indeed a defun,
>>>> if not, keep searching for the next/previous call.
>>>>
>>>>
>>>> > Maybe if there is an option for using either a regex or a function?
>>>>
>>>> Yes, instead of a regex you can pass a predicate function.
>>>>
>>>> >
>>>> > I am also not sure how forward-sexp can work with the current
>>>> treesit-search-forward-goto function as it does not take into consideration
>>>> the level. Is there perhaps a way to move forward/backward, but do not jump
>>>> to parents or children?
>>>>
>>>> If you want to move in the same level, perhaps you can use
>>>> treesit-next/prev/sibling?
>>>>
>>>> Yuan
>>>
>>>

[-- Attachment #2: Type: text/html, Size: 7454 bytes --]

bug#58711: Treesit hangs when calling treesit-search-forward

[Yuan Fu]

> On Nov 10, 2022, at 12:43 PM, Wilhelm Kirschbaum <wilhelm@floatpays.co.za> wrote:
> 
> Given the following code
> 
>  <.foo>  ;; component, start_component
>  <bar>  ;; tag, start_tag
>   </bar>;; [cursor here 1)] end_tag 
> </.foo> ;; [cursor here 2] end_component
> 
> and ast
> 
> (fragment [0, 0] - [5, 0]
>   (component [0, 0] - [3, 7]
>     (start_component [0, 0] - [0, 6]
>       (component_name [0, 1] - [0, 5]
>         (function [0, 2] - [0, 5])))
>     (tag [1, 2] - [2, 8]
>       (start_tag [1, 2] - [1, 7]
>         (tag_name [1, 3] - [1, 6]))
>       (end_tag [2, 2] - [2, 8]
>         (tag_name [2, 4] - [2, 7])))
>     (end_component [3, 0] - [3, 7]
>       (component_name [3, 2] - [3, 6]
>         (function [3, 3] - [3, 6])))))
> 
> I do not know how to reliably move from cursor position 1 to start of <bar> and from cursor position 2 to start of <.foo>

The snippet below should take you to the corresponding open tag.

(let ((end-node (treesit-search-forward-goto
                 (treesit-node-at (point))
                 (rx (or "end_tag" "end_component" "end_slot")) t t)))
  ;; Go to the corresponding start tag.
  (goto-char (treesit-node-start (treesit-node-parent end-node))))

> 
> as (treesit-search-forward (treesit-node-at (point)) (rx (or "end_tag" "end_component" "end_slot")) t) on position 1 will not look backwards, but find the <./foo> end_component. 
> 
> if i am at the point after the node, how do I find the node before the point? once we have the node before the point we can find the correct parent and then find the prev sibling to goto. 
> 
> Hope this makes sense. 

Yeah, if there is a function that gives you the node right before point, treesit-search-forward would work as expected. I’ll see how to add this functionality.

Yuan

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 2083 bytes --]

Thank you. I will spend some time over the weekend to try and have a look.

On Fri, 11 Nov 2022 at 00:07, Yuan Fu <casouri@gmail.com> wrote:

>
>
> > On Nov 10, 2022, at 12:43 PM, Wilhelm Kirschbaum <
> wilhelm@floatpays.co.za> wrote:
> >
> > Given the following code
> >
> >  <.foo>  ;; component, start_component
> >  <bar>  ;; tag, start_tag
> >   </bar>;; [cursor here 1)] end_tag
> > </.foo> ;; [cursor here 2] end_component
> >
> > and ast
> >
> > (fragment [0, 0] - [5, 0]
> >   (component [0, 0] - [3, 7]
> >     (start_component [0, 0] - [0, 6]
> >       (component_name [0, 1] - [0, 5]
> >         (function [0, 2] - [0, 5])))
> >     (tag [1, 2] - [2, 8]
> >       (start_tag [1, 2] - [1, 7]
> >         (tag_name [1, 3] - [1, 6]))
> >       (end_tag [2, 2] - [2, 8]
> >         (tag_name [2, 4] - [2, 7])))
> >     (end_component [3, 0] - [3, 7]
> >       (component_name [3, 2] - [3, 6]
> >         (function [3, 3] - [3, 6])))))
> >
> > I do not know how to reliably move from cursor position 1 to start of
> <bar> and from cursor position 2 to start of <.foo>
>
> The snippet below should take you to the corresponding open tag.
>
> (let ((end-node (treesit-search-forward-goto
>                  (treesit-node-at (point))
>                  (rx (or "end_tag" "end_component" "end_slot")) t t)))
>   ;; Go to the corresponding start tag.
>   (goto-char (treesit-node-start (treesit-node-parent end-node))))
>
> >
> > as (treesit-search-forward (treesit-node-at (point)) (rx (or "end_tag"
> "end_component" "end_slot")) t) on position 1 will not look backwards, but
> find the <./foo> end_component.
> >
> > if i am at the point after the node, how do I find the node before the
> point? once we have the node before the point we can find the correct
> parent and then find the prev sibling to goto.
> >
> > Hope this makes sense.
>
> Yeah, if there is a function that gives you the node right before point,
> treesit-search-forward would work as expected. I’ll see how to add this
> functionality.
>
> Yuan

[-- Attachment #2: Type: text/html, Size: 2781 bytes --]

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 4037 bytes --]

This works for me for all cases I managed to test:

( I have to keep track of the EOL and BOL to ensure that we don't jump to
parents or children. )

(defun heex--treesit-largest-node-at-point (&optional node)
  "Find the largest node at point or from specified NODE."
  (save-excursion
    (forward-comment (point-max))
    (let ((node-list
           (cl-loop for node = (or node (treesit-node-at (point)))
                    then (treesit-node-parent node)
                    while (and node (not (equal (treesit-node-type node)
"fragment")))
                    if (eq (treesit-node-start node)
                           (point))
                    collect node)))
      (car (last node-list)))))

(defun heex--treesit-backward-sexp ()
  "Forward sexp for Heex using treesit."
  (let ((node
         (save-excursion
           (forward-comment (- (point-max)))
           (let ((bol (pos-bol))
                 (end-node (treesit-search-forward-goto
                            (treesit-node-at (point))
                            (rx (or "end_tag" "end_component" "end_slot"))
t t)))
             (if (and end-node (> (treesit-node-end end-node) bol))
                 (treesit-node-start (treesit-node-parent end-node)))))))
    (when node (goto-char node))))

(defun heex--treesit-forward-sexp ()
  "Forward sexp for Heex using treesit."
  (let* ((node (heex--treesit-largest-node-at-point))
         (sibling (treesit-node-next-sibling node)))
    (if sibling
        (progn
          (goto-char (treesit-node-start sibling))
          (forward-comment (- (point-max))))
      (when node
        (pcase (treesit-node-type node)
          ((or "end_tag" "end_component" "end_slot") nil)
          (_ (goto-char (treesit-node-end node))))))))

On Fri, 11 Nov 2022 at 08:30, Wilhelm Kirschbaum <wilhelm@floatpays.co.za>
wrote:

> Thank you. I will spend some time over the weekend to try and have a look.
>
> On Fri, 11 Nov 2022 at 00:07, Yuan Fu <casouri@gmail.com> wrote:
>
>>
>>
>> > On Nov 10, 2022, at 12:43 PM, Wilhelm Kirschbaum <
>> wilhelm@floatpays.co.za> wrote:
>> >
>> > Given the following code
>> >
>> >  <.foo>  ;; component, start_component
>> >  <bar>  ;; tag, start_tag
>> >   </bar>;; [cursor here 1)] end_tag
>> > </.foo> ;; [cursor here 2] end_component
>> >
>> > and ast
>> >
>> > (fragment [0, 0] - [5, 0]
>> >   (component [0, 0] - [3, 7]
>> >     (start_component [0, 0] - [0, 6]
>> >       (component_name [0, 1] - [0, 5]
>> >         (function [0, 2] - [0, 5])))
>> >     (tag [1, 2] - [2, 8]
>> >       (start_tag [1, 2] - [1, 7]
>> >         (tag_name [1, 3] - [1, 6]))
>> >       (end_tag [2, 2] - [2, 8]
>> >         (tag_name [2, 4] - [2, 7])))
>> >     (end_component [3, 0] - [3, 7]
>> >       (component_name [3, 2] - [3, 6]
>> >         (function [3, 3] - [3, 6])))))
>> >
>> > I do not know how to reliably move from cursor position 1 to start of
>> <bar> and from cursor position 2 to start of <.foo>
>>
>> The snippet below should take you to the corresponding open tag.
>>
>> (let ((end-node (treesit-search-forward-goto
>>                  (treesit-node-at (point))
>>                  (rx (or "end_tag" "end_component" "end_slot")) t t)))
>>   ;; Go to the corresponding start tag.
>>   (goto-char (treesit-node-start (treesit-node-parent end-node))))
>>
>> >
>> > as (treesit-search-forward (treesit-node-at (point)) (rx (or "end_tag"
>> "end_component" "end_slot")) t) on position 1 will not look backwards, but
>> find the <./foo> end_component.
>> >
>> > if i am at the point after the node, how do I find the node before the
>> point? once we have the node before the point we can find the correct
>> parent and then find the prev sibling to goto.
>> >
>> > Hope this makes sense.
>>
>> Yeah, if there is a function that gives you the node right before point,
>> treesit-search-forward would work as expected. I’ll see how to add this
>> functionality.
>>
>> Yuan
>
>

[-- Attachment #2: Type: text/html, Size: 5364 bytes --]

bug#58711: Treesit hangs when calling treesit-search-forward

[Yuan Fu]

> On Nov 13, 2022, at 10:32 PM, Wilhelm Kirschbaum <wilhelm@floatpays.co.za> wrote:
> 
> This works for me for all cases I managed to test:
> 
> ( I have to keep track of the EOL and BOL to ensure that we don't jump to parents or children. )
> 
> (defun heex--treesit-largest-node-at-point (&optional node)
>   "Find the largest node at point or from specified NODE."
>   (save-excursion
>     (forward-comment (point-max))
>     (let ((node-list
>            (cl-loop for node = (or node (treesit-node-at (point)))
>                     then (treesit-node-parent node)
>                     while (and node (not (equal (treesit-node-type node) "fragment")))
>                     if (eq (treesit-node-start node)
>                            (point))
>                     collect node)))
>       (car (last node-list)))))

For largest node at point, you can probably use something like

(treesit-parent-while (or node (treesit-node-at (point)))
                      (lambda (n) (eq (treesit-node-start n) (point))))

Yuan

bug#58711: Treesit hangs when calling treesit-search-forward

[Wilhelm Kirschbaum]

[-- Attachment #1: Type: text/plain, Size: 1194 bytes --]

Thank you, the treesit-parent-while is better.

On Mon, 14 Nov 2022 at 10:45, Yuan Fu <casouri@gmail.com> wrote:

>
>
> > On Nov 13, 2022, at 10:32 PM, Wilhelm Kirschbaum <
> wilhelm@floatpays.co.za> wrote:
> >
> > This works for me for all cases I managed to test:
> >
> > ( I have to keep track of the EOL and BOL to ensure that we don't jump
> to parents or children. )
> >
> > (defun heex--treesit-largest-node-at-point (&optional node)
> >   "Find the largest node at point or from specified NODE."
> >   (save-excursion
> >     (forward-comment (point-max))
> >     (let ((node-list
> >            (cl-loop for node = (or node (treesit-node-at (point)))
> >                     then (treesit-node-parent node)
> >                     while (and node (not (equal (treesit-node-type node)
> "fragment")))
> >                     if (eq (treesit-node-start node)
> >                            (point))
> >                     collect node)))
> >       (car (last node-list)))))
>
> For largest node at point, you can probably use something like
>
> (treesit-parent-while (or node (treesit-node-at (point)))
>                       (lambda (n) (eq (treesit-node-start n) (point))))
>
> Yuan

[-- Attachment #2: Type: text/html, Size: 1789 bytes --]