MPS: treesit

MPS: treesit

[Gerd Möllmann]

I'm trying to make treesit work with igc, and I'm using c-ts-mode for
that purposes. Or I'm trying to because in a full
debug/checking/mps-debug build I immediately get a crash. I'm BTW
looking at/using treesit for the first time, so please bear with me.

treesit_ensure_parsed gets called, and passes a TSParser * to
ts_parser_parse.

  static void
  treesit_ensure_parsed (Lisp_Object parser)
  {
    struct buffer *buffer = XBUFFER (XTS_PARSER (parser)->buffer);

    /* Before we parse, catch up with the narrowing situation.  */
    treesit_check_buffer_size (buffer);
    /* This function has to run before we check for need_reparse flag,
       because it might set the flag to true.  */
    treesit_sync_visible_region (parser);

    /* Make sure this comes before everything else, see comment
       (ref:notifier-inside-ensure-parsed) for more detail.  */
    if (!XTS_PARSER (parser)->need_reparse)
      return;

    TSParser *treesit_parser = XTS_PARSER (parser)->parser;
    TSTree *tree = XTS_PARSER (parser)->tree;
    TSInput input = XTS_PARSER (parser)->input;

    TSTree *new_tree = ts_parser_parse (treesit_parser, tree, input);

  (lldb) p treesit_parser
  (TSParser *) 0x0000000154838e00

Down the stack I see some functions being called in the treesitter dylib

  frame #5: 0x0000000101464e58 libtree-sitter.0.dylib`ts_parser_parse + 2928
  frame #4: 0x000000010146160c libtree-sitter.0.dylib`ts_lexer_start + 84
  frame #3: 0x000000010146167c libtree-sitter.0.dylib`ts_lexer__get_chunk + 40
  frame #2: 0x00000001003ad184 emacs`treesit_read_buffer(parser=0x0000000105049ea0, byte_index=0, position=(row = 0, column = 0), bytes_read=0x0000000154838e9c) at treesit.c:1111:27

until I land here:

  static const char*
  treesit_read_buffer (void *parser, uint32_t byte_index,
                       TSPoint position, uint32_t *bytes_read)
  {
    struct buffer *buffer = XBUFFER (((struct Lisp_TS_Parser *) parser)->buffer);

Here, XBUFFER aborts with an invalid buffer because of --enable-checking=all.

  (lldb) p parser
  (void *) 0x0000000105049ea0

Note that parser here is not the treesit_parser some frames up that was
passed to the treesitter lib.

Can somone help and explain how this works in treesit.c? Where is the
Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
to the treesiter dylib translated to the Lisp_TS_Parser *?

TIA

Re: MPS: treesit

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Can somone help and explain how this works in treesit.c? Where is the
> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
> to the treesiter dylib translated to the Lisp_TS_Parser *?

Forget that question, I see it, TSInput and so on. But I'd be interested
if there are other such constructs in treesit?

Re: MPS: treesit

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Can somone help and explain how this works in treesit.c? Where is the
>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>> to the treesiter dylib translated to the Lisp_TS_Parser *?
>
> Forget that question, I see it, TSInput and so on. But I'd be interested
> if there are other such constructs in treesit?

Pushed a fix to scratch/igc, but the last question would still be
interesting, just in case. It's likely that I don't use all of
what treesit does because I have no idea what it offers.

Re: MPS: treesit

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Helmut Eller <eller.helmut@gmail.com>
> Date: Tue, 25 Jun 2024 09:44:34 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> > Can somone help and explain how this works in treesit.c? Where is the
> > Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
> > to the treesiter dylib translated to the Lisp_TS_Parser *?
> 
> Forget that question, I see it, TSInput and so on. But I'd be interested
> if there are other such constructs in treesit?

Yuan, can you help Gerd with understanding this?

Re: MPS: treesit

[Yuan Fu]

> On Jun 25, 2024, at 12:44 AM, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
>> Can somone help and explain how this works in treesit.c? Where is the
>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>> to the treesiter dylib translated to the Lisp_TS_Parser *?
> 
> Forget that question, I see it, TSInput and so on. But I'd be interested
> if there are other such constructs in treesit?
> 

If you mean tree-sitter struct that’s exposed to lisp, there are Lisp_TS_Parser, Lisp_TS_Node, and Lisp_TS_Query. Lisp_TS_Node is a node in the parser tree, Lisp_TS_Query is a lazily compiled query. Tree-sitter queries, like regexp, are compiled from string to tree-sitter’s internal struct before being used. HTH.

Yuan

Re: MPS: treesit

[Yuan Fu]

> On Jun 25, 2024, at 8:23 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Helmut Eller <eller.helmut@gmail.com>
>> Date: Tue, 25 Jun 2024 09:44:34 +0200
>> 
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>> 
>>> Can somone help and explain how this works in treesit.c? Where is the
>>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>>> to the treesiter dylib translated to the Lisp_TS_Parser *?
>> 
>> Forget that question, I see it, TSInput and so on. But I'd be interested
>> if there are other such constructs in treesit?
> 
> Yuan, can you help Gerd with understanding this?

Of course 😊

Yuan

Re: MPS: treesit

[Gerd Möllmann]

Yuan Fu <casouri@gmail.com> writes:

>> On Jun 25, 2024, at 12:44 AM, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>> 
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>> 
>>> Can somone help and explain how this works in treesit.c? Where is the
>>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>>> to the treesiter dylib translated to the Lisp_TS_Parser *?
>> 
>> Forget that question, I see it, TSInput and so on. But I'd be interested
>> if there are other such constructs in treesit?
>> 
>
> If you mean tree-sitter struct that’s exposed to lisp, there are
> Lisp_TS_Parser, Lisp_TS_Node, and Lisp_TS_Query. Lisp_TS_Node is a
> node in the parser tree, Lisp_TS_Query is a lazily compiled query.
> Tree-sitter queries, like regexp, are compiled from string to
> tree-sitter’s internal struct before being used. HTH.

Hi Yuan!

In the TSInput case that I could fix, TSInput::payload, a void *, is set
to point to a Lisp_TS_Parser, and the read callback in treesit.c gets
access to that parser because tree-sitter passes it the payload pointer.

What happened with igc is that the Lisp_TS_Parser object was moved in
memory, which is normal because MPS is a copying collector, but the
TSInput::payload pointer wasn't changed accordingly because I didn't
know that a reference was hiding there.

My question is if there is a second case like that. A pointer to a
Lisp_TS_xy or a Lisp_Object for such an object that is hiding in a place
that isn't apparent. At least for someoneo like me :-).

Re: MPS: treesit

[Yuan Fu]

> On Jun 25, 2024, at 9:54 PM, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>>> On Jun 25, 2024, at 12:44 AM, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>>> 
>>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>> 
>>>> Can somone help and explain how this works in treesit.c? Where is the
>>>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>>>> to the treesiter dylib translated to the Lisp_TS_Parser *?
>>> 
>>> Forget that question, I see it, TSInput and so on. But I'd be interested
>>> if there are other such constructs in treesit?
>>> 
>> 
>> If you mean tree-sitter struct that’s exposed to lisp, there are
>> Lisp_TS_Parser, Lisp_TS_Node, and Lisp_TS_Query. Lisp_TS_Node is a
>> node in the parser tree, Lisp_TS_Query is a lazily compiled query.
>> Tree-sitter queries, like regexp, are compiled from string to
>> tree-sitter’s internal struct before being used. HTH.
> 
> Hi Yuan!
> 
> In the TSInput case that I could fix, TSInput::payload, a void *, is set
> to point to a Lisp_TS_Parser, and the read callback in treesit.c gets
> access to that parser because tree-sitter passes it the payload pointer.
> 
> What happened with igc is that the Lisp_TS_Parser object was moved in
> memory, which is normal because MPS is a copying collector, but the
> TSInput::payload pointer wasn't changed accordingly because I didn't
> know that a reference was hiding there.
> 
> My question is if there is a second case like that. A pointer to a
> Lisp_TS_xy or a Lisp_Object for such an object that is hiding in a place
> that isn't apparent. At least for someoneo like me :-).

Ah, I see. I don’t think there’s another one.

Yuan

Re: MPS: treesit

[Gerd Möllmann]

Yuan Fu <casouri@gmail.com> writes:

>> My question is if there is a second case like that. A pointer to a
>> Lisp_TS_xy or a Lisp_Object for such an object that is hiding in a place
>> that isn't apparent. At least for someoneo like me :-).
>
> Ah, I see. I don’t think there’s another one.

Very good. Thanks for checking!