* MPS: treesit
@ 2024-06-25 5:27 Gerd Möllmann
2024-06-25 7:44 ` Gerd Möllmann
0 siblings, 1 reply; 9+ messages in thread
From: Gerd Möllmann @ 2024-06-25 5:27 UTC (permalink / raw)
To: Emacs Devel; +Cc: Eli Zaretskii, Helmut Eller
I'm trying to make treesit work with igc, and I'm using c-ts-mode for
that purposes. Or I'm trying to because in a full
debug/checking/mps-debug build I immediately get a crash. I'm BTW
looking at/using treesit for the first time, so please bear with me.
treesit_ensure_parsed gets called, and passes a TSParser * to
ts_parser_parse.
static void
treesit_ensure_parsed (Lisp_Object parser)
{
struct buffer *buffer = XBUFFER (XTS_PARSER (parser)->buffer);
/* Before we parse, catch up with the narrowing situation. */
treesit_check_buffer_size (buffer);
/* This function has to run before we check for need_reparse flag,
because it might set the flag to true. */
treesit_sync_visible_region (parser);
/* Make sure this comes before everything else, see comment
(ref:notifier-inside-ensure-parsed) for more detail. */
if (!XTS_PARSER (parser)->need_reparse)
return;
TSParser *treesit_parser = XTS_PARSER (parser)->parser;
TSTree *tree = XTS_PARSER (parser)->tree;
TSInput input = XTS_PARSER (parser)->input;
TSTree *new_tree = ts_parser_parse (treesit_parser, tree, input);
(lldb) p treesit_parser
(TSParser *) 0x0000000154838e00
Down the stack I see some functions being called in the treesitter dylib
frame #5: 0x0000000101464e58 libtree-sitter.0.dylib`ts_parser_parse + 2928
frame #4: 0x000000010146160c libtree-sitter.0.dylib`ts_lexer_start + 84
frame #3: 0x000000010146167c libtree-sitter.0.dylib`ts_lexer__get_chunk + 40
frame #2: 0x00000001003ad184 emacs`treesit_read_buffer(parser=0x0000000105049ea0, byte_index=0, position=(row = 0, column = 0), bytes_read=0x0000000154838e9c) at treesit.c:1111:27
until I land here:
static const char*
treesit_read_buffer (void *parser, uint32_t byte_index,
TSPoint position, uint32_t *bytes_read)
{
struct buffer *buffer = XBUFFER (((struct Lisp_TS_Parser *) parser)->buffer);
Here, XBUFFER aborts with an invalid buffer because of --enable-checking=all.
(lldb) p parser
(void *) 0x0000000105049ea0
Note that parser here is not the treesit_parser some frames up that was
passed to the treesitter lib.
Can somone help and explain how this works in treesit.c? Where is the
Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
to the treesiter dylib translated to the Lisp_TS_Parser *?
TIA
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: MPS: treesit
2024-06-25 5:27 MPS: treesit Gerd Möllmann
@ 2024-06-25 7:44 ` Gerd Möllmann
2024-06-25 8:28 ` Gerd Möllmann
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Gerd Möllmann @ 2024-06-25 7:44 UTC (permalink / raw)
To: Emacs Devel; +Cc: Eli Zaretskii, Helmut Eller
Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> Can somone help and explain how this works in treesit.c? Where is the
> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
> to the treesiter dylib translated to the Lisp_TS_Parser *?
Forget that question, I see it, TSInput and so on. But I'd be interested
if there are other such constructs in treesit?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: MPS: treesit
2024-06-25 7:44 ` Gerd Möllmann
@ 2024-06-25 8:28 ` Gerd Möllmann
2024-06-25 15:23 ` Eli Zaretskii
2024-06-26 4:35 ` Yuan Fu
2 siblings, 0 replies; 9+ messages in thread
From: Gerd Möllmann @ 2024-06-25 8:28 UTC (permalink / raw)
To: Emacs Devel; +Cc: Eli Zaretskii, Helmut Eller
Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Can somone help and explain how this works in treesit.c? Where is the
>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>> to the treesiter dylib translated to the Lisp_TS_Parser *?
>
> Forget that question, I see it, TSInput and so on. But I'd be interested
> if there are other such constructs in treesit?
Pushed a fix to scratch/igc, but the last question would still be
interesting, just in case. It's likely that I don't use all of
what treesit does because I have no idea what it offers.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: MPS: treesit
2024-06-25 7:44 ` Gerd Möllmann
2024-06-25 8:28 ` Gerd Möllmann
@ 2024-06-25 15:23 ` Eli Zaretskii
2024-06-26 4:35 ` Yuan Fu
2024-06-26 4:35 ` Yuan Fu
2 siblings, 1 reply; 9+ messages in thread
From: Eli Zaretskii @ 2024-06-25 15:23 UTC (permalink / raw)
To: Gerd Möllmann, Yuan Fu; +Cc: emacs-devel, eller.helmut
> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, Helmut Eller <eller.helmut@gmail.com>
> Date: Tue, 25 Jun 2024 09:44:34 +0200
>
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> > Can somone help and explain how this works in treesit.c? Where is the
> > Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
> > to the treesiter dylib translated to the Lisp_TS_Parser *?
>
> Forget that question, I see it, TSInput and so on. But I'd be interested
> if there are other such constructs in treesit?
Yuan, can you help Gerd with understanding this?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: MPS: treesit
2024-06-25 15:23 ` Eli Zaretskii
@ 2024-06-26 4:35 ` Yuan Fu
0 siblings, 0 replies; 9+ messages in thread
From: Yuan Fu @ 2024-06-26 4:35 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Gerd Möllmann, emacs-devel, eller.helmut
> On Jun 25, 2024, at 8:23 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, Helmut Eller <eller.helmut@gmail.com>
>> Date: Tue, 25 Jun 2024 09:44:34 +0200
>>
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>>> Can somone help and explain how this works in treesit.c? Where is the
>>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>>> to the treesiter dylib translated to the Lisp_TS_Parser *?
>>
>> Forget that question, I see it, TSInput and so on. But I'd be interested
>> if there are other such constructs in treesit?
>
> Yuan, can you help Gerd with understanding this?
Of course 😊
Yuan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: MPS: treesit
2024-06-25 7:44 ` Gerd Möllmann
2024-06-25 8:28 ` Gerd Möllmann
2024-06-25 15:23 ` Eli Zaretskii
@ 2024-06-26 4:35 ` Yuan Fu
2024-06-26 4:54 ` Gerd Möllmann
2 siblings, 1 reply; 9+ messages in thread
From: Yuan Fu @ 2024-06-26 4:35 UTC (permalink / raw)
To: Gerd Möllmann; +Cc: Emacs Devel, Eli Zaretskii, Helmut Eller
> On Jun 25, 2024, at 12:44 AM, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Can somone help and explain how this works in treesit.c? Where is the
>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>> to the treesiter dylib translated to the Lisp_TS_Parser *?
>
> Forget that question, I see it, TSInput and so on. But I'd be interested
> if there are other such constructs in treesit?
>
If you mean tree-sitter struct that’s exposed to lisp, there are Lisp_TS_Parser, Lisp_TS_Node, and Lisp_TS_Query. Lisp_TS_Node is a node in the parser tree, Lisp_TS_Query is a lazily compiled query. Tree-sitter queries, like regexp, are compiled from string to tree-sitter’s internal struct before being used. HTH.
Yuan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: MPS: treesit
2024-06-26 4:35 ` Yuan Fu
@ 2024-06-26 4:54 ` Gerd Möllmann
2024-06-26 5:18 ` Yuan Fu
0 siblings, 1 reply; 9+ messages in thread
From: Gerd Möllmann @ 2024-06-26 4:54 UTC (permalink / raw)
To: Yuan Fu; +Cc: Emacs Devel, Eli Zaretskii, Helmut Eller
Yuan Fu <casouri@gmail.com> writes:
>> On Jun 25, 2024, at 12:44 AM, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>>
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>>> Can somone help and explain how this works in treesit.c? Where is the
>>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>>> to the treesiter dylib translated to the Lisp_TS_Parser *?
>>
>> Forget that question, I see it, TSInput and so on. But I'd be interested
>> if there are other such constructs in treesit?
>>
>
> If you mean tree-sitter struct that’s exposed to lisp, there are
> Lisp_TS_Parser, Lisp_TS_Node, and Lisp_TS_Query. Lisp_TS_Node is a
> node in the parser tree, Lisp_TS_Query is a lazily compiled query.
> Tree-sitter queries, like regexp, are compiled from string to
> tree-sitter’s internal struct before being used. HTH.
Hi Yuan!
In the TSInput case that I could fix, TSInput::payload, a void *, is set
to point to a Lisp_TS_Parser, and the read callback in treesit.c gets
access to that parser because tree-sitter passes it the payload pointer.
What happened with igc is that the Lisp_TS_Parser object was moved in
memory, which is normal because MPS is a copying collector, but the
TSInput::payload pointer wasn't changed accordingly because I didn't
know that a reference was hiding there.
My question is if there is a second case like that. A pointer to a
Lisp_TS_xy or a Lisp_Object for such an object that is hiding in a place
that isn't apparent. At least for someoneo like me :-).
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: MPS: treesit
2024-06-26 4:54 ` Gerd Möllmann
@ 2024-06-26 5:18 ` Yuan Fu
2024-06-26 5:25 ` Gerd Möllmann
0 siblings, 1 reply; 9+ messages in thread
From: Yuan Fu @ 2024-06-26 5:18 UTC (permalink / raw)
To: Gerd Möllmann; +Cc: Emacs Devel, Eli Zaretskii, Helmut Eller
> On Jun 25, 2024, at 9:54 PM, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>
> Yuan Fu <casouri@gmail.com> writes:
>
>>> On Jun 25, 2024, at 12:44 AM, Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>>>
>>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>>
>>>> Can somone help and explain how this works in treesit.c? Where is the
>>>> Lisp_TS_Parser stored and how/where is the TSParser * that gets passed
>>>> to the treesiter dylib translated to the Lisp_TS_Parser *?
>>>
>>> Forget that question, I see it, TSInput and so on. But I'd be interested
>>> if there are other such constructs in treesit?
>>>
>>
>> If you mean tree-sitter struct that’s exposed to lisp, there are
>> Lisp_TS_Parser, Lisp_TS_Node, and Lisp_TS_Query. Lisp_TS_Node is a
>> node in the parser tree, Lisp_TS_Query is a lazily compiled query.
>> Tree-sitter queries, like regexp, are compiled from string to
>> tree-sitter’s internal struct before being used. HTH.
>
> Hi Yuan!
>
> In the TSInput case that I could fix, TSInput::payload, a void *, is set
> to point to a Lisp_TS_Parser, and the read callback in treesit.c gets
> access to that parser because tree-sitter passes it the payload pointer.
>
> What happened with igc is that the Lisp_TS_Parser object was moved in
> memory, which is normal because MPS is a copying collector, but the
> TSInput::payload pointer wasn't changed accordingly because I didn't
> know that a reference was hiding there.
>
> My question is if there is a second case like that. A pointer to a
> Lisp_TS_xy or a Lisp_Object for such an object that is hiding in a place
> that isn't apparent. At least for someoneo like me :-).
Ah, I see. I don’t think there’s another one.
Yuan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: MPS: treesit
2024-06-26 5:18 ` Yuan Fu
@ 2024-06-26 5:25 ` Gerd Möllmann
0 siblings, 0 replies; 9+ messages in thread
From: Gerd Möllmann @ 2024-06-26 5:25 UTC (permalink / raw)
To: Yuan Fu; +Cc: Emacs Devel, Eli Zaretskii, Helmut Eller
Yuan Fu <casouri@gmail.com> writes:
>> My question is if there is a second case like that. A pointer to a
>> Lisp_TS_xy or a Lisp_Object for such an object that is hiding in a place
>> that isn't apparent. At least for someoneo like me :-).
>
> Ah, I see. I don’t think there’s another one.
Very good. Thanks for checking!
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2024-06-26 5:25 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-06-25 5:27 MPS: treesit Gerd Möllmann
2024-06-25 7:44 ` Gerd Möllmann
2024-06-25 8:28 ` Gerd Möllmann
2024-06-25 15:23 ` Eli Zaretskii
2024-06-26 4:35 ` Yuan Fu
2024-06-26 4:35 ` Yuan Fu
2024-06-26 4:54 ` Gerd Möllmann
2024-06-26 5:18 ` Yuan Fu
2024-06-26 5:25 ` Gerd Möllmann
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.