Treesitter injection support

Treesitter injection support

[Pranshu Sharma via Emacs development discussions.]

I'm making cperl clone using treesitter, and have done all of
highlighting apart from regex and pod.

For regexp, I need different grammer to highlight it, and using the
treesit-parser-set-included-ranges doesn't work.  An example:

preq knowledge:

's/bi?g/small/' replaces instances of 'bg' and 'big' with 'small', and
's/([0-9]+)/$1 + 1/e' incrimental all number (the 'e' at the end tells
perl to evaluate the code).

the parse tree of 's/([0-9]+)/$1 + 1/e' is:
(substitution_regexp operator: s '
     content: (regexp_content not-interpolated not-interpolated) '
     (replacement
      (scalar $ (varname)))
     ' modifiers: (substitution_regexp_modifiers))

(replacement) needs to be conditionally parsed as perl over here because
of the 'e' modifier.  Now I cannot use range for this, because say if I
had:

's/(([0-9]+),)+/s#([0-9]+)#$1 + 1#e/e;'
                           ^^^^^^          Perl code
                ^^^^^^^^^^^^^^^^^^^        Perl code
                             

The replacement contains another replacment which contains perl code, so
it overlaps

So I won't have any way to highlight.  It seems making this work could
be possible using nested parsers with their own setting each using own
local treesit-range-settings, but this seems really hard with
treesit-range-settings being a buffer local variable.

-- 
Pranshu Sharma <https://p.bauherren.ovh>

Re: Treesitter injection support

[Yuan Fu]

> On Jan 2, 2025, at 6:48 AM, Pranshu Sharma <pranshu@bauherren.ovh> wrote:
> 
> 
> I'm making cperl clone using treesitter, and have done all of
> highlighting apart from regex and pod.
> 
> For regexp, I need different grammer to highlight it, and using the
> treesit-parser-set-included-ranges doesn't work.  An example:
> 
> preq knowledge:
> 
> 's/bi?g/small/' replaces instances of 'bg' and 'big' with 'small', and
> 's/([0-9]+)/$1 + 1/e' incrimental all number (the 'e' at the end tells
> perl to evaluate the code).
> 
> the parse tree of 's/([0-9]+)/$1 + 1/e' is:
> (substitution_regexp operator: s '
>     content: (regexp_content not-interpolated not-interpolated) '
>     (replacement
>      (scalar $ (varname)))
>     ' modifiers: (substitution_regexp_modifiers))
> 
> (replacement) needs to be conditionally parsed as perl over here because
> of the 'e' modifier.  Now I cannot use range for this, because say if I
> had:
> 
> 's/(([0-9]+),)+/s#([0-9]+)#$1 + 1#e/e;'
>                           ^^^^^^          Perl code
>                ^^^^^^^^^^^^^^^^^^^        Perl code
> 
> 
> The replacement contains another replacment which contains perl code, so
> it overlaps
> 
> So I won't have any way to highlight.  It seems making this work could
> be possible using nested parsers with their own setting each using own
> local treesit-range-settings, but this seems really hard with
> treesit-range-settings being a buffer local variable.
> 
> -- 
> Pranshu Sharma <https://p.bauherren.ovh>

Ok, so the problem is nested parsers. I don’t think the overlap would cause any problem. Right now treesit-range-settings can only give you one nested layer. I’ll need to make it support nesting a parser inside a local parser of the same language. I’ll work on that once I wrap up the thing I’m working on right now :-)

Yuan

Re: Treesitter injection support

[Pranshu Sharma via Emacs development discussions.]

Yuan Fu <casouri@gmail.com> writes:

>> On Jan 2, 2025, at 6:48 AM, Pranshu Sharma <pranshu@bauherren.ovh> wrote:
>> 
>> 
>> I'm making cperl clone using treesitter, and have done all of
>> highlighting apart from regex and pod.
>> 
>> For regexp, I need different grammer to highlight it, and using the
>> treesit-parser-set-included-ranges doesn't work.  An example:
>> 
>> preq knowledge:
>> 
>> 's/bi?g/small/' replaces instances of 'bg' and 'big' with 'small', and
>> 's/([0-9]+)/$1 + 1/e' incrimental all number (the 'e' at the end tells
>> perl to evaluate the code).
>> 
>> the parse tree of 's/([0-9]+)/$1 + 1/e' is:
>> (substitution_regexp operator: s '
>>     content: (regexp_content not-interpolated not-interpolated) '
>>     (replacement
>>      (scalar $ (varname)))
>>     ' modifiers: (substitution_regexp_modifiers))
>> 
>> (replacement) needs to be conditionally parsed as perl over here because
>> of the 'e' modifier.  Now I cannot use range for this, because say if I
>> had:
>> 
>> 's/(([0-9]+),)+/s#([0-9]+)#$1 + 1#e/e;'
>>                           ^^^^^^          Perl code
>>                ^^^^^^^^^^^^^^^^^^^        Perl code
>> 
>> 
>> The replacement contains another replacment which contains perl code, so
>> it overlaps
>> 
>> So I won't have any way to highlight.  It seems making this work could
>> be possible using nested parsers with their own setting each using own
>> local treesit-range-settings, but this seems really hard with
>> treesit-range-settings being a buffer local variable.
>> 
>
> Ok, so the problem is nested parsers. I don’t think the overlap would
> cause any problem. Right now treesit-range-settings can only give you
> one nested layer. I’ll need to make it support nesting a parser inside
> a local parser of the same language. I’ll work on that once I wrap up
> the thing I’m working on right now :-)

Thanks, this definetly seems like the problem.  Also the
treesit-range-settings seems kind of unstable, example when I purposly
leave closed string before it, and close the string, it doesn't reparse.

-- 
Pranshu Sharma <https://p.bauherren.ovh>

Re: Treesitter injection support

[Yuan Fu]

> On Jan 4, 2025, at 8:33 AM, Pranshu Sharma <pranshu@bauherren.ovh> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>>> On Jan 2, 2025, at 6:48 AM, Pranshu Sharma <pranshu@bauherren.ovh> wrote:
>>> 
>>> 
>>> I'm making cperl clone using treesitter, and have done all of
>>> highlighting apart from regex and pod.
>>> 
>>> For regexp, I need different grammer to highlight it, and using the
>>> treesit-parser-set-included-ranges doesn't work.  An example:
>>> 
>>> preq knowledge:
>>> 
>>> 's/bi?g/small/' replaces instances of 'bg' and 'big' with 'small', and
>>> 's/([0-9]+)/$1 + 1/e' incrimental all number (the 'e' at the end tells
>>> perl to evaluate the code).
>>> 
>>> the parse tree of 's/([0-9]+)/$1 + 1/e' is:
>>> (substitution_regexp operator: s '
>>>    content: (regexp_content not-interpolated not-interpolated) '
>>>    (replacement
>>>     (scalar $ (varname)))
>>>    ' modifiers: (substitution_regexp_modifiers))
>>> 
>>> (replacement) needs to be conditionally parsed as perl over here because
>>> of the 'e' modifier.  Now I cannot use range for this, because say if I
>>> had:
>>> 
>>> 's/(([0-9]+),)+/s#([0-9]+)#$1 + 1#e/e;'
>>>                          ^^^^^^          Perl code
>>>               ^^^^^^^^^^^^^^^^^^^        Perl code
>>> 
>>> 
>>> The replacement contains another replacment which contains perl code, so
>>> it overlaps
>>> 
>>> So I won't have any way to highlight.  It seems making this work could
>>> be possible using nested parsers with their own setting each using own
>>> local treesit-range-settings, but this seems really hard with
>>> treesit-range-settings being a buffer local variable.
>>> 
>> 
>> Ok, so the problem is nested parsers. I don’t think the overlap would
>> cause any problem. Right now treesit-range-settings can only give you
>> one nested layer. I’ll need to make it support nesting a parser inside
>> a local parser of the same language. I’ll work on that once I wrap up
>> the thing I’m working on right now :-)
> 
> Thanks, this definetly seems like the problem.  Also the
> treesit-range-settings seems kind of unstable, example when I purposly
> leave closed string before it, and close the string, it doesn't reparse.
> 
> -- 
> Pranshu Sharma <https://p.bauherren.ovh>

Can you show me a concrete example (reproduce recipe)? I can look into it.

Yuan