A few questions about c++-ts-mode.

A few questions about c++-ts-mode.

[Ergus]

Hi

There are a few differences between c++-ts-mode and c++-mode that I
don't know if should be reported or they are like that due to some
design choice.

1. namespaces are not fontified like in c-mode (i.e in std::string the
   `std` used to have font-lock-constant-face while `string` used to have
   font-lock-type-face). It seems like tresitter identifies properly them
   like (namespace_identifier) and (type_identifier)

2. Doc string comments are fontified like normal comments. It looks like
   treesiter does not recognize them at all. Is this something we need to
   report to treesitter or we need to workaround it in emacs side?

3. macros like `#if defined(something)` fontifies the whole macro with
   same font (the `identifier` used to have no colors in c++-mode)


Somehow related:

Now that there is treesitter, eglot, project.el, improved xref and
completions + flymake... does anyone still use CEDET? If not... is there
some plan to remove it in the future? There is few documentation around
mentioning it and created confusion in new users...

Thanks in advance,
Ergus

Re: A few questions about c++-ts-mode.

[Yuan Fu]

> On May 7, 2024, at 3:59 PM, Ergus <spacibba@aol.com> wrote:
> 
> Hi
> 
> There are a few differences between c++-ts-mode and c++-mode that I
> don't know if should be reported or they are like that due to some
> design choice.

AFAIK, the design goal of c++-ts-mode is to follow c++-mode whenever possible.

> 
> 1. namespaces are not fontified like in c-mode (i.e in std::string the
>  `std` used to have font-lock-constant-face while `string` used to have
>  font-lock-type-face). It seems like tresitter identifies properly them
>  like (namespace_identifier) and (type_identifier)

Then we probably want to fontify namespaces in constant face. Or perhaps add a namespace-face that inherits from constant face.

> 2. Doc string comments are fontified like normal comments. It looks like
>  treesiter does not recognize them at all. Is this something we need to
>  report to treesitter or we need to workaround it in emacs side?

I’m working on that. I think the best way is to workaround it in Emacs.

> 3. macros like `#if defined(something)` fontifies the whole macro with
>  same font (the `identifier` used to have no colors in c++-mode)

We should update the fontification to leave out the identifier.

Yuan

Re: A few questions about c++-ts-mode.

[Ergus]

On Tue, May 07, 2024 at 05:19:53PM GMT, Yuan Fu wrote:
>
>
>> On May 7, 2024, at 3:59 PM, Ergus <spacibba@aol.com> wrote:
>>
>> Hi
>>
>> There are a few differences between c++-ts-mode and c++-mode that I
>> don't know if should be reported or they are like that due to some
>> design choice.
>
>AFAIK, the design goal of c++-ts-mode is to follow c++-mode whenever possible.
>
>>
>> 1. namespaces are not fontified like in c-mode (i.e in std::string the
>>  `std` used to have font-lock-constant-face while `string` used to have
>>  font-lock-type-face). It seems like tresitter identifies properly them
>>  like (namespace_identifier) and (type_identifier)
>
>Then we probably want to fontify namespaces in constant face. Or perhaps add a namespace-face that inherits from constant face.
>
>> 2. Doc string comments are fontified like normal comments. It looks like
>>  treesiter does not recognize them at all. Is this something we need to
>>  report to treesitter or we need to workaround it in emacs side?
>
>I’m working on that. I think the best way is to workaround it in Emacs.
>
>> 3. macros like `#if defined(something)` fontifies the whole macro with
>>  same font (the `identifier` used to have no colors in c++-mode)
>
>We should update the fontification to leave out the identifier.
>
>Yuan

Hi Yuan:

Forgot to mention that

4. enum and enum class indentation is not working either

struct send {
      |
};

class send {
      |
};

enum send {
|
};

enum class send {
      |
};

Re: A few questions about c++-ts-mode.

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Tue, 7 May 2024 17:19:53 -0700
> Cc: emacs-devel@gnu.org
> 
> > 1. namespaces are not fontified like in c-mode (i.e in std::string the
> >  `std` used to have font-lock-constant-face while `string` used to have
> >  font-lock-type-face). It seems like tresitter identifies properly them
> >  like (namespace_identifier) and (type_identifier)
> 
> Then we probably want to fontify namespaces in constant face. Or perhaps add a namespace-face that inherits from constant face.

I think using font-lock-constant-face is fine.  Patches welcome.

> > 2. Doc string comments are fontified like normal comments. It looks like
> >  treesiter does not recognize them at all. Is this something we need to
> >  report to treesitter or we need to workaround it in emacs side?
> 
> I’m working on that. I think the best way is to workaround it in Emacs.

Please note that some of those specially-formatted comments are
explicitly supported by the grammar.

> > 3. macros like `#if defined(something)` fontifies the whole macro with
> >  same font (the `identifier` used to have no colors in c++-mode)
> 
> We should update the fontification to leave out the identifier.

We seem to have the technology already, but we are overwriting it?
Observe:

     (preproc_defined) @font-lock-preprocessor-face
     (preproc_defined (identifier) @font-lock-variable-name-face)
     [,@c-ts-mode--preproc-keywords] @font-lock-preprocessor-face)

Re: A few questions about c++-ts-mode.

[Ergus]

Hi Eli and Yuan

On May 8, 2024 3:35:52 PM GMT+02:00, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Tue, 7 May 2024 17:19:53 -0700
>> Cc: emacs-devel@gnu.org
>> 
>> > 1. namespaces are not fontified like in c-mode (i.e in std::string the
>> >  `std` used to have font-lock-constant-face while `string` used to have
>> >  font-lock-type-face). It seems like tresitter identifies properly them
>> >  like (namespace_identifier) and (type_identifier)
>> 
>> Then we probably want to fontify namespaces in constant face. Or perhaps add a namespace-face that inherits from constant face.
>
>I think using font-lock-constant-face is fine.  Patches welcome.
>
>> > 2. Doc string comments are fontified like normal comments. It looks like
>> >  treesiter does not recognize them at all. Is this something we need to
>> >  report to treesitter or we need to workaround it in emacs side?
>> 
>> I’m working on that. I think the best way is to workaround it in Emacs.
>
>Please note that some of those specially-formatted comments are
>explicitly supported by the grammar.
>

It looks like tree sitter already supports this using language injection:

<https://github.com/tree-sitter/tree-sitter-cpp/issues/116>

I am not sure if we can handle this same way in emacs. Or if it is a better or worst approach than using a made in home workaround.


>> > 3. macros like `#if defined(something)` fontifies the whole macro with
>> >  same font (the `identifier` used to have no colors in c++-mode)
>> 
>> We should update the fontification to leave out the identifier.
>
>We seem to have the technology already, but we are overwriting it?
>Observe:
>
>     (preproc_defined) @font-lock-preprocessor-face
>     (preproc_defined (identifier) @font-lock-variable-name-face)
>     [,@c-ts-mode--preproc-keywords] @font-lock-preprocessor-face)
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: A few questions about c++-ts-mode.

[Yuan Fu]

> On May 8, 2024, at 2:05 PM, Ergus <spacibba@aol.com> wrote:
> 
> Hi Eli and Yuan
> 
> On May 8, 2024 3:35:52 PM GMT+02:00, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Yuan Fu <casouri@gmail.com>
>>> Date: Tue, 7 May 2024 17:19:53 -0700
>>> Cc: emacs-devel@gnu.org
>>> 
>>>> 1. namespaces are not fontified like in c-mode (i.e in std::string the
>>>> `std` used to have font-lock-constant-face while `string` used to have
>>>> font-lock-type-face). It seems like tresitter identifies properly them
>>>> like (namespace_identifier) and (type_identifier)
>>> 
>>> Then we probably want to fontify namespaces in constant face. Or perhaps add a namespace-face that inherits from constant face.
>> 
>> I think using font-lock-constant-face is fine.  Patches welcome.
>> 
>>>> 2. Doc string comments are fontified like normal comments. It looks like
>>>> treesiter does not recognize them at all. Is this something we need to
>>>> report to treesitter or we need to workaround it in emacs side?
>>> 
>>> I’m working on that. I think the best way is to workaround it in Emacs.
>> 
>> Please note that some of those specially-formatted comments are
>> explicitly supported by the grammar.
>> 
> 
> It looks like tree sitter already supports this using language injection:
> 
> <https://github.com/tree-sitter/tree-sitter-cpp/issues/116>
> 
> I am not sure if we can handle this same way in emacs. Or if it is a better or worst approach than using a made in home workaround.

Emacs supports language injection (in the name of embedded parser), see 38.7 Parsing Text in Multiple Languages in elisp manual.

The workaround we were talking about is just fontifynig the /** comments with font-lock-doc-face, nothing fancy. For more elaborate effect you’d need doxygen grammar. 

Yuan

Re: A few questions about c++-ts-mode.

[Yuan Fu]

> On May 7, 2024, at 6:26 PM, Ergus <spacibba@aol.com> wrote:
> 
> On Tue, May 07, 2024 at 05:19:53PM GMT, Yuan Fu wrote:
>> 
>> 
>>> On May 7, 2024, at 3:59 PM, Ergus <spacibba@aol.com> wrote:
>>> 
>>> Hi
>>> 
>>> There are a few differences between c++-ts-mode and c++-mode that I
>>> don't know if should be reported or they are like that due to some
>>> design choice.
>> 
>> AFAIK, the design goal of c++-ts-mode is to follow c++-mode whenever possible.
>> 
>>> 
>>> 1. namespaces are not fontified like in c-mode (i.e in std::string the
>>> `std` used to have font-lock-constant-face while `string` used to have
>>> font-lock-type-face). It seems like tresitter identifies properly them
>>> like (namespace_identifier) and (type_identifier)
>> 
>> Then we probably want to fontify namespaces in constant face. Or perhaps add a namespace-face that inherits from constant face.
>> 
>>> 2. Doc string comments are fontified like normal comments. It looks like
>>> treesiter does not recognize them at all. Is this something we need to
>>> report to treesitter or we need to workaround it in emacs side?
>> 
>> I’m working on that. I think the best way is to workaround it in Emacs.
>> 
>>> 3. macros like `#if defined(something)` fontifies the whole macro with
>>> same font (the `identifier` used to have no colors in c++-mode)
>> 
>> We should update the fontification to leave out the identifier.
>> 
>> Yuan
> 
> Hi Yuan:
> 
> Forgot to mention that
> 
> 4. enum and enum class indentation is not working either
> 
> struct send {
>     |
> };
> 
> class send {
>     |
> };
> 
> enum send {
> |
> };
> 
> enum class send {
>     |
> };
> 

Could you open a bug report for this? I’ve fixed the other since they are relatively simple.

Yuan

Re: A few questions about c++-ts-mode.

[Eli Zaretskii]

> Date: Wed, 08 May 2024 23:05:10 +0200
> From: Ergus <spacibba@aol.com>
> 
> >> > 2. Doc string comments are fontified like normal comments. It looks like
> >> >  treesiter does not recognize them at all. Is this something we need to
> >> >  report to treesitter or we need to workaround it in emacs side?
> >> 
> >> I’m working on that. I think the best way is to workaround it in Emacs.
> >
> >Please note that some of those specially-formatted comments are
> >explicitly supported by the grammar.
> >
> 
> It looks like tree sitter already supports this using language injection:
> 
> <https://github.com/tree-sitter/tree-sitter-cpp/issues/116>
> 
> I am not sure if we can handle this same way in emacs. Or if it is a better or worst approach than using a made in home workaround.

If the only solution is language injection, then doing that our usual
way (with regexps etc.) is easier and more flexible.  I thought I saw
some commits in the C++ grammar that actually support that in the
grammar itself.  If there is such support, we should use it, IMO.

Re: A few questions about c++-ts-mode.

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Wed, 8 May 2024 17:16:11 -0700
> Cc: "Ergus via Emacs development discussions." <emacs-devel@gnu.org>,
>  Eli Zaretskii <eliz@gnu.org>
> 
> The workaround we were talking about is just fontifynig the /** comments with font-lock-doc-face, nothing fancy.

I think this should be a good-enough solution for now.

Re: A few questions about c++-ts-mode.

[Filippo Argiolas]

> > It looks like tree sitter already supports this using language injection:
> >
> > <https://github.com/tree-sitter/tree-sitter-cpp/issues/116>
> >
> > I am not sure if we can handle this same way in emacs. Or if it is a better or worst approach than using a made in home workaround.
>
> If the only solution is language injection, then doing that our usual
> way (with regexps etc.) is easier and more flexible.  I thought I saw
> some commits in the C++ grammar that actually support that in the
> grammar itself.  If there is such support, we should use it, IMO.
>

Sorry to intrude, may I ask what's the problem with language
injection? it seems the perfect use case for this.
According to Yuan this should already be supported with emacs (at
least the parser part, you would probably need to add some specific
rule for fontification).
Yuan, a complete example would be great!

Another use case in c-ts-mode would be fontifying format strings in
printf functions. There was some discussion about doing that in the Go
grammar and there too they concluded injections was the canonical
treesitter way for stuff like this and dropped the pull request.

Thanks,
Filippo

Re: A few questions about c++-ts-mode.

[Eli Zaretskii]

> From: Filippo Argiolas <filippo.argiolas@gmail.com>
> Date: Thu, 9 May 2024 07:34:58 +0200
> Cc: Ergus <spacibba@aol.com>, emacs-devel@gnu.org, casouri@gmail.com
> 
> > If the only solution is language injection, then doing that our usual
> > way (with regexps etc.) is easier and more flexible.  I thought I saw
> > some commits in the C++ grammar that actually support that in the
> > grammar itself.  If there is such support, we should use it, IMO.
> >
> 
> Sorry to intrude, may I ask what's the problem with language
> injection? it seems the perfect use case for this.

I didn't say there was a problem, I said that doing it with regexp
etc. would be easier and more flexible.

Language injection AFAIU requires an additional grammar library, which
means someone must compile it, the end-user must ensure it's
installed, etc.  IOW, it makes the disadvantages of tree-sitter more
prominent, for the benefit of a very minor feature.  From where I
stand, that makes the balance tip in favor of the simpler solution,
which users could more easily activate and customize.

Re: A few questions about c++-ts-mode.

[Filippo Argiolas]

On Thu, May 9, 2024 at 7:53 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Filippo Argiolas <filippo.argiolas@gmail.com>
> > Date: Thu, 9 May 2024 07:34:58 +0200
> > Cc: Ergus <spacibba@aol.com>, emacs-devel@gnu.org, casouri@gmail.com
> >
> > > If the only solution is language injection, then doing that our usual
> > > way (with regexps etc.) is easier and more flexible.  I thought I saw
> > > some commits in the C++ grammar that actually support that in the
> > > grammar itself.  If there is such support, we should use it, IMO.
> > >
> >
> > Sorry to intrude, may I ask what's the problem with language
> > injection? it seems the perfect use case for this.
>
> I didn't say there was a problem, I said that doing it with regexp
> etc. would be easier and more flexible.
>
> Language injection AFAIU requires an additional grammar library, which
> means someone must compile it, the end-user must ensure it's
> installed, etc.  IOW, it makes the disadvantages of tree-sitter more
> prominent, for the benefit of a very minor feature.  From where I
> stand, that makes the balance tip in favor of the simpler solution,
> which users could more easily activate and customize.

Thanks for the clarification!
I thought there was some intrinsic limitation either in the injection
feature or in emacs support for it.
Agree that requiring an extra grammar just for highlighting a couple
of @keywords would be too cumbersome.
OTOH from what I can tell injections seem popular in the nvim world so
that must not be that cumbersome for the users.

By the way, about printf format strings I was experimenting a bit here
[1] but refrained from proposing it upstream with the assumption that
it would be refused like for the tree-sitter-go pull request[2].

1. https://github.com/tree-sitter/tree-sitter-c/commit/fecec922c21b79ef83851f5028910bc316fcd8e1
2. https://github.com/tree-sitter/tree-sitter-go/pull/88

Re: A few questions about c++-ts-mode.

[Filippo Argiolas]

On Thu, May 9, 2024 at 6:30 AM Yuan Fu <casouri@gmail.com> wrote:
> Could you open a bug report for this? I’ve fixed the other since they are relatively simple.

I saw the recent commits, as far as I know doxygen supports also other
comment styles besides "/**".
At my workplace e.g. they use the Qt style "/*!" and "///" and "//!"
are also listed here: https://www.doxygen.nl/manual/docblocks.html
It would be nice to support the other variants too.

Re: A few questions about c++-ts-mode.

[Eli Zaretskii]

> From: Filippo Argiolas <filippo.argiolas@gmail.com>
> Date: Thu, 9 May 2024 08:11:53 +0200
> Cc: spacibba@aol.com, emacs-devel@gnu.org, casouri@gmail.com
> 
> > Language injection AFAIU requires an additional grammar library, which
> > means someone must compile it, the end-user must ensure it's
> > installed, etc.  IOW, it makes the disadvantages of tree-sitter more
> > prominent, for the benefit of a very minor feature.  From where I
> > stand, that makes the balance tip in favor of the simpler solution,
> > which users could more easily activate and customize.
> 
> Thanks for the clarification!
> I thought there was some intrinsic limitation either in the injection
> feature or in emacs support for it.
> Agree that requiring an extra grammar just for highlighting a couple
> of @keywords would be too cumbersome.
> OTOH from what I can tell injections seem popular in the nvim world so
> that must not be that cumbersome for the users.

nvim also requires users to have the exact version of the grammar
libraries they pin to support, something that IMNSHO is harsh on
users, and therefore we don't do.

IOW, what nvim or other projects do is not necessarily what we should
do.  We need to make our own decisions.

Re: A few questions about c++-ts-mode.

[Ergus]

On Wed, May 08, 2024 at 09:29:34PM GMT, Yuan Fu wrote:
>
>
>> On May 7, 2024, at 6:26 PM, Ergus <spacibba@aol.com> wrote:
>>
>> On Tue, May 07, 2024 at 05:19:53PM GMT, Yuan Fu wrote:
>>>
>>>
>>>> On May 7, 2024, at 3:59 PM, Ergus <spacibba@aol.com> wrote:
>>>>
>>>> Hi
>>>>
>>>> There are a few differences between c++-ts-mode and c++-mode that I
>>>> don't know if should be reported or they are like that due to some
>>>> design choice.
>>>
>>> AFAIK, the design goal of c++-ts-mode is to follow c++-mode whenever possible.
>>>
>>>>
>>>> 1. namespaces are not fontified like in c-mode (i.e in std::string the
>>>> `std` used to have font-lock-constant-face while `string` used to have
>>>> font-lock-type-face). It seems like tresitter identifies properly them
>>>> like (namespace_identifier) and (type_identifier)
>>>
>>> Then we probably want to fontify namespaces in constant face. Or perhaps add a namespace-face that inherits from constant face.
>>>
>>>> 2. Doc string comments are fontified like normal comments. It looks like
>>>> treesiter does not recognize them at all. Is this something we need to
>>>> report to treesitter or we need to workaround it in emacs side?
>>>
>>> I’m working on that. I think the best way is to workaround it in Emacs.
>>>
>>>> 3. macros like `#if defined(something)` fontifies the whole macro with
>>>> same font (the `identifier` used to have no colors in c++-mode)
>>>
>>> We should update the fontification to leave out the identifier.
>>>
>>> Yuan
>>
>> Hi Yuan:
>>
>> Forgot to mention that
>>
>> 4. enum and enum class indentation is not working either
>>
>> struct send {
>>     |
>> };
>>
>> class send {
>>     |
>> };
>>
>> enum send {
>> |
>> };
>>
>> enum class send {
>>     |
>> };
>>
>
>Could you open a bug report for this? I’ve fixed the other since they are relatively simple.
>
>Yuan

Hi Yuan:

Sorry for the delay.

I saw your fixes, and it looks like 2 and 3 are fixed; very thanks.

But 1 is still there.

I will report the indentation issue, but it seems to happen only somethimes,
so I'm looking for a pattern to reproduce it.

Best,
Ergus

Re: A few questions about c++-ts-mode.

[Filippo Argiolas]

On Thu, May 9, 2024 at 7:53 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Filippo Argiolas <filippo.argiolas@gmail.com>
> > Date: Thu, 9 May 2024 07:34:58 +0200
> > Cc: Ergus <spacibba@aol.com>, emacs-devel@gnu.org, casouri@gmail.com
> >
> > > If the only solution is language injection, then doing that our usual
> > > way (with regexps etc.) is easier and more flexible.  I thought I saw
> > > some commits in the C++ grammar that actually support that in the
> > > grammar itself.  If there is such support, we should use it, IMO.
> > >
> >
> > Sorry to intrude, may I ask what's the problem with language
> > injection? it seems the perfect use case for this.
>
> I didn't say there was a problem, I said that doing it with regexp
> etc. would be easier and more flexible.
>
> Language injection AFAIU requires an additional grammar library, which
> means someone must compile it, the end-user must ensure it's
> installed, etc.  IOW, it makes the disadvantages of tree-sitter more
> prominent, for the benefit of a very minor feature.  From where I
> stand, that makes the balance tip in favor of the simpler solution,
> which users could more easily activate and customize.

I know you disagree with this path, and I tend to agree, but a little
experimenting I guess won't hurt.
I did some experiment to see how really easy and flexible this would
be with current code and language injections.
It's pretty basic but I must say I'm pretty pleased about how easy
this was. Tree-sitter is really powerful and nice to work with.
Still not happy about the faces I'm using but it's a start.
Grammars are from [1] for doxygen and [2] for printf format strings.

(defvar fa/ts-font-lock-settings
  (treesit-font-lock-rules
   :language 'printf
   :feature 'format
   :override 'prepend
   '((format) @font-lock-constant-face)
   :language 'doxygen
   :feature 'doxy
   :override 'prepend
   '((tag_name) @font-lock-doc-markup-face
     (brief_header) @font-lock-doc-face
     (storageclass) @font-lock-keyword-face
     (type) @font-lock-type-face
     (identifier) @font-lock-variable-name-face
     (function_link) @font-lock-function-name-face
     (link) @link
     (emphasis) @bold
     (code_block (code_block_content) @default)
     (code_word (code) @default)
     ((tag_name) @bold (:match ".note" @bold))
     ((tag_name) @warning (:match ".warning" @warning))
     ((tag_name) @error (:match ".error" @error)))
))

(add-hook 'c-ts-mode-hook
          (lambda () (setq-local treesit-font-lock-settings
                 (append treesit-font-lock-settings
                     fa/ts-font-lock-settings))))
(defvar fa/ts-range-settings
  (treesit-range-rules
   :embed 'printf
   :local t
   :host 'c
   '((call_expression (argument_list (string_literal (string_content)
@capture))))
   :embed 'doxygen
   :local t
   :host 'c
   '(((comment) @capture (:match "/\\*[\\*!]" @capture)))))

(add-hook 'c-ts-mode-hook
          (lambda ()
            (setq-local treesit-range-settings (append treesit-range-settings
                                                       fa/ts-range-settings))))

1. https://github.com/tree-sitter-grammars/tree-sitter-doxygen
2. https://github.com/pstuifzand/tree-sitter-printf

Re: A few questions about c++-ts-mode.

[Filippo Argiolas]

On Mon, May 13, 2024 at 8:23 AM Filippo Argiolas
<filippo.argiolas@gmail.com> wrote:
>    '(((comment) @capture (:match "/\\*[\\*!]" @capture)))))

Just noticed this is a bit buggy, something like this should be more robust:
   `(((comment) @capture (:match ,(rx bos (or "/**" "/*!")) @capture)))))