Help sought from C++ expert: is this `value' a type or something else (what)?

Help sought from C++ expert: is this `value' a type or something else (what)?

[Alan Mackenzie]

Hello, Emacs.

In the following test file, templates-9.cc from the CC Mode test suite,
on the indicated line, what is the syntactic significance of `value'?
Traditionally, it has fontified with the default face, but recently after
some (not yet committed) changes in CC Mode, it has started getting
font-lock-type-face.

I don't understand the file at all, hardly, and can't work out whether
the old (non-) fontification was correct, or whether the new
fontification of f-l-type-face is correct.  Help would be appreciated.

Just as a matter of interest, in c++-ts-mode a treesit-query-error gets
thrown.  It isn't clear to me whether the file is still valid C++.

/////////////////////////////////////////////////////////////////////////
typedef std::pair<int
		  , std::pair<long
			      , std::pair<char*
					  , std::pair<nil,nil>
					 >
			     >
		 > list_of_types;

typedef std::pair<
    int, std::pair<
	long, std::pair<
	    char*, std::pair<nil,nil>
	>
    >
> list_of_types;

typedef typename if_true
<
    boost::is_same
    <
	boost::add_pointer<X>
	, int*
    >::value     // <===================================================
>::template then
<
    boost::remove_reference<X>
    // else
    , X
>::type modified_X;
/////////////////////////////////////////////////////////////////////////

Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Help sought from C++ expert: is this `value' a type or something else (what)?

[Eli Zaretskii]

> Date: Thu, 19 Jan 2023 14:53:22 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> Just as a matter of interest, in c++-ts-mode a treesit-query-error gets
> thrown.

I don't see any such error.  Maybe this is again your outdated grammar
library?

Difficulty building tree-sitter grammars [was: Help sought from C++ expert: ....]

[Alan Mackenzie]

Hello, Eli.

On Thu, Jan 19, 2023 at 16:59:59 +0200, Eli Zaretskii wrote:
> > Date: Thu, 19 Jan 2023 14:53:22 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > Just as a matter of interest, in c++-ts-mode a treesit-query-error gets
> > thrown.

> I don't see any such error.  Maybe this is again your outdated grammar
> library?

Heh!  Maybe you're right.  So time to get around to installing one from
github.  So I look at the Emacs-29 NEWS for instructions on how to do
this.  These instructions are glib and insufficiently helpful.

(i) The smallest problem is actually downloading the source of, e.g.,
tree-sitter-cpp.  github doesn't make it obvious how to download stuff,
and it took me over 10 minutes to find the GUI thing to activate the
download.  Then this download was a file.zip.  The instructions don't
mention that, on GNU, a package called zip is required to unzip this.

(ii) The instructions glibly say "To compile such a library, compile the
files "scanner.c" and "parser.c" (sometimes named "scanner.cc" and
"parser.cc") in the "src" subdirectory of the library's source tree
using the C or C++ compiler,...".  A Python hacker may well not even
know that the C/C++ compiler is called "gcc", far less that the
necessary command line option -c is needed to compile the two source
files.  Personally, this bit didn't present me any great difficulty.

(iii) The instructions continue with "..., then link these two files
into a shared library named "libtree-sitter-LANG.so, ...".  How,
exactly?  I've guessed that I can also do this with gcc, and need the
option -o libtree-sitter-cpp.so, but I also need some flags meaning
"link" and "build a .so" and some standard run-time library besides the
two object files.  I still haven't worked out what these are.  The
Python hacker pictured above will be totally lost here.

(iv) Then we have precise instructions on where to put the newly built
..so file.  This is good!

This process has so far taken me over an hour, which is too long for
something which should be purely routine.  It is likely to take a
typical Emacs user even longer.

I suggest that these instructions in NEWS should be enhanced with (i)
Tips on downloading stuff from github; (ii) A sample command line for
compiling the C/C++ source files; (iii) A sample command line for
linking these to the needed .so file.  Possibly two or three versions of
these would be needed for the different environments Emacs runs in.

And now, on with the tedious drudgery of working out how to link
tree-sitter-cpp.so using gcc.  :-(

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Difficulty building tree-sitter grammars [was: Help sought from C++ expert: ....]

[Dmitry Gutov]

Hi Alan,

On 19/01/2023 19:14, Alan Mackenzie wrote:
> I suggest that these instructions in NEWS should be enhanced with (i)
> Tips on downloading stuff from github; (ii) A sample command line for
> compiling the C/C++ source files; (iii) A sample command line for
> linking these to the needed .so file.  Possibly two or three versions of
> these would be needed for the different environments Emacs runs in.

Not to detract from your feedback on the documentation, but 'M-x 
treesit-install-language-grammar' even without existing presets these, 
turned out to be very helpful in my experience.

E.g., for c++ I choose 'cpp', answer yes to building manually, paste the 
github repository url when asked, press RET several times, and voila.

Re: Difficulty building tree-sitter grammars [was: Help sought from C++ expert: ....]

[Eli Zaretskii]

> Date: Thu, 19 Jan 2023 19:53:03 +0200
> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> Not to detract from your feedback on the documentation, but 'M-x 
> treesit-install-language-grammar' even without existing presets these, 
> turned out to be very helpful in my experience.

And of course, that command is mentioned in NEWS, right next to the
instructions how to compile manually.

Re: Difficulty building tree-sitter grammars [was: Help sought from C++ expert: ....]

[Dmitry Gutov]

On 19/01/2023 20:06, Eli Zaretskii wrote:
>> Date: Thu, 19 Jan 2023 19:53:03 +0200
>> Cc:emacs-devel@gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> Not to detract from your feedback on the documentation, but 'M-x
>> treesit-install-language-grammar' even without existing presets these,
>> turned out to be very helpful in my experience.
> And of course, that command is mentioned in NEWS, right next to the
> instructions how to compile manually.

I see your latest change, and it seems very wise ;-)

Re: Difficulty building tree-sitter grammars [was: Help sought from C++ expert: ....]

[Eli Zaretskii]

> Date: Thu, 19 Jan 2023 17:14:43 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Just as a matter of interest, in c++-ts-mode a treesit-query-error gets
> > > thrown.
> 
> > I don't see any such error.  Maybe this is again your outdated grammar
> > library?
> 
> Heh!  Maybe you're right.  So time to get around to installing one from
> github.  So I look at the Emacs-29 NEWS for instructions on how to do
> this.  These instructions are glib and insufficiently helpful.

Nothing is ever good enough, eh?

> (i) The smallest problem is actually downloading the source of, e.g.,
> tree-sitter-cpp.  github doesn't make it obvious how to download stuff,
> and it took me over 10 minutes to find the GUI thing to activate the
> download.  Then this download was a file.zip.  The instructions don't
> mention that, on GNU, a package called zip is required to unzip this.
> 
> (ii) The instructions glibly say "To compile such a library, compile the
> files "scanner.c" and "parser.c" (sometimes named "scanner.cc" and
> "parser.cc") in the "src" subdirectory of the library's source tree
> using the C or C++ compiler,...".  A Python hacker may well not even
> know that the C/C++ compiler is called "gcc", far less that the
> necessary command line option -c is needed to compile the two source
> files.  Personally, this bit didn't present me any great difficulty.
> 
> (iii) The instructions continue with "..., then link these two files
> into a shared library named "libtree-sitter-LANG.so, ...".  How,
> exactly?  I've guessed that I can also do this with gcc, and need the
> option -o libtree-sitter-cpp.so, but I also need some flags meaning
> "link" and "build a .so" and some standard run-time library besides the
> two object files.  I still haven't worked out what these are.  The
> Python hacker pictured above will be totally lost here.

If you don't know how to do this given what's in NEWS, then I'm sorry,
but these instructions are not for you.  You will have to wait until
someone else (your distro?) produces the grammar library and lets you
download and install it like you download and install all the image
libraries and the rest of the optional stuff for Emacs.

I'm not going to make NEWS a step-by-step tutorial for how to compile
a bunch of files into a shared library.  That is too much to expect
from us.  It is NOT our job to teach people how to do that.

> (iv) Then we have precise instructions on where to put the newly built
> ..so file.  This is good!

This is the only part that is specific to Emacs, so I deliberately
included there all the details, since you are unlikely to find it
elsewhere.

> This process has so far taken me over an hour, which is too long for
> something which should be purely routine.  It is likely to take a
> typical Emacs user even longer.

Typical Emacs users who don't know how to build libraries from sources
will need someone else to do the job for them.  Exactly like they do
with GMP or librsvg or HarfBuzz or any other library that Emacs can
link against.  Eventually, there's no way around this.

> I suggest that these instructions in NEWS should be enhanced with (i)
> Tips on downloading stuff from github; (ii) A sample command line for
> compiling the C/C++ source files; (iii) A sample command line for
> linking these to the needed .so file.  Possibly two or three versions of
> these would be needed for the different environments Emacs runs in.

Sorry, but NO!!

Re: Difficulty building tree-sitter grammars [was: Help sought from C++ expert: ....]

[Alan Mackenzie]

Hello, Eli.

On Thu, Jan 19, 2023 at 19:53:58 +0200, Eli Zaretskii wrote:
> > Date: Thu, 19 Jan 2023 17:14:43 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > Just as a matter of interest, in c++-ts-mode a treesit-query-error gets
> > > > thrown.

> > > I don't see any such error.  Maybe this is again your outdated grammar
> > > library?

> > Heh!  Maybe you're right.  So time to get around to installing one from
> > github.  So I look at the Emacs-29 NEWS for instructions on how to do
> > this.  These instructions are glib and insufficiently helpful.

> Nothing is ever good enough, eh?

Emacs isn't good enough.  That's why we're continually working on it.
;-)

[ .... ]

> If you don't know how to do this given what's in NEWS, then I'm sorry,
> but these instructions are not for you.  You will have to wait until
> someone else (your distro?) produces the grammar library and lets you
> download and install it like you download and install all the image
> libraries and the rest of the optional stuff for Emacs.

> I'm not going to make NEWS a step-by-step tutorial for how to compile
> a bunch of files into a shared library.  That is too much to expect
> from us.  It is NOT our job to teach people how to do that.

Believe it or not, I've never built a shared library in my life (aside
from invoking makefiles) and I'm sure I'm far from being alone.

> > (iv) Then we have precise instructions on where to put the newly built
> > ..so file.  This is good!

> This is the only part that is specific to Emacs, so I deliberately
> included there all the details, since you are unlikely to find it
> elsewhere.

> > This process has so far taken me over an hour, which is too long for
> > something which should be purely routine.  It is likely to take a
> > typical Emacs user even longer.

> Typical Emacs users who don't know how to build libraries from sources
> will need someone else to do the job for them.  Exactly like they do
> with GMP or librsvg or HarfBuzz or any other library that Emacs can
> link against.  Eventually, there's no way around this.

These libraries come with makefiles, so there is no need to get into the
details of how to build them.

> > I suggest that these instructions in NEWS should be enhanced with (i)
> > Tips on downloading stuff from github; (ii) A sample command line for
> > compiling the C/C++ source files; (iii) A sample command line for
> > linking these to the needed .so file.  Possibly two or three versions of
> > these would be needed for the different environments Emacs runs in.

> Sorry, but NO!!

At Dmitry pointed out, it turns out there's a makefile-equivalent for
these grammar libraries, namely treesit-install-language-grammar.  This
worked well for me.

Why is it not mentioned first in NEWS, something like "Typically you can
build and install a grammar with ....." followed by "To build a grammar
outside of Emacs ..." and the description of compiling linking and so on?
Then I would have seen it, and the noise of this thread would have been
avoided.  Again, I doubt I'm the only user who'll not see past the end of
a first unconditional instruction.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Difficulty building tree-sitter grammars [was: Help sought from C++ expert: ....]

[Eli Zaretskii]

> Date: Thu, 19 Jan 2023 20:56:13 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > I'm not going to make NEWS a step-by-step tutorial for how to compile
> > a bunch of files into a shared library.  That is too much to expect
> > from us.  It is NOT our job to teach people how to do that.
> 
> Believe it or not, I've never built a shared library in my life (aside
> from invoking makefiles) and I'm sure I'm far from being alone.

Like I said: then building the libraries is not something you should
try on your own.

> These libraries come with makefiles, so there is no need to get into the
> details of how to build them.

Only some of them come with Makefiles, the majority doesn't.  Which is
why those Makefiles aren't mentioned in NEWS: you cannot rely on their
being there.

> At Dmitry pointed out, it turns out there's a makefile-equivalent for
> these grammar libraries, namely treesit-install-language-grammar.  This
> worked well for me.
> 
> Why is it not mentioned first in NEWS, something like "Typically you can
> build and install a grammar with ....." followed by "To build a grammar
> outside of Emacs ..." and the description of compiling linking and so on?

Isn't that already so?

But that command cannot solve all the problems for users who have no
experience in building from sources.  So some of them will still need
someone else to do this.

Re: Help sought from C++ expert: is this `value' a type or something else (what)?

[Michael Welsh Duggan]

Alan Mackenzie <acm@muc.de> writes:

> In the following test file, templates-9.cc from the CC Mode test suite,
> on the indicated line, what is the syntactic significance of `value'?

Here `value' is a value, specifically a boolean struct member.  This is
boost's implementation of what became, in C++11, `std::is_same', which
can be found documented here:

  https://en.cppreference.com/w/cpp/types/is_same

> Traditionally, it has fontified with the default face, but recently after
> some (not yet committed) changes in CC Mode, it has started getting
> font-lock-type-face.
>
> I don't understand the file at all, hardly, and can't work out whether
> the old (non-) fontification was correct, or whether the new
> fontification of f-l-type-face is correct.  Help would be appreciated.

Given a name `Foo', the compiler doesn't necessarily know if `Foo::bar'
is a type or a value, because it doesn't necessarily know what type Foo
is - it could be a template argument, for example, and won't be known
until an instance is referenced.  The assumption is that it is a value.
If it was a type, this is pointed out to the compiler by calling it
`typename Foo::bar'.  For more on this, see:

  https://en.cppreference.com/w/cpp/language/dependent_name

From that page, "In a declaration or a definition of a template,
including alias templates, a name that is not a member of the current
instantiation and is dependent on a template parameter is not considered
to be a type unless the keyword `typename' is used or unless it was
already established as a type name, e.g. with a typedef declaration or
by being used to name a base class."


> Just as a matter of interest, in c++-ts-mode a treesit-query-error gets
> thrown.  It isn't clear to me whether the file is still valid C++.

It looks valid.  The `template' before the `then' is, like `typename' a
disambiguator for the dependent name `then', stating that it is a
template name.

> /////////////////////////////////////////////////////////////////////////
> typedef std::pair<int
> 		  , std::pair<long
> 			      , std::pair<char*
> 					  , std::pair<nil,nil>
> 					 >
> 			     >
> 		 > list_of_types;
>
> typedef std::pair<
>     int, std::pair<
> 	long, std::pair<
> 	    char*, std::pair<nil,nil>
> 	>
>     >
>> list_of_types;
>
> typedef typename if_true
> <
>     boost::is_same
>     <
> 	boost::add_pointer<X>
> 	, int*
>     >::value     // <===================================================
>>::template then
> <
>     boost::remove_reference<X>
>     // else
>     , X
>>::type modified_X;
> /////////////////////////////////////////////////////////////////////////
>
> Thanks!

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Help sought from C++ expert: is this `value' a type or something else (what)?

[Alan Mackenzie]

Hello, Michael.

On Thu, Jan 19, 2023 at 10:41:15 -0500, Michael Welsh Duggan wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > In the following test file, templates-9.cc from the CC Mode test suite,
> > on the indicated line, what is the syntactic significance of `value'?

> Here `value' is a value, specifically a boolean struct member.  This is
> boost's implementation of what became, in C++11, `std::is_same', which
> can be found documented here:

>   https://en.cppreference.com/w/cpp/types/is_same

> > Traditionally, it has fontified with the default face, but recently after
> > some (not yet committed) changes in CC Mode, it has started getting
> > font-lock-type-face.

> > I don't understand the file at all, hardly, and can't work out whether
> > the old (non-) fontification was correct, or whether the new
> > fontification of f-l-type-face is correct.  Help would be appreciated.

> Given a name `Foo', the compiler doesn't necessarily know if `Foo::bar'
> is a type or a value, because it doesn't necessarily know what type Foo
> is - it could be a template argument, for example, and won't be known
> until an instance is referenced.  The assumption is that it is a value.
> If it was a type, this is pointed out to the compiler by calling it
> `typename Foo::bar'.  For more on this, see:

>   https://en.cppreference.com/w/cpp/language/dependent_name

Thanks, I've glanced over that page, now.

> >From that page, "In a declaration or a definition of a template,
> including alias templates, a name that is not a member of the current
> instantiation and is dependent on a template parameter is not considered
> to be a type unless the keyword `typename' is used or unless it was
> already established as a type name, e.g. with a typedef declaration or
> by being used to name a base class."

That's helpful.

> > Just as a matter of interest, in c++-ts-mode a treesit-query-error gets
> > thrown.  It isn't clear to me whether the file is still valid C++.

> It looks valid.  The `template' before the `then' is, like `typename' a
> disambiguator for the dependent name `then', stating that it is a
> template name.

Thanks.  I can now read the file semi-fluently, whereas before it
appeared to me just as a random assemblage of C++ keywords and
operators.  ;-(

> > /////////////////////////////////////////////////////////////////////////
> > typedef std::pair<int
> > 		  , std::pair<long
> > 			      , std::pair<char*
> > 					  , std::pair<nil,nil>
> > 					 >
> > 			     >
> > 		 > list_of_types;
> >
> > typedef std::pair<
> >     int, std::pair<
> > 	long, std::pair<
> > 	    char*, std::pair<nil,nil>
> > 	>
> >     >
> >> list_of_types;
> >
> > typedef typename if_true
> > <
> >     boost::is_same
> >     <
> > 	boost::add_pointer<X>
> > 	, int*
> >     >::value     // <===================================================
> >>::template then
> > <
> >     boost::remove_reference<X>
> >     // else
> >     , X
> >>::type modified_X;
> > /////////////////////////////////////////////////////////////////////////
> >
> > Thanks!

> -- 
> Michael Welsh Duggan
> (md5i@md5i.com)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Help sought from C++ expert: is this `value' a type or something else (what)?

[Óscar Fuentes]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Emacs.
>
> In the following test file, templates-9.cc from the CC Mode test suite,
> on the indicated line, what is the syntactic significance of `value'?
> Traditionally, it has fontified with the default face, but recently after
> some (not yet committed) changes in CC Mode, it has started getting
> font-lock-type-face.
>
> I don't understand the file at all, hardly, and can't work out whether
> the old (non-) fontification was correct, or whether the new
> fontification of f-l-type-face is correct.  Help would be appreciated.
>
> Just as a matter of interest, in c++-ts-mode a treesit-query-error gets
> thrown.  It isn't clear to me whether the file is still valid C++.

It is not valid C++ because it misses quite a few declarations.

The parser has no way of knowing what `value' is unless it can see the
relevant declarations. And even then, in the general case, it is not
possible to really know unless your "parser" is, actually, a C++ front
end. So both c++-mode and c++-ts-mode are out of their breadth here.

IMAO, a reasonable heuristic is to consider it a value unless it is
preceded by `typename'.

Re: Help sought from C++ expert: is this `value' a type or something else (what)?

[Alan Mackenzie]

Hello, Óscar.

On Thu, Jan 19, 2023 at 17:22:41 +0100, Óscar Fuentes wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Hello, Emacs.
> >
> > In the following test file, templates-9.cc from the CC Mode test suite,
> > on the indicated line, what is the syntactic significance of `value'?
> > Traditionally, it has fontified with the default face, but recently after
> > some (not yet committed) changes in CC Mode, it has started getting
> > font-lock-type-face.
> >
> > I don't understand the file at all, hardly, and can't work out whether
> > the old (non-) fontification was correct, or whether the new
> > fontification of f-l-type-face is correct.  Help would be appreciated.
> >
> > Just as a matter of interest, in c++-ts-mode a treesit-query-error gets
> > thrown.  It isn't clear to me whether the file is still valid C++.

> It is not valid C++ because it misses quite a few declarations.

Yes.  Michael Duggan has identified the file as part of an early boost
source, and clearly there are bits missing.

> The parser has no way of knowing what `value' is unless it can see the
> relevant declarations. And even then, in the general case, it is not
> possible to really know unless your "parser" is, actually, a C++ front
> end. So both c++-mode and c++-ts-mode are out of their breadth here.

c++-ts-mode is a little disappointing, here.  It displays virtually
everything (including "value") in font-lock-type-face.

I've managed to track down the bug in my uncommitted CC Mode changes and
fix it.

> IMAO, a reasonable heuristic is to consider it a value unless it is
> preceded by `typename'.

Yes.  That is the general consensus from all who answered me, so a big
thank you to yourself and the other people!

-- 
Alan Mackenzie (Nuremberg, Germany).