bug#71562: treesit-install-language-grammar fails to install typescript

bug#71562: treesit-install-language-grammar fails to install typescript

[Steinar Bang]

Platform: emacs 29.3 from backports debian package (1:29.3+1-3~bpo12+1)
          debian 12.5 "bookworm" (current stable) on amd64

The 'M-x treesit-install-language-grammar RET typescript RET' fails with:

 ⛔ Warning (treesit): Error encountered when installing language grammar: (file-missing Setting current directory No such file or directory /tmp/treesit-workdirN2u4XF/repo/src)

I used defaults on all questions because I had no idea what else to put
there.

I think the reason the command fails is that the repository
 https://github.com/tree-sitter/tree-sitter-typescript
actually contains two tree-sitter parsers for typescript and tsx:
 https://github.com/tree-sitter/tree-sitter-typescript/tree/master/typescript
 https://github.com/tree-sitter/tree-sitter-typescript/tree/master/tsx

bug#71562: treesit-install-language-grammar fails to install typescript

[Dmitry Gutov]

On 14/06/2024 21:18, Steinar Bang wrote:
> Platform: emacs 29.3 from backports debian package (1:29.3+1-3~bpo12+1)
>            debian 12.5 "bookworm" (current stable) on amd64
> 
> The 'M-x treesit-install-language-grammar RET typescript RET' fails
> with:
> 
>   ⛔ Warning (treesit): Error encountered when installing language
> grammar: (file-missing Setting current directory No such file or
> directory /tmp/treesit-workdirN2u4XF/repo/src)
> 
> I used defaults on all questions because I had no idea what else to put
> there.
> 
> I think the reason the command fails is that the repository
>   https://github.com/tree-sitter/tree-sitter-typescript
> actually contains two tree-sitter parsers for typescript and tsx:
>   https://github.com/tree-sitter/tree-sitter-typescript/tree/master/typescript
>   https://github.com/tree-sitter/tree-sitter-typescript/tree/master/tsx

FWIW, this is more or less intended, since we don't store the exact urls 
for the grammars (in the default settings), and just infer them 
according to the common scheme.

But we could change that in some cases, and perhaps typescript is 
popular enough for us to do that.

bug#71562: treesit-install-language-grammar fails to install typescript

[Eli Zaretskii]

> Date: Sat, 15 Jun 2024 01:18:50 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/06/2024 21:18, Steinar Bang wrote:
> > Platform: emacs 29.3 from backports debian package (1:29.3+1-3~bpo12+1)
> >            debian 12.5 "bookworm" (current stable) on amd64
> > 
> > The 'M-x treesit-install-language-grammar RET typescript RET' fails
> > with:
> > 
> >   ⛔ Warning (treesit): Error encountered when installing language
> > grammar: (file-missing Setting current directory No such file or
> > directory /tmp/treesit-workdirN2u4XF/repo/src)
> > 
> > I used defaults on all questions because I had no idea what else to put
> > there.
> > 
> > I think the reason the command fails is that the repository
> >   https://github.com/tree-sitter/tree-sitter-typescript
> > actually contains two tree-sitter parsers for typescript and tsx:
> >   https://github.com/tree-sitter/tree-sitter-typescript/tree/master/typescript
> >   https://github.com/tree-sitter/tree-sitter-typescript/tree/master/tsx
> 
> FWIW, this is more or less intended, since we don't store the exact urls 
> for the grammars (in the default settings), and just infer them 
> according to the common scheme.
> 
> But we could change that in some cases, and perhaps typescript is 
> popular enough for us to do that.

We could look for directories names 'src' which have either parser.c
or parser.cc, and take it from there, instead of hard-coding the place
of the 'src' directory in the hierarchy.

I've added Yuan to the discussion.

bug#71562: treesit-install-language-grammar fails to install typescript

[Dmitry Gutov]

On 15/06/2024 09:31, Eli Zaretskii wrote:
> We could look for directories names 'src' which have either parser.c
> or parser.cc, and take it from there, instead of hard-coding the place
> of the 'src' directory in the hierarchy.

In the present case we have two problems:

* https://github.com/tree-sitter/tree-sitter-typescript/ contains two 
different files called parser.c in two different subdirectories (in 
typescript/src and tsx/src).

* The repository for the 'tsx' grammar does not match the common scheme 
- instead it's the same repository as for 'typescript'.

bug#71562: treesit-install-language-grammar fails to install typescript

[Eli Zaretskii]

> Date: Sat, 15 Jun 2024 22:30:25 +0300
> Cc: sb@dod.no, 71562@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 15/06/2024 09:31, Eli Zaretskii wrote:
> > We could look for directories names 'src' which have either parser.c
> > or parser.cc, and take it from there, instead of hard-coding the place
> > of the 'src' directory in the hierarchy.
> 
> In the present case we have two problems:
> 
> * https://github.com/tree-sitter/tree-sitter-typescript/ contains two 
> different files called parser.c in two different subdirectories (in 
> typescript/src and tsx/src).
> 
> * The repository for the 'tsx' grammar does not match the common scheme 
> - instead it's the same repository as for 'typescript'.

I'm saying that we should run the build commands in every subdirectory
called 'src' that has parser.c or parser.cc in it.  Each shared
library thus created should be installed.  The name of the library is
derived from the parent of the 'src' subdirectory.

bug#71562: treesit-install-language-grammar fails to install typescript

[Yuan Fu]

> On Jun 15, 2024, at 10:16 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Sat, 15 Jun 2024 22:30:25 +0300
>> Cc: sb@dod.no, 71562@debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> On 15/06/2024 09:31, Eli Zaretskii wrote:
>>> We could look for directories names 'src' which have either parser.c
>>> or parser.cc, and take it from there, instead of hard-coding the place
>>> of the 'src' directory in the hierarchy.
>> 
>> In the present case we have two problems:
>> 
>> * https://github.com/tree-sitter/tree-sitter-typescript/ contains two 
>> different files called parser.c in two different subdirectories (in 
>> typescript/src and tsx/src).
>> 
>> * The repository for the 'tsx' grammar does not match the common scheme 
>> - instead it's the same repository as for 'typescript'.
> 
> I'm saying that we should run the build commands in every subdirectory
> called 'src' that has parser.c or parser.cc in it.  Each shared
> library thus created should be installed.  The name of the library is
> derived from the parent of the 'src' subdirectory.

treesit-install-language-grammar works fine if you pass “typescript/src” instead of the default “src” to it, for the third argument, so I don’t think there’s anything to fix here. Installing multiple things “smartly” might be more confusing and error-prone than helpful, IMO.

For reference, the recipe for typescript is

(typescript "https://github.com/tree-sitter/tree-sitter-typescript” nil "typescript/src" nil nil)

Yuan

bug#71562: treesit-install-language-grammar fails to install typescript

[Steinar Bang]

>>>>> Yuan Fu <casouri@gmail.com>:

> treesit-install-language-grammar works fine if you pass “typescript/src” instead of the default “src” to it, for the third argument, so I don’t think there’s anything to fix here. Installing multiple things “smartly” might be more confusing and error-prone than helpful, IMO.

FWIW the repo contains two grammars: typescript and tsx

And both grammars are needed (not just typescript).

bug#71562: treesit-install-language-grammar fails to install typescript

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Dmitry Gutov <dmitry@gutov.dev>,
>   71562@debbugs.gnu.org
> Date: Wed, 19 Jun 2024 20:51:21 +0200
> 
> >>>>> Yuan Fu <casouri@gmail.com>:
> 
> > treesit-install-language-grammar works fine if you pass “typescript/src” instead of the default “src” to it, for the third argument, so I don’t think there’s anything to fix here. Installing multiple things “smartly” might be more confusing and error-prone than helpful, IMO.
> 
> FWIW the repo contains two grammars: typescript and tsx
> 
> And both grammars are needed (not just typescript).

Yes.  Which is why I said that we should find _all_ the subdirectories
called 'src' which have a parser, and produce a shared library in each
one of them.

bug#71562: treesit-install-language-grammar fails to install typescript

[Dmitry Gutov]

On 16/06/2024 08:16, Eli Zaretskii wrote:
> I'm saying that we should run the build commands in every subdirectory
> called 'src' that has parser.c or parser.cc in it.  Each shared
> library thus created should be installed.  The name of the library is
> derived from the parent of the 'src' subdirectory.

I suppose the downside is when you pass 'tsx' as the language you still 
get nothing.

bug#71562: treesit-install-language-grammar fails to install typescript

[Eli Zaretskii]

> Date: Wed, 19 Jun 2024 22:51:28 +0300
> Cc: casouri@gmail.com, sb@dod.no, 71562@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 16/06/2024 08:16, Eli Zaretskii wrote:
> > I'm saying that we should run the build commands in every subdirectory
> > called 'src' that has parser.c or parser.cc in it.  Each shared
> > library thus created should be installed.  The name of the library is
> > derived from the parent of the 'src' subdirectory.
> 
> I suppose the downside is when you pass 'tsx' as the language you still 
> get nothing.

I don't understand: pass 'tsx' as language to what interface and in
which situation?

bug#71562: treesit-install-language-grammar fails to install typescript

[Dmitry Gutov]

On 20/06/2024 07:54, Eli Zaretskii wrote:
>> Date: Wed, 19 Jun 2024 22:51:28 +0300
>> Cc:casouri@gmail.com,sb@dod.no,71562@debbugs.gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 16/06/2024 08:16, Eli Zaretskii wrote:
>>> I'm saying that we should run the build commands in every subdirectory
>>> called 'src' that has parser.c or parser.cc in it.  Each shared
>>> library thus created should be installed.  The name of the library is
>>> derived from the parent of the 'src' subdirectory.
>> I suppose the downside is when you pass 'tsx' as the language you still
>> get nothing.
> I don't understand: pass 'tsx' as language to what interface and in
> which situation?

1. Bob is a user who wants to edit a tsx file.
2. He tries the tsx-ts-mode and sees the error "TSX grammar not found".
3. He calls 'M-x treesit-install-language-grammar' and inputs 'tsx' as 
the language.

bug#71562: treesit-install-language-grammar fails to install typescript

[Eli Zaretskii]

> Date: Thu, 20 Jun 2024 13:28:23 +0300
> Cc: casouri@gmail.com, sb@dod.no, 71562@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 20/06/2024 07:54, Eli Zaretskii wrote:
> >> Date: Wed, 19 Jun 2024 22:51:28 +0300
> >> Cc:casouri@gmail.com,sb@dod.no,71562@debbugs.gnu.org
> >> From: Dmitry Gutov<dmitry@gutov.dev>
> >>
> >> On 16/06/2024 08:16, Eli Zaretskii wrote:
> >>> I'm saying that we should run the build commands in every subdirectory
> >>> called 'src' that has parser.c or parser.cc in it.  Each shared
> >>> library thus created should be installed.  The name of the library is
> >>> derived from the parent of the 'src' subdirectory.
> >> I suppose the downside is when you pass 'tsx' as the language you still
> >> get nothing.
> > I don't understand: pass 'tsx' as language to what interface and in
> > which situation?
> 
> 1. Bob is a user who wants to edit a tsx file.
> 2. He tries the tsx-ts-mode and sees the error "TSX grammar not found".
> 3. He calls 'M-x treesit-install-language-grammar' and inputs 'tsx' as 
> the language.

And what do you think should happen next?  Or where's the problem in
the above 3 steps?

bug#71562: treesit-install-language-grammar fails to install typescript

[Dmitry Gutov]

On 20/06/2024 13:45, Eli Zaretskii wrote:
>> Date: Thu, 20 Jun 2024 13:28:23 +0300
>> Cc:casouri@gmail.com,sb@dod.no,71562@debbugs.gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 20/06/2024 07:54, Eli Zaretskii wrote:
>>>> Date: Wed, 19 Jun 2024 22:51:28 +0300
>>>> Cc:casouri@gmail.com,sb@dod.no,71562@debbugs.gnu.org
>>>> From: Dmitry Gutov<dmitry@gutov.dev>
>>>>
>>>> On 16/06/2024 08:16, Eli Zaretskii wrote:
>>>>> I'm saying that we should run the build commands in every subdirectory
>>>>> called 'src' that has parser.c or parser.cc in it.  Each shared
>>>>> library thus created should be installed.  The name of the library is
>>>>> derived from the parent of the 'src' subdirectory.
>>>> I suppose the downside is when you pass 'tsx' as the language you still
>>>> get nothing.
>>> I don't understand: pass 'tsx' as language to what interface and in
>>> which situation?
>> 1. Bob is a user who wants to edit a tsx file.
>> 2. He tries the tsx-ts-mode and sees the error "TSX grammar not found".
>> 3. He calls 'M-x treesit-install-language-grammar' and inputs 'tsx' as
>> the language.
> And what do you think should happen next?  Or where's the problem in
> the above 3 steps?

Next he presses RET a few times and the installation fails.

If we added two entries to treesit-language-source-alist by default, 
OTOH (one for typescript, and another for tsx), then Bob would be able 
to install the grammar easily.

bug#71562: treesit-install-language-grammar fails to install typescript

[Eli Zaretskii]

> Date: Thu, 20 Jun 2024 14:18:42 +0300
> Cc: casouri@gmail.com, sb@dod.no, 71562@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 20/06/2024 13:45, Eli Zaretskii wrote:
> >> Date: Thu, 20 Jun 2024 13:28:23 +0300
> >> Cc:casouri@gmail.com,sb@dod.no,71562@debbugs.gnu.org
> >> From: Dmitry Gutov<dmitry@gutov.dev>
> >>
> >> On 20/06/2024 07:54, Eli Zaretskii wrote:
> >>>> Date: Wed, 19 Jun 2024 22:51:28 +0300
> >>>> Cc:casouri@gmail.com,sb@dod.no,71562@debbugs.gnu.org
> >>>> From: Dmitry Gutov<dmitry@gutov.dev>
> >>>>
> >>>> On 16/06/2024 08:16, Eli Zaretskii wrote:
> >>>>> I'm saying that we should run the build commands in every subdirectory
> >>>>> called 'src' that has parser.c or parser.cc in it.  Each shared
> >>>>> library thus created should be installed.  The name of the library is
> >>>>> derived from the parent of the 'src' subdirectory.
> >>>> I suppose the downside is when you pass 'tsx' as the language you still
> >>>> get nothing.
> >>> I don't understand: pass 'tsx' as language to what interface and in
> >>> which situation?
> >> 1. Bob is a user who wants to edit a tsx file.
> >> 2. He tries the tsx-ts-mode and sees the error "TSX grammar not found".
> >> 3. He calls 'M-x treesit-install-language-grammar' and inputs 'tsx' as
> >> the language.
> > And what do you think should happen next?  Or where's the problem in
> > the above 3 steps?
> 
> Next he presses RET a few times and the installation fails.

Why would it fail, if my suggestion is implemented?

> If we added two entries to treesit-language-source-alist by default, 
> OTOH (one for typescript, and another for tsx), then Bob would be able 
> to install the grammar easily.

But then we'd need to maintain a DB of all the "abnormal" repositories
(and there a few of them), which is less reliable.

bug#71562: treesit-install-language-grammar fails to install typescript

[Dmitry Gutov]

On 20/06/2024 14:31, Eli Zaretskii wrote:

>>> And what do you think should happen next?  Or where's the problem in
>>> the above 3 steps?
>>
>> Next he presses RET a few times and the installation fails.
> 
> Why would it fail, if my suggestion is implemented?

Because your suggestion is for the case where Bob specifies 'typescript' 
on step 3.

>> If we added two entries to treesit-language-source-alist by default,
>> OTOH (one for typescript, and another for tsx), then Bob would be able
>> to install the grammar easily.
> 
> But then we'd need to maintain a DB of all the "abnormal" repositories
> (and there a few of them),

There are not too many of them, and out of those TypeScript (and TSX 
with it) is more popular than most of the others as a programming 
language. So even if we add these two it wouldn't follow that we must 
have the rest.

 > which is less reliable.

I suppose we could do both - both update the recipe, and scan the 
directory tree for files named parser.c.

If we aren't concerned about side-effects, such as for example when the 
user wanted to stay on the previous version of the tsx grammar they had 
installed (because the major mode doesn't work with the newer one), but 
they need to update the typescript grammar, and the tsx grammar is 
overwritten in the process too.

bug#71562: treesit-install-language-grammar fails to install typescript

[Eli Zaretskii]

> Date: Thu, 20 Jun 2024 18:03:32 +0300
> Cc: casouri@gmail.com, sb@dod.no, 71562@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 20/06/2024 14:31, Eli Zaretskii wrote:
> 
> >>> And what do you think should happen next?  Or where's the problem in
> >>> the above 3 steps?
> >>
> >> Next he presses RET a few times and the installation fails.
> > 
> > Why would it fail, if my suggestion is implemented?
> 
> Because your suggestion is for the case where Bob specifies 'typescript' 
> on step 3.

No, it was for _any_ grammar, regardless of its name.

> >> If we added two entries to treesit-language-source-alist by default,
> >> OTOH (one for typescript, and another for tsx), then Bob would be able
> >> to install the grammar easily.
> > 
> > But then we'd need to maintain a DB of all the "abnormal" repositories
> > (and there a few of them),
> 
> There are not too many of them, and out of those TypeScript (and TSX 
> with it) is more popular than most of the others as a programming 
> language. So even if we add these two it wouldn't follow that we must 
> have the rest.

They are not too many, but they tend to grow, and maintaining such a
list would be an extra burden.  Why do that when a simpler and more
reliable solution exists?

> If we aren't concerned about side-effects, such as for example when the 
> user wanted to stay on the previous version of the tsx grammar they had 
> installed (because the major mode doesn't work with the newer one), but 
> they need to update the typescript grammar, and the tsx grammar is 
> overwritten in the process too.

I don't see why that would be a problem.  The grammars that come in
pairs share common parts anyway, so I think updating them all is
actually slightly safer.

bug#71562: treesit-install-language-grammar fails to install typescript

[Dmitry Gutov]

On 20/06/2024 18:59, Eli Zaretskii wrote:
>> Date: Thu, 20 Jun 2024 18:03:32 +0300
>> Cc: casouri@gmail.com, sb@dod.no, 71562@debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 20/06/2024 14:31, Eli Zaretskii wrote:
>>
>>>>> And what do you think should happen next?  Or where's the problem in
>>>>> the above 3 steps?
>>>>
>>>> Next he presses RET a few times and the installation fails.
>>>
>>> Why would it fail, if my suggestion is implemented?
>>
>> Because your suggestion is for the case where Bob specifies 'typescript'
>> on step 3.
> 
> No, it was for _any_ grammar, regardless of its name.

It will work only in the case where Bob specifies 'typescript', not 'tsx'.

>>>> If we added two entries to treesit-language-source-alist by default,
>>>> OTOH (one for typescript, and another for tsx), then Bob would be able
>>>> to install the grammar easily.
>>>
>>> But then we'd need to maintain a DB of all the "abnormal" repositories
>>> (and there a few of them),
>>
>> There are not too many of them, and out of those TypeScript (and TSX
>> with it) is more popular than most of the others as a programming
>> language. So even if we add these two it wouldn't follow that we must
>> have the rest.
> 
> They are not too many, but they tend to grow, and maintaining such a
> list would be an extra burden.  Why do that when a simpler and more
> reliable solution exists?

See the previous mentioned problems with it.

>> If we aren't concerned about side-effects, such as for example when the
>> user wanted to stay on the previous version of the tsx grammar they had
>> installed (because the major mode doesn't work with the newer one), but
>> they need to update the typescript grammar, and the tsx grammar is
>> overwritten in the process too.
> 
> I don't see why that would be a problem.  The grammars that come in
> pairs share common parts anyway, so I think updating them all is
> actually slightly safer.

That might be true, though probably not in all cases.

Our major modes for typescript and tsx do share some parts of the 
definition.