Treesit says grammar is unavailable when it

Treesit says grammar is unavailable when it

[mprodrigues]

Hello,

I've been trying to test the new tree-sitter modes, but I'm having some
difficulties and would like to debug this. I have tried 
dockerfile-ts-mode,
python-ts-mode and yaml-ts-mode, of these only dockerfile worked, for
the other two I get:

Warning (treesit): Cannot activate tree-sitter, because language grammar
for python is unavailable (not-found): (libtree-sitter-python 
libtree-sitter-python.so)
No such file or directory

but I have yaml and python's grammar under ~/.emacs.d/tree-sitter
I tried compiling the grammars myself and use emacs's own
treesit-install-language-grammar to no avail. How can I debug this?

My emacs version is 30.0.50

Best regards,
Mateus Rodrigues

Re: Treesit says grammar is unavailable when it

[Eli Zaretskii]

> Date: Fri, 10 Mar 2023 09:24:25 +0000
> From: mprodrigues@posteo.net
> 
> I've been trying to test the new tree-sitter modes, but I'm having some
> difficulties and would like to debug this. I have tried 
> dockerfile-ts-mode,
> python-ts-mode and yaml-ts-mode, of these only dockerfile worked, for
> the other two I get:
> 
> Warning (treesit): Cannot activate tree-sitter, because language grammar
> for python is unavailable (not-found): (libtree-sitter-python 
> libtree-sitter-python.so)
> No such file or directory
> 
> but I have yaml and python's grammar under ~/.emacs.d/tree-sitter
> I tried compiling the grammars myself and use emacs's own
> treesit-install-language-grammar to no avail. How can I debug this?

What is the value of user-emacs-directory in your Emacs sessions?

Also, please show the full contents of the ~/.emacs.d/tree-sitter
directory.

To debug this, I'd recommend stepping with a debugger into the
function treesit_load_language, and trying to figure out why it fails
to find the shared libraries you installed.

Re: Treesit says grammar is unavailable when it

[Gernot Kieseritzky]

Hello!

> Warning (treesit): Cannot activate tree-sitter, because language grammar
> for python is unavailable (not-found): (libtree-sitter-python libtree-sitter-python.so)
> No such file or directory

I had the same error when using "libtree-sitter-yaml.so" and, finally,
had some time for a deeper investigation. I tried the Emacs built-in
debugger (M-x debug-on-entry treesit_load_language) but I found the
error only occurs inside the C function "treesit-available-p". Since
this error occurred directly when Emacs loads the shared library
resting inside of my ".emacs.d/tree-sitter/" directory I resorted to
debugging the dynamic linker by starting Emacs with

$ LD_DEBUG=1; emacs 2> linker.log

and then just proceeded to activate "yaml-ts-mode". Then I hit this:

listdc++.so.6: error: version lookup error: version 'GLIBCXX_3.4.29'
not found (required by ~/.emacs.d/tree-sitter/libtree-sitter-yaml.so)

Turns out the yaml module is using C++ for its scanner component (see
https://github.com/ikatyang/tree-sitter-yaml) unlike all the other
plugins I am using (C, C++, JS and TS). But I compiled this module on
my own -- on the same machine -- so how can my system's C++ standard
library suddenly turn incompatible??

Well, it turns out I am using Alex Murray's Emacs snap package on my
Ubuntu 22.04 system being based on the "core20" snap base package
(i.e. it is based on Ubuntu 20). So this Emacs snap provides an older
set of build essentials including an incompatible C++ standard
library. So as a workaround I would need to cross-compile this module
using an Ubuntu 20 compatible environment. But the best solution is
probably to ask Alex Murray to add all available tree sitter parsers
in his snap.

Best regards,
Gernot Kieseritzky

Re: Treesit says grammar is unavailable when it

[Yuan Fu]

On 12/4/23 4:43 AM, Gernot Kieseritzky wrote:
> Hello!
>
>> Warning (treesit): Cannot activate tree-sitter, because language grammar
>> for python is unavailable (not-found): (libtree-sitter-python libtree-sitter-python.so)
>> No such file or directory
> I had the same error when using "libtree-sitter-yaml.so" and, finally,
> had some time for a deeper investigation. I tried the Emacs built-in
> debugger (M-x debug-on-entry treesit_load_language) but I found the
> error only occurs inside the C function "treesit-available-p". Since
> this error occurred directly when Emacs loads the shared library
> resting inside of my ".emacs.d/tree-sitter/" directory I resorted to
> debugging the dynamic linker by starting Emacs with
>
> $ LD_DEBUG=1; emacs 2> linker.log
>
> and then just proceeded to activate "yaml-ts-mode". Then I hit this:
>
> listdc++.so.6: error: version lookup error: version 'GLIBCXX_3.4.29'
> not found (required by ~/.emacs.d/tree-sitter/libtree-sitter-yaml.so)
>
> Turns out the yaml module is using C++ for its scanner component (see
> https://github.com/ikatyang/tree-sitter-yaml) unlike all the other
> plugins I am using (C, C++, JS and TS). But I compiled this module on
> my own -- on the same machine -- so how can my system's C++ standard
> library suddenly turn incompatible??
>
> Well, it turns out I am using Alex Murray's Emacs snap package on my
> Ubuntu 22.04 system being based on the "core20" snap base package
> (i.e. it is based on Ubuntu 20). So this Emacs snap provides an older
> set of build essentials including an incompatible C++ standard
> library. So as a workaround I would need to cross-compile this module
> using an Ubuntu 20 compatible environment. But the best solution is
> probably to ask Alex Murray to add all available tree sitter parsers
> in his snap.
>
> Best regards,
> Gernot Kieseritzky

Thanks for sharing this! Eli, this sort of thing might be good reasons 
to also print the error message from ldopen, rather than simply saying 
not found?

Yuan

Re: Treesit says grammar is unavailable when it

[Eli Zaretskii]

> Date: Tue, 12 Dec 2023 19:33:19 -0800
> From: Yuan Fu <casouri@gmail.com>
> 
> Thanks for sharing this! Eli, this sort of thing might be good reasons 
> to also print the error message from ldopen, rather than simply saying 
> not found?

I don't mind, but IME relying too much on these error messages is not
always a good idea, since sometimes they are too generic and say
nothing except "this failed".  But if it will help in some cases,
let's do it.

However, I would like to see the patch, since we seem to be signaling
an error already, and the error includes the error message from
dynlib_open?  So what am I missing?

Re: Treesit says grammar is unavailable when it

[Yuan Fu]

> On Dec 13, 2023, at 4:00 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Tue, 12 Dec 2023 19:33:19 -0800
>> From: Yuan Fu <casouri@gmail.com>
>> 
>> Thanks for sharing this! Eli, this sort of thing might be good reasons 
>> to also print the error message from ldopen, rather than simply saying 
>> not found?
> 
> I don't mind, but IME relying too much on these error messages is not
> always a good idea, since sometimes they are too generic and say
> nothing except "this failed".  But if it will help in some cases,
> let's do it.
> 
> However, I would like to see the patch, since we seem to be signaling
> an error already, and the error includes the error message from
> dynlib_open?  So what am I missing?

We were showing error messages from dlopen, and then removed them for brevity, see e8df6c311fcf59bf23d31b9db2bb8fec9d78fbe7

Maybe we can fold the extra error messages by default?

Yuan

Re: Treesit says grammar is unavailable when it

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Thu, 14 Dec 2023 00:14:01 -0800
> Cc: gernotk@gmail.com,
>  emacs-devel@gnu.org
> 
> >> Thanks for sharing this! Eli, this sort of thing might be good reasons 
> >> to also print the error message from ldopen, rather than simply saying 
> >> not found?
> > 
> > I don't mind, but IME relying too much on these error messages is not
> > always a good idea, since sometimes they are too generic and say
> > nothing except "this failed".  But if it will help in some cases,
> > let's do it.
> > 
> > However, I would like to see the patch, since we seem to be signaling
> > an error already, and the error includes the error message from
> > dynlib_open?  So what am I missing?
> 
> We were showing error messages from dlopen, and then removed them for brevity, see e8df6c311fcf59bf23d31b9db2bb8fec9d78fbe7

If these messages are useful, we can revert that commit.

> Maybe we can fold the extra error messages by default?

I'm not sure I understand the details, but the idea is certainly okay.

Re: Treesit says grammar is unavailable when it

[Madhu]

* Eli Zaretskii <83r0jpqjd9.fsf @gnu.org> :
Wrote on Thu, 14 Dec 2023 10:27:30 +0200:
>> > However, I would like to see the patch, since we seem to be
>> > signaling an error already, and the error includes the error
>> > message from dynlib_open?  So what am I missing?

>> We were showing error messages from dlopen, and then removed them for
>> brevity, see e8df6c311fcf59bf23d31b9db2bb8fec9d78fbe7
>
> If these messages are useful, we can revert that commit.
>
>> Maybe we can fold the extra error messages by default?
>
> I'm not sure I understand the details, but the idea is certainly okay.

I still think writing a lisp interface to dlopen and dlclose would iron
out a lot of things in the details and make it useful for treesitter
related stuff.  Or is the locking down the ffi interface done for FSF
reasons?

On another note, I also went through the same process with yaml-ts-mode,
A pointer to where the treesitter plugin is available is missing from
the docs.  Then one has to turn to third parties for this information
and to find the providers and toolchain providers and do the dance to
their tunes.

But most disappointing thing was that the modes come with absolutely no
navigation functions. These are structured files, and emacs has provides
way to navigate the structures.  This is a general quality defect from
to "prog" abstractions.  (when the facililties are provided they are
inadequate and a dedicated mode would have taken the diligence to do it
right)

Re: Treesit says grammar is unavailable when it

[Eli Zaretskii]

> From: Madhu <enometh@meer.net>
> Date: Fri, 15 Dec 2023 08:13:14 +0530
> 
> I still think writing a lisp interface to dlopen and dlclose would iron
> out a lot of things in the details and make it useful for treesitter
> related stuff.  Or is the locking down the ffi interface done for FSF
> reasons?

First, this particular aspect of Emacs is unrelated to the issue at
hand, so I suggest a new thread with a different Subject.

And second, we indeed don't want to allow loading arbitrary shared
libraries into Emacs, except via the emacs-modules interfaces.  It
mostly will make no sense anyway, since you will not be able to use
the loaded library without some glue code.  The grammar libraries are
a rare exception, because the tree-sitter library handles the glue.

I also object to having a direct dlopen interface exposed to Lisp on
the grounds that Emacs (unlike Guile, perhaps) is not a library for
interfacing the OS via Lisp bindings.  Emacs is a text-editing and
text-processing environment, and therefore should hide the OS features
behind suitable higher-level APIs.  Like emacs-modules do, by
extending 'load' and similar.  That makes these features more
portable, and allows writing Lisp programs that work on systems where
dlopen per se is not available.

> On another note, I also went through the same process with yaml-ts-mode,
> A pointer to where the treesitter plugin is available is missing from
> the docs.  Then one has to turn to third parties for this information
> and to find the providers and toolchain providers and do the dance to
> their tunes.

This is a job for downstream distros.  I explained in the past several
times why I object to making this our job.

> But most disappointing thing was that the modes come with absolutely no
> navigation functions. These are structured files, and emacs has provides
> way to navigate the structures.  This is a general quality defect from
> to "prog" abstractions.  (when the facililties are provided they are
> inadequate and a dedicated mode would have taken the diligence to do it
> right)

We have some navigation capabilities, like movement by defuns.
However, any reasonable extensions and improvements in this area will
surely be welcome, but I think we need a more detailed feature
requests for that.

Thanks.

Re: Treesit says grammar is unavailable when it

[Yuan Fu]

> On Dec 14, 2023, at 12:27 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Thu, 14 Dec 2023 00:14:01 -0800
>> Cc: gernotk@gmail.com,
>> emacs-devel@gnu.org
>> 
>>>> Thanks for sharing this! Eli, this sort of thing might be good reasons 
>>>> to also print the error message from ldopen, rather than simply saying 
>>>> not found?
>>> 
>>> I don't mind, but IME relying too much on these error messages is not
>>> always a good idea, since sometimes they are too generic and say
>>> nothing except "this failed".  But if it will help in some cases,
>>> let's do it.
>>> 
>>> However, I would like to see the patch, since we seem to be signaling
>>> an error already, and the error includes the error message from
>>> dynlib_open?  So what am I missing?
>> 
>> We were showing error messages from dlopen, and then removed them for brevity, see e8df6c311fcf59bf23d31b9db2bb8fec9d78fbe7
> 
> If these messages are useful, we can revert that commit.

I forgot to ask, should this be on emacs-29 or master?

> 
>> Maybe we can fold the extra error messages by default?
> 
> I'm not sure I understand the details, but the idea is certainly okay.

I’m thinking about applying text property to the extra messages so they appear as “…”, and is expanded when clicked (or hit enter), like what we have for printing large objects in the scratch buffer.

Yuan

Re: Treesit says grammar is unavailable when it

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 16 Dec 2023 22:31:02 -0800
> Cc: gernotk@gmail.com,
>  emacs-devel@gnu.org
> 
> > On Dec 14, 2023, at 12:27 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> We were showing error messages from dlopen, and then removed them for brevity, see e8df6c311fcf59bf23d31b9db2bb8fec9d78fbe7
> > 
> > If these messages are useful, we can revert that commit.
> 
> I forgot to ask, should this be on emacs-29 or master?

On master, I think.

> >> Maybe we can fold the extra error messages by default?
> > 
> > I'm not sure I understand the details, but the idea is certainly okay.
> 
> I’m thinking about applying text property to the extra messages so they appear as “…”, and is expanded when clicked (or hit enter), like what we have for printing large objects in the scratch buffer.

Sounds like over-engineering to me.  I don't think we do anything like
that in any other case where we emit error messages?

Re: Treesit says grammar is unavailable when it

[Yuan Fu]

> On Dec 17, 2023, at 12:32 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sat, 16 Dec 2023 22:31:02 -0800
>> Cc: gernotk@gmail.com,
>> emacs-devel@gnu.org
>> 
>>> On Dec 14, 2023, at 12:27 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>>> We were showing error messages from dlopen, and then removed them for brevity, see e8df6c311fcf59bf23d31b9db2bb8fec9d78fbe7
>>> 
>>> If these messages are useful, we can revert that commit.
>> 
>> I forgot to ask, should this be on emacs-29 or master?
> 
> On master, I think.
> 
>>>> Maybe we can fold the extra error messages by default?
>>> 
>>> I'm not sure I understand the details, but the idea is certainly okay.
>> 
>> I’m thinking about applying text property to the extra messages so they appear as “…”, and is expanded when clicked (or hit enter), like what we have for printing large objects in the scratch buffer.
> 
> Sounds like over-engineering to me.  I don't think we do anything like
> that in any other case where we emit error messages?

Yeah, I might be overthinking this. Reverted on master.

Yuan