Versioned Tree-sitter parser libraries

Versioned Tree-sitter parser libraries

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 709 bytes --]

A number of Tree-sitter parsers include a Makefile that will install, e.g., /usr/lib/libtree-sitter-json.so.0.0.  However, I notice that Emacs 29.0.90 will find it only if there exists a symlink /usr/lib/libtree-sitter-json.so.

Is this desirable?

The background to this question is that I’m thinking about packaging Tree-sitter parsers as RPMs for Fedora (https://pagure.io/tree-sitter-json/blob/rawhide/f/tree-sitter-json.spec).  Typical Fedora practice (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_devel_packages) is to assume that the unversioned symlink isn’t required for regular use, and to place it in a …-devel subpackage that wouldn’t usually be installed.

-- 
Peter Oliver

Re: Versioned Tree-sitter parser libraries

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 15:20:51 +0100 (BST)
> From: Peter Oliver <p.d.oliver@mavit.org.uk>
> 
> A number of Tree-sitter parsers include a Makefile that will install, e.g., /usr/lib/libtree-sitter-json.so.0.0.  However, I notice that Emacs 29.0.90 will find it only if there exists a symlink /usr/lib/libtree-sitter-json.so.
> 
> Is this desirable?

If we were to load a versioned shared library, how would Emacs know
which version to load?

> The background to this question is that I’m thinking about packaging Tree-sitter parsers as RPMs for Fedora (https://pagure.io/tree-sitter-json/blob/rawhide/f/tree-sitter-json.spec).  Typical Fedora practice (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_devel_packages) is to assume that the unversioned symlink isn’t required for regular use, and to place it in a …-devel subpackage that wouldn’t usually be installed.

That practice is fine for when a program is linked against the
library, because the -lFOO linker switch is supposed to take care of
that, and Emacs doesn't need to know which version is being used.  But
here we load the shared libraries at run time, so we must know their
names.  These names are explicitly mentioned in the Emacs sources.

Re: Versioned Tree-sitter parser libraries

[Yuan Fu]

> On Apr 16, 2023, at 7:49 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Date: Sun, 16 Apr 2023 15:20:51 +0100 (BST)
>> From: Peter Oliver <p.d.oliver@mavit.org.uk>
>> 
>> A number of Tree-sitter parsers include a Makefile that will install, e.g., /usr/lib/libtree-sitter-json.so.0.0.  However, I notice that Emacs 29.0.90 will find it only if there exists a symlink /usr/lib/libtree-sitter-json.so.
>> 
>> Is this desirable?
> 
> If we were to load a versioned shared library, how would Emacs know
> which version to load?

Another thing is that tree-sitter grammars don’t really have versions (API or ABI versions), except for a tree-sitter version.

> 
>> The background to this question is that I’m thinking about packaging Tree-sitter parsers as RPMs for Fedora (https://pagure.io/tree-sitter-json/blob/rawhide/f/tree-sitter-json.spec).  Typical Fedora practice (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_devel_packages) is to assume that the unversioned symlink isn’t required for regular use, and to place it in a …-devel subpackage that wouldn’t usually be installed.
> 
> That practice is fine for when a program is linked against the
> library, because the -lFOO linker switch is supposed to take care of
> that, and Emacs doesn't need to know which version is being used.  But
> here we load the shared libraries at run time, so we must know their
> names.  These names are explicitly mentioned in the Emacs sources.


Since Emacs requests/hopes the package manager or the system will provide the grammars, it seems a good idea to support the common way libraries tend to be named. Even maybe just a additional look up for only xxx.0.0, since grammars don’t have ABI versions, at least for now.

Yuan

Re: Versioned Tree-sitter parser libraries

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 16 Apr 2023 20:03:50 -0700
> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>  emacs-devel@gnu.org
> 
> > If we were to load a versioned shared library, how would Emacs know
> > which version to load?
> 
> Another thing is that tree-sitter grammars don’t really have versions (API or ABI versions), except for a tree-sitter version.

AFAIR, there is an ABI version, and we even expose it to Lisp, no?

> >> The background to this question is that I’m thinking about packaging Tree-sitter parsers as RPMs for Fedora (https://pagure.io/tree-sitter-json/blob/rawhide/f/tree-sitter-json.spec).  Typical Fedora practice (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_devel_packages) is to assume that the unversioned symlink isn’t required for regular use, and to place it in a …-devel subpackage that wouldn’t usually be installed.
> > 
> > That practice is fine for when a program is linked against the
> > library, because the -lFOO linker switch is supposed to take care of
> > that, and Emacs doesn't need to know which version is being used.  But
> > here we load the shared libraries at run time, so we must know their
> > names.  These names are explicitly mentioned in the Emacs sources.
> 
> 
> Since Emacs requests/hopes the package manager or the system will provide the grammars, it seems a good idea to support the common way libraries tend to be named. Even maybe just a additional look up for only xxx.0.0, since grammars don’t have ABI versions, at least for now.

I don't think I understand what you suggest in practical terms.  What
does it mean "additional look up for only xxx.0.0"? and why only
xxx.0.0?

Re: Versioned Tree-sitter parser libraries

[Yuan Fu]

> On Apr 17, 2023, at 10:03 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 16 Apr 2023 20:03:50 -0700
>> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>> emacs-devel@gnu.org
>> 
>>> If we were to load a versioned shared library, how would Emacs know
>>> which version to load?
>> 
>> Another thing is that tree-sitter grammars don’t really have versions (API or ABI versions), except for a tree-sitter version.
> 
> AFAIR, there is an ABI version, and we even expose it to Lisp, no?

There is a version, but it’s kind of a internal/protocol version between tree-sitter library and tree-sitter grammars, and I don’t think it’s a standard ABI version per se.

> 
>>>> The background to this question is that I’m thinking about packaging Tree-sitter parsers as RPMs for Fedora (https://pagure.io/tree-sitter-json/blob/rawhide/f/tree-sitter-json.spec).  Typical Fedora practice (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_devel_packages) is to assume that the unversioned symlink isn’t required for regular use, and to place it in a …-devel subpackage that wouldn’t usually be installed.
>>> 
>>> That practice is fine for when a program is linked against the
>>> library, because the -lFOO linker switch is supposed to take care of
>>> that, and Emacs doesn't need to know which version is being used.  But
>>> here we load the shared libraries at run time, so we must know their
>>> names.  These names are explicitly mentioned in the Emacs sources.
>> 
>> 
>> Since Emacs requests/hopes the package manager or the system will provide the grammars, it seems a good idea to support the common way libraries tend to be named. Even maybe just a additional look up for only xxx.0.0, since grammars don’t have ABI versions, at least for now.
> 
> I don't think I understand what you suggest in practical terms.  What
> does it mean "additional look up for only xxx.0.0"? and why only
> xxx.0.0?

I was suggesting that we do recognize ABI versions, but since tree-sitter grammars don’t really have a ABI version, and the version extension in filenames are just there by convention (of Fedora), we can expect the version extension to be always 0.0, should there be one.

Yuan

Re: Versioned Tree-sitter parser libraries

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 17 Apr 2023 11:41:49 -0700
> Cc: Peter Oliver <p.d.oliver@mavit.org.uk>,
>  emacs-devel@gnu.org
> 
> > I don't think I understand what you suggest in practical terms.  What
> > does it mean "additional look up for only xxx.0.0"? and why only
> > xxx.0.0?
> 
> I was suggesting that we do recognize ABI versions, but since tree-sitter grammars don’t really have a ABI version, and the version extension in filenames are just there by convention (of Fedora), we can expect the version extension to be always 0.0, should there be one.

If this will help distros, I don't mind.

But this should be limited to Posix hosts, since on Windows the story
is different, and on macOS it is also different (in another way),
AFAIR.

Re: Versioned Tree-sitter parser libraries

[Peter Oliver]

[-- Attachment #1: Type: text/plain, Size: 509 bytes --]

On Mon, 17 Apr 2023, Yuan Fu wrote:

> I was suggesting that we do recognize ABI versions, but since tree-sitter grammars don’t really have a ABI version, and the version extension in filenames are just there by convention (of Fedora), we can expect the version extension to be always 0.0, should there be one.

It’s the Tree-sitter project that have chosen the numbers in their Makefiles, not Fedora, by the way.  https://github.com/tree-sitter/tree-sitter-json/blob/master/Makefile#L42

-- 
Peter Oliver

Re: Versioned Tree-sitter parser libraries

[Yuan Fu]

> On Apr 17, 2023, at 1:17 PM, Peter Oliver <p.d.oliver@mavit.org.uk> wrote:
> 
> On Mon, 17 Apr 2023, Yuan Fu wrote:
> 
>> I was suggesting that we do recognize ABI versions, but since tree-sitter grammars don’t really have a ABI version, and the version extension in filenames are just there by convention (of Fedora), we can expect the version extension to be always 0.0, should there be one.
> 
> It’s the Tree-sitter project that have chosen the numbers in their Makefiles, not Fedora, by the way.  https://github.com/tree-sitter/tree-sitter-json/blob/master/Makefile#L42
> 
> -- 
> Peter Oliver

Now Emacs should recognize .0 and .0.0 suffixes. I pushed the change to emacs-29.

Yuan

Re: Versioned Tree-sitter parser libraries

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 24 Apr 2023 00:00:20 -0700
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org
> 
> > On Apr 17, 2023, at 1:17 PM, Peter Oliver <p.d.oliver@mavit.org.uk> wrote:
> > 
> > On Mon, 17 Apr 2023, Yuan Fu wrote:
> > 
> >> I was suggesting that we do recognize ABI versions, but since tree-sitter grammars don’t really have a ABI version, and the version extension in filenames are just there by convention (of Fedora), we can expect the version extension to be always 0.0, should there be one.
> > 
> > It’s the Tree-sitter project that have chosen the numbers in their Makefiles, not Fedora, by the way.  https://github.com/tree-sitter/tree-sitter-json/blob/master/Makefile#L42
> > 
> > -- 
> > Peter Oliver
> 
> Now Emacs should recognize .0 and .0.0 suffixes. I pushed the change to emacs-29.

You did?  Because I don't see it when I update from Git.

Re: Versioned Tree-sitter parser libraries

[Yuan Fu]

> On Apr 24, 2023, at 4:05 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Mon, 24 Apr 2023 00:00:20 -0700
>> Cc: Eli Zaretskii <eliz@gnu.org>,
>> emacs-devel@gnu.org
>> 
>>> On Apr 17, 2023, at 1:17 PM, Peter Oliver <p.d.oliver@mavit.org.uk> wrote:
>>> 
>>> On Mon, 17 Apr 2023, Yuan Fu wrote:
>>> 
>>>> I was suggesting that we do recognize ABI versions, but since tree-sitter grammars don’t really have a ABI version, and the version extension in filenames are just there by convention (of Fedora), we can expect the version extension to be always 0.0, should there be one.
>>> 
>>> It’s the Tree-sitter project that have chosen the numbers in their Makefiles, not Fedora, by the way.  https://github.com/tree-sitter/tree-sitter-json/blob/master/Makefile#L42
>>> 
>>> -- 
>>> Peter Oliver
>> 
>> Now Emacs should recognize .0 and .0.0 suffixes. I pushed the change to emacs-29.
> 
> You did?  Because I don't see it when I update from Git.

Oops, seems it didn’t go through. It should be on there now.

Yuan

Re: Versioned Tree-sitter parser libraries

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 24 Apr 2023 10:39:57 -0700
> Cc: p.d.oliver@mavit.org.uk,
>  emacs-devel@gnu.org
> 
> >> Now Emacs should recognize .0 and .0.0 suffixes. I pushed the change to emacs-29.
> > 
> > You did?  Because I don't see it when I update from Git.
> 
> Oops, seems it didn’t go through. It should be on there now.

Thanks, I see it now.  However, this is not needed on MS-Windows, so I
#ifdef'ed it away.