Hosting tree-sitter pre-built language definitions on ELPA?

Hosting tree-sitter pre-built language definitions on ELPA?

[Yuan Fu]

Tree-sitter needs language definitions to parse different languages. These language definitions come in the form of dynamic libraries. We can’t bundle them with Emacs since their version must match that of tree-sitter library, and we don’t bundle the tree-sitter library; also they are machine-dependent.

I wonder if we can host pre-built language definitions for languages we provide tree-sitter support OOTB on ELPA, so users can easily download them?

Yuan

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Po Lu]

Yuan Fu <casouri@gmail.com> writes:

> Tree-sitter needs language definitions to parse different
> languages. These language definitions come in the form of dynamic
> libraries. We can’t bundle them with Emacs since their version must
> match that of tree-sitter library, and we don’t bundle the tree-sitter
> library; also they are machine-dependent.

If that's a problem, perhaps we could import the tree-sitter library
into the Emacs tree, like the XMenu library currently is.

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Yuan Fu]

> On Oct 1, 2022, at 10:15 PM, Po Lu <luangruo@yahoo.com> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>> Tree-sitter needs language definitions to parse different
>> languages. These language definitions come in the form of dynamic
>> libraries. We can’t bundle them with Emacs since their version must
>> match that of tree-sitter library, and we don’t bundle the tree-sitter
>> library; also they are machine-dependent.
> 
> If that's a problem, perhaps we could import the tree-sitter library
> into the Emacs tree, like the XMenu library currently is.

Tree-sitter is designed to be embedded, so it’s small and has no dependencies. But tree-sitter could release a new version and then language definitions now uses that new version, while Emacs is still using the old version of tree-sitter. IMHO it is best to keep them (tree-sitter library and language definitions) external since that’ll be easier to maintain, and Emacs don’t depend on tree-sitter to function.

Yuan

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel <emacs-devel@gnu.org>
> Date: Sun, 02 Oct 2022 13:15:08 +0800
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
> > Tree-sitter needs language definitions to parse different
> > languages. These language definitions come in the form of dynamic
> > libraries. We can’t bundle them with Emacs since their version must
> > match that of tree-sitter library, and we don’t bundle the tree-sitter
> > library; also they are machine-dependent.
> 
> If that's a problem, perhaps we could import the tree-sitter library
> into the Emacs tree, like the XMenu library currently is.

Again, I don't see why tree-sitter should be handled differently from
other optional libraries.

We might discover good reasons for that after we gain some actual
experience, but up-front I see no reason to assume any special issues.
IOW, I think we are "putting the cart before the horse" here.

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[chad]

[-- Attachment #1: Type: text/plain, Size: 1433 bytes --]

On Sun, Oct 2, 2022 at 2:19 AM Eli Zaretskii <eliz@gnu.org> wrote:

> Again, I don't see why tree-sitter should be handled differently from
> other optional libraries.
>

I'm not sure if this is Yuan Fu's concern, but for me:

If I want to build a changed tree-sitter language definition myself, the
second-ish step is "install Node.js", and I stop there. (Whether this is a
sign of taste or curmudgeonliness is left as an exercise for the reader. :-)

On Sun, Oct 2, 2022 at 2:19 AM Eli Zaretskii <eliz@gnu.org> wrote:

> We might discover good reasons for that after we gain some actual
> experience, but up-front I see no reason to assume any special issues.
> IOW, I think we are "putting the cart before the horse" here.


This is an excellent point; it seems likely that we can gain useful
experience with tree-sitter in emacs without having pre-built a
distribution channel for version-specific binary pieces. In practice, I
suspect that the concern is responding to the current situation, where
tree-sitter experimentation is largely confined to people who already hack
on tree-sitter itself (rather than just wanting to use it), OR to people
who, in practice, download pre-compiled binaries from github. At least,
that's the situation I find myself in.

For now, I suspect that something akin to the Windows snapshots on
alpha.gnu.org might be a workable compromise if the problem actually comes
up.

Hope that helps.
~Chad

[-- Attachment #2: Type: text/html, Size: 2172 bytes --]

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sun, 2 Oct 2022 09:37:04 -0400
> Cc: Po Lu <luangruo@yahoo.com>, casouri@gmail.com, emacs-devel@gnu.org
> 
> If I want to build a changed tree-sitter language definition myself, the second-ish step is "install Node.js", and
> I stop there. (Whether this is a sign of taste or curmudgeonliness is left as an exercise for the reader. :-)

We are not talking about changing the language definitions, we are
talking about installing the ones provided by tree-sitter.  The use
case you mention is akin to your wanting to modify librsvg -- in which
case you'd need to start by installing Rust.

> For now, I suspect that something akin to the Windows snapshots on alpha.gnu.org might be a workable
> compromise if the problem actually comes up.

Windows snapshots are provided as a replacement for the GNU/Linux
distros, which are not very widespread on Windows, if they exist at
all.  If Emacs installations could be downloaded from, say, the MS
store, I doubt if we'd bother providing binaries.

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[chad]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

On Sun, Oct 2, 2022 at 9:52 AM Eli Zaretskii <eliz@gnu.org> wrote:

> We are not talking about changing the language definitions, we are
> talking about installing the ones provided by tree-sitter.  The use
> case you mention is akin to your wanting to modify librsvg -- in which
> case you'd need to start by installing Rust.
>

I _think_ I disagree about that analogy. If I want to use tree-sitter with
a language that is not already somehow installed on my computer, I need a
language definition for it. Right now, I can either install Node.js and
build it myself, or download a binary from github. Is there another option?
IIUC, the current expectation is NOT that an emacs+tree-sitter release will
bundle language definitions for every language we expect the users are
likely to want, or even for, say, the top 10 languages already supported by
emacs outside of treesit. Is that understanding incorrect?

~Chad

[-- Attachment #2: Type: text/html, Size: 1274 bytes --]

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Sun, 2 Oct 2022 12:36:23 -0400
> Cc: luangruo@yahoo.com, casouri@gmail.com, emacs-devel@gnu.org
> 
> On Sun, Oct 2, 2022 at 9:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  We are not talking about changing the language definitions, we are
>  talking about installing the ones provided by tree-sitter.  The use
>  case you mention is akin to your wanting to modify librsvg -- in which
>  case you'd need to start by installing Rust.
> 
> I _think_ I disagree about that analogy. If I want to use tree-sitter with a language that is not already somehow
> installed on my computer, I need a language definition for it. Right now, I can either install Node.js and build it
> myself, or download a binary from github. Is there another option?

Yes, you can download the C source of the language definition and
compile it yourself.

> IIUC, the current expectation is NOT that
> an emacs+tree-sitter release will bundle language definitions for every language we expect the users are
> likely to want, or even for, say, the top 10 languages already supported by emacs outside of treesit. Is that
> understanding incorrect? 

Yes.  I don't think it's fair to expect the Emacs project to provide
such degree of hand-holding.  It would place too much burden on our
insufficient resources.

If downloading and building the tree-sitter language definitions is
too much for many users, I expect the Emacs distros to provide
prebuilt packages for that.

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Yuan Fu]

> On Oct 2, 2022, at 10:06 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: chad <yandros@gmail.com>
>> Date: Sun, 2 Oct 2022 12:36:23 -0400
>> Cc: luangruo@yahoo.com, casouri@gmail.com, emacs-devel@gnu.org
>> 
>> On Sun, Oct 2, 2022 at 9:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> We are not talking about changing the language definitions, we are
>> talking about installing the ones provided by tree-sitter.  The use
>> case you mention is akin to your wanting to modify librsvg -- in which
>> case you'd need to start by installing Rust.
>> 
>> I _think_ I disagree about that analogy. If I want to use tree-sitter with a language that is not already somehow
>> installed on my computer, I need a language definition for it. Right now, I can either install Node.js and build it
>> myself, or download a binary from github. Is there another option?
> 
> Yes, you can download the C source of the language definition and
> compile it yourself.

Yeah, for the record, you don’t need nodejs to compile the language definitions, only a C compiler. (Sometimes a C++ compiler.)

> 
>> IIUC, the current expectation is NOT that
>> an emacs+tree-sitter release will bundle language definitions for every language we expect the users are
>> likely to want, or even for, say, the top 10 languages already supported by emacs outside of treesit. Is that
>> understanding incorrect? 
> 
> Yes.  I don't think it's fair to expect the Emacs project to provide
> such degree of hand-holding.  It would place too much burden on our
> insufficient resources.
> 
> If downloading and building the tree-sitter language definitions is
> too much for many users, I expect the Emacs distros to provide
> prebuilt packages for that.

I brought this up because someone suggested to bundle language definitions with Emacs for better access. If we decide to not provide hand-holding, I don’t mind doing less work ;-)

Yuan

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sun, 2 Oct 2022 15:17:35 -0700
> Cc: chad <yandros@gmail.com>,
>  Po Lu <luangruo@yahoo.com>,
>  emacs-devel@gnu.org
> 
> > Yes.  I don't think it's fair to expect the Emacs project to provide
> > such degree of hand-holding.  It would place too much burden on our
> > insufficient resources.
> > 
> > If downloading and building the tree-sitter language definitions is
> > too much for many users, I expect the Emacs distros to provide
> > prebuilt packages for that.
> 
> I brought this up because someone suggested to bundle language definitions with Emacs for better access. If we decide to not provide hand-holding, I don’t mind doing less work ;-)

I think at least at first we should not make this our job.  We could
revisit that decision later, based on actual experience and user
feedback, but up front I don't see why we should do this as part of
the Emacs project.

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Yuan Fu]

> On Oct 3, 2022, at 8:56 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Sun, 2 Oct 2022 15:17:35 -0700
>> Cc: chad <yandros@gmail.com>,
>> Po Lu <luangruo@yahoo.com>,
>> emacs-devel@gnu.org
>> 
>>> Yes.  I don't think it's fair to expect the Emacs project to provide
>>> such degree of hand-holding.  It would place too much burden on our
>>> insufficient resources.
>>> 
>>> If downloading and building the tree-sitter language definitions is
>>> too much for many users, I expect the Emacs distros to provide
>>> prebuilt packages for that.
>> 
>> I brought this up because someone suggested to bundle language definitions with Emacs for better access. If we decide to not provide hand-holding, I don’t mind doing less work ;-)
> 
> I think at least at first we should not make this our job.  We could
> revisit that decision later, based on actual experience and user
> feedback, but up front I don't see why we should do this as part of
> the Emacs project.

I agree.

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[chad]

[-- Attachment #1: Type: text/plain, Size: 303 bytes --]

On Sun, Oct 2, 2022 at 1:06 PM Eli Zaretskii <eliz@gnu.org> wrote:

> [...]
> Yes, you can download the C source of the language definition and
> compile it yourself.
>

Ah, yes; I had forgotten that it went through C. Mea culpa.

Thanks for taking the time to clarify/remind me. I appreciate it.
~Chad

[-- Attachment #2: Type: text/html, Size: 665 bytes --]

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: chad <yandros@gmail.com>
>> Date: Sun, 2 Oct 2022 09:37:04 -0400
>> Cc: Po Lu <luangruo@yahoo.com>, casouri@gmail.com, emacs-devel@gnu.org
>> 
>> If I want to build a changed tree-sitter language definition myself,
>> the second-ish step is "install Node.js", and
>> I stop there. (Whether this is a sign of taste or curmudgeonliness
>> is left as an exercise for the reader. :-)
>
> We are not talking about changing the language definitions, we are
> talking about installing the ones provided by tree-sitter.  

Language definitions are not "provided by tree-sitter". They are
provided by other projects that use/support tree-sitter. Such projects
typically provide language parsers for several IDEs. And they typically
provide binary installs.

-- 
-- Stephe

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: chad <yandros@gmail.com>,  luangruo@yahoo.com,  casouri@gmail.com,
>   emacs-devel@gnu.org
> Date: Wed, 12 Oct 2022 03:54:05 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: chad <yandros@gmail.com>
> >> Date: Sun, 2 Oct 2022 09:37:04 -0400
> >> Cc: Po Lu <luangruo@yahoo.com>, casouri@gmail.com, emacs-devel@gnu.org
> >> 
> >> If I want to build a changed tree-sitter language definition myself,
> >> the second-ish step is "install Node.js", and
> >> I stop there. (Whether this is a sign of taste or curmudgeonliness
> >> is left as an exercise for the reader. :-)
> >
> > We are not talking about changing the language definitions, we are
> > talking about installing the ones provided by tree-sitter.  
> 
> Language definitions are not "provided by tree-sitter". They are
> provided by other projects that use/support tree-sitter.

OK, "provided by the tree-sitter site".

> Such projects typically provide language parsers for several
> IDEs. And they typically provide binary installs.

The ones available from the tree-sitter site are in source form.

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 1 Oct 2022 21:47:11 -0700
> 
> Tree-sitter needs language definitions to parse different languages. These language definitions come in the form of dynamic libraries. We can’t bundle them with Emacs since their version must match that of tree-sitter library, and we don’t bundle the tree-sitter library; also they are machine-dependent.
> 
> I wonder if we can host pre-built language definitions for languages we provide tree-sitter support OOTB on ELPA, so users can easily download them?

We could, but how is this different from any other optional library
Emacs uses?  Users who want to build Emacs with an optional library
are expected to install that library from whatever 3rd-party sources
that make them available.  What is different here?  Is it really that
hard to download them from the official tree-sitter site?

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Stefan Monnier]

Yuan Fu [2022-10-01 21:47:11] wrote:
> I wonder if we can host pre-built language definitions for languages we
> provide tree-sitter support OOTB on ELPA, so users can easily download them?

[ Coming late to this discussion, sorry.  ]

We could consider it, but as Eli mentioned, it's not clear what we'll
need, so I think it's best to wait.  FWIW, there are similar questions
for Emacs modules distributed via (Non)GNU ELPA (such as the `libpq`
module), and currently we just don't do it.

Note also that the native ELisp compiler brings GCC as a dependency of
Emacs, so if that becomes the norm, we might prefer to just build the
language definition locally from its C version (which is not actually
its source), and maybe we'll want to host *those* i.e. the pre-digested
language definitions in their C form.

Hosting them on a gnu.org machine would make it easier to make their
installation (semi)automatic since we usually don't like to download
random code directly from third parties with which we have no
shared agreement.  But if the language definitions can be installed as
Debian packages, it might not be necessary at all.


        Stefan

Re: Hosting tree-sitter pre-built language definitions on ELPA?

[Yuan Fu]

> On Oct 11, 2022, at 1:51 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
> Yuan Fu [2022-10-01 21:47:11] wrote:
>> I wonder if we can host pre-built language definitions for languages we
>> provide tree-sitter support OOTB on ELPA, so users can easily download them?
> 
> [ Coming late to this discussion, sorry.  ]
> 
> We could consider it, but as Eli mentioned, it's not clear what we'll
> need, so I think it's best to wait.  FWIW, there are similar questions
> for Emacs modules distributed via (Non)GNU ELPA (such as the `libpq`
> module), and currently we just don't do it.
> 
> Note also that the native ELisp compiler brings GCC as a dependency of
> Emacs, so if that becomes the norm, we might prefer to just build the
> language definition locally from its C version (which is not actually
> its source), and maybe we'll want to host *those* i.e. the pre-digested
> language definitions in their C form.
> 
> Hosting them on a gnu.org machine would make it easier to make their
> installation (semi)automatic since we usually don't like to download
> random code directly from third parties with which we have no
> shared agreement.  But if the language definitions can be installed as
> Debian packages, it might not be necessary at all.

Sounds like a plan. Before we notice, they might have all the languages bundled for emacs build ;-)

Yuan