Where to place third-party C source code?

Where to place third-party C source code?

[Jorge Araya Navarro]

Hello!

I was wondering if placing third-party C source code that is used in any feature I would like to implement in the Emacs project is "against the rules", sort of speak.

I ask because I would like to attempt to integrate tree-sitter (a parser generator tool and an incremental parsing library; I'm aware there are a couple of projects that gives Emacs lisp bindings for this project, too; I'm just wondering how the world could be if this thing was shipped in Vanilla Emacs), according to its documentation[1]:

    [...] you can use the library in a larger project by adding one source file to the project. This source file needs three directories to be in the include path when compiled:

    source file:
    - tree-sitter/lib/src/lib.c 

    include directories: 
    - tree-sitter/lib/src
    - tree-sitter/lib/include
    - tree-sitter/lib/utf8proc

The instructions seem straightforward but reading the docs included with the Emacs source code there is no mention wherever this is "fair game" (or not) and where should such code be placed

[1]: https://tree-sitter.github.io/tree-sitter/using-parsers

Re: Where to place third-party C source code?

[Eli Zaretskii]

> Date: Fri, 27 Sep 2019 21:29:52 +0000 (UTC)
> From: Jorge Araya Navarro <jorgejavieran@yahoo.com.mx>
> 
> I was wondering if placing third-party C source code that is used in any feature I would like to implement in the Emacs project is "against the rules", sort of speak.

I don't understand the question.  Any feature supported by Emacs that
needs C-level support has some C code in one of the Emacs C source
files.  There's no "third-party" code, everything is part of Emacs
proper.

>     [...] you can use the library in a larger project by adding one source file to the project. This source file needs three directories to be in the include path when compiled:
> 
>     source file:
>     - tree-sitter/lib/src/lib.c 
> 
>     include directories: 
>     - tree-sitter/lib/src
>     - tree-sitter/lib/include
>     - tree-sitter/lib/utf8proc

I don't see why we would need this method, since tree-sitter is a
library, and Emacs can be linked against that library.  What you quote
is an alternative method, but why would we need such an alternative?

Of course, this is all putting the wagon ahead of the horse: we should
first discuss whether we want to have Emacs be able to link to that
library and provide the related features.  An alternative would be to
have an unbundled module that uses the Emacs module API.

Re: Where to place third-party C source code?

[Jorge Javier Araya Navarro]

[-- Attachment #1: Type: text/plain, Size: 2655 bytes --]



El sábado 28 de septiembre del 2019 a las 0031 horas, Eli Zaretskii escribió:

>> Date: Fri, 27 Sep 2019 21:29:52 +0000 (UTC)
>> From: Jorge Araya Navarro <jorgejavieran@yahoo.com.mx>
>>
>> I was wondering if placing third-party C source code that is used in any feature I would like to implement in the Emacs project is "against the rules", sort of speak.
>
> I don't understand the question.  Any feature supported by Emacs that
> needs C-level support has some C code in one of the Emacs C source
> files.  There's no "third-party" code, everything is part of Emacs
> proper.
>
>>     [...] you can use the library in a larger project by adding one source file to the project. This source file needs three directories to be in the include path when compiled:
>>
>>     source file:
>>     - tree-sitter/lib/src/lib.c
>>
>>     include directories:
>>     - tree-sitter/lib/src
>>     - tree-sitter/lib/include
>>     - tree-sitter/lib/utf8proc
>
> I don't see why we would need this method, since tree-sitter is a
> library, and Emacs can be linked against that library.  What you quote
> is an alternative method, but why would we need such an alternative?

Well, yes, I realized that adding an option to configure.ac would allow the compiler to find the
source code of Tree Sitter (like `--with-tree-sitter=/some/path/tree-sitter' or who knows)

> Of course, this is all putting the wagon ahead of the horse: we should
> first discuss whether we want to have Emacs be able to link to that
> library and provide the related features.  An alternative would be to
> have an unbundled module that uses the Emacs module API.

Ah, yes. There is one project that provides tree-sitter like a dynamic module using the Emacs module
API[1], but my concern is: why should vanilla Emacs require their final users to download a bunch of
packages to make the user experience better when we could, like, literally, provide them from the
get-go? IIRC one pain-point of Emacs for (new?) users is how much configuration is needed to have a
better editing experience.

We could leverage projects like tree-sitter to improve the user experience in Emacs out-of-the-box,
integrating tree-sitter with Emacs and ship the grammars of some programming languages that Emacs is
already shipped with (like Python and JavaScript) would improve the experience of editing code in
those languages first, and second in any other language supported with a third-party elisp packages,
without mentioning what this could mean in terms of the tooling available for package authors.

[1]: https://github.com/ubolonton/emacs-tree-sitter

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Where to place third-party C source code?

[Eli Zaretskii]

> From: Jorge Javier Araya Navarro <jorgejavieran@yahoo.com.mx>
> Cc: emacs-devel@gnu.org
> Date: Sat, 28 Sep 2019 01:33:40 -0600
> 
> > I don't see why we would need this method, since tree-sitter is a
> > library, and Emacs can be linked against that library.  What you quote
> > is an alternative method, but why would we need such an alternative?
> 
> Well, yes, I realized that adding an option to configure.ac would allow the compiler to find the
> source code of Tree Sitter (like `--with-tree-sitter=/some/path/tree-sitter' or who knows)

Why would building Emacs require access to the tree-sitter's source
code, even if we decide to provide this feature in core Emacs?  Isn't
linking against a library good enough?

> > Of course, this is all putting the wagon ahead of the horse: we should
> > first discuss whether we want to have Emacs be able to link to that
> > library and provide the related features.  An alternative would be to
> > have an unbundled module that uses the Emacs module API.
> 
> Ah, yes. There is one project that provides tree-sitter like a dynamic module using the Emacs module
> API[1], but my concern is: why should vanilla Emacs require their final users to download a bunch of
> packages to make the user experience better when we could, like, literally, provide them from the
> get-go? IIRC one pain-point of Emacs for (new?) users is how much configuration is needed to have a
> better editing experience.

There's no contradiction between better experience and having this
particular feature be implemented as a module.  We've added the module
API precisely for cases like this one, where some feature needed
access to the Emacs internals on the C level.

Making this part of the Emacs core would need to write some C code in
Emacs itself, and link against the library, it still won't need any
access to the sources of tree-sitter.  We could do that, but we
need first to be sure that this is worth our while, i.e. that having
that feature through an external module will somehow be insufficient,
or if most Emacs users will want to have that.  AFAIU, none of these
reasons were brought up and justified yet.  Improving the user
experience is not of itself a sufficient reason, because every
additional feature basically does that in some sense.  We need a more
serious reason, one that will justify the additional maintenance
burden that will put on the Emacs project in the long run.

Re: Where to place third-party C source code?

[Stefan Monnier]

> Ah, yes. There is one project that provides tree-sitter like a dynamic
> module using the Emacs module API[1], but my concern is: why should
> vanilla Emacs require their final users to download a bunch of
> packages to make the user experience better when we could, like,
> literally, provide them from the get-go? IIRC one pain-point of Emacs
> for (new?) users is how much configuration is needed to have a better
> editing experience.

I'm vaguely familiar with tree-sitter but haven't spent the time
investigating the situation enough to have a clear idea of what happens
with it.  Could you clarify the points below?

1- Other than the fact that it's not available "out of the box" is there
   some other problem with the existing dynamic module?  I.e. would it
   be OK for Emacs to simply use this existing module as-is, just
   bundling it into the distribution tarball, or would it still not be
   as good as a "native" binding?

2- Say I find an SML grammar for tree-sitter, what extra
   (i.e. Emacs-specific) work do I need to do (as maintainer of
   sml-mode) in order for sml-mode users to benefit from it?  and what
   kind of features would they get from it (I assume: indentation, of
   course, and probably some navigation, anything else)?

3- How does tree-sitter compare with the LSP-route (via eglot-mode
   or lsp-mode)?


        Stefan

Re: Where to place third-party C source code?

[yyoncho]

[-- Attachment #1: Type: text/plain, Size: 343 bytes --]

Hi Stefan,


> 3- How does tree-sitter compare with the LSP-route (via eglot-mode
>    or lsp-mode)?
>

lsp protocol is not going to support full-featured highlighting but only
semantic
because it won't be fast enough.

Related: https://github.com/microsoft/vscode/issues/77140 and
https://github.com/Microsoft/vscode/issues/585

Thanks,
Ivan

[-- Attachment #2: Type: text/html, Size: 836 bytes --]

RE: Using incremental parsing in Emacs

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

在 2020年1月3日 +0800 PM11:43，arthur miller <arthur.miller@live.com>，写道：
> Microsoft has been doing this since 1990's and they ended up with "language server protocol".  Maybe Emacs core could implement some kind of support for implementing lsp:s or implement some better support to interchange data with existing lsp:s.
>

It seems that they(VSCode team) are also trying something like the tree-sitter. I forward another discussion to you.
在 2019年12月27日 +0800 AM12:52，yyoncho <yyoncho@gmail.com>，写道：
> > Hi Stefan,
> >
> > > 3- How does tree-sitter compare with the LSP-route (via eglot-mode
> > >    or lsp-mode)?
> >
> > lsp protocol is not going to support full-featured highlighting but only semantic
> > because it won't be fast enough.
> >
> > Related: https://github.com/microsoft/vscode/issues/77140 and
> > https://github.com/Microsoft/vscode/issues/585
> >
> > Thanks,
> > Ivan

[-- Attachment #2: Type: text/html, Size: 2494 bytes --]

Re: Using incremental parsing in Emacs

[Tobias Bading]

[-- Attachment #1: Type: text/plain, Size: 848 bytes --]

> On 4. Jan 2020, at 04:26, HaiJun Zhang <netjune@outlook.com> wrote:
> [...]
> 在 2019年12月27日 +0800 AM12:52，yyoncho <yyoncho@gmail.com>，写道：
>> [...]
>> lsp protocol is not going to support full-featured highlighting but only semantic
>> because it won't be fast enough. 
>> 
>> Related: https://github.com/microsoft/vscode/issues/77140 and 
>> https://github.com/Microsoft/vscode/issues/585

I’m not 100% sure because I don’t like too colorful source code and it’s been a while since I did the configuration, but I’m using the awesome
https://github.com/MaskRay/ccls and
https://github.com/MaskRay/emacs-ccls
(in combination with lsp-mode)
on a daily basis and I’m quite sure those are able to fontify member variables and what not.

Mickeysoft decisions never stopped anyone...

Happy hacking,
Tobias

[-- Attachment #2: Type: text/html, Size: 2052 bytes --]

Re: Using incremental parsing in Emacs

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

What is VSCode, and how does it relate to Emacs development?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Using incremental parsing in Emacs

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sat, 04 Jan 2020 18:48:48 -0500
> Cc: eliz@gnu.org, emacs-devel@gnu.org, arthur.miller@live.com,
>  phillip.lord@russet.org.uk
> 
> What is VSCode, and how does it relate to Emacs development?

It's another programming editor, developed by Microsoft.

  https://en.wikipedia.org/wiki/Visual_Studio_Code