Getting tree-sitter grammars in Guix

Getting tree-sitter grammars in Guix

[Demis Balbach]

[-- Attachment #1: Type: text/plain, Size: 3202 bytes --]


Hello, I was wondering what the "correct" way would be to get grammars
for the tree-sitter library available in Guix when using tree-sitter
with Emacs 29+.

There is https://github.com/emacs-tree-sitter/tree-sitter-langs, but I
don't think it has been packaged in Guix yet. I started packaging the
grammar for JS/TS:

--8<---------------cut here---------------start------------->8---
(define-public tree-sitter-javascript
  (let ((commit "7a29d06274b7cf87d643212a433d970b73969016")
        (revision "0")
        (version "0.20.0"))
    (package
     (name "tree-sitter-javascript")
     (version (git-version version revision commit))
     (source
      (origin
       (method git-fetch)
       (uri (git-reference
             (url "https://github.com/tree-sitter/tree-sitter-javascript")
             (commit commit)))
       (sha256
        (base32 "1pk6d9g6a7bzhxmwnvfiycarcgz76wq2rgfqr0xjh7y7swfw5hvw"))
       (file-name (git-file-name name version))))
     (build-system gnu-build-system)
     (native-inputs
      (list gcc))
     (arguments
      `(#:tests?
        #f
        #:phases
        (modify-phases
         %standard-phases
         (delete 'configure)
         (delete 'install)
         (delete 'build)
         (add-after 'unpack 'create-dirs
                    (lambda _ (mkdir "lib")))
         (add-after 'create-dirs 'compile
                    (lambda* (#:key inputs outputs #:allow-other-keys)
                      (let* ((out (assoc-ref outputs "out"))
                             (gcc (string-append (assoc-ref inputs "gcc") "/bin")))
                        (copy-file "grammar.js" "src/grammar.js")
                        (with-directory-excursion
                         (string-append "./src")
                         (invoke (string-append gcc "/gcc") "-fPIC" "-c" "-I." "parser.c")
                         (invoke (string-append gcc "/gcc") "-fPIC" "-c" "-I." "scanner.c")
                         (invoke (string-append gcc "/gcc")
                                 "-fPIC" "-shared" "parser.o" "scanner.o" "-o"
                                 "libtree-sitter-javascript.so")))))
         (add-after 'compile 'symlink
                    (lambda* (#:key inputs outputs #:allow-other-keys)
                      (let* ((out (assoc-ref outputs "out"))
                             (lib (string-append out "/lib")))
                        (install-file "src/libtree-sitter-javascript.so" lib)))))))
      (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
     (synopsis "JavaScript and JSX grammar for tree-sitter.")
     (description "JavaScript and JSX grammar for tree-sitter.")
     (license license:expat))))
--8<---------------cut here---------------end--------------->8---

But I'm not sure if packaging each grammar is the correct way. I
hesitate to continue working on this because there are not other
grammars packaged in Guix (yet). With tree-sitter being built into Emacs
29+ I expected grammars to be packaged already (hence why I'm asking if
this approach is correct).

Thanks in advance!

-- 
Best regards / Mit freundlichen Grüßen,
Demis Balbach

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Getting tree-sitter grammars in Guix

[Katherine Cox-Buday]

Demis Balbach <db@minikn.xyz> writes:

> Hello, I was wondering what the "correct" way would be to get grammars
> for the tree-sitter library available in Guix when using tree-sitter
> with Emacs 29+.
>
> There is https://github.com/emacs-tree-sitter/tree-sitter-langs, but I
> don't think it has been packaged in Guix yet. I started packaging the
> grammar for JS/TS:
>
> (define-public tree-sitter-javascript
>   (let ((commit "7a29d06274b7cf87d643212a433d970b73969016")
>         (revision "0")
>         (version "0.20.0"))
>     (package
>      (name "tree-sitter-javascript")
>      (version (git-version version revision commit))
>      (source
>       (origin
>        (method git-fetch)
>        (uri (git-reference
>              (url "https://github.com/tree-sitter/tree-sitter-javascript")
>              (commit commit)))
>        (sha256
>         (base32 "1pk6d9g6a7bzhxmwnvfiycarcgz76wq2rgfqr0xjh7y7swfw5hvw"))
>        (file-name (git-file-name name version))))
>      (build-system gnu-build-system)
>      (native-inputs
>       (list gcc))
>      (arguments
>       `(#:tests?
>         #f
>         #:phases
>         (modify-phases
>          %standard-phases
>          (delete 'configure)
>          (delete 'install)
>          (delete 'build)
>          (add-after 'unpack 'create-dirs
>                     (lambda _ (mkdir "lib")))
>          (add-after 'create-dirs 'compile
>                     (lambda* (#:key inputs outputs #:allow-other-keys)
>                       (let* ((out (assoc-ref outputs "out"))
>                              (gcc (string-append (assoc-ref inputs "gcc") "/bin")))
>                         (copy-file "grammar.js" "src/grammar.js")
>                         (with-directory-excursion
>                          (string-append "./src")
>                          (invoke (string-append gcc "/gcc") "-fPIC" "-c" "-I." "parser.c")
>                          (invoke (string-append gcc "/gcc") "-fPIC" "-c" "-I." "scanner.c")
>                          (invoke (string-append gcc "/gcc")
>                                  "-fPIC" "-shared" "parser.o" "scanner.o" "-o"
>                                  "libtree-sitter-javascript.so")))))
>          (add-after 'compile 'symlink
>                     (lambda* (#:key inputs outputs #:allow-other-keys)
>                       (let* ((out (assoc-ref outputs "out"))
>                              (lib (string-append out "/lib")))
>                         (install-file "src/libtree-sitter-javascript.so" lib)))))))
>       (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
>      (synopsis "JavaScript and JSX grammar for tree-sitter.")
>      (description "JavaScript and JSX grammar for tree-sitter.")
>      (license license:expat))))
>
> But I'm not sure if packaging each grammar is the correct way. I
> hesitate to continue working on this because there are not other
> grammars packaged in Guix (yet). With tree-sitter being built into
> Emacs 29+ I expected grammars to be packaged already (hence why I'm
> asking if this approach is correct).


This seems correct to me, but you might get more informed opinions on
guix-devel@gnu.org (CCed), since this is a packaging question.

>
> Thanks in advance!

-- 
Katherine

Getting tree-sitter grammars in Guix

[Simon Tournier]

Hi,

On Tue, 10 Jan 2023 at 18:29, Demis Balbach <db@minikn.xyz> wrote:

> Hello, I was wondering what the "correct" way would be to get grammars
> for the tree-sitter library available in Guix when using tree-sitter
> with Emacs 29+.

This raises a question about the bootstrap.

From my understanding, the inputs of Tree-sitter is the file
’grammar.js’ and then using Node.js, it generates C files.  It is often
these C files which are considered as the Tree-sitter parser.

Many Tree-sitter parsers are available [1].  For example, Bash [2].  The
grammar is described by ’grammar.js’ [3] and then using it, the file
’src/parser.c’ [4] is generated (note the size 3.46MB!).  Then, using
this file ’src/parser.c’ is generated a library which is called by the
editor tool (say Emacs tree-sitter [5]).

I am not aware if someone has tried to rebuild using Guix only from the
grammar.js to a working tree-sitter editing experience.

Well, considering the state of Node.js and Guix, it appears to me
difficult to debootstrap.   And maybe, we should accept a large
unverified seed for Tree-sitter.  Well, I do not know.

What people think?  What do we do with Tree-sitter?


1: <https://tree-sitter.github.io/tree-sitter/>
2: <https://github.com/tree-sitter/tree-sitter-bash>
3: <https://github.com/tree-sitter/tree-sitter-bash/blob/master/grammar.js>
4: <https://github.com/tree-sitter/tree-sitter-bash/blob/master/src/parser.c>
5: <https://emacs-tree-sitter.github.io/>


Cheers,
simon

Re: Getting tree-sitter grammars in Guix

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 3986 bytes --]

On 2023-01-10 18:29, Demis Balbach wrote:

> Hello, I was wondering what the "correct" way would be to get grammars
> for the tree-sitter library available in Guix when using tree-sitter
> with Emacs 29+.
>
> There is https://github.com/emacs-tree-sitter/tree-sitter-langs, but I
> don't think it has been packaged in Guix yet. I started packaging the
> grammar for JS/TS:
>
> --8<---------------cut here---------------start------------->8---
> (define-public tree-sitter-javascript
>   (let ((commit "7a29d06274b7cf87d643212a433d970b73969016")
>         (revision "0")
>         (version "0.20.0"))
>     (package
>      (name "tree-sitter-javascript")
>      (version (git-version version revision commit))
>      (source
>       (origin
>        (method git-fetch)
>        (uri (git-reference
>              (url "https://github.com/tree-sitter/tree-sitter-javascript")
>              (commit commit)))
>        (sha256
>         (base32 "1pk6d9g6a7bzhxmwnvfiycarcgz76wq2rgfqr0xjh7y7swfw5hvw"))
>        (file-name (git-file-name name version))))
>      (build-system gnu-build-system)
>      (native-inputs
>       (list gcc))
>      (arguments
>       `(#:tests?
>         #f
>         #:phases
>         (modify-phases
>          %standard-phases
>          (delete 'configure)
>          (delete 'install)
>          (delete 'build)
>          (add-after 'unpack 'create-dirs
>                     (lambda _ (mkdir "lib")))
>          (add-after 'create-dirs 'compile
>                     (lambda* (#:key inputs outputs #:allow-other-keys)
>                       (let* ((out (assoc-ref outputs "out"))
>                              (gcc (string-append (assoc-ref inputs "gcc") "/bin")))
>                         (copy-file "grammar.js" "src/grammar.js")
>                         (with-directory-excursion
>                          (string-append "./src")
>                          (invoke (string-append gcc "/gcc") "-fPIC" "-c" "-I." "parser.c")
>                          (invoke (string-append gcc "/gcc") "-fPIC" "-c" "-I." "scanner.c")
>                          (invoke (string-append gcc "/gcc")
>                                  "-fPIC" "-shared" "parser.o" "scanner.o" "-o"
>                                  "libtree-sitter-javascript.so")))))
>          (add-after 'compile 'symlink
>                     (lambda* (#:key inputs outputs #:allow-other-keys)
>                       (let* ((out (assoc-ref outputs "out"))
>                              (lib (string-append out "/lib")))
>                         (install-file "src/libtree-sitter-javascript.so" lib)))))))
>       (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
>      (synopsis "JavaScript and JSX grammar for tree-sitter.")
>      (description "JavaScript and JSX grammar for tree-sitter.")
>      (license license:expat))))
> --8<---------------cut here---------------end--------------->8---
>
> But I'm not sure if packaging each grammar is the correct way. I
> hesitate to continue working on this because there are not other
> grammars packaged in Guix (yet). With tree-sitter being built into Emacs
> 29+ I expected grammars to be packaged already (hence why I'm asking if
> this approach is correct).
>
> Thanks in advance!

Hi Demis!

We have some work for tree-sitter done by muradm and he made an example
of the function, which returns a package for a tree-sitter grammar for
particular language, I think that we want to do something like this:
https://yhetil.org/guix-devel/87sfhvm927.fsf@encom.net/3-a.txt

Thank you very much for your snippet, I'll experiment a little more with
tree-sitter locally and report back with more details.

Also, I applied a patch, which enables built-in treesitter in Emacs
from this thread:
https://yhetil.org/guix-patches/87a6217tw5.fsf@riseup.net/

So I hope we will get tree-sitter support in Guix soon!

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Getting tree-sitter grammars in Guix

[Demis Balbach]

[-- Attachment #1: Type: text/plain, Size: 4440 bytes --]

On 2023-01-22 16:38, Simon Tournier wrote:

> Hi,
>
> On Tue, 10 Jan 2023 at 18:29, Demis Balbach <db@minikn.xyz> wrote:
>
>> Hello, I was wondering what the "correct" way would be to get grammars
>> for the tree-sitter library available in Guix when using tree-sitter
>> with Emacs 29+.
>
> This raises a question about the bootstrap.
>
> From my understanding, the inputs of Tree-sitter is the file
> ’grammar.js’ and then using Node.js, it generates C files.  It is often
> these C files which are considered as the Tree-sitter parser.
>
> Many Tree-sitter parsers are available [1].  For example, Bash [2].  The
> grammar is described by ’grammar.js’ [3] and then using it, the file
> ’src/parser.c’ [4] is generated (note the size 3.46MB!).  Then, using
> this file ’src/parser.c’ is generated a library which is called by the
> editor tool (say Emacs tree-sitter [5]).
>
> I am not aware if someone has tried to rebuild using Guix only from the
> grammar.js to a working tree-sitter editing experience.
>
> Well, considering the state of Node.js and Guix, it appears to me
> difficult to debootstrap.   And maybe, we should accept a large
> unverified seed for Tree-sitter.  Well, I do not know.
>
> What people think?  What do we do with Tree-sitter?
>
>
> 1: <https://tree-sitter.github.io/tree-sitter/>
> 2: <https://github.com/tree-sitter/tree-sitter-bash>
> 3: <https://github.com/tree-sitter/tree-sitter-bash/blob/master/grammar.js>
> 4: <https://github.com/tree-sitter/tree-sitter-bash/blob/master/src/parser.c>
> 5: <https://emacs-tree-sitter.github.io/>
>
>
> Cheers,
> simon

Hello Simon,

Thanks for the input. I came across this:

https://lists.gnu.org/archive/html/guix-devel/2022-12/msg00073.html

Maybe this can be used as a foundation. I tested it locally and it works
fine, but I don't know how it needs to be altered in order to be pushed
upstream.

On that note, how would one package an emacs package that leverages
tree-sitter? I want to package https://github.com/isamert/jsdoc.el which
requires tree-sitter. I wrote a package definition where I substituted

My understanding is that I need to provide emacs with tree-sitter
support as an input for that to work. Here's a package definition I wrote:

--8<---------------cut here---------------start------------->8---
(define-public emacs-jsdoc
  (let ((commit "2e7c02ff2dc422bc21c405bd90a7092c2f599630")
        (revision "0")
        (version "0.2"))
    (package
     (name "emacs-jsdoc")
     (version (git-version version revision commit))
     (source
      (origin
       (method git-fetch)
       (uri (git-reference
             (url "https://github.com/isamert/jsdoc.el")
             (commit commit)))
       (sha256
        (base32 "07sz5lpyqv7ixvnnzdfjkj7f0ykdz31lkljp13pvlf36a6sff4rc"))
       (file-name (git-file-name name version))))
     (build-system emacs-build-system)
     (arguments
      `(#:phases
        (modify-phases
         %standard-phases
         (add-after 'unpack 'patch-bin
                    (lambda* (#:key inputs outputs #:allow-other-keys)
                      (substitute* "jsdoc.el"
                                   (("'tree-sitter") "'treesit")))))))
     (inputs
      (list
       ;; tree-sitter
       emacs-next-pgtk-stable
       emacs-s
       emacs-dash))
     (home-page "https://github.com/isamert/jsdoc.el")
     (synopsis "Inserts JSDoc function comments/typedefs easily.")
     (description "Inserts JSDoc function comments/typedefs easily.
It also tries to infer types by itself while doing that.
Type inference is quite primitive.")
     (license license:gpl3+))))     
--8<---------------cut here---------------end--------------->8---

My understanding is that I need to provide emacs with tree-sitter
support as an input for this to work, which I did, but it'll fail with

--8<---------------cut here---------------start------------->8---
Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "treesit")
--8<---------------cut here---------------end--------------->8---

Maybe someone can help me here. I tried looking at other package
definitions, but I don't know if there are any emacs packages that
require tree-sitter packaged in Guix yet.

Thanks in advance.

-- 
Best regards / Mit freundlichen Grüßen,
Demis Balbach

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Getting tree-sitter grammars in Guix

[Simon Tournier]

Hi,

On sam., 04 févr. 2023 at 20:27, Demis Balbach <db@minikn.xyz> wrote:

> My understanding is that I need to provide emacs with tree-sitter
> support as an input for this to work, which I did, but it'll fail with
>
> --8<---------------cut here---------------start------------->8---
> Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "treesit")
> --8<---------------cut here---------------end--------------->8---
>
> Maybe someone can help me here. I tried looking at other package
> definitions, but I don't know if there are any emacs packages that
> require tree-sitter packaged in Guix yet.

Could you share your definition of Emacs variant allowing tree-sitter?

Please note it is not clear for me if the tree-sitter parsers should be
provided by Guix since 1. they are auto-generated and so it is against
the effort to debootstrap and 2. they are often very large.

Cheers,
simon

Re: Getting tree-sitter grammars in Guix

[Pierre Langlois]

Hi all!

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On sam., 04 févr. 2023 at 20:27, Demis Balbach <db@minikn.xyz> wrote:
>
>> My understanding is that I need to provide emacs with tree-sitter
>> support as an input for this to work, which I did, but it'll fail with
>>
>> --8<---------------cut here---------------start------------->8---
>> Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "treesit")
>> --8<---------------cut here---------------end--------------->8---
>>
>> Maybe someone can help me here. I tried looking at other package
>> definitions, but I don't know if there are any emacs packages that
>> require tree-sitter packaged in Guix yet.
>
> Could you share your definition of Emacs variant allowing tree-sitter?
>
> Please note it is not clear for me if the tree-sitter parsers should be
> provided by Guix since 1. they are auto-generated and so it is against
> the effort to debootstrap and 2. they are often very large.

Seeing this discussion just now, you'll probably want to take a look at
the series I've been working on which addresses this (at least I hope
so, reviews on the source-only bootstrapping welcome!).

https://issues.guix.gnu.org/49946#215

I still need to work on it and address feedback, however I've not had
any time for Guix the last couple of months.

Hope this helps avoid work duplication! If people need this work in
sonner than later and I'm not able to see it through in time, I'm happy
for someone else to pick it up. Although hopefully I'll have some time
next weekend.

Thanks!
Pierre

>
> Cheers,
> simon

Re: Getting tree-sitter grammars in Guix

[zimoun]

Hi Pierre,

On Mon, 06 Feb 2023 at 21:01, Pierre Langlois <pierre.langlois@gmx.com> wrote:

> https://issues.guix.gnu.org/49946#215

Oh cool, this tree-sitter-build-system looks great!  Thanks.

Sorry, I have overlooked this patch (and the Tree-sitter story :-)).

Cheers,
simon

Re: Getting tree-sitter grammars in Guix

[Demis Balbach]

[-- Attachment #1: Type: text/plain, Size: 1640 bytes --]

On 2023-02-06 18:25, Simon Tournier wrote:

> Hi,
>
> On sam., 04 févr. 2023 at 20:27, Demis Balbach <db@minikn.xyz> wrote:
>
>> My understanding is that I need to provide emacs with tree-sitter
>> support as an input for this to work, which I did, but it'll fail with
>>
>> --8<---------------cut here---------------start------------->8---
>> Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "treesit")
>> --8<---------------cut here---------------end--------------->8---
>>
>> Maybe someone can help me here. I tried looking at other package
>> definitions, but I don't know if there are any emacs packages that
>> require tree-sitter packaged in Guix yet.
>
> Could you share your definition of Emacs variant allowing tree-sitter?
>
> Please note it is not clear for me if the tree-sitter parsers should be
> provided by Guix since 1. they are auto-generated and so it is against
> the effort to debootstrap and 2. they are often very large.
>
> Cheers,
> simon

Hello Simon,

as far as I understand emacs is already packaged with tree-sitter
support in Guix (I didn't know this when I wrote my last message).

Regarding packaging emacs packages that leverage tree-sitter, one just
has to provide the correct emacs version as an argument, like:

--8<---------------cut here---------------start------------->8---
;; ...
(arguments
`(#:emacs ,emacs-next-pgtk
 ;; ...
))
--8<---------------cut here---------------end--------------->8---

Thanks to Andrew Troping for showing me that.

-- 
Best regards / Mit freundlichen Grüßen,
Demis Balbach

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]