[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois <pierre.langlois@gmx.com>]

[-- Attachment #1: Type: text/plain, Size: 10653 bytes --]


Andrew Tropin <andrew@trop.in> writes:

> [[PGP Signed Part:Undecided]]
> On 2023-02-10 17:02, Pierre Langlois wrote:
>
>> Pierre Langlois <pierre.langlois@gmx.com> writes:
>>
>>> [[PGP Signed Part:Undecided]]
>>> Hi Andrew, thanks for pushing this along! It's great to see things
>>> getting merged.
>>>
>>> Andrew Tropin <andrew@trop.in> writes:
>>>
>>>> [[PGP Signed Part:Undecided]]
>>>> On 2023-02-09 18:04, Andrew Tropin wrote:
>>>>
>>>>> On 2023-02-09 13:39, zimoun wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>>>
>>>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>>>
>>>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>>>> right?
>>>>>>
>>>>>>
>>>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>>
>>>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>>
>>>>>> Leaving out all the others, right?
>>>>>
>>>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>>>> addresses some things from reviews and one commit, which adds html
>>>>> grammar.
>>>>>
>>>>> The html grammar is added for the testing purposes.  It relies on
>>>>> generated parser.c and scanner.c and we will need to repackage it using
>>>>> grammar.js instead.  I'm not sure if a separate build system is needed
>>>>> for this, I guess we can just rewrite tree-sitter-grammar function,
>>>>> which generates packages as in example with tree-sitter-grammar-html:
>>>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>>>>
>>>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>>>>> read it carefully, evaluate and either introduce new build system or
>>>>> just move all needed parts to tree-sitter-grammar function.  WDYT?
>>>>> After we done with it we can package all other grammars.
>>>>
>>>> Ok, I realized that the proper build process for tree-sitter grammars is
>>>> a little harder than I expected, tree-sitter-build system make sense.  I
>>>> reviewed it, made a small change:
>>>
>>> Ah great, I was going to comment to try and push for us to keep the
>>> build system. I originally went with a template package and inheritance,
>>> but Maxime suggested moving to a build-system which ended up making the
>>> package definitions a *lot* nicer IMO (see previous discussion here
>>> https://issues.guix.gnu.org/49946#144). It also allows us to deal with
>>> grammars that depend on each other more nicely I think.
>>>
>>>>
>>>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>>>>  ;; Commentary:
>>>>  ;;
>>>>  ;; Build procedures for tree-sitter grammar packages.  This is the
>>>> -;; builder-side code, which builds on top fo the node build-system.
>>>> +;; builder-side code, which builds on top of the node build-system.
>>>>  ;;
>>>>  ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>>>>  ;; shared object.  The `tree-sitter generate' command invokes `node' in order
>>>> @@ -114,7 +114,7 @@ (define (compile-language dir)
>>>>                     "-fno-exceptions"
>>>>                     "-O2"
>>>>                     "-g"
>>>> -                   "-o" ,(string-append lib "/" lang ".so")
>>>> +                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>>>>                     ;; An additional `scanner.{c,cc}' file is sometimes
>>>>                     ;; provided.
>>>>                     ,@(cond
>>>>
>>>>
>>>> rewrote html grammar to use this build system and made it work with
>>>> built-in treesit package.  Also, tried examples of c and cpp grammars
>>>> from patches in this thread.
>>>>
>>>> If you ok with it, I'll push the build system to master and update the
>>>> html grammar accordingly.
>>
>> Oh, I forgot to say, this change to the build system LGTM! I'm really
>> happy to see it merged soon :-). The path change will probably break the
>> emacs-28-based tree-sitter support, but that's OK, it's better for the
>> build-system to be made to target emacs 29's builtin support. I'm sure I
>> can work around for emacs 28.
>
> Actually, I think we can build grammars with both names, just providing
> two .so files instead of one.  If you won't find a better workaround we
> can go this way.

That's no problem, I was able to work-around it quite easily.  The
emacs-tree-sitter-langs package expects all the grammars supported to be
in a single directory, so we have to bundle them.  So we can quite
easily create symlinks from the bundle with the expected names,
something like:

--8<---------------cut here---------------start------------->8---
(define-public emacs-tree-sitter-langs-grammar-bundle
  (package
    (name "emacs-tree-sitter-langs-grammar-bundle")
    (source #f)
    (version (package-version tree-sitter))
    (build-system trivial-build-system)
    (inputs
     ;; FIXME: Support for some languages is still left to package.
     (list tree-sitter-bash
           tree-sitter-c
           tree-sitter-c-sharp
           tree-sitter-cpp
           tree-sitter-css
           tree-sitter-elixir
           tree-sitter-elm
           tree-sitter-go
           tree-sitter-haskell
           tree-sitter-html
           tree-sitter-java
           tree-sitter-javascript
           tree-sitter-json
           tree-sitter-julia
           tree-sitter-lua
           tree-sitter-ocaml
           tree-sitter-php
           tree-sitter-python
           tree-sitter-r
           tree-sitter-rust
           tree-sitter-ruby
           tree-sitter-typescript))
    (arguments
     (list #:builder
           (with-imported-modules '((guix build union)
                                    (guix build utils))
             #~(begin
                 (use-modules (ice-9 match)
                              (srfi srfi-1)
                              (guix build union)
                              (guix build utils))
                 (mkdir-p #$output)
                 (for-each
                  (lambda (lib)
                    (symlink lib
                             (string-append #$output "/"
                                            (substring (basename lib)
                                                       (string-length "libtree-sitter-")))))
                  (append-map (match-lambda
                                ((name directory)
                                 (find-files directory "libtree-sitter-.*\\.so$")))
                              '#$(package-inputs this-package)))))))
    (synopsis #f)
    (description #f)
    (home-page #f)
    (license #f)))
--8<---------------cut here---------------end--------------->8---

Giving us:

--8<---------------cut here---------------start------------->8---
/gnu/store/fr9h59wgn55ilfifvm5df7xzxfwh38pc-emacs-tree-sitter-langs-grammar-bundle-0.20.7
├── bash.so -> /gnu/store/9qzvcdlpryjl44klfq0i2liqf6lsp5jq-tree-sitter-bash-0.19.0/lib/tree-sitter/libtree-sitter-bash.so
├── cpp.so -> /gnu/store/my02kq2dr6h48nmrl2dnfnm5rggx837w-tree-sitter-cpp-0.20.0-1.5ead1e2/lib/tree-sitter/libtree-sitter-cpp.so
├── c_sharp.so -> /gnu/store/mp8jvbhx5xlgj5bxa52zhmk5x8i7md5h-tree-sitter-c-sharp-0.19.1/lib/tree-sitter/libtree-sitter-c_sharp.so
├── c.so -> /gnu/store/y9ln76yx68q495vx8bnqdy87x7k8ihy5-tree-sitter-c-0.20.2/lib/tree-sitter/libtree-sitter-c.so
├── css.so -> /gnu/store/fwji59sdwvqpqyb94m55qw1ak92cmys9-tree-sitter-css-0.19.0/lib/tree-sitter/libtree-sitter-css.so
├── elixir.so -> /gnu/store/f6ismnxf7hlh1sq87zzarw56avvrzsl9-tree-sitter-elixir-0.19.0-1.de20391/lib/tree-sitter/libtree-sitter-elixir.so
├── elm.so -> /gnu/store/56zjl2ljvba3z1j6zfmpfa8mq19a3g3a-tree-sitter-elm-5.6.3/lib/tree-sitter/libtree-sitter-elm.so
├── go.so -> /gnu/store/qfy5sw6za96wkszbi21adxbxzggvjz3g-tree-sitter-go-0.19.1-1.05900fa/lib/tree-sitter/libtree-sitter-go.so
├── haskell.so -> /gnu/store/cq3chmqcb8g1nf5mzb6yhzdd6x9gvg1v-tree-sitter-haskell-0.14.0-1.e30bdfd/lib/tree-sitter/libtree-sitter-haskell.so
├── html.so -> /gnu/store/5pqfyjrg3yyvxaxidk690ffls3yb7wbi-tree-sitter-html-0.19.0/lib/tree-sitter/libtree-sitter-html.so
├── javascript.so -> /gnu/store/28s09v3dfb1c9bdkq2791z9abxnih66p-tree-sitter-javascript-0.20.0/lib/tree-sitter/libtree-sitter-javascript.so
├── java.so -> /gnu/store/i7jlqf5hbv0rhlkp4qlyc2d5ndw69dx4-tree-sitter-java-0.20.1/lib/tree-sitter/libtree-sitter-java.so
├── json.so -> /gnu/store/5dv4r74p6gd9w2ncs6pjrhz5hbw47262-tree-sitter-json-0.19.0/lib/tree-sitter/libtree-sitter-json.so
├── julia.so -> /gnu/store/5k37g1sdsllgh64p0w6ggabsni6jqlkr-tree-sitter-julia-0.19.0/lib/tree-sitter/libtree-sitter-julia.so
├── lua.so -> /gnu/store/nqzn4a6kgb2rx4y44pxdm4sqf3pzcpz1-tree-sitter-lua-0.0.14/lib/tree-sitter/libtree-sitter-lua.so
├── ocaml_interface.so -> /gnu/store/3h7krcj3xxclirb8afxh65ipabw1821l-tree-sitter-ocaml-0.19.0-1.0348562/lib/tree-sitter/libtree-sitter-ocaml_interface.so
├── ocaml.so -> /gnu/store/3h7krcj3xxclirb8afxh65ipabw1821l-tree-sitter-ocaml-0.19.0-1.0348562/lib/tree-sitter/libtree-sitter-ocaml.so
├── php.so -> /gnu/store/ymxf5m8jhihbrag7v2pghgydj3byp7wh-tree-sitter-php-0.19.0-1.435fa00/lib/tree-sitter/libtree-sitter-php.so
├── python.so -> /gnu/store/ggmzicwfxb7gz1rr9lfkx8cak62bfw7v-tree-sitter-python-0.19.1-1.ed0fe62/lib/tree-sitter/libtree-sitter-python.so
├── r.so -> /gnu/store/y9dxbnbb5dyf0rq3kpar7ip4w3lq6sb9-tree-sitter-r-0.0.1-1.80efda5/lib/tree-sitter/libtree-sitter-r.so
├── ruby.so -> /gnu/store/ky8n30dw16ck6byaqnhbf9ib7xp7j0yw-tree-sitter-ruby-0.20.0/lib/tree-sitter/libtree-sitter-ruby.so
├── rust.so -> /gnu/store/25zdpwrgq1xibhv7xpg64i4g71xah6g1-tree-sitter-rust-0.20.1/lib/tree-sitter/libtree-sitter-rust.so
├── tsx.so -> /gnu/store/wqpcphz855yjrginwqrymd3xzzxb8k8l-tree-sitter-typescript-0.20.1/lib/tree-sitter/libtree-sitter-tsx.so
└── typescript.so -> /gnu/store/wqpcphz855yjrginwqrymd3xzzxb8k8l-tree-sitter-typescript-0.20.1/lib/tree-sitter/libtree-sitter-typescript.so
--8<---------------cut here---------------end--------------->8---

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]