[bug#49946] [PATCH 00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

[bug#49946] [PATCH 00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 5333 bytes --]

Hi Guix!

Here's a patch series that adds support for the Tree-sitter library,
command-line tool and its support in emacs:
https://tree-sitter.github.io/tree-sitter

The series in split into multiple parts:

1. Add the tree-sitter package, which is both a command-line tool for
   generating parsers and a runtime library.  The runtime is written in
   C but the tooling in Rust.

  * gnu: rust-chunked-transfer: Update to 1.4.0.
  * gnu: Add rust-fdlimit.
  * gnu: Add rust-html-escape.
  * gnu: Add rust-smallbitvec.
  * gnu: rust-tiny-http: Update to 0.8.
  * gnu: rust-spin: Update to 0.7.
  * gnu: Add tree-sitter.

2. Add support for native addons to the node build-system.

  * gnu: node: Patch /usr/bin/env in node-gyp.
  * guix: node-build-system: Support compiling addons with node-gyp.

  I've not touched the node build system before so it'd be great to get
  some review here!

  The idea is that when a package has a `binding.gyp' file then there's
  native code that needs to be compiled against node.  This uses
  node-gyp, which is a build system written in python.
  Normally, there shouldn't be anything to do, the package.json file
  should invoke `node-gyp` tp compile C code.  However, `node-gyp' needs
  node headers to compile against, packaged as a tarball, which it tries
  to download.  Instead, we can run a `node-gyp --tarball <> configure'
  step to manually provide the tarball, which we can package separately
  for any given node version.

  I could have added this into the packages themselves, but I thought it
  made more sense to be a part of the build system, let me know what you
  think!

3. Packaging grammars for multiple languages.

  Tree-sitter grammars are Nodejs packages, they consist of a grammar.js
  file which the tree-sitter command-line program uses to generate C
  code for the parser/scanner.  Using npm as a packaging mechanism for
  this seems overkill, but it's true some grammars depend on each other
  (cpp on c, typescript and javascript).

  * gnu: Add node-nan.
  * gnu: Add tree-sitter-c.
  * gnu: Add tree-sitter-cpp.
  * gnu: Add tree-sitter-css.
  * gnu: Add tree-sitter-go.
  * gnu: Add tree-sitter-html.
  * gnu: Add tree-sitter-java.
  * gnu: Add tree-sitter-javascript.
  * gnu: Add tree-sitter-json.
  * gnu: Add tree-sitter-julia.
  * gnu: Add tree-sitter-php.
  * gnu: Add tree-sitter-python.
  * gnu: Add tree-sitter-rust.
  * gnu: Add tree-sitter-typescript.

4. Finally, emacs support!

  emacs-tree-sitter can be used for syntax highlighting, replacing
  font-lock.  The tree-sitter runtime library is built as a Rust crate
  and exposed to emacs using a module thanks to the rust-emacs crate.

  Then emacs-tree-sitter-core provides just the core APIs as exposed by
  the rust module, then emacs-tree-sitter integrates it with emacs'
  syntax highlighting (and probably other things).

  Finally, we bundle together the various language grammars into a
  single emacs-tree-sitter-langs package so that they can get loaded
  along with their respective major modes.  Note that there are grammars
  left to support which I've not yet managed to package, but what we
  have already is a good start I think :-). So far I've tested C++,
  Rust, Python and JavaScript and JSON.

  * gnu: Add rust-bindgen@0.56.
  * gnu: Add rust-tree-sitter.
  * gnu: rust-emacs-module: Update to 0.16.
  * gnu: rust-emacs-macros: Update to 0.17.
  * gnu: rust-emacs: Update to 0.17.
  * gnu: Add emacs-tree-sitter-core.
  * gnu: Add emacs-tree-sitter.
  * gnu: Add emacs-tree-sitter-langs.


Let me know what you think! This rebuilds all node packages, but there
aren't that many so hopefully that'll be ok on master.

Thanks,
Pierre

Pierre Langlois (31):
  gnu: rust-chunked-transfer: Update to 1.4.0.
  gnu: Add rust-fdlimit.
  gnu: Add rust-html-escape.
  gnu: Add rust-smallbitvec.
  gnu: rust-tiny-http: Update to 0.8.
  gnu: rust-spin: Update to 0.7.
  gnu: Add tree-sitter.
  gnu: node: Patch /usr/bin/env in node-gyp.
  guix: node-build-system: Support compiling addons with node-gyp.
  gnu: Add node-nan.
  gnu: Add tree-sitter-c.
  gnu: Add tree-sitter-cpp.
  gnu: Add tree-sitter-css.
  gnu: Add tree-sitter-go.
  gnu: Add tree-sitter-html.
  gnu: Add tree-sitter-java.
  gnu: Add tree-sitter-javascript.
  gnu: Add tree-sitter-json.
  gnu: Add tree-sitter-julia.
  gnu: Add tree-sitter-php.
  gnu: Add tree-sitter-python.
  gnu: Add tree-sitter-rust.
  gnu: Add tree-sitter-typescript.
  gnu: Add rust-bindgen@0.56.
  gnu: Add rust-tree-sitter.
  gnu: rust-emacs-module: Update to 0.16.
  gnu: rust-emacs-macros: Update to 0.17.
  gnu: rust-emacs: Update to 0.17.
  gnu: Add emacs-tree-sitter-core.
  gnu: Add emacs-tree-sitter.
  gnu: Add emacs-tree-sitter-langs.

 gnu/local.mk                     |   1 +
 gnu/packages/crates-io.scm       | 290 +++++++++++--
 gnu/packages/node-xyz.scm        |  30 ++
 gnu/packages/node.scm            |  29 +-
 gnu/packages/tree-sitter.scm     | 716 +++++++++++++++++++++++++++++++
 guix/build-system/node.scm       |  16 +
 guix/build/node-build-system.scm |  15 +
 7 files changed, 1064 insertions(+), 33 deletions(-)
 create mode 100644 gnu/packages/tree-sitter.scm

-- 
2.32.0

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH 01/31] gnu: rust-chunked-transfer: Update to 1.4.0.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-chunked-transfer-1): New variable.
(rust-chunked-transfer-0.3): Inherit from rust-chunked-transfer-1.
---
 gnu/packages/crates-io.scm | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 24cc772ba7..01a0dff724 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -18,6 +18,7 @@
 ;;; Copyright © 2021 Antero Mejr <antero@kodmin.com>
 ;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2021 Vinicius Monego <monego@posteo.net>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -7825,19 +7826,21 @@ Python arrow.humanize.")
 TimeZone trait for @code{rust-chrono}.")
     (license (list license:expat license:asl2.0))))

-(define-public rust-chunked-transfer-0.3
+(define-public rust-chunked-transfer-1
   (package
     (name "rust-chunked-transfer")
-    (version "0.3.1")
+    (version "1.4.0")
     (source
      (origin
        (method url-fetch)
        (uri (crate-uri "chunked_transfer" version))
        (file-name (string-append name "-" version ".tar.gz"))
        (sha256
-        (base32 "11yghnd24w0i9p8g368c3pg7qh9nfz7kgri6pywja9pnmakj13a9"))))
+        (base32 "0bkdlsrszfcscw3j6yhs7kj6jbp8id47jjk6h9k58px47na5gy7z"))))
     (build-system cargo-build-system)
-    (arguments `(#:skip-build? #t))
+    (arguments
+     `(#:cargo-development-inputs
+       (("rust-criterion" ,rust-criterion-0.3))))
     (home-page "https://github.com/frewsxcv/rust-chunked-transfer")
     (synopsis "Encoder and decoder for HTTP chunked transfer coding")
     (description
@@ -7845,6 +7848,19 @@ TimeZone trait for @code{rust-chrono}.")
 coding.")
     (license license:asl2.0)))

+(define-public rust-chunked-transfer-0.3
+  (package (inherit rust-chunked-transfer-1)
+    (name "rust-chunked-transfer")
+    (version "0.3.1")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "chunked_transfer" version))
+       (file-name (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32 "11yghnd24w0i9p8g368c3pg7qh9nfz7kgri6pywja9pnmakj13a9"))))
+    (arguments `(#:skip-build? #t))))
+
 (define-public rust-ci-info-0.3
   (package
     (name "rust-ci-info")
--
2.32.0

[bug#49946] [PATCH 02/31] gnu: Add rust-fdlimit.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-fdlimit-0.1): New variable.
---
 gnu/packages/crates-io.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 01a0dff724..d6b52a217f 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -15930,6 +15930,28 @@ floats.")
 it.")
     (license (list license:expat license:asl2.0))))

+(define-public rust-fdlimit-0.1
+  (package
+    (name "rust-fdlimit")
+    (version "0.1.4")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "fdlimit" version))
+       (file-name
+        (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "0g30d6gqkrwy8ylwdy7pqm443iq0p5dmnpz4ks41pirl7dclm98d"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-libc" ,rust-libc-0.2))))
+    (home-page "https://github.com/paritytech/fdlimit")
+    (synopsis "Raise the open file descriptors limit")
+    (description "Utility crate for raising file descriptors limit.")
+    (license license:asl2.0)))
+
 (define-public rust-femme-2
   (package
     (name "rust-femme")
--
2.32.0

[bug#49946] [PATCH 03/31] gnu: Add rust-html-escape.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-html-escape-0.2): New variable.
---
 gnu/packages/crates-io.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index d6b52a217f..4a9db4943b 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -20824,6 +20824,32 @@ Hash-based Message Authentication Code algorithm} for SHA1.")
      "This package provides a library for HTML entity encoding and decoding.")
     (license (list license:asl2.0 license:expat license:mpl2.0))))

+(define-public rust-html-escape-0.2
+  (package
+    (name "rust-html-escape")
+    (version "0.2.9")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "html-escape" version))
+       (file-name
+        (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "1dxw9lpckrqzzqgbkw64ckbajr4b7xxnjdn8adgzqf2mm40shvl1"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-utf8-width" ,rust-utf8-width-0.1))
+       #:cargo-development-inputs
+       (("rust-bencher" ,rust-bencher-0.1))))
+    (home-page "https://magiclen.org/html-escape")
+    (synopsis "Library for encoding and escaping special characters in HTML")
+    (description
+     "This package provides a library for encoding and escaping special
+characters in HTML, decoding and unescaping HTML entities as well.")
+    (license license:expat)))
+
 (define-public rust-http-0.2
   (package
     (name "rust-http")
--
2.32.0

[bug#49946] [PATCH 04/31] gnu: Add rust-smallbitvec.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-smallbitvec-2): New variable.
---
 gnu/packages/crates-io.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 4a9db4943b..2274e4f12b 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -44087,6 +44087,29 @@ I/O programming.")
         ("rust-futures-core-preview" ,rust-futures-core-preview-0.3)
         ("rust-futures-io-preview" ,rust-futures-io-preview-0.3))))))

+(define-public rust-smallbitvec-2
+  (package
+    (name "rust-smallbitvec")
+    (version "2.5.1")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "smallbitvec" version))
+       (file-name (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "0plrbldsjpwip3afbzd8fgrnvdhizcg5z4ncfqs4q6x4qjflzkkm"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-development-inputs
+       (("rust-bit-vec" ,rust-bit-vec-0.4)
+        ("rust-rand" ,rust-rand-0.4))))
+    (home-page "https://github.com/servo/smallbitvec")
+    (synopsis "A bit vector optimized for size and inline storage")
+    (description "This package provides a bit vector optimized for size and
+inline storage")
+    (license (list license:expat license:asl2.0))))
+
 (define-public rust-smallvec-1
   (package
     (name "rust-smallvec")
--
2.32.0

[bug#49946] [PATCH 05/31] gnu: rust-tiny-http: Update to 0.8.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-tiny-http-0.8): New variable.
(rust-tiny-http-0.6): Inherit from rust-tiny-http-0.8.
---
 gnu/packages/crates-io.scm | 39 ++++++++++++++++++++++++++++++++------
 1 file changed, 33 insertions(+), 6 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 2274e4f12b..bb31356ceb 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -48810,8 +48810,39 @@ closures after a delay or at a given timestamp.")
 manipulation in Rust.")
     (license license:expat)))

-(define-public rust-tiny-http-0.6
+(define-public rust-tiny-http-0.8
   (package
+    (name "rust-tiny-http")
+    (version "0.8.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "tiny_http" version))
+       (file-name
+        (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "0fcdwpb2ghk671qjjrk6048hs3yp7f681hxpr68gamk00181prcw"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-ascii" ,rust-ascii-1)
+        ("rust-chrono" ,rust-chrono-0.4)
+        ("rust-chunked-transfer" ,rust-chunked-transfer-1)
+        ("rust-log" ,rust-log-0.4)
+        ("rust-openssl" ,rust-openssl-0.10)
+        ("rust-url" ,rust-url-2))
+       #:cargo-development-inputs
+       (("rust-fdlimit" ,rust-fdlimit-0.1)
+        ("rust-rustc-serialize" ,rust-rustc-serialize-0.3)
+        ("rust-sha1" ,rust-sha1-0.6))))
+    (home-page "https://github.com/tiny-http/tiny-http")
+    (synopsis "Low level HTTP server library")
+    (description "This package provides a low level HTTP server library.")
+    (license license:asl2.0)))
+
+(define-public rust-tiny-http-0.6
+  (package (inherit rust-tiny-http-0.8)
     (name "rust-tiny-http")
     (version "0.6.2")
     (source
@@ -48830,11 +48861,7 @@ manipulation in Rust.")
         ("rust-chunked-transfer" ,rust-chunked-transfer-0.3)
         ("rust-log" ,rust-log-0.4)
         ("rust-openssl" ,rust-openssl-0.10)
-        ("rust-url" ,rust-url-1))))
-    (home-page "https://github.com/tiny-http/tiny-http")
-    (synopsis "Low level HTTP server library")
-    (description "This package provides a low level HTTP server library.")
-    (license license:asl2.0)))
+        ("rust-url" ,rust-url-1))))))

 (define-public rust-tiny-keccak-2
   (package
--
2.32.0

[bug#49946] [PATCH 06/31] gnu: rust-spin: Update to 0.7.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-spin-0.7): New variable.
(rust-spin-0.5): Inherit from rust-spin-0.7.
---
 gnu/packages/crates-io.scm | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index bb31356ceb..b551e6921d 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -44496,10 +44496,10 @@ track of where each new file and line starts.")
      "An RSpec inspired minimal testing framework for Rust.")
     (license license:expat)))

-(define-public rust-spin-0.5
+(define-public rust-spin-0.7
   (package
     (name "rust-spin")
-    (version "0.5.2")
+    (version "0.7.1")
     (source
       (origin
         (method url-fetch)
@@ -44507,8 +44507,11 @@ track of where each new file and line starts.")
         (file-name (string-append name "-" version ".crate"))
         (sha256
          (base32
-          "0b84m6dbzrwf2kxylnw82d3dr8w06av7rfkr8s85fb5f43rwyqvf"))))
+          "0qjips9f6fsvkyd7wj3a4gzaqknn2q4kkb19957pl86im56pna0k"))))
     (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-lock-api" ,rust-lock-api-0.4))))
     (home-page "https://github.com/mvdnes/spin-rs")
     (synopsis "Synchronization primitives based on spinning")
     (description "This crate provides synchronization primitives based on
@@ -44516,6 +44519,21 @@ spinning.  They may contain data, are usable without @code{std},and static
 initializers are available.")
     (license license:expat)))

+(define-public rust-spin-0.5
+  (package
+    (inherit rust-spin-0.7)
+    (name "rust-spin")
+    (version "0.5.2")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "spin" version))
+        (file-name (string-append name "-" version ".crate"))
+        (sha256
+         (base32
+          "0b84m6dbzrwf2kxylnw82d3dr8w06av7rfkr8s85fb5f43rwyqvf"))))
+    (arguments '(#:cargo-inputs ()))))
+
 (define-public rust-spin-0.4
   (package
     (inherit rust-spin-0.5)
--
2.32.0

[bug#49946] [PATCH 07/31] gnu: Add tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Register it.
---
 gnu/local.mk                 |  1 +
 gnu/packages/tree-sitter.scm | 98 ++++++++++++++++++++++++++++++++++++
 2 files changed, 99 insertions(+)
 create mode 100644 gnu/packages/tree-sitter.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index 3f09b167b1..3763df0821 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -559,6 +559,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/tls.scm				\
   %D%/packages/tmux.scm				\
   %D%/packages/tor.scm				\
+  %D%/packages/tree-sitter.scm			\
   %D%/packages/tv.scm				\
   %D%/packages/uglifyjs.scm			\
   %D%/packages/uml.scm				\
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
new file mode 100644
index 0000000000..2c749dc0a8
--- /dev/null
+++ b/gnu/packages/tree-sitter.scm
@@ -0,0 +1,98 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages tree-sitter)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system cargo)
+  #:use-module (guix git-download)
+  #:use-module (guix packages)
+  #:use-module (gnu packages crates-io)
+  #:use-module (gnu packages crates-graphics))
+
+(define-public tree-sitter
+  (package
+    (name "tree-sitter")
+    (version "0.19.5")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "1qmb0sva28zv6r3c3j7xs9pc8bpwwhkb9vxxndw2zbdn9wkvmbmn"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(;; Running test requires downloading fixtures, see the
+       ;; script/fetch-fixtures script.
+       #:tests? #f
+       ;; FIXME: Installing the sources for the tree-sitter Rust bindings
+       ;; doesn't work out of the box due to tree-sitter having multiple
+       ;; Rust packages in the same repository (bindings and CLI).
+       #:install-source? #f
+       #:cargo-inputs
+       (("rust-ansi-term" ,rust-ansi-term-0.12)
+        ("rust-atty" ,rust-atty-0.2)
+        ("rust-clap" ,rust-clap-2)
+        ("rust-difference" ,rust-difference-2)
+        ("rust-dirs" ,rust-dirs-3)
+        ("rust-html-escape" ,rust-html-escape-0.2)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-smallbitvec" ,rust-smallbitvec-2)
+        ("rust-spin" ,rust-spin-0.7)
+        ("rust-tiny-http" ,rust-tiny-http-0.8)
+        ("rust-walkdir" ,rust-walkdir-2)
+        ("rust-webbrowser" ,rust-webbrowser-0.5)
+        ("rust-which" ,rust-which-4))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (add-after 'build 'build-lib
+           (lambda _ (invoke "make")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin"))
+                    (lib (string-append out "/lib")))
+               (mkdir-p bin)
+               (install-file "target/release/tree-sitter" bin)
+               (setenv "PREFIX" out)
+               (invoke "make" "install")))))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Incremental parsing system for programming tools")
+    (description "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @command{tree-sitter} tool as well as the runtime
+library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 08/31] gnu: node: Patch /usr/bin/env in node-gyp.

[Pierre Langlois]

* gnu/packages/node.scm (node)[arguments]: Fix /usr/bin/env shebang in
node-gyp.js.
(node-lts)[arguments]: Ditto.
---
 gnu/packages/node.scm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
index 36c45e9c7a..522d4943d0 100644
--- a/gnu/packages/node.scm
+++ b/gnu/packages/node.scm
@@ -6,7 +6,7 @@
 ;;; Copyright © 2017 Mike Gerwitz <mtg@gnu.org>
 ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2018, 2019, 2020, 2021 Marius Bakke <marius@gnu.org>
-;;; Copyright © 2020 Pierre Langlois <pierre.langlois@gmx.com>
+;;; Copyright © 2020, 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;; Copyright © 2020 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;;
@@ -120,6 +120,10 @@
                (("'/usr/bin/env'")
                 (string-append "'" (which "env") "'")))

+             ;; Fix /usr/bin/env shebang in node-gyp.
+             (substitute* "deps/npm/node_modules/node-gyp/bin/node-gyp.js"
+               (("#!/usr/bin/env") (string-append "#!" (which "env"))))
+
              ;; FIXME: These tests fail in the build container, but they don't
              ;; seem to be indicative of real problems in practice.
              (for-each delete-file
@@ -661,6 +665,10 @@ source files.")
                  (("'/usr/bin/env'")
                   (string-append "'" (which "env") "'")))

+               ;; Fix /usr/bin/env shebang in node-gyp.
+               (substitute* "deps/npm/node_modules/node-gyp/bin/node-gyp.js"
+                 (("#!/usr/bin/env") (string-append "#!" (which "env"))))
+
                ;; FIXME: These tests fail in the build container, but they don't
                ;; seem to be indicative of real problems in practice.
                (for-each delete-file
--
2.32.0

[bug#49946] [PATCH 09/31] guix: node-build-system: Support compiling addons with node-gyp.

[Pierre Langlois]

* gnu/packages/node.scm (node-headers): New function.
* guix/build-system/node.scm (python): New function.
(node-headers): New function.
(lower): Add node-headers and python to build inputs.
* guix/build/node-build-system.scm (configure-gyp): New function.
(%standard-phases): Add 'configure-gyp after 'configure.
---
 gnu/packages/node.scm            | 19 ++++++++++++++++++-
 guix/build-system/node.scm       | 16 ++++++++++++++++
 guix/build/node-build-system.scm | 15 +++++++++++++++
 3 files changed, 49 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
index 522d4943d0..d0f7c5cdb7 100644
--- a/gnu/packages/node.scm
+++ b/gnu/packages/node.scm
@@ -48,7 +48,24 @@
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
   #:use-module (gnu packages tls)
-  #:use-module (gnu packages web))
+  #:use-module (gnu packages web)
+  #:use-module (ice-9 match)
+  #:export (node-headers))
+
+(define (node-headers node)
+  "Return an <origin> object for a tarball with headers for the given node
+package version."
+  (let* ((version (package-version node))
+         (hash (match version
+                 ("10.24.0"
+                  "0h37zjwcpxjdqcxqjfj5zp1n5zjxaa0g8lsy83955afg5cca8p0n")
+                 ("14.16.0"
+                  "1vpdgq7kcw1a0w90lpvbvxbrc0n3pwjrs3sm42pjj7560clvji2b"))))
+    (origin
+      (method url-fetch)
+      (uri (string-append "https://nodejs.org/dist/v" version
+                          "/node-v" version "-headers.tar.gz"))
+      (sha256 (base32 hash)))))

 (define-public node
   (package
diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm
index 98f63f87ef..7828582a9a 100644
--- a/guix/build-system/node.scm
+++ b/guix/build-system/node.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2019 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -41,6 +42,19 @@
   (let ((node (resolve-interface '(gnu packages node))))
     (module-ref node 'node-lts)))

+(define (python)
+  "Return the python package."
+  ;; Lazily resolve the binding to avoid a circular dependency.
+  (let ((module (resolve-interface '(gnu packages python))))
+    (module-ref module 'python-wrapper)))
+
+(define (node-headers node)
+  "Return a tarball with headers for the given node, needed for packages that
+need to build native bindings using node-gyp."
+  ;; Lazily resolve the binding to avoid a circular dependency.
+  (let ((module (resolve-interface '(gnu packages node))))
+    ((module-ref module 'node-headers) node)))
+
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (node (default-node))
@@ -62,6 +76,8 @@
                         ;; Keep the standard inputs of 'gnu-build-system'.
                         ,@(standard-packages)))
          (build-inputs `(("node" ,node)
+                         ("node-headers" ,(node-headers node))
+                         ("python" ,(python))
                          ,@native-inputs))
          (outputs outputs)
          (build node-build)
diff --git a/guix/build/node-build-system.scm b/guix/build/node-build-system.scm
index 70a367618e..2993c49b2b 100644
--- a/guix/build/node-build-system.scm
+++ b/guix/build/node-build-system.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
 ;;; Copyright © 2016, 2020 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2019, 2021 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -101,6 +102,19 @@
     (invoke npm "--offline" "--ignore-scripts" "install")
     #t))

+(define* (configure-gyp #:key inputs #:allow-other-keys)
+  "Run 'node-gyp configure' if we see a 'binding.gyp' file.
+
+By default, 'node-gyp' will try to download node headers from the internet, we
+prevent this with the '--tarball' flag."
+  (let ((node-gyp (string-append (assoc-ref inputs "node")
+                                 "/lib/node_modules/npm/node_modules/node-gyp"
+                                 "/bin/node-gyp.js")))
+    (if (file-exists? "binding.gyp")
+        (invoke node-gyp "--tarball" (assoc-ref inputs "node-headers")
+                "configure"))
+    #t))
+
 (define* (build #:key inputs #:allow-other-keys)
   (let ((package-meta (call-with-input-file "package.json" read-json)))
     (if (and=> (assoc-ref package-meta "scripts")
@@ -147,6 +161,7 @@
     (add-after 'unpack 'set-home set-home)
     (add-before 'configure 'patch-dependencies patch-dependencies)
     (replace 'configure configure)
+    (add-after 'configure 'configure-gyp configure-gyp)
     (replace 'build build)
     (replace 'check check)
     (add-before 'install 'repack repack)
--
2.32.0

[bug#49946] [PATCH 10/31] gnu: Add node-nan.

[Pierre Langlois]

---
 gnu/packages/node-xyz.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/node-xyz.scm b/gnu/packages/node-xyz.scm
index cbb68f77d6..eb4b59d1ab 100644
--- a/gnu/packages/node-xyz.scm
+++ b/gnu/packages/node-xyz.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2020 Giacomo Leidi <goodoldpaul@autistici.org>
 ;;; Copyright © 2021 Noisytoot <noisytoot@disroot.org>
 ;;; Copyright © 2021 Charles <charles.b.jackson@protonmail.com>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -171,6 +172,35 @@ user-land JavaScript.")
 random number generator.")
     (license license:bsd-3)))

+(define-public node-nan
+  (package
+    (name "node-nan")
+    (version "2.15.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+               (url "https://github.com/nodejs/nan")
+               (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "18xslh9va5ld872scrp5y4251ax9s3c6qh0lnl1200lpzbsxy7yd"))))
+    (build-system node-build-system)
+    (arguments
+     '(#:tests? #f                      ; FIXME: tests depend on node-tap
+       #:phases
+       (modify-phases %standard-phases
+         ;; The default configure phase fails due to tap being missing, as we do
+         ;; not have tap packaged yet.  It is used only for tests.  This package
+         ;; still works as a dependency of node-glob and node-inflight.
+         (delete 'configure))))
+    (home-page "https://github.com/nodejs/nan")
+    (synopsis "Native Abstractions for Node.js")
+    (description "This package provides header files filled with macros and
+utilities for making add-on development for Node.js easier across versions.")
+    (license license:bsd-3)))
+
 (define-public node-oop
   ;; No releases, last commit was February 2013.
   (let ((commit "f9d87cda0958886955c14a0a716e57021ed295dc")
--
2.32.0

[bug#49946] [PATCH 11/31] gnu: Add tree-sitter-c.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-c): New variable.
---
 gnu/packages/tree-sitter.scm | 63 +++++++++++++++++++++++++++++++++++-
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 2c749dc0a8..c855cba347 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -19,10 +19,12 @@
 (define-module (gnu packages tree-sitter)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
+  #:use-module (guix build-system node)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (gnu packages crates-io)
-  #:use-module (gnu packages crates-graphics))
+  #:use-module (gnu packages crates-graphics)
+  #:use-module (gnu packages node-xyz))

 (define-public tree-sitter
   (package
@@ -96,3 +98,62 @@ can be embedded in any application.
 This package includes the @command{tree-sitter} tool as well as the runtime
 library.")
     (license license:expat)))
+
+(define-public tree-sitter-c
+  (package
+    (name "tree-sitter-c")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-c")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "1diys8yigvhm4ppbmp3a473yxjg2d5lk11y0ay7qprcz7233lakv"))))
+    (build-system node-build-system)
+    (native-inputs
+     `(("tree-sitter" ,tree-sitter)
+       ("node-nan" ,node-nan)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         ;; tree-sitter-cli is listed as a Node.js dependency, however the
+         ;; node tree-sitter-cli package is just a wrapper which downloads a
+         ;; tree-sitter binary, see
+         ;; https://github.com/tree-sitter/tree-sitter/tree/master/cli/npm
+         ;; Instead we remove it as a dependency so that we can use our own
+         ;; tree-sitter package.
+         (add-after 'unpack 'fix-configure
+           (lambda _
+             (with-atomic-file-replacement "package.json"
+               (lambda (in out)
+                 (use-modules ((guix build json)))
+                 (let ((package-meta (read-json in)))
+                   (assoc-remove! (assoc-ref package-meta "devDependencies")
+                                  "tree-sitter-cli")
+                   (write-json package-meta out))))))
+         (add-before 'build 'set-cc
+           (lambda _
+             (setenv "CC" "gcc")))
+         (add-before 'build 'make-files-writable
+           (lambda _
+             (for-each make-file-writable (find-files "src" ".*"))))
+         (add-after 'install 'install-native-lib
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out")
+                                       "/lib/tree-sitter")))
+               ;; Invoking `tree-sitter test' loads the grammar which
+               ;; compiles it to a .so binary that we install.
+               (invoke "tree-sitter" "test")
+               (mkdir-p lib)
+               (copy-recursively (string-append (getenv "HOME")
+                                                "/.tree-sitter/bin")
+                                 lib)))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-c")
+    (synopsis "Tree-sitter C grammar")
+    (description
+      "This package provides a C grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 12/31] gnu: Add tree-sitter-cpp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cpp): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c855cba347..ac4fe5c578 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -157,3 +157,27 @@ library.")
     (description
       "This package provides a C grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-cpp
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-cpp")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-cpp")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "08ywv6n80sa541rr08bqz4zyg7byvjcabp68lvxmcahjk8xzcgwk"))))
+    (native-inputs
+     `(("tree-sitter" ,tree-sitter)
+       ("tree-sitter-c" ,tree-sitter-c)
+       ("node-nan" ,node-nan)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-cpp")
+    (synopsis "Tree-sitter C++ grammar")
+    (description
+      "This package provides a C++ grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 13/31] gnu: Add tree-sitter-css.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-css): New variable.
---
 gnu/packages/tree-sitter.scm | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index ac4fe5c578..05929a9062 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -181,3 +181,23 @@ library.")
     (description
       "This package provides a C++ grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-css
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-css")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-css")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "014jrlgi7zfza9g38hsr4vlbi8964i5p7iglaih6qmzaiml7bja2"))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-css")
+    (synopsis "Tree-sitter CSS grammar")
+    (description
+      "This package provides a CSS grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 14/31] gnu: Add tree-sitter-go.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-go): New variable.
---
 gnu/packages/tree-sitter.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 05929a9062..f163efd7a3 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -22,6 +22,7 @@
   #:use-module (guix build-system node)
   #:use-module (guix git-download)
   #:use-module (guix packages)
+  #:use-module (guix utils)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages node-xyz))
@@ -201,3 +202,32 @@ library.")
     (description
       "This package provides a CSS grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-go
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-go")
+    (version "0.19.1")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-go")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "0nxs47vd2fc2fr0qlxq496y852rwg39flhg334s7dlyq7d3lcx4x"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-go")
+    (synopsis "Tree-sitter Go grammar")
+    (description
+      "This package provides a Golang grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 15/31] gnu: Add tree-sitter-html.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-html): New variable.
---
 gnu/packages/tree-sitter.scm | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index f163efd7a3..c2ca5a842a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -231,3 +231,23 @@ library.")
     (description
       "This package provides a Golang grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-html
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-html")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-html")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-html")
+    (synopsis "Tree-sitter HTML grammar")
+    (description
+      "This package provides a HTML grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 16/31] gnu: Add tree-sitter-java.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-java): New variable.
---
 gnu/packages/tree-sitter.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c2ca5a842a..e671c3fbf9 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -251,3 +251,32 @@ library.")
     (description
       "This package provides a HTML grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-java
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-java")
+    (version "0.19.1")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-java")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "07zw9ygb45hnvlx9qlz7rlz8hc3byjy03d24v72i5iyhpiiwlhvl"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-java")
+    (synopsis "Tree-sitter Java grammar")
+    (description
+      "This package provides a Java grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 17/31] gnu: Add tree-sitter-javascript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-javascript): New variable.
---
 gnu/packages/tree-sitter.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e671c3fbf9..e59a44c521 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -280,3 +280,33 @@ library.")
     (description
       "This package provides a Java grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-javascript
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-javascript")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-javascript")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "063va0s727yzhy1cz04fipzwwbq4af6fhgka6g970hk9yf7ggmnj"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
+    (synopsis "Tree-sitter Javascript grammar")
+    (description
+      "This package provides a Javascript grammar for the Tree-sitter
+library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 18/31] gnu: Add tree-sitter-json.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-json): New variable.
---
 gnu/packages/tree-sitter.scm | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e59a44c521..d6214de2ec 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -310,3 +310,23 @@ library.")
       "This package provides a Javascript grammar for the Tree-sitter
 library.")
     (license license:expat)))
+
+(define-public tree-sitter-json
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-json")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-json")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "06pjh31bv9ja9hlnykk257a6zh8bsxg2fqa54al7qk1r4n9ksnff"))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-json")
+    (synopsis "Tree-sitter JSON grammar")
+    (description
+      "This package provides a JSON grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 19/31] gnu: Add tree-sitter-julia.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-julia): New variable.
---
 gnu/packages/tree-sitter.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d6214de2ec..aa72348347 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -330,3 +330,32 @@ library.")
     (description
       "This package provides a JSON grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-julia
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-julia")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-julia")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "1pbnmvhy2gq4vg1b0sjzmjm4s2gsgdjh7h01yj8qrrqbcl29c463"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-julia")
+    (synopsis "Tree-sitter Julia grammar")
+    (description
+      "This package provides a Julia grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 20/31] gnu: Add tree-sitter-php.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-php): New variable.
---
 gnu/packages/tree-sitter.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index aa72348347..9f1a30f0cc 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -359,3 +359,32 @@ library.")
     (description
       "This package provides a Julia grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-php
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-php")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-php")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+           "17cmybgpprw7w9d2v7lmc6zmr90d70g0jqq279gzg0mpwfzla53s"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-php")
+    (synopsis "Tree-sitter PHP grammar")
+    (description
+      "This package provides a PHP grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 21/31] gnu: Add tree-sitter-python.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-python): New variable.
---
 gnu/packages/tree-sitter.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 9f1a30f0cc..d7c3468035 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -388,3 +388,32 @@ library.")
     (description
       "This package provides a PHP grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-python
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-python")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-python")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "04b85qxqs64x6nhbpcgrzkbilxaiwvr9yd3h065rynv7rsdg0hii"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-python")
+    (synopsis "Tree-sitter Python grammar")
+    (description
+      "This package provides a Python grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 22/31] gnu: Add tree-sitter-rust.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-rust): New variable.
---
 gnu/packages/tree-sitter.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d7c3468035..6c2b2e7577 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -23,6 +23,7 @@
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages algebra)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages node-xyz))
@@ -417,3 +418,27 @@ library.")
     (description
       "This package provides a Python grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-rust
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-rust")
+    (version "0.19.1")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-rust")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "118vkhv7n3sw8y9pi0987cgdcd74sjqwviijw01mhnk3bkyczi3l"))))
+    (native-inputs
+     `(("bc" ,bc)
+       ("tree-sitter" ,tree-sitter)
+       ("node-nan" ,node-nan)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-rust")
+    (synopsis "Tree-sitter Rust grammar")
+    (description
+      "This package provides a Rust grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 23/31] gnu: Add tree-sitter-typescript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-typescript): New variable.
---
 gnu/packages/tree-sitter.scm | 60 ++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 6c2b2e7577..0f1946a77e 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -442,3 +442,63 @@ library.")
     (description
       "This package provides a Rust grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-typescript
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-typescript")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-typescript")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "01pkmwwmbv6kxda0n6g4cfg72ldmpi9gmp11a5gygn472vhrh2xw"))))
+    (native-inputs
+     `(("tree-sitter" ,tree-sitter)
+       ("tree-sitter-javascript" ,tree-sitter-javascript)
+       ("node-nan" ,node-nan)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))
+         (add-after 'build 'build-gyp
+           (lambda* (#:key inputs #:allow-other-keys)
+             (let ((node (assoc-ref inputs "node")))
+               (invoke (string-append node "/lib/node_modules/npm/node_modules"
+                                           "/node-gyp/bin/node-gyp.js")
+                       "build"))))
+         (replace 'make-files-writable
+           (lambda _
+             (for-each make-file-writable
+                       (find-files "typescript" ".*"))
+             (for-each make-file-writable
+                       (find-files "tsx" ".*"))))
+         (replace 'install-native-lib
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out")
+                                       "/lib/tree-sitter")))
+               ;; Invoking `tree-sitter test' loads the grammar which
+               ;; compiles it to a .so binary that we install.
+               (with-directory-excursion "typescript"
+                 (invoke "tree-sitter" "test"))
+               (with-directory-excursion "tsx"
+                 (invoke "tree-sitter" "test"))
+               (mkdir-p lib)
+               (copy-recursively (string-append (getenv "HOME")
+                                                "/.tree-sitter/bin")
+                                 lib))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-typescript")
+    (synopsis "Tree-sitter Typescript grammar")
+    (description
+      "This package provides Typescript and TSX grammars for the Tree-sitter
+library.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 24/31] gnu: Add rust-bindgen@0.56.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-bindgen-0.56): New variable.
---
 gnu/packages/crates-io.scm | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index b551e6921d..44f8885adb 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -4482,6 +4482,43 @@ bindings to C and C++ libraries.")
         ("rust-shlex" ,rust-shlex-0.1)
         ("rust-which" ,rust-which-3))))))

+(define-public rust-bindgen-0.56
+  (package
+    (inherit rust-bindgen-0.57)
+    (name "rust-bindgen")
+    (version "0.56.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "bindgen" version))
+       (file-name (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "0fajmgk2064ca1z9iq1jjkji63qwwz38z3d67kv6xdy0xgdpk8rd"))))
+    (arguments
+     `(#:skip-build? #t
+       #:cargo-inputs
+       (("rust-bitflags" ,rust-bitflags-1)
+        ("rust-cexpr" ,rust-cexpr-0.4)
+        ("rust-cfg-if" ,rust-cfg-if-0.1)
+        ("rust-clang-sys" ,rust-clang-sys-1)
+        ("rust-clap" ,rust-clap-2)
+        ("rust-env-logger" ,rust-env-logger-0.8)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-lazycell" ,rust-lazycell-1)
+        ("rust-log" ,rust-log-0.4)
+        ("rust-peeking-take-while" ,rust-peeking-take-while-0.1)
+        ("rust-proc-macro2" ,rust-proc-macro2-1)
+        ("rust-quote" ,rust-quote-1)
+        ("rust-regex" ,rust-regex-1)
+        ("rust-rustc-hash" ,rust-rustc-hash-1)
+        ("rust-shlex" ,rust-shlex-0.1)
+        ("rust-which" ,rust-which-3))
+       #:cargo-development-inputs
+       (("rust-clap" ,rust-clap-2)
+        ("rust-diff" ,rust-diff-0.1)
+        ("rust-shlex" ,rust-shlex-0.1))))))
+
 (define-public rust-bindgen-0.55
   (package
     (inherit rust-bindgen-0.57)
--
2.32.0

[bug#49946] [PATCH 25/31] gnu: Add rust-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 0f1946a77e..4817b293ce 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -20,6 +20,7 @@
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system node)
+  #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
@@ -101,6 +102,32 @@ This package includes the @command{tree-sitter} tool as well as the runtime
 library.")
     (license license:expat)))

+(define-public rust-tree-sitter-0.19
+  (package
+    (name "rust-tree-sitter")
+    (version "0.19.5")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "tree-sitter" version))
+        (file-name (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1h6adq5kqf4izzsklch5lfxx2aisxga463zz7w44rgwnck16wwmd"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:tests? #f  ;; Running tests misinterprets comments as doc-tests.
+       #:cargo-inputs
+       (("rust-cc" ,rust-cc-1)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-regex" ,rust-regex-1)
+        ("rust-spin" ,rust-spin-0.7))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Rust bindings to the Tree-sitter parsing library")
+    (description "This package provides Rust bindings to the Tree-sitter
+parsing library.")
+    (license license:expat)))
+
 (define-public tree-sitter-c
   (package
     (name "tree-sitter-c")
--
2.32.0

[bug#49946] [PATCH 26/31] gnu: rust-emacs-module: Update to 0.16.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-module-0.16): New variable.
(rust-emacs-module-0.10): Inherit from rust-emacs-module-0.10.
---
 gnu/packages/crates-io.scm | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 44f8885adb..57a287c5d2 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -14638,10 +14638,10 @@ Emacs' support for dynamic modules.")
     (description "This package provides proc macros for Emacs modules.")
     (license license:bsd-3)))

-(define-public rust-emacs-module-0.10
+(define-public rust-emacs-module-0.16
   (package
     (name "rust-emacs-module")
-    (version "0.10.0")
+    (version "0.16.2")
     (source
       (origin
         (method url-fetch)
@@ -14650,11 +14650,11 @@ Emacs' support for dynamic modules.")
          (string-append name "-" version ".tar.gz"))
         (sha256
          (base32
-          "1gf9lz735xbkyir53dyv362drfx3nin5an5cx39kd8q8kjjwix5g"))))
+          "0s8qrh0ggjmqr04zkcf7s4ijmpd44rjcag78npnq64jv10lxvsry"))))
     (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
-       (("rust-bindgen" ,rust-bindgen-0.48))))
+       (("rust-bindgen" ,rust-bindgen-0.56))))
     (inputs
      `(("clang" ,clang)))
     (home-page "https://github.com/ubolonton/emacs-module-rs")
@@ -14663,6 +14663,23 @@ Emacs' support for dynamic modules.")
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

+(define-public rust-emacs-module-0.10
+  (package (inherit rust-emacs-module-0.16)
+    (name "rust-emacs-module")
+    (version "0.10.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs_module" version))
+        (file-name
+         (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1gf9lz735xbkyir53dyv362drfx3nin5an5cx39kd8q8kjjwix5g"))))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-bindgen" ,rust-bindgen-0.48))))))
+
 (define-public rust-embed-resource-1
   (package
     (name "rust-embed-resource")
--
2.32.0

[bug#49946] [PATCH 27/31] gnu: rust-emacs-macros: Update to 0.17.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-macros-0.17): New variable.
(rust-emacs-macros-0.11): Inherit from rust-emacs-macros-0.17.
---
 gnu/packages/crates-io.scm | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 57a287c5d2..365cd6dcb1 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -14614,8 +14614,31 @@ signing, and verification in pure Rust.")
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

-(define-public rust-emacs-macros-0.11
+(define-public rust-emacs-macros-0.17
   (package
+    (name "rust-emacs-macros")
+    (version "0.17.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs-macros" version))
+        (file-name (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "0qg1dcn5acbirq617qq2fgg9adswif2dnr292s3qnq62wzgnyrb9"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-darling" ,rust-darling-0.10)
+        ("rust-quote" ,rust-quote-1)
+        ("rust-syn" ,rust-syn-1))))
+    (home-page "https://github.com/ubolonton/emacs-module-rs")
+    (synopsis "Proc macros for Emacs modules")
+    (description "This package provides proc macros for Emacs modules.")
+    (license license:bsd-3)))
+
+(define-public rust-emacs-macros-0.11
+  (package (inherit rust-emacs-macros-0.17)
     (name "rust-emacs-macros")
     (version "0.11.0")
     (source
@@ -14627,16 +14650,11 @@ Emacs' support for dynamic modules.")
         (sha256
          (base32
           "0390y8vafxdi334hhgrzvcqjq3n5ckcmvilqcfp8vajjq8irrly6"))))
-    (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
        (("rust-darling" ,rust-darling-0.9)
         ("rust-quote" ,rust-quote-0.6)
-        ("rust-syn" ,rust-syn-0.15))))
-    (home-page "https://github.com/ubolonton/emacs-module-rs")
-    (synopsis "Proc macros for Emacs modules")
-    (description "This package provides proc macros for Emacs modules.")
-    (license license:bsd-3)))
+        ("rust-syn" ,rust-syn-0.15))))))

 (define-public rust-emacs-module-0.16
   (package
--
2.32.0

[bug#49946] [PATCH 28/31] gnu: rust-emacs: Update to 0.17.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-0.17): New variable.
(rust-emacs-0.11): Inherit from rust-emacs-0.17.
---
 gnu/packages/crates-io.scm | 38 +++++++++++++++++++++++++++++++-------
 1 file changed, 31 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 365cd6dcb1..6b7dbe689c 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -14583,10 +14583,10 @@ signing, and verification in pure Rust.")
 @code{Right} is a general purpose sum type with two cases.")
     (license (list license:expat license:asl2.0))))

-(define-public rust-emacs-0.11
+(define-public rust-emacs-0.17
   (package
     (name "rust-emacs")
-    (version "0.11.0")
+    (version "0.17.0")
     (source
       (origin
         (method url-fetch)
@@ -14595,16 +14595,18 @@ signing, and verification in pure Rust.")
          (string-append name "-" version ".tar.gz"))
         (sha256
          (base32
-          "1xn6nr8m3p24irlmv3i7c9n95yb10qc6aikwqv99qlmhbky5x0z7"))))
+          "02f31apvz03dr0dwn5wyirdmqkk2d635iki0kmsh291xr6dysy2q"))))
     (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
-       (("rust-ctor" ,rust-ctor-0.1)
-        ("rust-emacs-macros" ,rust-emacs-macros-0.11)
-        ("rust-emacs-module" ,rust-emacs-module-0.10)
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-ctor" ,rust-ctor-0.1)
+        ("rust-emacs-macros" ,rust-emacs-macros-0.17)
+        ("rust-emacs-module" ,rust-emacs-module-0.16)
         ("rust-failure" ,rust-failure-0.1)
         ("rust-failure-derive" ,rust-failure-derive-0.1)
-        ("rust-lazy-static" ,rust-lazy-static-1))))
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-thiserror" ,rust-thiserror-1))))
     (inputs
      `(("clang" ,clang)))
     (home-page "https://github.com/ubolonton/emacs-module-rs")
@@ -14614,6 +14616,28 @@ signing, and verification in pure Rust.")
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

+(define-public rust-emacs-0.11
+  (package (inherit rust-emacs-0.17)
+    (name "rust-emacs")
+    (version "0.11.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs" version))
+        (file-name
+         (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1xn6nr8m3p24irlmv3i7c9n95yb10qc6aikwqv99qlmhbky5x0z7"))))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-ctor" ,rust-ctor-0.1)
+        ("rust-emacs-macros" ,rust-emacs-macros-0.11)
+        ("rust-emacs-module" ,rust-emacs-module-0.10)
+        ("rust-failure" ,rust-failure-0.1)
+        ("rust-failure-derive" ,rust-failure-derive-0.1)
+        ("rust-lazy-static" ,rust-lazy-static-1))))))
+
 (define-public rust-emacs-macros-0.17
   (package
     (name "rust-emacs-macros")
--
2.32.0

[bug#49946] [PATCH 29/31] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
(emacs-tree-sitter-core): New variable.
---
 gnu/packages/tree-sitter.scm | 79 ++++++++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 4817b293ce..a9c1714936 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -19,6 +19,7 @@
 (define-module (gnu packages tree-sitter)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
+  #:use-module (guix build-system emacs)
   #:use-module (guix build-system node)
   #:use-module (guix download)
   #:use-module (guix git-download)
@@ -27,6 +28,7 @@
   #:use-module (gnu packages algebra)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages crates-graphics)
+  #:use-module (gnu packages llvm)
   #:use-module (gnu packages node-xyz))

 (define-public tree-sitter
@@ -529,3 +531,80 @@ library.")
       "This package provides Typescript and TSX grammars for the Tree-sitter
 library.")
     (license license:expat)))
+
+;; Local package definition solely for building the native emacs module
+;; written in Rust.
+(define tree-sitter-emacs-module
+  (package
+    (name "tree-sitter-emacs-module")
+    (version "0.15.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "15y0wjnck8rbfhl0xrl71ci7clbcp11lhqil5l8ykprsdjv0c2as"))))
+    (build-system cargo-build-system)
+    (inputs
+     `(("clang" ,clang)))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-emacs" ,rust-emacs-0.17)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-tree-sitter" ,rust-tree-sitter-0.19))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (add-after 'delete-cargo.lock 'do-not-fetch-from-github
+           (lambda _
+             (substitute* "Cargo.toml"
+               (("\\[patch.*") "")
+               (("git = .*") ""))))
+         (add-after 'do-not-fetch-from-github 'chdir
+           (lambda _ (chdir "core")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out") "/lib")))
+               (install-file "../target/release/libtsc_dyn.so" lib)))))))
+    (home-page #f)
+    (synopsis #f)
+    (description #f)
+    (license license:expat)))
+
+(define-public emacs-tree-sitter-core
+  (package
+    (name "emacs-tree-sitter-core")
+    (version (package-version tree-sitter-emacs-module))
+    (source (package-source tree-sitter-emacs-module))
+    (build-system emacs-build-system)
+    (native-inputs
+     `(("tree-sitter-emacs-module" ,tree-sitter-emacs-module)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'install 'install-module
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out")))
+                   (module (string-append (assoc-ref inputs "tree-sitter-emacs-module")
+                                          "/lib/libtsc_dyn.so")))
+               ;; Writing "LOCAL" in this file prevents the package from
+               ;; trying to download the module from the internet.
+               (call-with-output-file (string-append elpa "/DYN-VERSION")
+                 (lambda (port) (display "LOCAL" port)))
+               (substitute* "tsc-dyn-get.el"
+                 (("defcustom tsc-dyn-dir tsc--dir")
+                  (string-append "defcustom tsc-dyn-dir \"" elpa "\"")))
+               (copy-file module (string-append elpa "/tsc-dyn.so"))))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp, core library")
+    (description "This package provides core APIs of the Emacs binding for
+Tree-sitter, an incremental parsing system.")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 30/31] gnu: Add emacs-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index a9c1714936..22551d4482 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -608,3 +608,30 @@ library.")
     (description "This package provides core APIs of the Emacs binding for
 Tree-sitter, an incremental parsing system.")
     (license license:expat)))
+
+(define-public emacs-tree-sitter
+  (package
+    (name "emacs-tree-sitter")
+    (version (package-version emacs-tree-sitter-core))
+    (source (package-source emacs-tree-sitter-core))
+    (build-system emacs-build-system)
+    (propagated-inputs
+     `(("emacs-tree-sitter-core" ,emacs-tree-sitter-core)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "lisp"))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp")
+    (description "This package provides Emacs bindings for Tree-sitter, an
+incremental parsing library.  It aims to be the foundation for a new breed of
+Emacs packages that understand code structurally.  For example:
+
+@enumerate
+@item Faster, fine-grained code highlighting.
+@item More flexible code folding.
+@item Structural editing (like Paredit, or even better) for non-Lisp code.
+@item More informative indexing for imenu.
+@end enumerate")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 31/31] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter-langs): New variable.
---
 gnu/packages/tree-sitter.scm | 79 ++++++++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 22551d4482..bfdfacf122 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -635,3 +635,82 @@ Emacs packages that understand code structurally.  For example:
 @item More informative indexing for imenu.
 @end enumerate")
     (license license:expat)))
+
+(define-public emacs-tree-sitter-langs
+  (package
+    (name "emacs-tree-sitter-langs")
+    (version "0.10.2")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/tree-sitter-langs")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0lym09jnlb7kvqkwkjbrqpc24fkpv99f8r9139sw8g537yj5f8rq"))))
+    (build-system emacs-build-system)
+    ;; FIXME: Support for some languages, such as bash, is still missing from
+    ;; the bundle.
+    (inputs
+     `(("tree-sitter-c" ,tree-sitter-c)
+       ("tree-sitter-cpp" ,tree-sitter-cpp)
+       ("tree-sitter-css" ,tree-sitter-css)
+       ("tree-sitter-go" ,tree-sitter-go)
+       ("tree-sitter-html" ,tree-sitter-html)
+       ("tree-sitter-java" ,tree-sitter-java)
+       ("tree-sitter-javascript" ,tree-sitter-javascript)
+       ("tree-sitter-json" ,tree-sitter-json)
+       ("tree-sitter-julia" ,tree-sitter-julia)
+       ("tree-sitter-php" ,tree-sitter-php)
+       ("tree-sitter-python" ,tree-sitter-python)
+       ("tree-sitter-rust" ,tree-sitter-rust)
+       ("tree-sitter-typescript" ,tree-sitter-typescript)))
+    (propagated-inputs
+     `(("emacs-tree-sitter" ,emacs-tree-sitter)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-before 'build 'fake-bundle
+           (lambda _
+             ;; The BUNDLE-VERSION file prevents the package from
+             ;; downloading libraries at load time.
+             (call-with-output-file "bin/BUNDLE-VERSION"
+               (lambda (port) (display ,version port)))))
+         (add-after 'install 'install-libs
+           (lambda* (#:key outputs inputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+               (define (install-lib lang)
+                 (let ((input (assoc-ref inputs (string-append "tree-sitter-" lang))))
+                   (copy-file (string-append input "/lib/tree-sitter/" lang ".so")
+                              (string-append elpa "/bin/" lang ".so"))))
+               (mkdir-p (string-append elpa "/bin"))
+               ;; The BUNDLE-VERSION file prevents the package from
+               ;; downloading libraries at load time.
+               (call-with-output-file (string-append elpa "/bin/BUNDLE-VERSION")
+                 (lambda (port) (display ,version port)))
+               (for-each (lambda (input)
+                           (use-modules (ice-9 regex))
+                           (cond
+                            ((string-match "^tree-sitter-(.*)$" (car input))
+                             => (lambda (m)
+                                  (install-lib (match:substring m 1))))))
+                         inputs))))
+         (add-after 'install 'install-queries
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+                (copy-recursively "queries" (string-append elpa "/queries"))))))))
+    (home-page "https://ubolonton.github.io/emacs-tree-sitter/languages/")
+    (synopsis "Language support bundle for Tree-sitter")
+    (description "This package is a convenient language bundle for
+Tree-sitter.  For each supported language, this package provides:
+
+@enumerate
+@item Pre-compiled grammar binaries.
+@item An optional highlights.scm file that provides highlighting patterns.
+This is mainly intended for major modes that are not aware of tree-sitter.
+@item Optional query patterns for other minor modes that provide high-level
+functionalities on top of tree-sitter, such as code folding, evil text
+objects, ...etc.
+@end enumerate")
+    (license license:expat)))
--
2.32.0

[bug#49946] [PATCH 08/31] gnu: node: Patch /usr/bin/env in node-gyp.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]

Pierre Langlois schreef op ma 09-08-2021 om 00:33 [+0100]:
> @@ -120,6 +120,10 @@
>                 (("'/usr/bin/env'")
>                  (string-append "'" (which "env") "'")))
> 
> +             ;; Fix /usr/bin/env shebang in node-gyp.
> +             (substitute* "deps/npm/node_modules/node-gyp/bin/node-gyp.js"
> +               (("#!/usr/bin/env") (string-append "#!" (which "env"))))

For cross-compilation, this should most likely be
(string-append (assoc-ref inputs "coreutils") "/bin/env")
or something like that instead.  Likewise in other places.
The old code uses (which "env") in some cases, but those
are probably wrong (except where it is patched in tests).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [PATCH 11/31] gnu: Add tree-sitter-c.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 240 bytes --]

Pierre Langlois schreef op ma 09-08-2021 om 00:33 [+0100]:
> +             (setenv "CC" "gcc")))

Use (setenv "CC" ,(cc-for-target)) instead,
such that the cross-compiler TARGET-gcc is used
when cross-compiling.

reetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [PATCH 08/31] gnu: node: Patch /usr/bin/env in node-gyp.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> writes:

> [[PGP Signed Part:Undecided]]
> Pierre Langlois schreef op ma 09-08-2021 om 00:33 [+0100]:
>> @@ -120,6 +120,10 @@
>>                 (("'/usr/bin/env'")
>>                  (string-append "'" (which "env") "'")))
>> 
>> +             ;; Fix /usr/bin/env shebang in node-gyp.
>> +             (substitute* "deps/npm/node_modules/node-gyp/bin/node-gyp.js"
>> +               (("#!/usr/bin/env") (string-append "#!" (which "env"))))
>
> For cross-compilation, this should most likely be
> (string-append (assoc-ref inputs "coreutils") "/bin/env")
> or something like that instead.  Likewise in other places.
> The old code uses (which "env") in some cases, but those
> are probably wrong (except where it is patched in tests).

Good point, I didn't consider cross-compilation.  Actually, trying it,
it looks like our node package doesn't currently cross-compile
correctly.  I just managed to get it to cross-compile though, I'll
submit another patch for it!  In the meantime, with this series I agree
it's better for the new code to be correct from the begining though.

Thanks for taking a look!
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH 07/31] gnu: Add tree-sitter.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 3325 bytes --]


Pierre Langlois <pierre.langlois@gmx.com> writes:

> * gnu/packages/tree-sitter.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Register it.
> ---
>  gnu/local.mk                 |  1 +
>  gnu/packages/tree-sitter.scm | 98 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 99 insertions(+)
>  create mode 100644 gnu/packages/tree-sitter.scm
>
> diff --git a/gnu/local.mk b/gnu/local.mk
> index 3f09b167b1..3763df0821 100644
> --- a/gnu/local.mk
> +++ b/gnu/local.mk
> @@ -559,6 +559,7 @@ GNU_SYSTEM_MODULES =				\
>    %D%/packages/tls.scm				\
>    %D%/packages/tmux.scm				\
>    %D%/packages/tor.scm				\
> +  %D%/packages/tree-sitter.scm			\
>    %D%/packages/tv.scm				\
>    %D%/packages/uglifyjs.scm			\
>    %D%/packages/uml.scm				\
> diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
> new file mode 100644
> index 0000000000..2c749dc0a8
> --- /dev/null
> +++ b/gnu/packages/tree-sitter.scm
> @@ -0,0 +1,98 @@
> +;;; GNU Guix --- Functional package management for GNU
> +;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
> +;;;
> +;;; This file is part of GNU Guix.
> +;;;
> +;;; GNU Guix is free software; you can redistribute it and/or modify it
> +;;; under the terms of the GNU General Public License as published by
> +;;; the Free Software Foundation; either version 3 of the License, or (at
> +;;; your option) any later version.
> +;;;
> +;;; GNU Guix is distributed in the hope that it will be useful, but
> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;;; GNU General Public License for more details.
> +;;;
> +;;; You should have received a copy of the GNU General Public License
> +;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
> +
> +(define-module (gnu packages tree-sitter)
> +  #:use-module ((guix licenses) #:prefix license:)
> +  #:use-module (guix build-system cargo)
> +  #:use-module (guix git-download)
> +  #:use-module (guix packages)
> +  #:use-module (gnu packages crates-io)
> +  #:use-module (gnu packages crates-graphics))
> +
> +(define-public tree-sitter
> +  (package
> +    (name "tree-sitter")
> +    (version "0.19.5")
> +    (source
> +      (origin
> +        (method git-fetch)
> +        (uri (git-reference
> +              (url "https://github.com/tree-sitter/tree-sitter")
> +              (commit (string-append "v" version))))
> +        (file-name (git-file-name name version))
> +        (sha256
> +         (base32
> +          "1qmb0sva28zv6r3c3j7xs9pc8bpwwhkb9vxxndw2zbdn9wkvmbmn"))))
> +    (build-system cargo-build-system)

The tree-sitter package here both ships the `tree-sitter' command-line
program, written in Rust, and the runtime library, written in C.  I
think it would be better to split this into two packages, `tree-sitter'
for the runtime and `tree-sitter-cli` for the Rust program.

That way, a potential future Emacs will be able to link against
libtree-sitter *without* bringing in Rust as a hard dependency. I saw
that native tree-sitter support was being worked on on emacs-devel
list. One would still need Rust to actually use it, but at least it
remains an optional dependency.

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v2 00/33] gnu: Add tree-sitter.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 2071 bytes --]

Hi Guix!

Here's a V2 of the tree-sitter patch series. Changes:

- Split tree-sitter into tree-sitter enacs tree-sitter-cli.  This way
  applications can link against the tree-sitter runtime without
  depending on rust. Emacs may do that in the future :-).
- Fix the node build system changes to work with cross-compilation in
  the future.
- Use cc-for-target in tree-sitter-c.
- Add tree-sitter-bash, figured out how to package that one.

Thanks,
Pierre

Pierre Langlois (33):
  gnu: rust-chunked-transfer: Update to 1.4.0.
  gnu: Add rust-fdlimit.
  gnu: Add rust-html-escape.
  gnu: Add rust-smallbitvec.
  gnu: rust-tiny-http: Update to 0.8.
  gnu: rust-spin: Update to 0.7.
  gnu: Add tree-sitter.
  gnu: Add tree-sitter-cli.
  gnu: node: Patch /usr/bin/env in node-gyp.
  guix: node-build-system: Support compiling addons with node-gyp.
  gnu: Add node-nan.
  gnu: Add tree-sitter-c.
  gnu: Add tree-sitter-cpp.
  gnu: Add tree-sitter-bash.
  gnu: Add tree-sitter-css.
  gnu: Add tree-sitter-go.
  gnu: Add tree-sitter-html.
  gnu: Add tree-sitter-java.
  gnu: Add tree-sitter-javascript.
  gnu: Add tree-sitter-json.
  gnu: Add tree-sitter-julia.
  gnu: Add tree-sitter-php.
  gnu: Add tree-sitter-python.
  gnu: Add tree-sitter-rust.
  gnu: Add tree-sitter-typescript.
  gnu: Add rust-bindgen@0.56.
  gnu: Add rust-tree-sitter.
  gnu: rust-emacs-module: Update to 0.16.
  gnu: rust-emacs-macros: Update to 0.17.
  gnu: rust-emacs: Update to 0.17.
  gnu: Add emacs-tree-sitter-core.
  gnu: Add emacs-tree-sitter.
  gnu: Add emacs-tree-sitter-langs.

 gnu/local.mk                     |   1 +
 gnu/packages/crates-io.scm       | 290 +++++++++--
 gnu/packages/node-xyz.scm        |  30 ++
 gnu/packages/node.scm            |  31 +-
 gnu/packages/tree-sitter.scm     | 791 +++++++++++++++++++++++++++++++
 guix/build-system/node.scm       |  16 +
 guix/build/node-build-system.scm |  15 +
 7 files changed, 1141 insertions(+), 33 deletions(-)
 create mode 100644 gnu/packages/tree-sitter.scm

-- 
2.33.0

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v2 01/33] gnu: rust-chunked-transfer: Update to 1.4.0.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-chunked-transfer-1): New variable.
(rust-chunked-transfer-0.3): Inherit from rust-chunked-transfer-1.
---
 gnu/packages/crates-io.scm | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 087ffa1d27..3e93db1879 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -18,6 +18,7 @@
 ;;; Copyright © 2021 Antero Mejr <antero@kodmin.com>
 ;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
 ;;; Copyright © 2021 Vinicius Monego <monego@posteo.net>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -7884,19 +7885,21 @@ Python arrow.humanize.")
 TimeZone trait for @code{rust-chrono}.")
     (license (list license:expat license:asl2.0))))

-(define-public rust-chunked-transfer-0.3
+(define-public rust-chunked-transfer-1
   (package
     (name "rust-chunked-transfer")
-    (version "0.3.1")
+    (version "1.4.0")
     (source
      (origin
        (method url-fetch)
        (uri (crate-uri "chunked_transfer" version))
        (file-name (string-append name "-" version ".tar.gz"))
        (sha256
-        (base32 "11yghnd24w0i9p8g368c3pg7qh9nfz7kgri6pywja9pnmakj13a9"))))
+        (base32 "0bkdlsrszfcscw3j6yhs7kj6jbp8id47jjk6h9k58px47na5gy7z"))))
     (build-system cargo-build-system)
-    (arguments `(#:skip-build? #t))
+    (arguments
+     `(#:cargo-development-inputs
+       (("rust-criterion" ,rust-criterion-0.3))))
     (home-page "https://github.com/frewsxcv/rust-chunked-transfer")
     (synopsis "Encoder and decoder for HTTP chunked transfer coding")
     (description
@@ -7904,6 +7907,19 @@ TimeZone trait for @code{rust-chrono}.")
 coding.")
     (license license:asl2.0)))

+(define-public rust-chunked-transfer-0.3
+  (package (inherit rust-chunked-transfer-1)
+    (name "rust-chunked-transfer")
+    (version "0.3.1")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "chunked_transfer" version))
+       (file-name (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32 "11yghnd24w0i9p8g368c3pg7qh9nfz7kgri6pywja9pnmakj13a9"))))
+    (arguments `(#:skip-build? #t))))
+
 (define-public rust-ci-info-0.3
   (package
     (name "rust-ci-info")
--
2.33.0

[bug#49946] [PATCH v2 02/33] gnu: Add rust-fdlimit.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-fdlimit-0.1): New variable.
---
 gnu/packages/crates-io.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 3e93db1879..bfa14d5297 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -16008,6 +16008,28 @@ it.")
        (("rust-libc" ,rust-libc-0.2)
         ("rust-winapi" ,rust-winapi-0.3))))))

+(define-public rust-fdlimit-0.1
+  (package
+    (name "rust-fdlimit")
+    (version "0.1.4")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "fdlimit" version))
+       (file-name
+        (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "0g30d6gqkrwy8ylwdy7pqm443iq0p5dmnpz4ks41pirl7dclm98d"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-libc" ,rust-libc-0.2))))
+    (home-page "https://github.com/paritytech/fdlimit")
+    (synopsis "Raise the open file descriptors limit")
+    (description "Utility crate for raising file descriptors limit.")
+    (license license:asl2.0)))
+
 (define-public rust-femme-2
   (package
     (name "rust-femme")
--
2.33.0

[bug#49946] [PATCH v2 03/33] gnu: Add rust-html-escape.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-html-escape-0.2): New variable.
---
 gnu/packages/crates-io.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index bfa14d5297..31136a88e3 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -20921,6 +20921,32 @@ Hash-based Message Authentication Code algorithm} for SHA1.")
      "This package provides a library for HTML entity encoding and decoding.")
     (license (list license:asl2.0 license:expat license:mpl2.0))))

+(define-public rust-html-escape-0.2
+  (package
+    (name "rust-html-escape")
+    (version "0.2.9")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "html-escape" version))
+       (file-name
+        (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "1dxw9lpckrqzzqgbkw64ckbajr4b7xxnjdn8adgzqf2mm40shvl1"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-utf8-width" ,rust-utf8-width-0.1))
+       #:cargo-development-inputs
+       (("rust-bencher" ,rust-bencher-0.1))))
+    (home-page "https://magiclen.org/html-escape")
+    (synopsis "Library for encoding and escaping special characters in HTML")
+    (description
+     "This package provides a library for encoding and escaping special
+characters in HTML, decoding and unescaping HTML entities as well.")
+    (license license:expat)))
+
 (define-public rust-http-0.2
   (package
     (name "rust-http")
--
2.33.0

[bug#49946] [PATCH v2 04/33] gnu: Add rust-smallbitvec.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-smallbitvec-2): New variable.
---
 gnu/packages/crates-io.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 31136a88e3..de1b581cbd 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -44429,6 +44429,29 @@ I/O programming.")
         ("rust-futures-core-preview" ,rust-futures-core-preview-0.3)
         ("rust-futures-io-preview" ,rust-futures-io-preview-0.3))))))

+(define-public rust-smallbitvec-2
+  (package
+    (name "rust-smallbitvec")
+    (version "2.5.1")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "smallbitvec" version))
+       (file-name (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "0plrbldsjpwip3afbzd8fgrnvdhizcg5z4ncfqs4q6x4qjflzkkm"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-development-inputs
+       (("rust-bit-vec" ,rust-bit-vec-0.4)
+        ("rust-rand" ,rust-rand-0.4))))
+    (home-page "https://github.com/servo/smallbitvec")
+    (synopsis "A bit vector optimized for size and inline storage")
+    (description "This package provides a bit vector optimized for size and
+inline storage")
+    (license (list license:expat license:asl2.0))))
+
 (define-public rust-smallvec-1
   (package
     (name "rust-smallvec")
--
2.33.0

[bug#49946] [PATCH v2 05/33] gnu: rust-tiny-http: Update to 0.8.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-tiny-http-0.8): New variable.
(rust-tiny-http-0.6): Inherit from rust-tiny-http-0.8.
---
 gnu/packages/crates-io.scm | 39 ++++++++++++++++++++++++++++++++------
 1 file changed, 33 insertions(+), 6 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index de1b581cbd..07d0a7594e 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -49152,8 +49152,39 @@ closures after a delay or at a given timestamp.")
 manipulation in Rust.")
     (license license:expat)))

-(define-public rust-tiny-http-0.6
+(define-public rust-tiny-http-0.8
   (package
+    (name "rust-tiny-http")
+    (version "0.8.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "tiny_http" version))
+       (file-name
+        (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "0fcdwpb2ghk671qjjrk6048hs3yp7f681hxpr68gamk00181prcw"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-ascii" ,rust-ascii-1)
+        ("rust-chrono" ,rust-chrono-0.4)
+        ("rust-chunked-transfer" ,rust-chunked-transfer-1)
+        ("rust-log" ,rust-log-0.4)
+        ("rust-openssl" ,rust-openssl-0.10)
+        ("rust-url" ,rust-url-2))
+       #:cargo-development-inputs
+       (("rust-fdlimit" ,rust-fdlimit-0.1)
+        ("rust-rustc-serialize" ,rust-rustc-serialize-0.3)
+        ("rust-sha1" ,rust-sha1-0.6))))
+    (home-page "https://github.com/tiny-http/tiny-http")
+    (synopsis "Low level HTTP server library")
+    (description "This package provides a low level HTTP server library.")
+    (license license:asl2.0)))
+
+(define-public rust-tiny-http-0.6
+  (package (inherit rust-tiny-http-0.8)
     (name "rust-tiny-http")
     (version "0.6.2")
     (source
@@ -49172,11 +49203,7 @@ manipulation in Rust.")
         ("rust-chunked-transfer" ,rust-chunked-transfer-0.3)
         ("rust-log" ,rust-log-0.4)
         ("rust-openssl" ,rust-openssl-0.10)
-        ("rust-url" ,rust-url-1))))
-    (home-page "https://github.com/tiny-http/tiny-http")
-    (synopsis "Low level HTTP server library")
-    (description "This package provides a low level HTTP server library.")
-    (license license:asl2.0)))
+        ("rust-url" ,rust-url-1))))))

 (define-public rust-tiny-keccak-2
   (package
--
2.33.0

[bug#49946] [PATCH v2 06/33] gnu: rust-spin: Update to 0.7.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-spin-0.7): New variable.
(rust-spin-0.5): Inherit from rust-spin-0.7.
---
 gnu/packages/crates-io.scm | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 07d0a7594e..476d35c241 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -44838,10 +44838,10 @@ track of where each new file and line starts.")
      "An RSpec inspired minimal testing framework for Rust.")
     (license license:expat)))

-(define-public rust-spin-0.5
+(define-public rust-spin-0.7
   (package
     (name "rust-spin")
-    (version "0.5.2")
+    (version "0.7.1")
     (source
       (origin
         (method url-fetch)
@@ -44849,8 +44849,11 @@ track of where each new file and line starts.")
         (file-name (string-append name "-" version ".crate"))
         (sha256
          (base32
-          "0b84m6dbzrwf2kxylnw82d3dr8w06av7rfkr8s85fb5f43rwyqvf"))))
+          "0qjips9f6fsvkyd7wj3a4gzaqknn2q4kkb19957pl86im56pna0k"))))
     (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-lock-api" ,rust-lock-api-0.4))))
     (home-page "https://github.com/mvdnes/spin-rs")
     (synopsis "Synchronization primitives based on spinning")
     (description "This crate provides synchronization primitives based on
@@ -44858,6 +44861,21 @@ spinning.  They may contain data, are usable without @code{std},and static
 initializers are available.")
     (license license:expat)))

+(define-public rust-spin-0.5
+  (package
+    (inherit rust-spin-0.7)
+    (name "rust-spin")
+    (version "0.5.2")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "spin" version))
+        (file-name (string-append name "-" version ".crate"))
+        (sha256
+         (base32
+          "0b84m6dbzrwf2kxylnw82d3dr8w06av7rfkr8s85fb5f43rwyqvf"))))
+    (arguments '(#:cargo-inputs ()))))
+
 (define-public rust-spin-0.4
   (package
     (inherit rust-spin-0.5)
--
2.33.0

[bug#49946] [PATCH v2 07/33] gnu: Add tree-sitter.

[Pierre Langlois]

* gnu/package/tree-sitter.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Register it.
---
 gnu/local.mk                 |  1 +
 gnu/packages/tree-sitter.scm | 73 ++++++++++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+)
 create mode 100644 gnu/packages/tree-sitter.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index a0da1a96d8..0315106c30 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -559,6 +559,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/tls.scm				\
   %D%/packages/tmux.scm				\
   %D%/packages/tor.scm				\
+  %D%/packages/tree-sitter.scm			\
   %D%/packages/tv.scm				\
   %D%/packages/uglifyjs.scm			\
   %D%/packages/uml.scm				\
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
new file mode 100644
index 0000000000..e6e66cd447
--- /dev/null
+++ b/gnu/packages/tree-sitter.scm
@@ -0,0 +1,73 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages tree-sitter)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix git-download)
+  #:use-module (guix packages)
+  #:use-module (guix utils))
+
+(define-public tree-sitter
+  (package
+    (name "tree-sitter")
+    (version "0.19.5")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "1qmb0sva28zv6r3c3j7xs9pc8bpwwhkb9vxxndw2zbdn9wkvmbmn"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:tests? #f   ;; No check target.
+       #:phases
+       (modify-phases %standard-phases
+         (delete 'configure)
+         (add-before 'build 'set-cc
+           (lambda _
+             (setenv "CC" ,(cc-for-target))))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (lib (string-append out "/lib")))
+               (setenv "PREFIX" out)
+               (invoke "make" "install")))))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Incremental parsing system for programming tools")
+    (description "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @code{libtree-sitter} runtime library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 08/33] gnu: Add tree-sitter-cli.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cli): New variable.
---
 gnu/packages/tree-sitter.scm | 61 +++++++++++++++++++++++++++++++++++-
 1 file changed, 60 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e6e66cd447..d8d4a305da 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -18,10 +18,13 @@

 (define-module (gnu packages tree-sitter)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix git-download)
   #:use-module (guix packages)
-  #:use-module (guix utils))
+  #:use-module (guix utils)
+  #:use-module (gnu packages crates-graphics)
+  #:use-module (gnu packages crates-io))

 (define-public tree-sitter
   (package
@@ -71,3 +74,59 @@ can be embedded in any application.

 This package includes the @code{libtree-sitter} runtime library.")
     (license license:expat)))
+
+(define-public tree-sitter-cli
+  (package (inherit tree-sitter)
+    (name "tree-sitter-cli")
+    (build-system cargo-build-system)
+    (arguments
+     `(;; Running test requires downloading fixtures, see the
+       ;; script/fetch-fixtures script.
+       #:tests? #f
+       ;; FIXME: Installing the sources for the tree-sitter Rust bindings
+       ;; doesn't work out of the box due to tree-sitter having multiple
+       ;; Rust packages in the same repository (bindings and CLI).
+       #:install-source? #f
+       #:cargo-inputs
+       (("rust-ansi-term" ,rust-ansi-term-0.12)
+        ("rust-atty" ,rust-atty-0.2)
+        ("rust-clap" ,rust-clap-2)
+        ("rust-difference" ,rust-difference-2)
+        ("rust-dirs" ,rust-dirs-3)
+        ("rust-html-escape" ,rust-html-escape-0.2)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-smallbitvec" ,rust-smallbitvec-2)
+        ("rust-spin" ,rust-spin-0.7)
+        ("rust-tiny-http" ,rust-tiny-http-0.8)
+        ("rust-walkdir" ,rust-walkdir-2)
+        ("rust-webbrowser" ,rust-webbrowser-0.5)
+        ("rust-which" ,rust-which-4))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin"))
+                    (lib (string-append out "/lib")))
+               (mkdir-p bin)
+               (install-file "target/release/tree-sitter" bin)))))))
+    (synopsis "Incremental parsing system for programming tools")
+    (description "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @command{tree-sitter} command-line tool.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 09/33] gnu: node: Patch /usr/bin/env in node-gyp.

[Pierre Langlois]

* gnu/packages/node.scm (node)[arguments]: Fix /usr/bin/env shebang in
node-gyp.js.
(node-lts)[arguments]: Ditto.
---
 gnu/packages/node.scm | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
index 36c45e9c7a..530c04bc68 100644
--- a/gnu/packages/node.scm
+++ b/gnu/packages/node.scm
@@ -6,7 +6,7 @@
 ;;; Copyright © 2017 Mike Gerwitz <mtg@gnu.org>
 ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2018, 2019, 2020, 2021 Marius Bakke <marius@gnu.org>
-;;; Copyright © 2020 Pierre Langlois <pierre.langlois@gmx.com>
+;;; Copyright © 2020, 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;; Copyright © 2020 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
 ;;;
@@ -120,6 +120,11 @@
                (("'/usr/bin/env'")
                 (string-append "'" (which "env") "'")))

+             ;; Fix /usr/bin/env shebang in node-gyp.
+             (substitute* "deps/npm/node_modules/node-gyp/bin/node-gyp.js"
+               (("#!/usr/bin/env")
+                (string-append "#!" (assoc-ref inputs "coreutils") "/bin/env")))
+
              ;; FIXME: These tests fail in the build container, but they don't
              ;; seem to be indicative of real problems in practice.
              (for-each delete-file
@@ -661,6 +666,11 @@ source files.")
                  (("'/usr/bin/env'")
                   (string-append "'" (which "env") "'")))

+               ;; Fix /usr/bin/env shebang in node-gyp.
+               (substitute* "deps/npm/node_modules/node-gyp/bin/node-gyp.js"
+                 (("#!/usr/bin/env")
+                  (string-append "#!" (assoc-ref inputs "coreutils") "/bin/env")))
+
                ;; FIXME: These tests fail in the build container, but they don't
                ;; seem to be indicative of real problems in practice.
                (for-each delete-file
--
2.33.0

[bug#49946] [PATCH v2 10/33] guix: node-build-system: Support compiling addons with node-gyp.

[Pierre Langlois]

* gnu/packages/node.scm (node-headers): New function.
* guix/build-system/node.scm (python): New function.
(node-headers): New function.
(lower): Add node-headers and python to build inputs.
* guix/build/node-build-system.scm (configure-gyp): New function.
(%standard-phases): Add 'configure-gyp after 'configure.
---
 gnu/packages/node.scm            | 19 ++++++++++++++++++-
 guix/build-system/node.scm       | 16 ++++++++++++++++
 guix/build/node-build-system.scm | 15 +++++++++++++++
 3 files changed, 49 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
index 530c04bc68..a6dab75309 100644
--- a/gnu/packages/node.scm
+++ b/gnu/packages/node.scm
@@ -48,7 +48,24 @@
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
   #:use-module (gnu packages tls)
-  #:use-module (gnu packages web))
+  #:use-module (gnu packages web)
+  #:use-module (ice-9 match)
+  #:export (node-headers))
+
+(define (node-headers node)
+  "Return an <origin> object for a tarball with headers for the given node
+package version."
+  (let* ((version (package-version node))
+         (hash (match version
+                 ("10.24.0"
+                  "0h37zjwcpxjdqcxqjfj5zp1n5zjxaa0g8lsy83955afg5cca8p0n")
+                 ("14.16.0"
+                  "1vpdgq7kcw1a0w90lpvbvxbrc0n3pwjrs3sm42pjj7560clvji2b"))))
+    (origin
+      (method url-fetch)
+      (uri (string-append "https://nodejs.org/dist/v" version
+                          "/node-v" version "-headers.tar.gz"))
+      (sha256 (base32 hash)))))

 (define-public node
   (package
diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm
index 98f63f87ef..7828582a9a 100644
--- a/guix/build-system/node.scm
+++ b/guix/build-system/node.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2019 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -41,6 +42,19 @@
   (let ((node (resolve-interface '(gnu packages node))))
     (module-ref node 'node-lts)))

+(define (python)
+  "Return the python package."
+  ;; Lazily resolve the binding to avoid a circular dependency.
+  (let ((module (resolve-interface '(gnu packages python))))
+    (module-ref module 'python-wrapper)))
+
+(define (node-headers node)
+  "Return a tarball with headers for the given node, needed for packages that
+need to build native bindings using node-gyp."
+  ;; Lazily resolve the binding to avoid a circular dependency.
+  (let ((module (resolve-interface '(gnu packages node))))
+    ((module-ref module 'node-headers) node)))
+
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (node (default-node))
@@ -62,6 +76,8 @@
                         ;; Keep the standard inputs of 'gnu-build-system'.
                         ,@(standard-packages)))
          (build-inputs `(("node" ,node)
+                         ("node-headers" ,(node-headers node))
+                         ("python" ,(python))
                          ,@native-inputs))
          (outputs outputs)
          (build node-build)
diff --git a/guix/build/node-build-system.scm b/guix/build/node-build-system.scm
index 70a367618e..2993c49b2b 100644
--- a/guix/build/node-build-system.scm
+++ b/guix/build/node-build-system.scm
@@ -2,6 +2,7 @@
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
 ;;; Copyright © 2016, 2020 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2019, 2021 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -101,6 +102,19 @@
     (invoke npm "--offline" "--ignore-scripts" "install")
     #t))

+(define* (configure-gyp #:key inputs #:allow-other-keys)
+  "Run 'node-gyp configure' if we see a 'binding.gyp' file.
+
+By default, 'node-gyp' will try to download node headers from the internet, we
+prevent this with the '--tarball' flag."
+  (let ((node-gyp (string-append (assoc-ref inputs "node")
+                                 "/lib/node_modules/npm/node_modules/node-gyp"
+                                 "/bin/node-gyp.js")))
+    (if (file-exists? "binding.gyp")
+        (invoke node-gyp "--tarball" (assoc-ref inputs "node-headers")
+                "configure"))
+    #t))
+
 (define* (build #:key inputs #:allow-other-keys)
   (let ((package-meta (call-with-input-file "package.json" read-json)))
     (if (and=> (assoc-ref package-meta "scripts")
@@ -147,6 +161,7 @@
     (add-after 'unpack 'set-home set-home)
     (add-before 'configure 'patch-dependencies patch-dependencies)
     (replace 'configure configure)
+    (add-after 'configure 'configure-gyp configure-gyp)
     (replace 'build build)
     (replace 'check check)
     (add-before 'install 'repack repack)
--
2.33.0

[bug#49946] [PATCH v2 11/33] gnu: Add node-nan.

[Pierre Langlois]

---
 gnu/packages/node-xyz.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/node-xyz.scm b/gnu/packages/node-xyz.scm
index cbb68f77d6..eb4b59d1ab 100644
--- a/gnu/packages/node-xyz.scm
+++ b/gnu/packages/node-xyz.scm
@@ -3,6 +3,7 @@
 ;;; Copyright © 2020 Giacomo Leidi <goodoldpaul@autistici.org>
 ;;; Copyright © 2021 Noisytoot <noisytoot@disroot.org>
 ;;; Copyright © 2021 Charles <charles.b.jackson@protonmail.com>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -171,6 +172,35 @@ user-land JavaScript.")
 random number generator.")
     (license license:bsd-3)))

+(define-public node-nan
+  (package
+    (name "node-nan")
+    (version "2.15.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+               (url "https://github.com/nodejs/nan")
+               (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "18xslh9va5ld872scrp5y4251ax9s3c6qh0lnl1200lpzbsxy7yd"))))
+    (build-system node-build-system)
+    (arguments
+     '(#:tests? #f                      ; FIXME: tests depend on node-tap
+       #:phases
+       (modify-phases %standard-phases
+         ;; The default configure phase fails due to tap being missing, as we do
+         ;; not have tap packaged yet.  It is used only for tests.  This package
+         ;; still works as a dependency of node-glob and node-inflight.
+         (delete 'configure))))
+    (home-page "https://github.com/nodejs/nan")
+    (synopsis "Native Abstractions for Node.js")
+    (description "This package provides header files filled with macros and
+utilities for making add-on development for Node.js easier across versions.")
+    (license license:bsd-3)))
+
 (define-public node-oop
   ;; No releases, last commit was February 2013.
   (let ((commit "f9d87cda0958886955c14a0a716e57021ed295dc")
--
2.33.0

[bug#49946] [PATCH v2 12/33] gnu: Add tree-sitter-c.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-c): New variable.
---
 gnu/packages/tree-sitter.scm | 63 +++++++++++++++++++++++++++++++++++-
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d8d4a305da..bc04a4a5a7 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -20,11 +20,13 @@
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
+  #:use-module (guix build-system node)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
   #:use-module (gnu packages crates-graphics)
-  #:use-module (gnu packages crates-io))
+  #:use-module (gnu packages crates-io)
+  #:use-module (gnu packages node-xyz))

 (define-public tree-sitter
   (package
@@ -130,3 +132,62 @@ can be embedded in any application.

 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))
+
+(define-public tree-sitter-c
+  (package
+    (name "tree-sitter-c")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-c")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "1diys8yigvhm4ppbmp3a473yxjg2d5lk11y0ay7qprcz7233lakv"))))
+    (build-system node-build-system)
+    (native-inputs
+     `(("tree-sitter-cli" ,tree-sitter-cli)
+       ("node-nan" ,node-nan)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         ;; tree-sitter-cli is listed as a Node.js dependency, however the
+         ;; node tree-sitter-cli package is just a wrapper which downloads a
+         ;; tree-sitter binary, see
+         ;; https://github.com/tree-sitter/tree-sitter/tree/master/cli/npm
+         ;; Instead we remove it as a dependency so that we can use our own
+         ;; tree-sitter package.
+         (add-after 'unpack 'fix-configure
+           (lambda _
+             (with-atomic-file-replacement "package.json"
+               (lambda (in out)
+                 (use-modules ((guix build json)))
+                 (let ((package-meta (read-json in)))
+                   (assoc-remove! (assoc-ref package-meta "devDependencies")
+                                  "tree-sitter-cli")
+                   (write-json package-meta out))))))
+         (add-before 'build 'set-cc
+           (lambda _
+             (setenv "CC" ,(cc-for-target))))
+         (add-before 'build 'make-files-writable
+           (lambda _
+             (for-each make-file-writable (find-files "src" ".*"))))
+         (add-after 'install 'install-native-lib
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out")
+                                       "/lib/tree-sitter")))
+               ;; Invoking `tree-sitter test' loads the grammar which
+               ;; compiles it to a .so binary that we install.
+               (invoke "tree-sitter" "test")
+               (mkdir-p lib)
+               (copy-recursively (string-append (getenv "HOME")
+                                                "/.tree-sitter/bin")
+                                 lib)))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-c")
+    (synopsis "Tree-sitter C grammar")
+    (description
+      "This package provides a C grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 13/33] gnu: Add tree-sitter-cpp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cpp): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index bc04a4a5a7..07bdb54627 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -191,3 +191,27 @@ This package includes the @command{tree-sitter} command-line tool.")
     (description
       "This package provides a C grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-cpp
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-cpp")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-cpp")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "08ywv6n80sa541rr08bqz4zyg7byvjcabp68lvxmcahjk8xzcgwk"))))
+    (native-inputs
+     `(("tree-sitter-cli" ,tree-sitter-cli)
+       ("tree-sitter-c" ,tree-sitter-c)
+       ("node-nan" ,node-nan)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-cpp")
+    (synopsis "Tree-sitter C++ grammar")
+    (description
+      "This package provides a C++ grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 14/33] gnu: Add tree-sitter-bash.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-bash): New variable.
---
 gnu/packages/tree-sitter.scm | 41 ++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 07bdb54627..f98a7d392f 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -215,3 +215,44 @@ This package includes the @command{tree-sitter} command-line tool.")
     (description
       "This package provides a C++ grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-bash
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-bash")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-bash")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "18c030bb65r50i6z37iy7jb9z9i8i36y7b08dbc9bchdifqsijs5"))))
+    (native-inputs
+     `(("tree-sitter-cli" ,tree-sitter-cli)
+       ("node-nan" ,node-nan)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+          ;; The `prebuild' dependency is a way to intercept the build step
+          ;; and download a prebuilt binary instead.  If we remove it then the
+          ;; buid system falls back to building from source.
+          (add-after 'unpack 'remove-prebuild-dependency
+            (lambda _
+              (with-atomic-file-replacement "package.json"
+                (lambda (in out)
+                  (use-modules ((guix build json)))
+                  (let ((package-meta (read-json in)))
+                    (assoc-remove! (assoc-ref package-meta "dependencies")
+                                   "prebuild-install")
+                    (assoc-remove! (assoc-ref package-meta "devDependencies")
+                                   "prebuild")
+                    (write-json package-meta out))))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-bash")
+    (synopsis "Tree-sitter Bash grammar")
+    (description
+      "This package provides a Bash grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 15/33] gnu: Add tree-sitter-css.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-css): New variable.
---
 gnu/packages/tree-sitter.scm | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index f98a7d392f..592a61ed25 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -256,3 +256,23 @@ This package includes the @command{tree-sitter} command-line tool.")
     (description
       "This package provides a Bash grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-css
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-css")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-css")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "014jrlgi7zfza9g38hsr4vlbi8964i5p7iglaih6qmzaiml7bja2"))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-css")
+    (synopsis "Tree-sitter CSS grammar")
+    (description
+      "This package provides a CSS grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 16/33] gnu: Add tree-sitter-go.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-go): New variable.
---
 gnu/packages/tree-sitter.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 592a61ed25..5cb3a0e76b 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -276,3 +276,32 @@ This package includes the @command{tree-sitter} command-line tool.")
     (description
       "This package provides a CSS grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-go
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-go")
+    (version "0.19.1")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-go")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "0nxs47vd2fc2fr0qlxq496y852rwg39flhg334s7dlyq7d3lcx4x"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-go")
+    (synopsis "Tree-sitter Go grammar")
+    (description
+      "This package provides a Golang grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 17/33] gnu: Add tree-sitter-html.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-html): New variable.
---
 gnu/packages/tree-sitter.scm | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 5cb3a0e76b..e2a0757e30 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -305,3 +305,23 @@ This package includes the @command{tree-sitter} command-line tool.")
     (description
       "This package provides a Golang grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-html
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-html")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-html")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-html")
+    (synopsis "Tree-sitter HTML grammar")
+    (description
+      "This package provides a HTML grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 18/33] gnu: Add tree-sitter-java.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-java): New variable.
---
 gnu/packages/tree-sitter.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e2a0757e30..174f0923f8 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -325,3 +325,32 @@ This package includes the @command{tree-sitter} command-line tool.")
     (description
       "This package provides a HTML grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-java
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-java")
+    (version "0.19.1")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-java")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "07zw9ygb45hnvlx9qlz7rlz8hc3byjy03d24v72i5iyhpiiwlhvl"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-java")
+    (synopsis "Tree-sitter Java grammar")
+    (description
+      "This package provides a Java grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 19/33] gnu: Add tree-sitter-javascript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-javascript): New variable.
---
 gnu/packages/tree-sitter.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 174f0923f8..2be6b15cb0 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -354,3 +354,33 @@ This package includes the @command{tree-sitter} command-line tool.")
     (description
       "This package provides a Java grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-javascript
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-javascript")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-javascript")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "063va0s727yzhy1cz04fipzwwbq4af6fhgka6g970hk9yf7ggmnj"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
+    (synopsis "Tree-sitter Javascript grammar")
+    (description
+      "This package provides a Javascript grammar for the Tree-sitter
+library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 20/33] gnu: Add tree-sitter-json.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-json): New variable.
---
 gnu/packages/tree-sitter.scm | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 2be6b15cb0..74249b2aca 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -384,3 +384,23 @@ This package includes the @command{tree-sitter} command-line tool.")
       "This package provides a Javascript grammar for the Tree-sitter
 library.")
     (license license:expat)))
+
+(define-public tree-sitter-json
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-json")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-json")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "06pjh31bv9ja9hlnykk257a6zh8bsxg2fqa54al7qk1r4n9ksnff"))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-json")
+    (synopsis "Tree-sitter JSON grammar")
+    (description
+      "This package provides a JSON grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 21/33] gnu: Add tree-sitter-julia.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-julia): New variable.
---
 gnu/packages/tree-sitter.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 74249b2aca..36831b00b6 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -404,3 +404,32 @@ library.")
     (description
       "This package provides a JSON grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-julia
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-julia")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-julia")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "1pbnmvhy2gq4vg1b0sjzmjm4s2gsgdjh7h01yj8qrrqbcl29c463"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-julia")
+    (synopsis "Tree-sitter Julia grammar")
+    (description
+      "This package provides a Julia grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 22/33] gnu: Add tree-sitter-php.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-php): New variable.
---
 gnu/packages/tree-sitter.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 36831b00b6..d717bbf7b0 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -433,3 +433,32 @@ library.")
     (description
       "This package provides a Julia grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-php
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-php")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-php")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+           "17cmybgpprw7w9d2v7lmc6zmr90d70g0jqq279gzg0mpwfzla53s"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-php")
+    (synopsis "Tree-sitter PHP grammar")
+    (description
+      "This package provides a PHP grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 23/33] gnu: Add tree-sitter-python.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-python): New variable.
---
 gnu/packages/tree-sitter.scm | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d717bbf7b0..ae8bfcd9ff 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -462,3 +462,32 @@ library.")
     (description
       "This package provides a PHP grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-python
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-python")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-python")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "04b85qxqs64x6nhbpcgrzkbilxaiwvr9yd3h065rynv7rsdg0hii"))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-python")
+    (synopsis "Tree-sitter Python grammar")
+    (description
+      "This package provides a Python grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 24/33] gnu: Add tree-sitter-rust.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-rust): New variable.
---
 gnu/packages/tree-sitter.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index ae8bfcd9ff..8456a15607 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -24,6 +24,7 @@
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages algebra)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages node-xyz))
@@ -491,3 +492,27 @@ library.")
     (description
       "This package provides a Python grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-rust
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-rust")
+    (version "0.19.1")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-rust")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "118vkhv7n3sw8y9pi0987cgdcd74sjqwviijw01mhnk3bkyczi3l"))))
+    (native-inputs
+     `(("bc" ,bc)
+       ("tree-sitter-cli" ,tree-sitter-cli)
+       ("node-nan" ,node-nan)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-rust")
+    (synopsis "Tree-sitter Rust grammar")
+    (description
+      "This package provides a Rust grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 25/33] gnu: Add tree-sitter-typescript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-typescript): New variable.
---
 gnu/packages/tree-sitter.scm | 60 ++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 8456a15607..001322dd84 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -516,3 +516,63 @@ library.")
     (description
       "This package provides a Rust grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-typescript
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-typescript")
+    (version "0.19.0")
+    (source
+      (origin
+        (method git-fetch)
+        (uri (git-reference
+              (url "https://github.com/tree-sitter/tree-sitter-typescript")
+              (commit (string-append "v" version))))
+        (file-name (git-file-name name version))
+        (sha256
+         (base32
+          "01pkmwwmbv6kxda0n6g4cfg72ldmpi9gmp11a5gygn472vhrh2xw"))))
+    (native-inputs
+     `(("tree-sitter-cli" ,tree-sitter-cli)
+       ("tree-sitter-javascript" ,tree-sitter-javascript)
+       ("node-nan" ,node-nan)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))
+         (add-after 'build 'build-gyp
+           (lambda* (#:key inputs #:allow-other-keys)
+             (let ((node (assoc-ref inputs "node")))
+               (invoke (string-append node "/lib/node_modules/npm/node_modules"
+                                           "/node-gyp/bin/node-gyp.js")
+                       "build"))))
+         (replace 'make-files-writable
+           (lambda _
+             (for-each make-file-writable
+                       (find-files "typescript" ".*"))
+             (for-each make-file-writable
+                       (find-files "tsx" ".*"))))
+         (replace 'install-native-lib
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out")
+                                       "/lib/tree-sitter")))
+               ;; Invoking `tree-sitter test' loads the grammar which
+               ;; compiles it to a .so binary that we install.
+               (with-directory-excursion "typescript"
+                 (invoke "tree-sitter" "test"))
+               (with-directory-excursion "tsx"
+                 (invoke "tree-sitter" "test"))
+               (mkdir-p lib)
+               (copy-recursively (string-append (getenv "HOME")
+                                                "/.tree-sitter/bin")
+                                 lib))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-typescript")
+    (synopsis "Tree-sitter Typescript grammar")
+    (description
+      "This package provides Typescript and TSX grammars for the Tree-sitter
+library.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 26/33] gnu: Add rust-bindgen@0.56.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-bindgen-0.56): New variable.
---
 gnu/packages/crates-io.scm | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 476d35c241..f769a8e9a5 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -4541,6 +4541,43 @@ bindings to C and C++ libraries.")
         ("rust-shlex" ,rust-shlex-0.1)
         ("rust-which" ,rust-which-3))))))

+(define-public rust-bindgen-0.56
+  (package
+    (inherit rust-bindgen-0.57)
+    (name "rust-bindgen")
+    (version "0.56.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (crate-uri "bindgen" version))
+       (file-name (string-append name "-" version ".tar.gz"))
+       (sha256
+        (base32
+         "0fajmgk2064ca1z9iq1jjkji63qwwz38z3d67kv6xdy0xgdpk8rd"))))
+    (arguments
+     `(#:skip-build? #t
+       #:cargo-inputs
+       (("rust-bitflags" ,rust-bitflags-1)
+        ("rust-cexpr" ,rust-cexpr-0.4)
+        ("rust-cfg-if" ,rust-cfg-if-0.1)
+        ("rust-clang-sys" ,rust-clang-sys-1)
+        ("rust-clap" ,rust-clap-2)
+        ("rust-env-logger" ,rust-env-logger-0.8)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-lazycell" ,rust-lazycell-1)
+        ("rust-log" ,rust-log-0.4)
+        ("rust-peeking-take-while" ,rust-peeking-take-while-0.1)
+        ("rust-proc-macro2" ,rust-proc-macro2-1)
+        ("rust-quote" ,rust-quote-1)
+        ("rust-regex" ,rust-regex-1)
+        ("rust-rustc-hash" ,rust-rustc-hash-1)
+        ("rust-shlex" ,rust-shlex-0.1)
+        ("rust-which" ,rust-which-3))
+       #:cargo-development-inputs
+       (("rust-clap" ,rust-clap-2)
+        ("rust-diff" ,rust-diff-0.1)
+        ("rust-shlex" ,rust-shlex-0.1))))))
+
 (define-public rust-bindgen-0.55
   (package
     (inherit rust-bindgen-0.57)
--
2.33.0

[bug#49946] [PATCH v2 27/33] gnu: Add rust-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 001322dd84..3357fce4a5 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -21,6 +21,7 @@
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system node)
+  #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
@@ -134,6 +135,32 @@ can be embedded in any application.
 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))

+(define-public rust-tree-sitter-0.19
+  (package
+    (name "rust-tree-sitter")
+    (version "0.19.5")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "tree-sitter" version))
+        (file-name (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1h6adq5kqf4izzsklch5lfxx2aisxga463zz7w44rgwnck16wwmd"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:tests? #f  ;; Running tests misinterprets comments as doc-tests.
+       #:cargo-inputs
+       (("rust-cc" ,rust-cc-1)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-regex" ,rust-regex-1)
+        ("rust-spin" ,rust-spin-0.7))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Rust bindings to the Tree-sitter parsing library")
+    (description "This package provides Rust bindings to the Tree-sitter
+parsing library.")
+    (license license:expat)))
+
 (define-public tree-sitter-c
   (package
     (name "tree-sitter-c")
--
2.33.0

[bug#49946] [PATCH v2 28/33] gnu: rust-emacs-module: Update to 0.16.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-module-0.16): New variable.
(rust-emacs-module-0.10): Inherit from rust-emacs-module-0.10.
---
 gnu/packages/crates-io.scm | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index f769a8e9a5..e4892d5852 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -14697,10 +14697,10 @@ Emacs' support for dynamic modules.")
     (description "This package provides proc macros for Emacs modules.")
     (license license:bsd-3)))

-(define-public rust-emacs-module-0.10
+(define-public rust-emacs-module-0.16
   (package
     (name "rust-emacs-module")
-    (version "0.10.0")
+    (version "0.16.2")
     (source
       (origin
         (method url-fetch)
@@ -14709,11 +14709,11 @@ Emacs' support for dynamic modules.")
          (string-append name "-" version ".tar.gz"))
         (sha256
          (base32
-          "1gf9lz735xbkyir53dyv362drfx3nin5an5cx39kd8q8kjjwix5g"))))
+          "0s8qrh0ggjmqr04zkcf7s4ijmpd44rjcag78npnq64jv10lxvsry"))))
     (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
-       (("rust-bindgen" ,rust-bindgen-0.48))))
+       (("rust-bindgen" ,rust-bindgen-0.56))))
     (inputs
      `(("clang" ,clang)))
     (home-page "https://github.com/ubolonton/emacs-module-rs")
@@ -14722,6 +14722,23 @@ Emacs' support for dynamic modules.")
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

+(define-public rust-emacs-module-0.10
+  (package (inherit rust-emacs-module-0.16)
+    (name "rust-emacs-module")
+    (version "0.10.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs_module" version))
+        (file-name
+         (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1gf9lz735xbkyir53dyv362drfx3nin5an5cx39kd8q8kjjwix5g"))))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-bindgen" ,rust-bindgen-0.48))))))
+
 (define-public rust-embed-resource-1
   (package
     (name "rust-embed-resource")
--
2.33.0

[bug#49946] [PATCH v2 29/33] gnu: rust-emacs-macros: Update to 0.17.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-macros-0.17): New variable.
(rust-emacs-macros-0.11): Inherit from rust-emacs-macros-0.17.
---
 gnu/packages/crates-io.scm | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index e4892d5852..d7375d7e56 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -14673,8 +14673,31 @@ signing, and verification in pure Rust.")
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

-(define-public rust-emacs-macros-0.11
+(define-public rust-emacs-macros-0.17
   (package
+    (name "rust-emacs-macros")
+    (version "0.17.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs-macros" version))
+        (file-name (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "0qg1dcn5acbirq617qq2fgg9adswif2dnr292s3qnq62wzgnyrb9"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-darling" ,rust-darling-0.10)
+        ("rust-quote" ,rust-quote-1)
+        ("rust-syn" ,rust-syn-1))))
+    (home-page "https://github.com/ubolonton/emacs-module-rs")
+    (synopsis "Proc macros for Emacs modules")
+    (description "This package provides proc macros for Emacs modules.")
+    (license license:bsd-3)))
+
+(define-public rust-emacs-macros-0.11
+  (package (inherit rust-emacs-macros-0.17)
     (name "rust-emacs-macros")
     (version "0.11.0")
     (source
@@ -14686,16 +14709,11 @@ Emacs' support for dynamic modules.")
         (sha256
          (base32
           "0390y8vafxdi334hhgrzvcqjq3n5ckcmvilqcfp8vajjq8irrly6"))))
-    (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
        (("rust-darling" ,rust-darling-0.9)
         ("rust-quote" ,rust-quote-0.6)
-        ("rust-syn" ,rust-syn-0.15))))
-    (home-page "https://github.com/ubolonton/emacs-module-rs")
-    (synopsis "Proc macros for Emacs modules")
-    (description "This package provides proc macros for Emacs modules.")
-    (license license:bsd-3)))
+        ("rust-syn" ,rust-syn-0.15))))))

 (define-public rust-emacs-module-0.16
   (package
--
2.33.0

[bug#49946] [PATCH v2 30/33] gnu: rust-emacs: Update to 0.17.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-0.17): New variable.
(rust-emacs-0.11): Inherit from rust-emacs-0.17.
---
 gnu/packages/crates-io.scm | 38 +++++++++++++++++++++++++++++++-------
 1 file changed, 31 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index d7375d7e56..18538650f8 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -14642,10 +14642,10 @@ signing, and verification in pure Rust.")
 @code{Right} is a general purpose sum type with two cases.")
     (license (list license:expat license:asl2.0))))

-(define-public rust-emacs-0.11
+(define-public rust-emacs-0.17
   (package
     (name "rust-emacs")
-    (version "0.11.0")
+    (version "0.17.0")
     (source
       (origin
         (method url-fetch)
@@ -14654,16 +14654,18 @@ signing, and verification in pure Rust.")
          (string-append name "-" version ".tar.gz"))
         (sha256
          (base32
-          "1xn6nr8m3p24irlmv3i7c9n95yb10qc6aikwqv99qlmhbky5x0z7"))))
+          "02f31apvz03dr0dwn5wyirdmqkk2d635iki0kmsh291xr6dysy2q"))))
     (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
-       (("rust-ctor" ,rust-ctor-0.1)
-        ("rust-emacs-macros" ,rust-emacs-macros-0.11)
-        ("rust-emacs-module" ,rust-emacs-module-0.10)
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-ctor" ,rust-ctor-0.1)
+        ("rust-emacs-macros" ,rust-emacs-macros-0.17)
+        ("rust-emacs-module" ,rust-emacs-module-0.16)
         ("rust-failure" ,rust-failure-0.1)
         ("rust-failure-derive" ,rust-failure-derive-0.1)
-        ("rust-lazy-static" ,rust-lazy-static-1))))
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-thiserror" ,rust-thiserror-1))))
     (inputs
      `(("clang" ,clang)))
     (home-page "https://github.com/ubolonton/emacs-module-rs")
@@ -14673,6 +14675,28 @@ signing, and verification in pure Rust.")
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

+(define-public rust-emacs-0.11
+  (package (inherit rust-emacs-0.17)
+    (name "rust-emacs")
+    (version "0.11.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs" version))
+        (file-name
+         (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1xn6nr8m3p24irlmv3i7c9n95yb10qc6aikwqv99qlmhbky5x0z7"))))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-ctor" ,rust-ctor-0.1)
+        ("rust-emacs-macros" ,rust-emacs-macros-0.11)
+        ("rust-emacs-module" ,rust-emacs-module-0.10)
+        ("rust-failure" ,rust-failure-0.1)
+        ("rust-failure-derive" ,rust-failure-derive-0.1)
+        ("rust-lazy-static" ,rust-lazy-static-1))))))
+
 (define-public rust-emacs-macros-0.17
   (package
     (name "rust-emacs-macros")
--
2.33.0

[bug#49946] [PATCH v2 31/33] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
(emacs-tree-sitter-core): New variable.
---
 gnu/packages/tree-sitter.scm | 79 ++++++++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 3357fce4a5..13f7a63acc 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -19,6 +19,7 @@
 (define-module (gnu packages tree-sitter)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
+  #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system node)
   #:use-module (guix download)
@@ -28,6 +29,7 @@
   #:use-module (gnu packages algebra)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
+  #:use-module (gnu packages llvm)
   #:use-module (gnu packages node-xyz))

 (define-public tree-sitter
@@ -603,3 +605,80 @@ library.")
       "This package provides Typescript and TSX grammars for the Tree-sitter
 library.")
     (license license:expat)))
+
+;; Local package definition solely for building the native emacs module
+;; written in Rust.
+(define tree-sitter-emacs-module
+  (package
+    (name "tree-sitter-emacs-module")
+    (version "0.15.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "15y0wjnck8rbfhl0xrl71ci7clbcp11lhqil5l8ykprsdjv0c2as"))))
+    (build-system cargo-build-system)
+    (inputs
+     `(("clang" ,clang)))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-emacs" ,rust-emacs-0.17)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-tree-sitter" ,rust-tree-sitter-0.19))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (add-after 'delete-cargo.lock 'do-not-fetch-from-github
+           (lambda _
+             (substitute* "Cargo.toml"
+               (("\\[patch.*") "")
+               (("git = .*") ""))))
+         (add-after 'do-not-fetch-from-github 'chdir
+           (lambda _ (chdir "core")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out") "/lib")))
+               (install-file "../target/release/libtsc_dyn.so" lib)))))))
+    (home-page #f)
+    (synopsis #f)
+    (description #f)
+    (license license:expat)))
+
+(define-public emacs-tree-sitter-core
+  (package
+    (name "emacs-tree-sitter-core")
+    (version (package-version tree-sitter-emacs-module))
+    (source (package-source tree-sitter-emacs-module))
+    (build-system emacs-build-system)
+    (native-inputs
+     `(("tree-sitter-emacs-module" ,tree-sitter-emacs-module)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'install 'install-module
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out")))
+                   (module (string-append (assoc-ref inputs "tree-sitter-emacs-module")
+                                          "/lib/libtsc_dyn.so")))
+               ;; Writing "LOCAL" in this file prevents the package from
+               ;; trying to download the module from the internet.
+               (call-with-output-file (string-append elpa "/DYN-VERSION")
+                 (lambda (port) (display "LOCAL" port)))
+               (substitute* "tsc-dyn-get.el"
+                 (("defcustom tsc-dyn-dir tsc--dir")
+                  (string-append "defcustom tsc-dyn-dir \"" elpa "\"")))
+               (copy-file module (string-append elpa "/tsc-dyn.so"))))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp, core library")
+    (description "This package provides core APIs of the Emacs binding for
+Tree-sitter, an incremental parsing system.")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 32/33] gnu: Add emacs-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 13f7a63acc..0454d75ad4 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -682,3 +682,30 @@ library.")
     (description "This package provides core APIs of the Emacs binding for
 Tree-sitter, an incremental parsing system.")
     (license license:expat)))
+
+(define-public emacs-tree-sitter
+  (package
+    (name "emacs-tree-sitter")
+    (version (package-version emacs-tree-sitter-core))
+    (source (package-source emacs-tree-sitter-core))
+    (build-system emacs-build-system)
+    (propagated-inputs
+     `(("emacs-tree-sitter-core" ,emacs-tree-sitter-core)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "lisp"))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp")
+    (description "This package provides Emacs bindings for Tree-sitter, an
+incremental parsing library.  It aims to be the foundation for a new breed of
+Emacs packages that understand code structurally.  For example:
+
+@enumerate
+@item Faster, fine-grained code highlighting.
+@item More flexible code folding.
+@item Structural editing (like Paredit, or even better) for non-Lisp code.
+@item More informative indexing for imenu.
+@end enumerate")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 33/33] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter-langs): New variable.
---
 gnu/packages/tree-sitter.scm | 80 ++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 0454d75ad4..6bb52b68ac 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -709,3 +709,83 @@ Emacs packages that understand code structurally.  For example:
 @item More informative indexing for imenu.
 @end enumerate")
     (license license:expat)))
+
+(define-public emacs-tree-sitter-langs
+  (package
+    (name "emacs-tree-sitter-langs")
+    (version "0.10.3")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/tree-sitter-langs")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0fa0xi8sky5h3rscnvcci9djdi71qj30vivzzvx7w8pqkmsy1vq7"))))
+    (build-system emacs-build-system)
+    ;; FIXME: Support for some languages, such as bash, is still missing from
+    ;; the bundle.
+    (inputs
+     `(("tree-sitter-bash" ,tree-sitter-bash)
+       ("tree-sitter-c" ,tree-sitter-c)
+       ("tree-sitter-cpp" ,tree-sitter-cpp)
+       ("tree-sitter-css" ,tree-sitter-css)
+       ("tree-sitter-go" ,tree-sitter-go)
+       ("tree-sitter-html" ,tree-sitter-html)
+       ("tree-sitter-java" ,tree-sitter-java)
+       ("tree-sitter-javascript" ,tree-sitter-javascript)
+       ("tree-sitter-json" ,tree-sitter-json)
+       ("tree-sitter-julia" ,tree-sitter-julia)
+       ("tree-sitter-php" ,tree-sitter-php)
+       ("tree-sitter-python" ,tree-sitter-python)
+       ("tree-sitter-rust" ,tree-sitter-rust)
+       ("tree-sitter-typescript" ,tree-sitter-typescript)))
+    (propagated-inputs
+     `(("emacs-tree-sitter" ,emacs-tree-sitter)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-before 'build 'fake-bundle
+           (lambda _
+             ;; The BUNDLE-VERSION file prevents the package from
+             ;; downloading libraries at load time.
+             (call-with-output-file "bin/BUNDLE-VERSION"
+               (lambda (port) (display ,version port)))))
+         (add-after 'install 'install-libs
+           (lambda* (#:key outputs inputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+               (define (install-lib lang)
+                 (let ((input (assoc-ref inputs (string-append "tree-sitter-" lang))))
+                   (copy-file (string-append input "/lib/tree-sitter/" lang ".so")
+                              (string-append elpa "/bin/" lang ".so"))))
+               (mkdir-p (string-append elpa "/bin"))
+               ;; The BUNDLE-VERSION file prevents the package from
+               ;; downloading libraries at load time.
+               (call-with-output-file (string-append elpa "/bin/BUNDLE-VERSION")
+                 (lambda (port) (display ,version port)))
+               (for-each (lambda (input)
+                           (use-modules (ice-9 regex))
+                           (cond
+                            ((string-match "^tree-sitter-(.*)$" (car input))
+                             => (lambda (m)
+                                  (install-lib (match:substring m 1))))))
+                         inputs))))
+         (add-after 'install 'install-queries
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+                (copy-recursively "queries" (string-append elpa "/queries"))))))))
+    (home-page "https://ubolonton.github.io/emacs-tree-sitter/languages/")
+    (synopsis "Language support bundle for Tree-sitter")
+    (description "This package is a convenient language bundle for
+Tree-sitter.  For each supported language, this package provides:
+
+@enumerate
+@item Pre-compiled grammar binaries.
+@item An optional highlights.scm file that provides highlighting patterns.
+This is mainly intended for major modes that are not aware of tree-sitter.
+@item Optional query patterns for other minor modes that provide high-level
+functionalities on top of tree-sitter, such as code folding, evil text
+objects, ...etc.
+@end enumerate")
+    (license license:expat)))
--
2.33.0

[bug#49946] [PATCH v2 33/33] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]


Pierre Langlois <pierre.langlois@gmx.com> writes:

> * gnu/packages/tree-sitter.scm (emacs-tree-sitter-langs): New variable.
> ---
>  gnu/packages/tree-sitter.scm | 80 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 80 insertions(+)
>
> diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
> index 0454d75ad4..6bb52b68ac 100644
> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm
> @@ -709,3 +709,83 @@ Emacs packages that understand code structurally.  For example:
>  @item More informative indexing for imenu.
>  @end enumerate")
>      (license license:expat)))
> +
> +(define-public emacs-tree-sitter-langs
> +  (package
> +    (name "emacs-tree-sitter-langs")
> +    (version "0.10.3")
> +    (source (origin
> +              (method git-fetch)
> +              (uri (git-reference
> +                    (url "https://github.com/emacs-tree-sitter/tree-sitter-langs")
> +                    (commit version)))
> +              (file-name (git-file-name name version))
> +              (sha256
> +               (base32
> +                "0fa0xi8sky5h3rscnvcci9djdi71qj30vivzzvx7w8pqkmsy1vq7"))))
> +    (build-system emacs-build-system)
> +    ;; FIXME: Support for some languages, such as bash, is still missing from
> +    ;; the bundle.

Whoops, I forgot to update the comment after adding support for bash,
I'll fix that locally!

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH 08/31] gnu: node: Patch /usr/bin/env in node-gyp.

[Philip McGrath]

I'm interested in the node-gyp part of this, which has come up in some 
other software I'm trying to package. These comments come with the 
caveat that my experience with node.js and npm is fairly shallow.

On 8/10/21 2:28 PM, Maxime Devos wrote:
> Pierre Langlois schreef op ma 09-08-2021 om 00:33 [+0100]:
>> @@ -120,6 +120,10 @@
>>                  (("'/usr/bin/env'")
>>                   (string-append "'" (which "env") "'")))
>>
>> +             ;; Fix /usr/bin/env shebang in node-gyp.
>> +             (substitute* "deps/npm/node_modules/node-gyp/bin/node-gyp.js"
>> +               (("#!/usr/bin/env") (string-append "#!" (which "env"))))
> 
> For cross-compilation, this should most likely be
> (string-append (assoc-ref inputs "coreutils") "/bin/env")
> or something like that instead.  Likewise in other places.

Since the shebang line for node-gyp is specifically "#!/usr/bin/env 
node", I wonder if it should use the node built by this package, rather 
than a dynamic node.

More generally, I see that there are 355 directories installed under 
"lib/node_modules/npm/node_modules" (which corresponds to the "deps" 
path above). Most of them don't seem to be available as Guix packages 
that could be depended upon by other Guix node packages. I'd guess 
node-gyp may not be the only one with shebangs that ought to be patched.

On 8/8/21 6:29 PM, Pierre Langlois wrote:

 >    ... `node-gyp' needs

 >    node headers to compile against, packaged as a tarball, which it tries

 >    to download.  Instead, we can run a `node-gyp --tarball <> configure'

 >    step to manually provide the tarball, which we can package separately

 >    for any given node version.

There is also a --nodedir option, which I found could work with 
something like:

     (string-append "--nodedir=" (assoc-ref inputs "node"))

That seems like it might be better, though I don't know all the 
considerations for cross-compilation and such.

-Philip

[bug#49946] [PATCH 08/31] gnu: node: Patch /usr/bin/env in node-gyp.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 4878 bytes --]

Hi Philip,

Philip McGrath <philip@philipmcgrath.com> writes:

> I'm interested in the node-gyp part of this, which has come up in some other
> software I'm trying to package. These comments come with the caveat that my
> experience with node.js and npm is fairly shallow.

Thanks for your feedback!

>
> On 8/10/21 2:28 PM, Maxime Devos wrote:
>> Pierre Langlois schreef op ma 09-08-2021 om 00:33 [+0100]:
>>> @@ -120,6 +120,10 @@
>>>                  (("'/usr/bin/env'")
>>>                   (string-append "'" (which "env") "'")))
>>>
>>> +             ;; Fix /usr/bin/env shebang in node-gyp.
>>> +             (substitute* "deps/npm/node_modules/node-gyp/bin/node-gyp.js"
>>> +               (("#!/usr/bin/env") (string-append "#!" (which "env"))))
>> For cross-compilation, this should most likely be
>> (string-append (assoc-ref inputs "coreutils") "/bin/env")
>> or something like that instead.  Likewise in other places.
>
> Since the shebang line for node-gyp is specifically "#!/usr/bin/env node", I
> wonder if it should use the node built by this package, rather than a dynamic
> node.

Yeah we could do that, although I generally prefer to follow whatever
the script already does, there could be a good reason for them to use
`env' no?

> More generally, I see that there are 355 directories installed under
> "lib/node_modules/npm/node_modules" (which corresponds to the "deps" 
> path above). Most of them don't seem to be available as Guix packages that could
> be depended upon by other Guix node packages.

Yeah that's tricky, ideally we should remove all the node_modules deps
and package them separately, I wonder if anybody tried to do that
already. I would suspect it to be quite a lot of work, sometimes
unbundling stops being worth and when it's hard to maintain dependencies
manually.

Hopefully we can get there one day though! I don't want to deter anybody
from trying :-), I might give it a go on a raindy day.

> I'd guess node-gyp may not be the only one with shebangs that ought to
> be patched.

Yeah there could be others, although normally the patching phase from
the gnu build system should have taken care of most of them, hopefully
all, I'm not sure why it didn't work for /usr/bin/env though.

I would suggest we patch things as we encounter them, did you find
anymore issues when working on your package?

For instance, while working on a newer version of one of the packages in
this series, I saw we may need to patch GYP's python reference as well,
like so:

(substitute* "deps/npm/node_modules/node-gyp/gyp/gyp_main.py"
  (("#!/usr/bin/env python")
   (string-append "#!" (assoc-ref inputs "python") "/bin/python3")))

Only for node 14+. The reason seems to be that gyp still refers to
"python", but python2 is no longer a dependency for newer nodes. And it
seems GYP is perfectly happy with python3, and the shebang is fixed
upstream so a never node will be fine:
https://github.com/nodejs/node-gyp/pull/2355/files

Maybe updating node would be better than this fix though.

> On 8/8/21 6:29 PM, Pierre Langlois wrote:
>
>>    ... `node-gyp' needs
>
>>    node headers to compile against, packaged as a tarball, which it tries
>
>>    to download.  Instead, we can run a `node-gyp --tarball <> configure'
>
>>    step to manually provide the tarball, which we can package separately
>
>>    for any given node version.
>
> There is also a --nodedir option, which I found could work with something like:
>
>     (string-append "--nodedir=" (assoc-ref inputs "node"))
>
> That seems like it might be better, though I don't know all the considerations
> for cross-compilation and such.

Oh that's a good idea, I didn't really like having to download the
headers separately from the main package, especially given we run
snippet on the source to remove bundled dependencies.

Trying this out this approach does work, but I needed to:

  - Create a union directory with both node and libuv. The node package
    only has headers for V8/node, but we also need libuv, so doing
    something like this works:

    (union-build node-sources
                (list (assoc-ref inputs "node")
                      (assoc-ref inputs "libuv"))
                #:create-all-directories? #t
                #:log-port (%make-void-port "w"))

  - For some reason, --nodedir didn't really "configure" gyp to use that
    node directory, I think it's meant to be passed everytime you run
    any gyp command. Instead I found that you can use and environment
    variable:

    (setenv "npm_config_nodedir" node-sources)

And that works for the packages in this series!  That'll be much better
than before, I'll do it this way.

Thanks again for taking a look, I'll see if I can send updated patches
sometimes this weekend.

Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH 08/31] gnu: node: Patch /usr/bin/env in node-gyp.

[Philip McGrath]

Hi Pierre,

On 9/25/21 6:24 AM, Pierre Langlois wrote:
> Philip McGrath <philip@philipmcgrath.com> writes:
>> Since the shebang line for node-gyp is specifically "#!/usr/bin/env node", I
>> wonder if it should use the node built by this package, rather than a dynamic
>> node.
> 
> Yeah we could do that, although I generally prefer to follow whatever
> the script already does, there could be a good reason for them to use
> `env' no
I think it might be better to use `patch-shebang` from `(guix build 
utils)` rather than `substitute*` these by hand, and it seems that 
`patch-shebang` removed the indirection through `env`. My guess is most 
of these cases are to accommodate the fact that `node` and `python` are 
often installed to places other than `/usr/bin`.

 >> I'd guess node-gyp may not be the only one with shebangs that ought to

 >> be patched.

 >

 > Yeah there could be others, although normally the patching phase from

 > the gnu build system should have taken care of most of them, hopefully

 > all, I'm not sure why it didn't work for /usr/bin/env though.

 >

 > I would suggest we patch things as we encounter them, did you find

 > anymore issues when working on your package?


Looking at `gnu-build-system`, it seems that the `'patch-shebangs` phase 
only operates on files installed in the "/bin" and "/sbin" 
subdirectories of the package's outputs. That restriction doesn't make 
sense to me in general: for instance, what about "/libexec"? For Node 
specifically, this misses a lot of stuff under "/lib/node_modules" and 
"/lib/node_modules/npm/node_modules". I think I more general fix could 
subsume the `'patch-npm-shebang` and `'patch-node-shebang` phases in 
building Node, too.

 > For instance, while working on a newer version of one of the packages in

 > this series, I saw we may need to patch GYP's python reference as well,

 > like so:

 >

 > (substitute* "deps/npm/node_modules/node-gyp/gyp/gyp_main.py"

 >    (("#!/usr/bin/env python")

 >     (string-append "#!" (assoc-ref inputs "python") "/bin/python3")))

 >

 > Only for node 14+. The reason seems to be that gyp still refers to

 > "python", but python2 is no longer a dependency for newer nodes. And it

 > seems GYP is perfectly happy with python3, and the shebang is fixed

 > upstream so a never node will be fine:

 > https://github.com/nodejs/node-gyp/pull/2355/files


I think in some places (but perhaps not enough places) Guix uses 
`python-wrapper` to work around this ...

 >

 > Maybe updating node would be better than this fix though.

I'm not totally clear on whether the upstream fix is in 14.17.6 LTS, 
but, if so, that seems great!

> 
>> More generally, I see that there are 355 directories installed under
>> "lib/node_modules/npm/node_modules" (which corresponds to the "deps"
>> path above). Most of them don't seem to be available as Guix packages that could
>> be depended upon by other Guix node packages.
> 
> Yeah that's tricky, ideally we should remove all the node_modules deps
> and package them separately, I wonder if anybody tried to do that
> already. I would suspect it to be quite a lot of work, sometimes
> unbundling stops being worth and when it's hard to maintain dependencies
> manually.
> 
> Hopefully we can get there one day though! I don't want to deter anybody
> from trying :-), I might give it a go on a raindy day.

Since these are developed and released with Node, and apparently we can 
build them as part of the Node build process, I was thinking we could 
just make packages that point to these versions we're already building. 
It might be good to hear from someone who develops with node/npm, though 
... I just use it to install software that I can't find packaged elsewhere.

> 
>> On 8/8/21 6:29 PM, Pierre Langlois wrote:
>>
>>>     ... `node-gyp' needs
>>
>>>     node headers to compile against, packaged as a tarball, which it tries
>>
>>>     to download.  Instead, we can run a `node-gyp --tarball <> configure'
>>
>>>     step to manually provide the tarball, which we can package separately
>>
>>>     for any given node version.
>>
>> There is also a --nodedir option, which I found could work with something like:
>>
>>      (string-append "--nodedir=" (assoc-ref inputs "node"))
>>
>> That seems like it might be better, though I don't know all the considerations
>> for cross-compilation and such.
> 
> Oh that's a good idea, I didn't really like having to download the
> headers separately from the main package, especially given we run
> snippet on the source to remove bundled dependencies.
> 
> Trying this out this approach does work, but I needed to:
> 
>    - Create a union directory with both node and libuv. The node package
>      only has headers for V8/node, but we also need libuv, so doing
>      something like this works:
> 
>      (union-build node-sources
>                  (list (assoc-ref inputs "node")
>                        (assoc-ref inputs "libuv"))
>                  #:create-all-directories? #t
>                  #:log-port (%make-void-port "w"))

I found it worked to just add libuv as an input of packages built with 
node-gyp. I hadn't tried to change `node-build-system`, but I think that 
would be the place to do it.

> 
>    - For some reason, --nodedir didn't really "configure" gyp to use that
>      node directory, I think it's meant to be passed everytime you run
>      any gyp command. Instead I found that you can use and environment
>      variable:
> 
>      (setenv "npm_config_nodedir" node-sources)

That seems right. I believe there's a similar "npm_config_python" for 
the Python executable to use.

Alternatively, I think it's possible to configure these in 
$PREFIX/etc/npmrc: <https://docs.npmjs.com/cli/v7/configuring-npm/npmrc>

> 
> And that works for the packages in this series!  That'll be much better
> than before, I'll do it this way.
> 
> Thanks again for taking a look, I'll see if I can send updated patches
> sometimes this weekend.

Glad it was useful!

For patching the shebangs, here's a variant of node-lts that worked for 
me, though I think it would be even better to combine it with the 
existing phases:

```
(define-public patched-node
   (let ((node node-lts))
     (package
       (inherit node)
       (arguments
        (substitute-keyword-arguments (package-arguments node)
          ((#:phases standard-phases)
           `(modify-phases ,standard-phases
              (add-after 'patch-npm-shebang 'patch-more-shebangs
                (lambda* (#:key inputs outputs #:allow-other-keys)
                  (define (append-map f lst)
                    (apply append (map f lst)))
                  ;; from patch-shebangs
                  (define bin-directories
                    ;;(match-lambda
                    ;;  ((_ . dir)
                    (lambda (pr)
                      (let ((dir (cdr pr)))
                        (list (string-append dir "/bin")
                              (string-append dir "/sbin")))))
                  (define output-bindirs
                    (append-map bin-directories outputs))
                  (define input-bindirs
                    ;; Shebangs should refer to binaries of the target 
system---i.e., from
                    ;; "inputs", not from "native-inputs".
                    (append-map bin-directories inputs))
                  (define path
                    (append output-bindirs input-bindirs))
                  (with-directory-excursion
                      (string-append (assoc-ref outputs "out")
                                     "/lib/node_modules/npm/node_modules")
                    (for-each
                     ;;(cut patch-shebang <> path)
                     (lambda (file)
                       (patch-shebang file path))
                     ;; from patch-generated-file-shebangs
                     (find-files "."
                                 (lambda (file stat)
                                   (and (eq? 'regular (stat:type stat))
                                        (not (zero? (logand (stat:mode 
stat) #o100)))))
                                 #:stat lstat))))))))))))
```

-Philip

[bug#49946] [PATCH 08/31] gnu: node: Patch /usr/bin/env in node-gyp.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

Philip McGrath schreef op zo 26-09-2021 om 18:02 [-0400]:
> Looking at `gnu-build-system`, it seems that the `'patch-shebangs` phase 
> only operates on files installed in the "/bin" and "/sbin" 
> subdirectories of the package's outputs. That restriction doesn't make 
> sense to me in general: for instance, what about "/libexec"?

'libexec' is included on core-updates{,-frozen}.  I believe the idea of the
restriction is to avoid patching too much.  E.g., "autoconf" has a file
share/autoconf/build-aux/config.guess with a #!/bin/sh shebang.  It should
not be patched, otherwise tarballs created with "make dist" would include
a store path and hence be Guix-specific and architecture-specific.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [PATCH 08/31] gnu: node: Patch /usr/bin/env in node-gyp.

[Philip McGrath]

On 9/27/21 6:11 AM, Maxime Devos wrote:
> Philip McGrath schreef op zo 26-09-2021 om 18:02 [-0400]:
>> Looking at `gnu-build-system`, it seems that the `'patch-shebangs` phase
>> only operates on files installed in the "/bin" and "/sbin"
>> subdirectories of the package's outputs. That restriction doesn't make
>> sense to me in general: for instance, what about "/libexec"?
> 
> 'libexec' is included on core-updates{,-frozen}.  I believe the idea of the
> restriction is to avoid patching too much.  E.g., "autoconf" has a file
> share/autoconf/build-aux/config.guess with a #!/bin/sh shebang.  It should
> not be patched, otherwise tarballs created with "make dist" would include
> a store path and hence be Guix-specific and architecture-specific.

That makes some sense. I would have thought checking that the file is 
executable would catch most such cases, but, if this works for 
`gnu-build-system`, great.

As I look at potentially making a patch, another thing that seems odd is 
that `(gnu packages node)` exports node@10.24.0 as `node` (via 
`define-public`), but node@14.16.0 as `node-lts`. Normally, if I saw 
that there were packages `node` and `node-lts`, I'd assume that 
`node-lts` was *older*. It's especially confusing because, at the 
command line, `guix install node` refers to what in Scheme you have to 
write as `node-lts`.

I wonder if it was a mistake, and should have used `define` rather than 
`define-public`, since this code:

```
;; This should be the latest version of node that still builds without
;; depending on llhttp.
(define-public node-bootstrap
   (hidden-package node))
```

seems to be trying to hide the older node.

It looks like `node` has only a few dependents, and it seems like at 
least several of them only used it because it had the more obvious name. 
The `node-build-system` uses `node-lts` as the `(default-node)`.

Would it make sense to change the names? Or just to remove the 
`define-public` of `node`?

-Philip

[bug#49946] [PATCH 08/31] gnu: node: Patch /usr/bin/env in node-gyp.

[Philip McGrath]

On 9/25/21 6:24 AM, Pierre Langlois wrote:
> For instance, while working on a newer version of one of the packages in
> this series, I saw we may need to patch GYP's python reference as well,
> like so:
> 
> (substitute* "deps/npm/node_modules/node-gyp/gyp/gyp_main.py"
>    (("#!/usr/bin/env python")
>     (string-append "#!" (assoc-ref inputs "python") "/bin/python3")))
> 
> Only for node 14+. The reason seems to be that gyp still refers to
> "python", but python2 is no longer a dependency for newer nodes. And it
> seems GYP is perfectly happy with python3, and the shebang is fixed
> upstream so a never node will be fine:
> https://github.com/nodejs/node-gyp/pull/2355/files

I think this needs to be a `python` from `inputs` rather than 
`native-inputs`, for cross-compilation, IIUC.

I tried building node 14.18.0, and it ran into other issues, but there 
were still a number shebangs with `python` rather than `python3` in 
various places, though I think they'd be fine with `python-wrapper`.

-Philip

[bug#49946] [PATCH 0/3] guix: node-build-system: Support compiling addons with node-gyp.

[Philip McGrath]

Hi,

I've reworked the part of the patch series dealing with node-gyp.

I'd like to find an NPM addon package to submit as part of this series, too,
basically as a test case. If I can find one that doesn't raise too many other
complications, I may send this in as a separate patch, but feel free to try it
with tree-sitter, too.

There are a few things I'm still not sure about. I haven't made
node-build-system add libuv as an implicit input, because I think some
node-gyp addons don't actually need libuv, but maybe it's common enough that
it should be done automatically.

Likewise, I haven't tried to change the issue of `node` referring to
`node-bootstrap`, but I still think it should be changed.

These patches are also on GitLab at
<https://gitlab.com/philip1/guix-patches/-/tree/wip-node-npm-gyp>.

Improvements welcome!

-Philip


Philip McGrath (3):
  gnu: node: Avoid duplicating build phases.
  gnu: node: Update to 10.24.1 for bootstrapping.
  guix: node-build-system: Support compiling addons with node-gyp.

 gnu/packages/node.scm            | 187 ++++++++++---------------------
 guix/build-system/node.scm       |   7 +-
 guix/build/node-build-system.scm |   9 ++
 3 files changed, 74 insertions(+), 129 deletions(-)

-- 
2.30.2

[bug#49946] [PATCH 1/3] gnu: node: Avoid duplicating build phases.

[Philip McGrath]

* gnu/packages/node.scm (node)[arguments]: Split 'patch-files phase
into 'patch-hardcoded-program-references and
'delete-problematic-tests. Adapt those phases and 'configure to work
unmodified on node-lts.
(node, node-lts)[inputs]: Use bash-minimal rather than bash.
(node-lts)[arguments]: Inherit 'patch-hardcoded-program-references,
'delete-problemating-tests, and 'configure phases from the bootstrap
node. Remove the 'patch-files phase, keeping its remaining
non-inherited work in a new 'replace-llhttp-sources phase.
---
 gnu/packages/node.scm | 150 +++++++++++-------------------------------
 1 file changed, 39 insertions(+), 111 deletions(-)

diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
index f8ac95884c..34c2bfa9d4 100644
--- a/gnu/packages/node.scm
+++ b/gnu/packages/node.scm
@@ -9,6 +9,7 @@
 ;;; Copyright © 2020, 2021 Pierre Langlois <pierre.langlois@gmx.com>
 ;;; Copyright © 2020 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
+;;; Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -105,14 +106,22 @@
        #:test-target "test-ci-js"
        #:phases
        (modify-phases %standard-phases
-         (add-before 'configure 'patch-files
+         (add-before 'configure 'patch-hardcoded-program-references
            (lambda* (#:key inputs #:allow-other-keys)
+
              ;; Fix hardcoded /bin/sh references.
-             (substitute* '("lib/child_process.js"
-                            "lib/internal/v8_prof_polyfill.js"
-                            "test/parallel/test-child-process-spawnsync-shell.js"
-                            "test/parallel/test-stdio-closed.js"
-                            "test/sequential/test-child-process-emfile.js")
+             (substitute*
+                 (let ((common
+                        '("lib/child_process.js"
+                          "lib/internal/v8_prof_polyfill.js"
+                          "test/parallel/test-child-process-spawnsync-shell.js"
+                          "test/parallel/test-stdio-closed.js"
+                          "test/sequential/test-child-process-emfile.js"))
+                       ;; not in bootstap node:
+                       (sigxfsz "test/parallel/test-fs-write-sigxfsz.js"))
+                   (if (file-exists? sigxfsz)
+                       (cons sigxfsz common)
+                       common))
                (("'/bin/sh'")
                 (string-append "'" (assoc-ref inputs "bash") "/bin/sh'")))
 
@@ -122,18 +131,28 @@
                             "test/parallel/test-child-process-exec-env.js")
                (("'/usr/bin/env'")
                 (string-append "'" (assoc-ref inputs "coreutils")
-                               "/bin/env'")))
+                               "/bin/env'")))))
+         (add-after 'patch-hardcoded-program-references 'delete-problematic-tests
+           (lambda* (#:key inputs #:allow-other-keys)
+             (define (delete-file-if-exists pth)
+               (when (file-exists? pth)
+                 (delete-file pth)))
 
              ;; FIXME: These tests fail in the build container, but they don't
              ;; seem to be indicative of real problems in practice.
-             (for-each delete-file
+             (for-each delete-file-if-exists
                        '("test/parallel/test-cluster-master-error.js"
                          "test/parallel/test-cluster-master-kill.js"
                          ;; See also <https://github.com/nodejs/node/issues/25903>.
+                         ;; (no longer exists in node-lts)
                          "test/sequential/test-performance.js"))
 
-             ;; This requires a DNS resolver.
-             (delete-file "test/parallel/test-dns.js")
+             ;; These require a DNS resolver.
+             (for-each delete-file-if-exists
+                       '("test/parallel/test-dns.js"
+                         ;; not in the bootstrap node:
+                         "test/parallel/test-dns-lookupService-promises.js"))
+
 
              ;; FIXME: This test fails randomly:
              ;; https://github.com/nodejs/node/issues/31213
@@ -217,9 +236,13 @@
                (setenv "CXX" ,(cxx-for-target))
                (setenv "PKG_CONFIG" ,(pkg-config-for-target))
                (apply invoke
-                      (string-append (assoc-ref (or native-inputs inputs)
-                                                "python")
-                                     "/bin/python")
+                      (let ((python
+                             (string-append (assoc-ref (or native-inputs inputs)
+                                                       "python")
+                                            "/bin/python")))
+                        (if (file-exists? python)
+                            python
+                            (string-append python "3")))
                       "configure" flags))))
          (add-after 'patch-shebangs 'patch-npm-shebang
            (lambda* (#:key outputs #:allow-other-keys)
@@ -256,7 +279,7 @@
             (variable "NODE_PATH")
             (files '("lib/node_modules")))))
     (inputs
-     `(("bash" ,bash)
+     `(("bash" ,bash-minimal)
        ("coreutils" ,coreutils)
        ("c-ares" ,c-ares)
        ("http-parser" ,http-parser)
@@ -711,103 +734,8 @@ source files.")
                                    libuv "/lib:"
                                    zlib "/lib"
                                    "'],"))))))
-           (replace 'configure
-             ;; Node's configure script is actually a python script, so we can't
-             ;; run it with bash.
-             (lambda* (#:key outputs (configure-flags '()) native-inputs inputs
-                       #:allow-other-keys)
-               (let* ((prefix (assoc-ref outputs "out"))
-                      (xflags ,(if (%current-target-system)
-                                   `'("--cross-compiling"
-                                     ,(string-append
-                                       "--dest-cpu="
-                                       (match (%current-target-system)
-                                         ((? (cut string-prefix? "arm" <>))
-                                          "arm")
-                                         ((? (cut string-prefix? "aarch64" <>))
-                                          "arm64")
-                                         ((? (cut string-prefix? "i686" <>))
-                                          "ia32")
-                                         ((? (cut string-prefix? "x86_64" <>))
-                                          "x64")
-                                         ((? (cut string-prefix? "powerpc64" <>))
-                                          "ppc64")
-                                         (_ "unsupported"))))
-                                   ''()))
-                      (flags (cons
-                               (string-append "--prefix=" prefix)
-                               (append xflags configure-flags))))
-                 (format #t "build directory: ~s~%" (getcwd))
-                 (format #t "configure flags: ~s~%" flags)
-                 ;; Node's configure script expects the CC environment variable to
-                 ;; be set.
-                 (setenv "CC_host" "gcc")
-                 (setenv "CXX_host" "g++")
-                 (setenv "CC" ,(cc-for-target))
-                 (setenv "CXX" ,(cxx-for-target))
-                 (setenv "PKG_CONFIG" ,(pkg-config-for-target))
-                 (apply invoke
-                        (string-append (assoc-ref (or native-inputs inputs)
-                                                  "python")
-                                       "/bin/python3")
-                        "configure" flags))))
-           (replace 'patch-files
+           (add-after 'delete-problematic-tests 'replace-llhttp-sources
              (lambda* (#:key inputs #:allow-other-keys)
-               ;; Fix hardcoded /bin/sh references.
-               (substitute* '("lib/child_process.js"
-                              "lib/internal/v8_prof_polyfill.js"
-                              "test/parallel/test-child-process-spawnsync-shell.js"
-                              "test/parallel/test-fs-write-sigxfsz.js"
-                              "test/parallel/test-stdio-closed.js"
-                              "test/sequential/test-child-process-emfile.js")
-                 (("'/bin/sh'")
-                  (string-append "'" (assoc-ref inputs "bash") "/bin/sh'")))
-
-               ;; Fix hardcoded /usr/bin/env references.
-               (substitute* '("test/parallel/test-child-process-default-options.js"
-                              "test/parallel/test-child-process-env.js"
-                              "test/parallel/test-child-process-exec-env.js")
-                 (("'/usr/bin/env'")
-                  (string-append "'" (assoc-ref inputs "coreutils")
-                                 "/bin/env'")))
-
-               ;; FIXME: These tests fail in the build container, but they don't
-               ;; seem to be indicative of real problems in practice.
-               (for-each delete-file
-                         '("test/parallel/test-cluster-master-error.js"
-                           "test/parallel/test-cluster-master-kill.js"))
-
-               ;; These require a DNS resolver.
-               (for-each delete-file
-                         '("test/parallel/test-dns.js"
-                           "test/parallel/test-dns-lookupService-promises.js"))
-
-               ;; FIXME: This test fails randomly:
-               ;; https://github.com/nodejs/node/issues/31213
-               (delete-file "test/parallel/test-net-listen-after-destroying-stdin.js")
-
-               ;; FIXME: These tests fail on armhf-linux:
-               ;; https://github.com/nodejs/node/issues/31970
-               ,@(if (target-arm32?)
-                     '((for-each delete-file
-                                 '("test/parallel/test-zlib.js"
-                                   "test/parallel/test-zlib-brotli.js"
-                                   "test/parallel/test-zlib-brotli-flush.js"
-                                   "test/parallel/test-zlib-brotli-from-brotli.js"
-                                   "test/parallel/test-zlib-brotli-from-string.js"
-                                   "test/parallel/test-zlib-convenience-methods.js"
-                                   "test/parallel/test-zlib-random-byte-pipes.js"
-                                   "test/parallel/test-zlib-write-after-flush.js")))
-                     '())
-
-               ;; These tests have an expiry date: they depend on the validity of
-               ;; TLS certificates that are bundled with the source.  We want this
-               ;; package to be reproducible forever, so remove those.
-               ;; TODO: Regenerate certs instead.
-               (for-each delete-file
-                         '("test/parallel/test-tls-passphrase.js"
-                           "test/parallel/test-tls-server-verify.js"))
-
                ;; Replace pre-generated llhttp sources
                (let ((llhttp (assoc-ref inputs "llhttp")))
                  (copy-file (string-append llhttp "/src/llhttp.c")
@@ -834,7 +762,7 @@ source files.")
        ("python" ,python)
        ("util-linux" ,util-linux)))
     (inputs
-     `(("bash" ,bash)
+     `(("bash" ,bash-minimal)
        ("coreutils" ,coreutils)
        ("c-ares" ,c-ares)
        ("icu4c" ,icu4c-67)
-- 
2.30.2

[bug#49946] [PATCH 2/3] gnu: node: Update to 10.24.1 for bootstrapping.

[Philip McGrath]

* gnu/packages/node.scm (node): Update to 10.24.1.
---
 gnu/packages/node.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
index 34c2bfa9d4..6d9c3ccc71 100644
--- a/gnu/packages/node.scm
+++ b/gnu/packages/node.scm
@@ -57,14 +57,14 @@
 (define-public node
   (package
     (name "node")
-    (version "10.24.0")
+    (version "10.24.1")
     (source (origin
               (method url-fetch)
               (uri (string-append "https://nodejs.org/dist/v" version
                                   "/node-v" version ".tar.xz"))
               (sha256
                (base32
-                "1k1srdis23782hnd1ymgczs78x9gqhv77v0am7yb54gqcspp70hm"))
+                "032801kg24j04xmf09m0vxzlcz86sv21s24lv9l4cfv08k1c4byp"))
               (modules '((guix build utils)))
               (snippet
                `(begin
-- 
2.30.2

[bug#49946] [PATCH 3/3] guix: node-build-system: Support compiling addons with node-gyp.

[Philip McGrath]

* gnu/packages/node.scm (node)[arguments]: Replace 'patch-npm-shebang
and 'patch-node-shebang with a new 'patch-nested-shebangs that also
handles node-gyp and other shebangs under "/lib/node_modules".
[inputs]: Add Python for node-gyp as "python-for-target".
(node-lts)[inputs]: Likewise.
* guix/build-system/node.scm (lower): Add optional #:python argument.
* guix/build/node-build-system.scm (set-node-gyp-paths): New
function. Sets the "npm_config_nodedir" and "npm_config_python"
environment variables.
(%standard-phases): Add 'set-node-gyp-paths after 'set-paths.

Co-authored-by: Pierre Langlois <pierre.langlois@gmx.com>
---
 gnu/packages/node.scm            | 33 +++++++++++++++++---------------
 guix/build-system/node.scm       |  7 ++++++-
 guix/build/node-build-system.scm |  9 +++++++++
 3 files changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
index 6d9c3ccc71..805a4f18fc 100644
--- a/gnu/packages/node.scm
+++ b/gnu/packages/node.scm
@@ -244,21 +244,22 @@
                             python
                             (string-append python "3")))
                       "configure" flags))))
-         (add-after 'patch-shebangs 'patch-npm-shebang
-           (lambda* (#:key outputs #:allow-other-keys)
-             (let* ((bindir (string-append (assoc-ref outputs "out")
-                                           "/bin"))
-                    (npm    (string-append bindir "/npm"))
-                    (target (readlink npm)))
-               (with-directory-excursion bindir
-                 (patch-shebang target (list bindir))))))
-         (add-after 'install 'patch-node-shebang
-           (lambda* (#:key outputs #:allow-other-keys)
-             (let* ((bindir (string-append (assoc-ref outputs "out")
-                                           "/bin"))
-                    (npx    (readlink (string-append bindir "/npx"))))
-               (with-directory-excursion bindir
-                 (patch-shebang npx (list bindir)))))))))
+         (add-after 'patch-shebangs 'patch-nested-shebangs
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let* ((prefix (assoc-ref outputs "out"))
+                    (path (map (lambda (dir)
+                                 (string-append dir "/bin"))
+                               (list prefix
+                                     (assoc-ref inputs "python-for-target")))))
+               (with-directory-excursion (string-append prefix "/lib/node_modules")
+                 (for-each
+                  (lambda (file)
+                    (patch-shebang file path))
+                  (find-files "."
+                              (lambda (file stat)
+                                (and (eq? 'regular (stat:type stat))
+                                     (not (zero? (logand (stat:mode stat) #o100)))))
+                              #:stat lstat)))))))))
     (native-inputs
      `(;; Runtime dependencies for binaries used as a bootstrap.
        ("c-ares" ,c-ares)
@@ -281,6 +282,7 @@
     (inputs
      `(("bash" ,bash-minimal)
        ("coreutils" ,coreutils)
+       ("python-for-target" ,python-wrapper) ;; for node-gyp (supports python3)
        ("c-ares" ,c-ares)
        ("http-parser" ,http-parser)
        ("icu4c" ,icu4c)
@@ -764,6 +766,7 @@ source files.")
     (inputs
      `(("bash" ,bash-minimal)
        ("coreutils" ,coreutils)
+       ("python-for-target" ,python-wrapper) ;; for node-gyp (supports python3)
        ("c-ares" ,c-ares)
        ("icu4c" ,icu4c-67)
        ("libuv" ,libuv-for-node)
diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm
index 98f63f87ef..3e49e67ff6 100644
--- a/guix/build-system/node.scm
+++ b/guix/build-system/node.scm
@@ -1,6 +1,8 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2019 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
+;;; Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -24,6 +26,7 @@
   #:use-module (guix search-paths)
   #:use-module (guix build-system)
   #:use-module (guix build-system gnu)
+  #:use-module (guix build-system python)
   #:use-module (ice-9 match)
   #:export (%node-build-system-modules
             node-build
@@ -44,11 +47,12 @@
 (define* (lower name
                 #:key source inputs native-inputs outputs system target
                 (node (default-node))
+                (python (default-python)) ;; for node-gyp
                 #:allow-other-keys
                 #:rest arguments)
   "Return a bag for NAME."
   (define private-keywords
-    '(#:source #:target #:node #:inputs #:native-inputs))
+    '(#:source #:target #:node #:python #:inputs #:native-inputs))
 
   (and (not target)                    ;XXX: no cross-compilation
        (bag
@@ -62,6 +66,7 @@
                         ;; Keep the standard inputs of 'gnu-build-system'.
                         ,@(standard-packages)))
          (build-inputs `(("node" ,node)
+                         ("python" ,python)
                          ,@native-inputs))
          (outputs outputs)
          (build node-build)
diff --git a/guix/build/node-build-system.scm b/guix/build/node-build-system.scm
index 70a367618e..5e62eb4784 100644
--- a/guix/build/node-build-system.scm
+++ b/guix/build/node-build-system.scm
@@ -2,6 +2,8 @@
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
 ;;; Copyright © 2016, 2020 Jelle Licht <jlicht@fsfe.org>
 ;;; Copyright © 2019, 2021 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
+;;; Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -46,6 +48,12 @@
               (format #t "set HOME to ~s~%" (getenv "HOME")))))))
   #t)
 
+(define* (set-node-gyp-paths #:key inputs #:allow-other-keys)
+  "Initialize environment variables needed for building native addons."
+  (setenv "npm_config_nodedir" (assoc-ref inputs "node"))
+  (setenv "npm_config_python" (assoc-ref inputs "python"))
+  #t)
+
 (define (module-name module)
   (let* ((package.json (string-append module "/package.json"))
          (package-meta (call-with-input-file package.json read-json)))
@@ -144,6 +152,7 @@
 
 (define %standard-phases
   (modify-phases gnu:%standard-phases
+    (add-after 'set-paths 'set-node-gyp-paths set-node-gyp-paths)
     (add-after 'unpack 'set-home set-home)
     (add-before 'configure 'patch-dependencies patch-dependencies)
     (replace 'configure configure)
-- 
2.30.2

[bug#49946] [PATCH 1/3] gnu: node: Avoid duplicating build phases.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 1980 bytes --]

Hi Philip,

Philip McGrath <philip@philipmcgrath.com> writes:

> * gnu/packages/node.scm (node)[arguments]: Split 'patch-files phase
> into 'patch-hardcoded-program-references and
> 'delete-problematic-tests. Adapt those phases and 'configure to work
> unmodified on node-lts.
> (node, node-lts)[inputs]: Use bash-minimal rather than bash.
> (node-lts)[arguments]: Inherit 'patch-hardcoded-program-references,
> 'delete-problemating-tests, and 'configure phases from the bootstrap
> node. Remove the 'patch-files phase, keeping its remaining
> non-inherited work in a new 'replace-llhttp-sources phase.

While I agree that most of the time, factoring out common code is a good
thing, I'm not sure it applies in the case of patching tests. The list
of tests is specific to a version and it's likely for each version to
need fixes. Having a common phase that describes the tests to patch for
2 versions (3 if we add node 16) is harder to maintain than three phases
IMO, even though they'll look similar indeed. Having to change commmon
code can also cause unnessecary rebuilds.

For example, I started working updating node last weekend and saw these
test changes:

 - 14.16 -> 14.17: Delete test/parallel/test-https-agent-unref-socket.js,
                   requires networking
 - 16: Extra test needs /bin/sh patched test/parallel/test-stdin-from-file-spawn.js"
       A couple tests were renamed:
       test/parallel/test-cluster-master-error.js -> test/parallel/test-cluster-primary-error.js
       test/parallel/test-cluster-master-kill.js -> test/parallel/test-cluster-primary-kill.js

That being said, I definetely agree we should have a separate phase for
the replacement of the llhttp source, that's logically different from
patching tests, and is unlikely to change version to version.

Keeping the list of tests local to each packages allows to add node 16
while avoiding rebuilding the others, does this make sense? I could be
wrong here of course :-).

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH 0/3] guix: node-build-system: Support compiling addons with node-gyp.

[Pierre Langlois]

[-- Attachment #1.1: Type: text/plain, Size: 1992 bytes --]

Hi Philip,

Philip McGrath <philip@philipmcgrath.com> writes:

> Hi,
>
> I've reworked the part of the patch series dealing with node-gyp.

Nice, thanks for working on this! I'll take a look at the patches, sorry
I've not had too much time this week yet :-).

> I'd like to find an NPM addon package to submit as part of this series, too,
> basically as a test case. If I can find one that doesn't raise too many other
> complications, I may send this in as a separate patch, but feel free to try it
> with tree-sitter, too.

Cool, I'll give it a test with the tree-sitter series.

>
> There are a few things I'm still not sure about. I haven't made
> node-build-system add libuv as an implicit input, because I think some
> node-gyp addons don't actually need libuv, but maybe it's common enough that
> it should be done automatically.

I see, I assumed libuv was needed for all addons. I'm thinking given
node itself heavily depends on libuv anymays, it might be nice to add it
by default in the build system, but I don't really mind either way. I'll
check if I can just add libuv in the tree-sitter packages.

>
> Likewise, I haven't tried to change the issue of `node` referring to
> `node-bootstrap`, but I still think it should be changed.

I agree it would be good to change this. I started working on updating
node last weekend but haven't had time to work on it after that. I think
we should probably name node always as "node-<version>":

  node -> node-10
  node-lts -> node-14

Then keep the node and node-lts names as aliases:

  (define-public node node-14)     ;; The latest, but move it to node-16 later
  (define-public node-lts node-14)

However, moving the node variable to node-14 is likely to cause
rebuilds, so we'll have to check that.

Anyways, here's my WIP patch in case there's anything of interest
there. Note that it still has the bug you pointed out where I'm refering
to the host python instead of the target one (thanks for pointing it
out!).

Thanks,
Pierre


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-update-node.patch --]
[-- Type: text/x-patch, Size: 15097 bytes --]

From b43db52b9a1b55b5756268b710907813f07a42ba Mon Sep 17 00:00:00 2001
From: Pierre Langlois <pierre.langlois@gmx.com>
Date: Sun, 26 Sep 2021 18:58:53 +0100
Subject: [PATCH] update node

---
 gnu/packages/adns.scm |  46 ++++++++++-
 gnu/packages/node.scm | 175 +++++++++++++++++++++++++++++++++++++-----
 2 files changed, 198 insertions(+), 23 deletions(-)

diff --git a/gnu/packages/adns.scm b/gnu/packages/adns.scm
index b36ec18462..6fba9783e0 100644
--- a/gnu/packages/adns.scm
+++ b/gnu/packages/adns.scm
@@ -23,8 +23,10 @@
   #:use-module (guix licenses)
   #:use-module (guix packages)
   #:use-module (guix download)
+  #:use-module (guix git-download)
   #:use-module (guix build-system cmake)
   #:use-module (guix build-system gnu)
+  #:use-module (gnu packages autotools)
   #:use-module (gnu packages m4)
   #:use-module (gnu packages pkg-config))

@@ -92,7 +94,7 @@ multiple clients and programs with graphical user interfaces.")
   (package
     (inherit c-ares)
     (name "c-ares")
-    (version "1.17.1")
+    (version "1.17.2")
     (source (origin
               (method url-fetch)
               (uri (string-append
@@ -100,10 +102,46 @@ multiple clients and programs with graphical user interfaces.")
                     ".tar.gz"))
               (sha256
                (base32
-                "0h7wjfnk2092glqcp9mqaax7xx0s13m501z1gi0gsjl2vvvd0gfp"))))
+                "0gcincjvpll2qmlc906jx6mfq97s87mgi0zby0753ki0rr2ch0s8"))))
     (arguments
-     `(;; FIXME: Some tests require network access
-       #:tests? #f))))
+     `(#:phases
+       (modify-phases %standard-phases
+         (replace 'check
+           (lambda* (#:key tests? #:allow-other-keys)
+             (when tests?
+               (invoke "./test/arestest"
+                       (string-append
+                         ;; "Live" tests require network access.
+                         "--gtest_filter=-*.Live*:"
+                         ;; FIXME: This test fails in the build sandbox, but
+                         ;; otherwise passes.
+                         "AddressFamiliesAI/"
+                         "MockChannelTestAI.FamilyV4ServiceName/0"))))))))))
+
+(define-public c-ares-for-node
+  (let ((commit "6299d7be383de62da831a3c48f4017b70c664de8")
+        (revision "1"))
+    (package
+      (inherit c-ares/fixed)
+      (name "c-ares")
+      (version (git-version "1.17.2" revision commit))
+      (source
+       (origin
+         (method git-fetch)
+         (uri (git-reference
+               (url "https://github.com/c-ares/c-ares.git")
+               (commit commit)))
+         (file-name (git-file-name name version))
+         (sha256
+          (base32 "1rmk7m7lv5kmqxhb2dmq5fxk73iicg5rgsz2y855hk0a92xlrcsd"))))
+      (native-inputs
+       `(("pkg-config" ,pkg-config)
+         ("autoconf" ,autoconf)
+         ("automake" ,automake)
+         ("libtool" ,libtool))))))
+      ;; (arguments
+      ;;  `(;; FIXME: Some tests require network access
+      ;;    #:tests? #f)))))

 ;; gRPC requires a c-ares built with CMake in order to get the .cmake modules.
 ;; We can not build c-ares itself with CMake because that would introduce a
diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
index 71da2aa18f..79ffa8fca9 100644
--- a/gnu/packages/node.scm
+++ b/gnu/packages/node.scm
@@ -53,7 +53,7 @@
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-26))

-(define-public node
+(define-public node-10
   (package
     (name "node")
     (version "10.24.0")
@@ -284,7 +284,7 @@ devices.")
 ;; This should be the latest version of node that still builds without
 ;; depending on llhttp.
 (define-public node-bootstrap
-  (hidden-package node))
+  (hidden-package node-10))

 ;; Duplicate of node-semver
 (define-public node-semver-bootstrap
@@ -524,17 +524,17 @@ Node.js and web browsers.")
 (define-public node-llparse-bootstrap
   (package
     (name "node-llparse")
-    (version "7.1.0")
+    (version "7.1.1")
     (source
      (origin
        (method git-fetch)
        (uri (git-reference
-             (url "https://github.com/indutny/llparse.git")
+             (url "https://github.com/nodejs/llparse.git")
              (commit (string-append "v" version))))
        (file-name (git-file-name name version))
        (sha256
         (base32
-         "10da273iy2if88hp79cwms6c8qpsl1fkgzll6gmqyx5yxv5mkyp6"))
+         "0gzsa4nwrhvm7gz817l5r6v7i8lmqpnrg25smqiq6x8xgs8dlmgl"))
        (modules '((guix build utils)))
        (snippet
         '(begin
@@ -577,7 +577,7 @@ Node.js and web browsers.")
 parser definition into a C output.")
     (license license:expat)))

-(define-public llhttp-bootstrap
+(define-public llhttp-bootstrap-2
   (package
     (name "llhttp")
     (version "2.1.3")
@@ -648,17 +648,42 @@ parser definition into a C output.")
 source files.")
     (license license:expat)))

-(define-public node-lts
+(define-public llhttp-bootstrap
+  (package (inherit llhttp-bootstrap-2)
+    (name "llhttp")
+    (version "6.0.5")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/nodejs/llhttp.git")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1wc5xsa76my32k86cax540q0g4y85w7cikqvdjy3rkz2r7fjlyyk"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Fix imports for esbuild.
+                  ;; https://github.com/evanw/esbuild/issues/477
+                  (substitute* "src/llhttp/http.ts"
+                    (("\\* as assert") "assert"))
+                  (substitute* "Makefile"
+                    (("npx ts-node bin/generate.ts")
+                     "node bin/generate.js"))
+                  #t))))))
+
+(define-public node-14
   (package
-    (inherit node)
-    (version "14.16.0")
+    (inherit node-10)
+    (version "14.17.6")
     (source (origin
               (method url-fetch)
               (uri (string-append "https://nodejs.org/dist/v" version
                                   "/node-v" version ".tar.xz"))
               (sha256
                (base32
-                "19nz2mhmn6ikahxqyna1dn25pb5v3z9vsz9zb2flb6zp2yk4hxjf"))
+                "0pmd0haav2ychhcsw44klx6wfn8c7j1rsw08rc8hcm5i3h5wsn7l"))
               (modules '((guix build utils)))
               (snippet
                `(begin
@@ -675,7 +700,7 @@ source files.")
                     (("deps/zlib/zlib.gyp") ""))
                   #t))))
     (arguments
-     (substitute-keyword-arguments (package-arguments node)
+     (substitute-keyword-arguments (package-arguments node-10)
        ((#:configure-flags configure-flags)
         ''("--shared-cares"
            "--shared-libuv"
@@ -798,6 +823,8 @@ source files.")
                          '("test/parallel/test-dns.js"
                            "test/parallel/test-dns-lookupService-promises.js"))

+               (delete-file "test/parallel/test-https-agent-unref-socket.js")
+
                ;; FIXME: This test fails randomly:
                ;; https://github.com/nodejs/node/issues/31213
                (delete-file "test/parallel/test-net-listen-after-destroying-stdin.js")
@@ -822,9 +849,10 @@ source files.")
                ;; TODO: Regenerate certs instead.
                (for-each delete-file
                          '("test/parallel/test-tls-passphrase.js"
-                           "test/parallel/test-tls-server-verify.js"))
-
-               ;; Replace pre-generated llhttp sources
+                           "test/parallel/test-tls-server-verify.js"))))
+           ;; Replace pre-generated llhttp sources
+           (add-after 'patch-files 'replace-llhttp
+             (lambda* (#:key inputs #:allow-other-keys)
                (let ((llhttp (assoc-ref inputs "llhttp")))
                  (copy-file (string-append llhttp "/src/llhttp.c")
                             "deps/llhttp/src/llhttp.c")
@@ -836,7 +864,7 @@ source files.")
                             "deps/llhttp/include/llhttp.h"))))))))
     (native-inputs
      `(;; Runtime dependencies for binaries used as a bootstrap.
-       ("c-ares" ,c-ares)
+       ("c-ares" ,c-ares-for-node)
        ("google-brotli" ,google-brotli)
        ("icu4c" ,icu4c-67)
        ("libuv" ,libuv-for-node)
@@ -852,20 +880,129 @@ source files.")
     (inputs
      `(("bash" ,bash)
        ("coreutils" ,coreutils)
-       ("c-ares" ,c-ares)
+       ("c-ares" ,c-ares-for-node)
        ("icu4c" ,icu4c-67)
        ("libuv" ,libuv-for-node)
-       ("llhttp" ,llhttp-bootstrap)
+       ("llhttp" ,llhttp-bootstrap-2)
        ("google-brotli" ,google-brotli)
        ("nghttp2" ,nghttp2 "lib")
        ("openssl" ,openssl)
        ("zlib" ,zlib)))))

+(define-public node
+  (package
+    (inherit node-14)
+    (version "16.10.0")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "https://nodejs.org/dist/v" version
+                                  "/node-v" version ".tar.xz"))
+              (sha256
+               (base32
+                "04krpy0r8msv64rcf0vy2l2yzf0a401km8p5p7h12j9b4g51mp4p"))
+              (modules '((guix build utils)))
+              (snippet
+               `(begin
+                  ;; Remove bundled software, where possible
+                  (for-each delete-file-recursively
+                            '("deps/cares"
+                              "deps/icu-small"
+                              "deps/nghttp2"
+                              "deps/openssl"
+                              "deps/zlib"))
+                  (substitute* "Makefile"
+                    ;; Remove references to bundled software.
+                    (("deps/uv/uv.gyp") "")
+                    (("deps/zlib/zlib.gyp") ""))
+                  #t))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments node-14)
+       ((#:phases phases)
+        `(modify-phases ,phases
+           (replace 'patch-files
+             (lambda* (#:key native-inputs inputs #:allow-other-keys)
+               ;; Fix hardcoded /bin/sh references.
+               (substitute* '("lib/child_process.js"
+                              "lib/internal/v8_prof_polyfill.js"
+                              "test/parallel/test-child-process-spawnsync-shell.js"
+                              "test/parallel/test-fs-write-sigxfsz.js"
+                              "test/parallel/test-stdio-closed.js"
+                              "test/parallel/test-stdin-from-file-spawn.js"
+                              "test/sequential/test-child-process-emfile.js")
+                 (("'/bin/sh'")
+                  (string-append "'" (assoc-ref inputs "bash") "/bin/sh'")))
+
+               ;; Fix hardcoded /usr/bin/env references.
+               (substitute* '("test/parallel/test-child-process-default-options.js"
+                              "test/parallel/test-child-process-env.js"
+                              "test/parallel/test-child-process-exec-env.js")
+                 (("'/usr/bin/env'")
+                  (string-append "'" (assoc-ref inputs "coreutils")
+                                 "/bin/env'")))
+
+               ;; Fix /usr/bin/env shebang in node-gyp.
+               (substitute* "deps/npm/node_modules/node-gyp/bin/node-gyp.js"
+                 (("#!/usr/bin/env")
+                  (string-append "#!" (assoc-ref inputs "coreutils") "/bin/env")))
+
+               (substitute* "deps/npm/node_modules/node-gyp/gyp/gyp_main.py"
+                 (("#!/usr/bin/env python")
+                  (string-append "#!" (assoc-ref (or native-inputs inputs)
+                                                 "python")
+                                 "/bin/python3")))
+
+               ;; FIXME: These tests fail in the build container, but they don't
+               ;; seem to be indicative of real problems in practice.
+               (for-each delete-file
+                         '("test/parallel/test-cluster-primary-error.js"
+                           "test/parallel/test-cluster-primary-kill.js"))
+
+               ;; These require a DNS resolver.
+               (for-each delete-file
+                         '("test/parallel/test-dns.js"
+                           "test/parallel/test-dns-lookupService-promises.js"))
+
+               ;; FIXME: This test fails randomly:
+               ;; https://github.com/nodejs/node/issues/31213
+               (delete-file "test/parallel/test-net-listen-after-destroying-stdin.js")
+
+               ;; FIXME: These tests fail on armhf-linux:
+               ;; https://github.com/nodejs/node/issues/31970
+               ,@(if (target-arm32?)
+                     '((for-each delete-file
+                                 '("test/parallel/test-zlib.js"
+                                   "test/parallel/test-zlib-brotli.js"
+                                   "test/parallel/test-zlib-brotli-flush.js"
+                                   "test/parallel/test-zlib-brotli-from-brotli.js"
+                                   "test/parallel/test-zlib-brotli-from-string.js"
+                                   "test/parallel/test-zlib-convenience-methods.js"
+                                   "test/parallel/test-zlib-random-byte-pipes.js"
+                                   "test/parallel/test-zlib-write-after-flush.js")))
+                     '())
+
+               ;; These tests have an expiry date: they depend on the validity of
+               ;; TLS certificates that are bundled with the source.  We want this
+               ;; package to be reproducible forever, so remove those.
+               ;; TODO: Regenerate certs instead.
+               (for-each delete-file
+                         '("test/parallel/test-tls-passphrase.js"
+                           "test/parallel/test-tls-server-verify.js"))))))))
+    (native-inputs
+     (alist-replace "icu4c" (list icu4c-68)
+                    (package-native-inputs node-14)))
+    (inputs
+     (alist-replace "icu4c" (list icu4c-68)
+                    (alist-replace "llhttp" (list llhttp-bootstrap)
+                                   (package-inputs node-14))))))
+
+;; LTS release used by the node build system.
+(define-public node-lts node-14)
+
 (define-public libnode
-  (package/inherit node
+  (package/inherit node-10
     (name "libnode")
     (arguments
-     (substitute-keyword-arguments (package-arguments node)
+     (substitute-keyword-arguments (package-arguments node-10)
        ((#:configure-flags flags ''())
         `(cons* "--shared" "--without-npm" ,flags))
        ((#:phases phases '%standard-phases)
--
2.33.0

[bug#49946] [PATCH 3/3] guix: node-build-system: Support compiling addons with node-gyp.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 8025 bytes --]


Philip McGrath <philip@philipmcgrath.com> writes:

> * gnu/packages/node.scm (node)[arguments]: Replace 'patch-npm-shebang
> and 'patch-node-shebang with a new 'patch-nested-shebangs that also
> handles node-gyp and other shebangs under "/lib/node_modules".
> [inputs]: Add Python for node-gyp as "python-for-target".
> (node-lts)[inputs]: Likewise.
> * guix/build-system/node.scm (lower): Add optional #:python argument.
> * guix/build/node-build-system.scm (set-node-gyp-paths): New
> function. Sets the "npm_config_nodedir" and "npm_config_python"
> environment variables.
> (%standard-phases): Add 'set-node-gyp-paths after 'set-paths.

Nice! I'll test this with the tree-sitter series. I just had one comment
inline, otherwise it looks good to me.

Do you want me to integrate it into the tree-sitter series or submit it
separately? It might make its way upstream quicker separately, in which
case I'd suggest to send it again in a new bug for more visibility.

>
> Co-authored-by: Pierre Langlois <pierre.langlois@gmx.com>
> ---
>  gnu/packages/node.scm            | 33 +++++++++++++++++---------------
>  guix/build-system/node.scm       |  7 ++++++-
>  guix/build/node-build-system.scm |  9 +++++++++
>  3 files changed, 33 insertions(+), 16 deletions(-)
>
> diff --git a/gnu/packages/node.scm b/gnu/packages/node.scm
> index 6d9c3ccc71..805a4f18fc 100644
> --- a/gnu/packages/node.scm
> +++ b/gnu/packages/node.scm
> @@ -244,21 +244,22 @@
>                              python
>                              (string-append python "3")))
>                        "configure" flags))))
> -         (add-after 'patch-shebangs 'patch-npm-shebang
> -           (lambda* (#:key outputs #:allow-other-keys)
> -             (let* ((bindir (string-append (assoc-ref outputs "out")
> -                                           "/bin"))
> -                    (npm    (string-append bindir "/npm"))
> -                    (target (readlink npm)))
> -               (with-directory-excursion bindir
> -                 (patch-shebang target (list bindir))))))
> -         (add-after 'install 'patch-node-shebang
> -           (lambda* (#:key outputs #:allow-other-keys)
> -             (let* ((bindir (string-append (assoc-ref outputs "out")
> -                                           "/bin"))
> -                    (npx    (readlink (string-append bindir "/npx"))))
> -               (with-directory-excursion bindir
> -                 (patch-shebang npx (list bindir)))))))))
> +         (add-after 'patch-shebangs 'patch-nested-shebangs
> +           (lambda* (#:key inputs outputs #:allow-other-keys)
> +             (let* ((prefix (assoc-ref outputs "out"))
> +                    (path (map (lambda (dir)
> +                                 (string-append dir "/bin"))
> +                               (list prefix
> +                                     (assoc-ref inputs "python-for-target")))))
> +               (with-directory-excursion (string-append prefix "/lib/node_modules")
> +                 (for-each
> +                  (lambda (file)
> +                    (patch-shebang file path))
> +                  (find-files "."
> +                              (lambda (file stat)
> +                                (and (eq? 'regular (stat:type stat))
> +                                     (not (zero? (logand (stat:mode stat) #o100)))))
> +                              #:stat lstat)))))))))

Here you don't necessarily need with-directory-excursion. I see we also
have a executable-file? predicate function in (guix build utils), could
we use that? i.e:

  (for-each
    (lambda (file)
      (patch-shebang file path))
    (find-files (string-append prefix "/lib/node_modules") executable-file?))

>      (native-inputs
>       `(;; Runtime dependencies for binaries used as a bootstrap.
>         ("c-ares" ,c-ares)
> @@ -281,6 +282,7 @@
>      (inputs
>       `(("bash" ,bash-minimal)
>         ("coreutils" ,coreutils)
> +       ("python-for-target" ,python-wrapper) ;; for node-gyp (supports python3)
>         ("c-ares" ,c-ares)
>         ("http-parser" ,http-parser)
>         ("icu4c" ,icu4c)
> @@ -764,6 +766,7 @@ source files.")
>      (inputs
>       `(("bash" ,bash-minimal)
>         ("coreutils" ,coreutils)
> +       ("python-for-target" ,python-wrapper) ;; for node-gyp (supports python3)
>         ("c-ares" ,c-ares)
>         ("icu4c" ,icu4c-67)
>         ("libuv" ,libuv-for-node)
> diff --git a/guix/build-system/node.scm b/guix/build-system/node.scm
> index 98f63f87ef..3e49e67ff6 100644
> --- a/guix/build-system/node.scm
> +++ b/guix/build-system/node.scm
> @@ -1,6 +1,8 @@
>  ;;; GNU Guix --- Functional package management for GNU
>  ;;; Copyright © 2016 Jelle Licht <jlicht@fsfe.org>
>  ;;; Copyright © 2019 Timothy Sample <samplet@ngyro.com>
> +;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
> +;;; Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -24,6 +26,7 @@
>    #:use-module (guix search-paths)
>    #:use-module (guix build-system)
>    #:use-module (guix build-system gnu)
> +  #:use-module (guix build-system python)
>    #:use-module (ice-9 match)
>    #:export (%node-build-system-modules
>              node-build
> @@ -44,11 +47,12 @@
>  (define* (lower name
>                  #:key source inputs native-inputs outputs system target
>                  (node (default-node))
> +                (python (default-python)) ;; for node-gyp
>                  #:allow-other-keys
>                  #:rest arguments)
>    "Return a bag for NAME."
>    (define private-keywords
> -    '(#:source #:target #:node #:inputs #:native-inputs))
> +    '(#:source #:target #:node #:python #:inputs #:native-inputs))
>  
>    (and (not target)                    ;XXX: no cross-compilation
>         (bag
> @@ -62,6 +66,7 @@
>                          ;; Keep the standard inputs of 'gnu-build-system'.
>                          ,@(standard-packages)))
>           (build-inputs `(("node" ,node)
> +                         ("python" ,python)
>                           ,@native-inputs))
>           (outputs outputs)
>           (build node-build)
> diff --git a/guix/build/node-build-system.scm b/guix/build/node-build-system.scm
> index 70a367618e..5e62eb4784 100644
> --- a/guix/build/node-build-system.scm
> +++ b/guix/build/node-build-system.scm
> @@ -2,6 +2,8 @@
>  ;;; Copyright © 2015 David Thompson <davet@gnu.org>
>  ;;; Copyright © 2016, 2020 Jelle Licht <jlicht@fsfe.org>
>  ;;; Copyright © 2019, 2021 Timothy Sample <samplet@ngyro.com>
> +;;; Copyright © 2021 Pierre Langlois <pierre.langlois@gmx.com>
> +;;; Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -46,6 +48,12 @@
>                (format #t "set HOME to ~s~%" (getenv "HOME")))))))
>    #t)
>  
> +(define* (set-node-gyp-paths #:key inputs #:allow-other-keys)
> +  "Initialize environment variables needed for building native addons."
> +  (setenv "npm_config_nodedir" (assoc-ref inputs "node"))
> +  (setenv "npm_config_python" (assoc-ref inputs "python"))
> +  #t)
> +
>  (define (module-name module)
>    (let* ((package.json (string-append module "/package.json"))
>           (package-meta (call-with-input-file package.json read-json)))
> @@ -144,6 +152,7 @@
>  
>  (define %standard-phases
>    (modify-phases gnu:%standard-phases
> +    (add-after 'set-paths 'set-node-gyp-paths set-node-gyp-paths)
>      (add-after 'unpack 'set-home set-home)
>      (add-before 'configure 'patch-dependencies patch-dependencies)
>      (replace 'configure configure)


The build system changes are now so nice and simple :-). I guess we
don't really need to set npm_config_python, it should be able to find
python in the PATH, but it doesn't hurt.

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [WIP PATCH v3 00/26] gnu: Add tree-sitter for emacs (and neovim as well?).

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 2541 bytes --]

Hi Guix, hi Luis,

Here's a WIP V3 of my series for tree-sitter. I'm almost done with it, I
need to to another couple of passes to see if there's anything left to
cleanup.

I'm sharing it early as a WIP first, as I just noticed the other day
that Luis also contributed a patches for tree-sitter for neovim[0]. The
series I have here is quite a lot bigger though so I'm happy to rebase
it on top of yours, Luis :-). The package I had for tree-sitter looks
basically the same, I would then move it to a new file with all
tree-sitter related packages, if you're happy with that.

The other reason I'm posting this is this series, which while it only
supports emacs, actually builds grammars for some of the supported
languages, I picked a set that I think are popular, there are probably
more to add but I thought it was a good starting point. While the
grammars are originally in JSON, they are distributed as binaries it
seems so it'll be better if we can build them as part of Guix.

So I'm wondering, are these tree-sitter grammar packages also useful for
the neovim package for tree-sitter?

Anyways, see patches incoming in just a minute!

Thanks,
Pierre

[0]: https://issues.guix.gnu.org/53523

Pierre Langlois (26):
  gnu: Add rust-smallbitvec.
  gnu: Add rust-html-escape.
  gnu: Add rust-spin@0.7.
  gnu: Add tree-sitter.
  gnu: Add tree-sitter-cli.
  gnu: Add tree-sitter-c.
  gnu: Add tree-sitter-cpp.
  gnu: Add tree-sitter-bash.
  gnu: Add tree-sitter-css.
  gnu: Add tree-sitter-go.
  gnu: Add tree-sitter-html.
  gnu: Add tree-sitter-java.
  gnu: Add tree-sitter-javascript.
  gnu: Add tree-sitter-json.
  gnu: Add tree-sitter-julia.
  gnu: Add tree-sitter-php.
  gnu: Add tree-sitter-python.
  gnu: Add tree-sitter-rust.
  gnu: Add tree-sitter-typescript.
  gnu: Add rust-tree-sitter-for-emacs.
  gnu: rust-emacs-module: Update to 0.18.
  gnu: rust-emacs-macros: Update to 0.17.
  gnu: rust-emacs: Update to 0.18.
  gnu: Add emacs-tree-sitter-core.
  gnu: Add emacs-tree-sitter.
  gnu: Add emacs-tree-sitter-langs.

 gnu/local.mk                                  |   2 +
 gnu/packages/crates-io.scm                    | 154 +++-
 .../rust-tree-sitter-text-provider-fix.patch  |  98 +++
 gnu/packages/tree-sitter.scm                  | 808 ++++++++++++++++++
 4 files changed, 1044 insertions(+), 18 deletions(-)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
 create mode 100644 gnu/packages/tree-sitter.scm

-- 
2.34.0



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [WIP PATCH v3 01/26] gnu: Add rust-smallbitvec.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-smallbitvec-2): New variable.
---
 gnu/packages/crates-io.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index fa9a7d9ab5..d001e723ec 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -56099,6 +56099,28 @@ (define-public rust-sluice-0.4
         ("rust-futures-core-preview" ,rust-futures-core-preview-0.3)
         ("rust-futures-io-preview" ,rust-futures-io-preview-0.3))))))

+(define-public rust-smallbitvec-2
+  (package
+    (name "rust-smallbitvec")
+    (version "2.5.1")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "smallbitvec" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "0plrbldsjpwip3afbzd8fgrnvdhizcg5z4ncfqs4q6x4qjflzkkm"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-development-inputs
+       (("rust-bit-vec" ,rust-bit-vec-0.4)
+        ("rust-rand" ,rust-rand-0.4))))
+    (home-page "https://github.com/servo/smallbitvec")
+    (synopsis "A bit vector optimized for size and inline storage")
+    (description "This package provides a bit vector optimized for size and
+inline storage")
+    (license (list license:expat license:asl2.0))))
+
 (define-public rust-smallvec-1
   (package
     (name "rust-smallvec")
--
2.34.0

[bug#49946] [WIP PATCH v3 02/26] gnu: Add rust-html-escape.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-html-escape-0.2): New variable.
---
 gnu/packages/crates-io.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index d001e723ec..cff703c427 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -26933,6 +26933,30 @@ (define-public rust-htmlescape-0.3
      "This package provides a library for HTML entity encoding and decoding.")
     (license (list license:asl2.0 license:expat license:mpl2.0))))

+(define-public rust-html-escape-0.2
+  (package
+    (name "rust-html-escape")
+    (version "0.2.9")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "html-escape" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "1dxw9lpckrqzzqgbkw64ckbajr4b7xxnjdn8adgzqf2mm40shvl1"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-utf8-width" ,rust-utf8-width-0.1))
+       #:cargo-development-inputs
+       (("rust-bencher" ,rust-bencher-0.1))))
+    (home-page "https://magiclen.org/html-escape")
+    (synopsis "Library for encoding and escaping special characters in HTML")
+    (description
+     "This package provides a library for encoding and escaping special
+characters in HTML, decoding and unescaping HTML entities as well.")
+    (license license:expat)))
+
 (define-public rust-http-0.2
   (package
     (name "rust-http")
--
2.34.0

[bug#49946] [WIP PATCH v3 03/26] gnu: Add rust-spin@0.7.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-spin-0.7): New variable.
---
 gnu/packages/crates-io.scm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index cff703c427..ad27298404 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -56716,6 +56716,19 @@ (define-public rust-spin-0.9
 initializers are available.")
     (license license:expat)))

+(define-public rust-spin-0.7
+  (package
+    (inherit rust-spin-0.9)
+    (name "rust-spin")
+    (version "0.7.1")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "spin" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "0qjips9f6fsvkyd7wj3a4gzaqknn2q4kkb19957pl86im56pna0k"))))))
+
 (define-public rust-spin-0.5
   (package
     (inherit rust-spin-0.9)
--
2.34.0

[bug#49946] [WIP PATCH v3 04/26] gnu: Add tree-sitter.

[Pierre Langlois]

* gnu/package/tree-sitter.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Register it.
---
 gnu/local.mk                 |  1 +
 gnu/packages/tree-sitter.scm | 76 ++++++++++++++++++++++++++++++++++++
 2 files changed, 77 insertions(+)
 create mode 100644 gnu/packages/tree-sitter.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index 198c8f64a6..273d80a9c2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -584,6 +584,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/tmux.scm				\
   %D%/packages/toolkits.scm			\
   %D%/packages/tor.scm				\
+  %D%/packages/tree-sitter.scm			\
   %D%/packages/tv.scm				\
   %D%/packages/uglifyjs.scm			\
   %D%/packages/uml.scm				\
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
new file mode 100644
index 0000000000..e7d2ce113d
--- /dev/null
+++ b/gnu/packages/tree-sitter.scm
@@ -0,0 +1,76 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages tree-sitter)
+  #:use-module (guix gexp)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix git-download)
+  #:use-module (guix packages)
+  #:use-module (guix utils))
+
+(define-public tree-sitter
+  (package
+    (name "tree-sitter")
+    (version "0.20.4")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1yldgdbf3l5l4ki52abdf81nwkcbvg219gwr3ydcjwfsg7hf7zhz"))))
+    (build-system gnu-build-system)
+    (arguments
+     (list
+      #:tests? #f ;No check target.
+      #:phases
+      #~(modify-phases %standard-phases
+          (delete 'configure)
+          (add-before 'build 'set-cc
+            (lambda _
+              (setenv "CC"
+                      #$(cc-for-target))))
+          (replace 'install
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out")) (lib (string-append
+                                                           out "/lib")))
+                (setenv "PREFIX" out)
+                (invoke "make" "install")))))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Incremental parsing system for programming tools")
+    (description
+     "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @code{libtree-sitter} runtime library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 05/26] gnu: Add tree-sitter-cli.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cli): New variable.
---
 gnu/packages/tree-sitter.scm | 68 +++++++++++++++++++++++++++++++++++-
 1 file changed, 67 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e7d2ce113d..c2919e76b2 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -19,10 +19,13 @@
 (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix git-download)
   #:use-module (guix packages)
-  #:use-module (guix utils))
+  #:use-module (guix utils)
+  #:use-module (gnu packages crates-graphics)
+  #:use-module (gnu packages crates-io))

 (define-public tree-sitter
   (package
@@ -74,3 +77,66 @@ (define-public tree-sitter

 This package includes the @code{libtree-sitter} runtime library.")
     (license license:expat)))
+
+(define-public tree-sitter-cli
+  (package (inherit tree-sitter)
+    (name "tree-sitter-cli")
+    (build-system cargo-build-system)
+    (arguments
+     `(;; Running test requires downloading fixtures, see the
+       ;; script/fetch-fixtures script.
+       #:tests? #f
+       ;; FIXME: Installing the sources for the tree-sitter Rust bindings
+       ;; doesn't work out of the box due to tree-sitter having multiple
+       ;; Rust packages in the same repository (bindings and CLI).
+       #:install-source? #f
+       #:cargo-inputs
+       (("rust-ansi-term" ,rust-ansi-term-0.12)
+        ("rust-anyhow" ,rust-anyhow-1)
+        ("rust-atty" ,rust-atty-0.2)
+        ("rust-clap" ,rust-clap-2)
+        ("rust-difference" ,rust-difference-2)
+        ("rust-dirs" ,rust-dirs-3)
+        ("rust-html-escape" ,rust-html-escape-0.2)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-rustc-hash" ,rust-rustc-hash-1)
+        ("rust-serde" ,rust-serde-1)
+        ("rust-smallbitvec" ,rust-smallbitvec-2)
+        ("rust-spin" ,rust-spin-0.7)
+        ("rust-thiserror" ,rust-thiserror-1)
+        ("rust-tiny-http" ,rust-tiny-http-0.8)
+        ("rust-toml" ,rust-toml-0.5)
+        ("rust-walkdir" ,rust-walkdir-2)
+        ("rust-webbrowser" ,rust-webbrowser-0.5)
+        ("rust-which" ,rust-which-4))
+       #:cargo-development-inputs
+       (("rust-pretty-assertions" ,rust-pretty-assertions-0.7))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin"))
+                    (lib (string-append out "/lib")))
+               (mkdir-p bin)
+               (install-file "target/release/tree-sitter" bin)))))))
+    (synopsis "Incremental parsing system for programming tools")
+    (description "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @command{tree-sitter} command-line tool.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 07/26] gnu: Add tree-sitter-cpp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cpp): New variable.
---
 gnu/packages/tree-sitter.scm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index b62cb483c9..cc3bb2cec0 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -247,3 +247,21 @@ (define (source-file? basename)
     (description
       "This package provides a C grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-cpp
+  (package
+    (inherit tree-sitter-c)
+    (name "tree-sitter-cpp")
+    (version "0.20.0")
+    (source (tree-sitter-origin
+             name
+             version
+             "0hxcpdvyyig8njga1mxp4qcnbbnr1d0aiy27vahijwbh98b081nr"))
+    (inputs
+     (list node-nan
+           `(,tree-sitter-c "js")))
+    (home-page "https://github.com/tree-sitter/tree-sitter-cpp")
+    (synopsis "Tree-sitter C++ grammar")
+    (description
+     "This package provides a C++ grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 08/26] gnu: Add tree-sitter-bash.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-bash): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index cc3bb2cec0..827e1cdc4d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -265,3 +265,27 @@ (define-public tree-sitter-cpp
     (description
      "This package provides a C++ grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-bash
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-bash")
+    (version "0.19.0")
+    (source (tree-sitter-origin
+             name
+             version
+             "18c030bb65r50i6z37iy7jb9z9i8i36y7b08dbc9bchdifqsijs5"))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+          (replace 'delete-dependencies
+           (lambda _
+             (delete-dependencies '("tree-sitter-cli"
+                                    "node-gyp"
+                                    "prebuild"
+                                    "prebuild-install"))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-bash")
+    (synopsis "Tree-sitter Bash grammar")
+    (description
+      "This package provides a Bash grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 09/26] gnu: Add tree-sitter-css.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-css): New variable.
---
 gnu/packages/tree-sitter.scm | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 827e1cdc4d..36b712c189 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -289,3 +289,17 @@ (define-public tree-sitter-bash
     (description
       "This package provides a Bash grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-css
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-css")
+    (version "0.19.0")
+    (source (tree-sitter-origin
+             name
+             version
+             "014jrlgi7zfza9g38hsr4vlbi8964i5p7iglaih6qmzaiml7bja2"))
+    (home-page "https://github.com/tree-sitter/tree-sitter-css")
+    (synopsis "Tree-sitter CSS grammar")
+    (description
+      "This package provides a CSS grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 10/26] gnu: Add tree-sitter-go.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-go): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 36b712c189..47f102b0b3 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -303,3 +303,26 @@ (define-public tree-sitter-css
     (description
       "This package provides a CSS grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-go
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-go")
+    (version "0.19.1")
+    (source (tree-sitter-origin
+             name
+             version
+             "0nxs47vd2fc2fr0qlxq496y852rwg39flhg334s7dlyq7d3lcx4x"))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-go")
+    (synopsis "Tree-sitter Go grammar")
+    (description
+      "This package provides a Golang grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 11/26] gnu: Add tree-sitter-html.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-html): New variable.
---
 gnu/packages/tree-sitter.scm | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 47f102b0b3..e4679dc8ab 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -326,3 +326,17 @@ (define-public tree-sitter-go
     (description
       "This package provides a Golang grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-html
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-html")
+    (version "0.19.0")
+    (source (tree-sitter-origin
+             name
+             version
+             "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
+    (home-page "https://github.com/tree-sitter/tree-sitter-html")
+    (synopsis "Tree-sitter HTML grammar")
+    (description
+      "This package provides a HTML grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 12/26] gnu: Add tree-sitter-java.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-java): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e4679dc8ab..760a7871df 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -340,3 +340,26 @@ (define-public tree-sitter-html
     (description
       "This package provides a HTML grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-java
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-java")
+    (version "0.19.1")
+    (source (tree-sitter-origin
+             name
+             version
+             "07zw9ygb45hnvlx9qlz7rlz8hc3byjy03d24v72i5iyhpiiwlhvl"))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-java")
+    (synopsis "Tree-sitter Java grammar")
+    (description
+      "This package provides a Java grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 13/26] gnu: Add tree-sitter-javascript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-javascript): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 760a7871df..b728e6511b 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -363,3 +363,27 @@ (define-public tree-sitter-java
     (description
       "This package provides a Java grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-javascript
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-javascript")
+    (version "0.20.0")
+    (source (tree-sitter-origin
+             name
+             version
+             "175yrk382n2di0c2xn4gpv8y4n83x1lg4hqn04vabf0yqynlkq67"))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
+    (synopsis "Tree-sitter Javascript grammar")
+    (description
+      "This package provides a Javascript grammar for the Tree-sitter
+library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 14/26] gnu: Add tree-sitter-json.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-json): New variable.
---
 gnu/packages/tree-sitter.scm | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index b728e6511b..d35edc533d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -387,3 +387,17 @@ (define-public tree-sitter-javascript
       "This package provides a Javascript grammar for the Tree-sitter
 library.")
     (license license:expat)))
+
+(define-public tree-sitter-json
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-json")
+    (version "0.19.0")
+    (source (tree-sitter-origin
+             name
+             version
+             "06pjh31bv9ja9hlnykk257a6zh8bsxg2fqa54al7qk1r4n9ksnff"))
+    (home-page "https://github.com/tree-sitter/tree-sitter-json")
+    (synopsis "Tree-sitter JSON grammar")
+    (description
+      "This package provides a JSON grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 15/26] gnu: Add tree-sitter-julia.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-julia): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d35edc533d..aeb541224a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -401,3 +401,26 @@ (define-public tree-sitter-json
     (description
       "This package provides a JSON grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-julia
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-julia")
+    (version "0.19.0")
+    (source (tree-sitter-origin
+             name
+             version
+             "1pbnmvhy2gq4vg1b0sjzmjm4s2gsgdjh7h01yj8qrrqbcl29c463"))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-julia")
+    (synopsis "Tree-sitter Julia grammar")
+    (description
+      "This package provides a Julia grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 16/26] gnu: Add tree-sitter-php.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-php): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index aeb541224a..072d2bf588 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -424,3 +424,26 @@ (define-public tree-sitter-julia
     (description
       "This package provides a Julia grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-php
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-php")
+    (version "0.19.0")
+    (source (tree-sitter-origin
+             name
+             version
+             "17cmybgpprw7w9d2v7lmc6zmr90d70g0jqq279gzg0mpwfzla53s"))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-php")
+    (synopsis "Tree-sitter PHP grammar")
+    (description
+      "This package provides a PHP grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 17/26] gnu: Add tree-sitter-python.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-python): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 072d2bf588..05d6ade77d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -447,3 +447,26 @@ (define-public tree-sitter-php
     (description
       "This package provides a PHP grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-python
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-python")
+    (version "0.19.1")
+    (source (tree-sitter-origin
+             name
+             version
+             "16i5mcpz4ld0rybffb0nxim4crb9fhnnq20b4bkjs7jjr3zcx7vq"))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+         ;; The parse-example script tries to clone git repositories.
+         (add-after 'unpack 'remove-parse-example
+           (lambda _
+             (substitute* "package.json"
+               ((" && script\\/parse-examples") ""))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-python")
+    (synopsis "Tree-sitter Python grammar")
+    (description
+      "This package provides a Python grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 18/26] gnu: Add tree-sitter-rust.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-rust): New variable.
---
 gnu/packages/tree-sitter.scm | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 05d6ade77d..63e558f50f 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -26,6 +26,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages algebra)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages node-xyz)
@@ -470,3 +471,20 @@ (define-public tree-sitter-python
     (description
       "This package provides a Python grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-rust
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-rust")
+    (version "0.20.1")
+    (source (tree-sitter-origin
+             name
+             version
+             "174j5pxwf80c4xniri39l3a6bb7nq96g2s8hh5sgv4i7xvbpfsmg"))
+    (native-inputs
+     (modify-inputs (package-native-inputs tree-sitter-c)
+       (prepend bc)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-rust")
+    (synopsis "Tree-sitter Rust grammar")
+    (description
+      "This package provides a Rust grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 19/26] gnu: Add tree-sitter-typescript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-typescript): New variable.
---
 gnu/packages/tree-sitter.scm | 87 ++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 63e558f50f..2dcf4dae87 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -488,3 +488,90 @@ (define-public tree-sitter-rust
     (description
       "This package provides a Rust grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-typescript
+  (let ((commit "111b07762e86efab9a918b7c721f720c37e76b0a")
+        (revision "1"))
+    (package (inherit tree-sitter-c)
+      (name "tree-sitter-typescript")
+      (version (git-version "0.20.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-typescript")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32 "1364bm3wbqqzvp03cvpx1w89bbqynb1v62i46gy7f6rhib9bf63a"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (for-each
+                     (lambda (lang)
+                       (with-directory-excursion lang
+                         (delete-file "src/grammar.json")
+                         (delete-file "src/node-types.json")
+                         (delete-file "src/parser.c")
+                         (delete-file-recursively "src/tree_sitter")))
+                     '("typescript" "tsx"))
+                    #t))))
+      (inputs
+       (list node-nan
+             `(,tree-sitter-javascript "js")))
+      (arguments
+       (substitute-keyword-arguments (package-arguments tree-sitter-c)
+        ((#:phases phases)
+         #~(modify-phases #$phases
+             ;; The parse-example script tries to clone git repositories.
+             (add-after 'unpack 'remove-parse-example
+               (lambda _
+                 (substitute* "package.json"
+                   ((" && script\\/parse-examples") ""))))
+             (add-after 'build 'build-gyp
+               (lambda* (#:key inputs #:allow-other-keys)
+                 (invoke
+                  (search-input-file
+                   inputs
+                   "/lib/node_modules/npm/bin/node-gyp-bin/node-gyp")
+                  "build")))
+             (replace 'tree-sitter-generate
+               (lambda _
+                 (for-each (lambda (dir)
+                             (with-directory-excursion dir
+                               (invoke "tree-sitter" "generate")))
+                           '("typescript" "tsx"))))
+             (replace 'build-and-install-lib
+               (lambda* (#:key outputs #:allow-other-keys)
+                 (use-modules (guix build json))
+                 (let ((name (assoc-ref
+                              (call-with-input-file "package.json" read-json)
+                              "name"))
+                       (lib (string-append #$output "/lib/tree-sitter/")))
+                   (mkdir-p lib)
+                   (define (source-file? basename)
+                     "document me"
+                     (let ((path (string-append src "/" basename)))
+                       (if (file-exists? path)
+                           path
+                           #f)))
+                   (for-each
+                    (lambda (lang)
+                      (let ((src (string-append #$output:js
+                                                "/lib/node_modules/"
+                                                name "/" lang "/src")))
+                        (invoke #$(cc-for-target)
+                                "-shared"
+                                "-fPIC"
+                                "-fno-exceptions"
+                                "-O2"
+                                "-g"
+                                "-o" (string-append lib "/" lang ".so")
+                                (string-append src "/scanner.c")
+                                (string-append src "/parser.c"))))
+                    '("typescript" "tsx")))))))))
+      (home-page "https://github.com/tree-sitter/tree-sitter-typescript")
+      (synopsis "Tree-sitter Typescript grammar")
+      (description
+        "This package provides Typescript and TSX grammars for the Tree-sitter
+library.")
+      (license license:expat))))
--
2.34.0

[bug#49946] [WIP PATCH v3 20/26] gnu: Add rust-tree-sitter-for-emacs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter-for-emacs): New variable.
* gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
---
 gnu/local.mk                                  |  1 +
 .../rust-tree-sitter-text-provider-fix.patch  | 98 +++++++++++++++++++
 gnu/packages/tree-sitter.scm                  | 45 +++++++++
 3 files changed, 144 insertions(+)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 273d80a9c2..f2b6f0afdd 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1811,6 +1811,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/rust-nettle-sys-disable-vendor.patch	 \
   %D%/packages/patches/rust-openssl-sys-no-vendor.patch	\
   %D%/packages/patches/rust-shell2batch-lint-fix.patch		\
+  %D%/packages/patches/rust-tree-sitter-text-provider-fix.patch	\
   %D%/packages/patches/rust-wl-clipboard-rs-newer-wl.patch      \
   %D%/packages/patches/sbc-fix-build-non-x86.patch		\
   %D%/packages/patches/sbcl-aserve-add-HTML-5-elements.patch	\
diff --git a/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
new file mode 100644
index 0000000000..beda2d8391
--- /dev/null
+++ b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
@@ -0,0 +1,98 @@
+From 475b822f47bdc58d832533448b6f6d9818554f37 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Tu=E1=BA=A5n-Anh=20Nguy=E1=BB=85n?= <ubolonton@gmail.com>
+Date: Sun, 25 Jul 2021 13:11:52 +0700
+Subject: [PATCH] Allow TextProvider's iterators to generate owned text
+
+---
+ binding_rust/lib.rs | 33 ++++++++++++++++++---------------
+ 1 file changed, 18 insertions(+), 15 deletions(-)
+
+diff --git a/lib/binding_rust/lib.rs b/lib/binding_rust/lib.rs
+index 068a60285..79c3991ed 100644
+--- a/binding_rust/lib.rs
++++ b/binding_rust/lib.rs
+@@ -8,6 +8,7 @@ pub mod allocations;
+ use std::os::unix::io::AsRawFd;
+
+ use std::{
++    borrow::Cow,
+     char, error,
+     ffi::CStr,
+     fmt, hash, iter,
+@@ -162,7 +163,8 @@ pub struct QueryCaptures<'a, 'tree: 'a, T: TextProvider<'a>> {
+ }
+
+ pub trait TextProvider<'a> {
+-    type I: Iterator<Item = &'a [u8]> + 'a;
++    type I: Iterator<Item = Cow<'a, [u8]>>;
++
+     fn text(&mut self, node: Node) -> Self::I;
+ }
+
+@@ -1799,19 +1801,19 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+         buffer2: &mut Vec<u8>,
+         text_provider: &mut impl TextProvider<'a>,
+     ) -> bool {
+-        fn get_text<'a, 'b: 'a, I: Iterator<Item = &'b [u8]>>(
++        fn get_text<'a, 'b: 'a, I: Iterator<Item = Cow<'b, [u8]>>>(
+             buffer: &'a mut Vec<u8>,
+             mut chunks: I,
+-        ) -> &'a [u8] {
+-            let first_chunk = chunks.next().unwrap_or(&[]);
++        ) -> Cow<'a, [u8]> {
++            let first_chunk = chunks.next().unwrap_or(Cow::Owned(vec![0u8; 0]));
+             if let Some(next_chunk) = chunks.next() {
+                 buffer.clear();
+-                buffer.extend_from_slice(first_chunk);
+-                buffer.extend_from_slice(next_chunk);
++                buffer.extend_from_slice(&first_chunk);
++                buffer.extend_from_slice(&next_chunk);
+                 for chunk in chunks {
+-                    buffer.extend_from_slice(chunk);
++                    buffer.extend_from_slice(&chunk);
+                 }
+-                buffer.as_slice()
++                Cow::Borrowed(buffer.as_slice())
+             } else {
+                 first_chunk
+             }
+@@ -1835,7 +1837,7 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+                 TextPredicate::CaptureMatchString(i, r, is_positive) => {
+                     let node = self.nodes_for_capture_index(*i).next().unwrap();
+                     let text = get_text(buffer1, text_provider.text(node));
+-                    r.is_match(text) == *is_positive
++                    r.is_match(&text) == *is_positive
+                 }
+             })
+     }
+@@ -1946,23 +1948,24 @@ impl<'cursor, 'tree> fmt::Debug for QueryMatch<'cursor, 'tree> {
+     }
+ }
+
+-impl<'a, F, I> TextProvider<'a> for F
++impl<'a, F, I, T> TextProvider<'a> for F
+ where
+     F: FnMut(Node) -> I,
+-    I: Iterator<Item = &'a [u8]> + 'a,
++    T: Into<Cow<'a, [u8]>>,
++    I: Iterator<Item = T>,
+ {
+-    type I = I;
++    type I = iter::Map<I, fn(T) -> Cow<'a, [u8]>>;
+
+     fn text(&mut self, node: Node) -> Self::I {
+-        (self)(node)
++        (self)(node).map(T::into)
+     }
+ }
+
+ impl<'a> TextProvider<'a> for &'a [u8] {
+-    type I = iter::Once<&'a [u8]>;
++    type I = iter::Once<Cow<'a, [u8]>>;
+
+     fn text(&mut self, node: Node) -> Self::I {
+-        iter::once(&self[node.byte_range()])
++        iter::once(Cow::Borrowed(&self[node.byte_range()]))
+     }
+ }
+
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 2dcf4dae87..c86244a300 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -23,9 +23,11 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system node)
   #:use-module (guix gexp)
+  #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages)
   #:use-module (gnu packages algebra)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
@@ -146,6 +148,49 @@ (define-public tree-sitter-cli
 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))

+;; We need to apply a patch in order to compile the rust bindings against the
+;; emacs tree-sitter module.
+;; See https://github.com/tree-sitter/tree-sitter/pull/1294
+(define-public rust-tree-sitter-for-emacs
+  (package
+    (name "rust-tree-sitter")
+    (version "0.20.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "tree-sitter" version))
+        (file-name (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1yg4p54hsfsxxknjq492i8b4rvibzpl2zdvr2bwvwakqgah05v33"))
+        (patches (search-patches "rust-tree-sitter-text-provider-fix.patch"))
+        (modules '((guix build utils)))
+        (snippet
+         '(begin
+            ;; Force
+            (delete-file-recursively "src")
+            (delete-file "binding_rust/build.rs")
+            (with-output-to-file "binding_rust/build.rs"
+              (lambda _
+                (format #t "fn main() {~@
+                        println!(\"cargo:rustc-link-lib=tree-sitter\");~@
+                        }~%")))
+            #f))))
+    (build-system cargo-build-system)
+    (inputs (list tree-sitter))
+    (arguments
+     `(#:tests? #f  ;; Running tests misinterprets comments as doc-tests.
+       #:cargo-inputs
+       (("rust-cc" ,rust-cc-1)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-regex" ,rust-regex-1)
+        ("rust-spin" ,rust-spin-0.7))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Rust bindings to the Tree-sitter parsing library")
+    (description "This package provides Rust bindings to the Tree-sitter
+parsing library.")
+    (license license:expat)))
+
 (define (tree-sitter-origin name version hash)
   (origin
     (method git-fetch)
--
2.34.0

[bug#49946] [WIP PATCH v3 21/26] gnu: rust-emacs-module: Update to 0.18.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-module-0.18): New variable.
(rust-emacs-module-0.10): Inherit from rust-emacs-module-0.18.
---
 gnu/packages/crates-io.scm | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index ad27298404..c039591c56 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -19090,10 +19090,10 @@ (define-public rust-emacs-macros-0.11
     (description "This package provides proc macros for Emacs modules.")
     (license license:bsd-3)))

-(define-public rust-emacs-module-0.10
+(define-public rust-emacs-module-0.18
   (package
     (name "rust-emacs-module")
-    (version "0.10.0")
+    (version "0.18.0")
     (source
       (origin
         (method url-fetch)
@@ -19102,11 +19102,11 @@ (define-public rust-emacs-module-0.10
          (string-append name "-" version ".tar.gz"))
         (sha256
          (base32
-          "1gf9lz735xbkyir53dyv362drfx3nin5an5cx39kd8q8kjjwix5g"))))
+          "1ypjyyv2ca3vza4sia91ckxamgfk63yd8frkvg3d4ph4fk4pn1mk"))))
     (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
-       (("rust-bindgen" ,rust-bindgen-0.48))))
+       (("rust-bindgen" ,rust-bindgen-0.59))))
     (inputs
      (list clang))
     (home-page "https://github.com/ubolonton/emacs-module-rs")
@@ -19115,6 +19115,23 @@ (define-public rust-emacs-module-0.10
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

+(define-public rust-emacs-module-0.10
+  (package (inherit rust-emacs-module-0.18)
+    (name "rust-emacs-module")
+    (version "0.10.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs_module" version))
+        (file-name
+         (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1gf9lz735xbkyir53dyv362drfx3nin5an5cx39kd8q8kjjwix5g"))))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-bindgen" ,rust-bindgen-0.48))))))
+
 (define-public rust-email-0.0.20
   (package
     (name "rust-email")
--
2.34.0

[bug#49946] [WIP PATCH v3 22/26] gnu: rust-emacs-macros: Update to 0.17.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-macros-0.17): New variable.
(rust-emacs-macros-0.11): Inherit from rust-emacs-macros-0.17.
---
 gnu/packages/crates-io.scm | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index c039591c56..ea2e9c9328 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -19066,8 +19066,31 @@ (define-public rust-emacs-0.11
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

-(define-public rust-emacs-macros-0.11
+(define-public rust-emacs-macros-0.17
   (package
+    (name "rust-emacs-macros")
+    (version "0.17.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs-macros" version))
+        (file-name (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "0qg1dcn5acbirq617qq2fgg9adswif2dnr292s3qnq62wzgnyrb9"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-darling" ,rust-darling-0.10)
+        ("rust-quote" ,rust-quote-1)
+        ("rust-syn" ,rust-syn-1))))
+    (home-page "https://github.com/ubolonton/emacs-module-rs")
+    (synopsis "Proc macros for Emacs modules")
+    (description "This package provides proc macros for Emacs modules.")
+    (license license:bsd-3)))
+
+(define-public rust-emacs-macros-0.11
+  (package (inherit rust-emacs-macros-0.17)
     (name "rust-emacs-macros")
     (version "0.11.0")
     (source
@@ -19079,16 +19102,11 @@ (define-public rust-emacs-macros-0.11
         (sha256
          (base32
           "0390y8vafxdi334hhgrzvcqjq3n5ckcmvilqcfp8vajjq8irrly6"))))
-    (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
        (("rust-darling" ,rust-darling-0.9)
         ("rust-quote" ,rust-quote-0.6)
-        ("rust-syn" ,rust-syn-0.15))))
-    (home-page "https://github.com/ubolonton/emacs-module-rs")
-    (synopsis "Proc macros for Emacs modules")
-    (description "This package provides proc macros for Emacs modules.")
-    (license license:bsd-3)))
+        ("rust-syn" ,rust-syn-0.15))))))

 (define-public rust-emacs-module-0.18
   (package
--
2.34.0

[bug#49946] [WIP PATCH v3 23/26] gnu: rust-emacs: Update to 0.18.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-0.18): New variable.
(rust-emacs-0.11): Inherit from rust-emacs-0.18.
---
 gnu/packages/crates-io.scm | 38 +++++++++++++++++++++++++++++++-------
 1 file changed, 31 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index ea2e9c9328..f9ea194438 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -19035,10 +19035,10 @@ (define-public rust-elf-0.0.10
      "This package provides a pure-Rust library for parsing ELF files.")
     (license (list license:expat license:asl2.0))))

-(define-public rust-emacs-0.11
+(define-public rust-emacs-0.18
   (package
     (name "rust-emacs")
-    (version "0.11.0")
+    (version "0.18.0")
     (source
       (origin
         (method url-fetch)
@@ -19047,16 +19047,18 @@ (define-public rust-emacs-0.11
          (string-append name "-" version ".tar.gz"))
         (sha256
          (base32
-          "1xn6nr8m3p24irlmv3i7c9n95yb10qc6aikwqv99qlmhbky5x0z7"))))
+          "0r860i73b2680i2fhdl2l1wwvvmf2zksncpckgkksdcx310ak5v7"))))
     (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
-       (("rust-ctor" ,rust-ctor-0.1)
-        ("rust-emacs-macros" ,rust-emacs-macros-0.11)
-        ("rust-emacs-module" ,rust-emacs-module-0.10)
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-ctor" ,rust-ctor-0.1)
+        ("rust-emacs-macros" ,rust-emacs-macros-0.17)
+        ("rust-emacs-module" ,rust-emacs-module-0.18)
         ("rust-failure" ,rust-failure-0.1)
         ("rust-failure-derive" ,rust-failure-derive-0.1)
-        ("rust-lazy-static" ,rust-lazy-static-1))))
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-thiserror" ,rust-thiserror-1))))
     (inputs
      (list clang))
     (home-page "https://github.com/ubolonton/emacs-module-rs")
@@ -19066,6 +19068,28 @@ (define-public rust-emacs-0.11
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

+(define-public rust-emacs-0.11
+  (package (inherit rust-emacs-0.18)
+    (name "rust-emacs")
+    (version "0.11.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs" version))
+        (file-name
+         (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1xn6nr8m3p24irlmv3i7c9n95yb10qc6aikwqv99qlmhbky5x0z7"))))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-ctor" ,rust-ctor-0.1)
+        ("rust-emacs-macros" ,rust-emacs-macros-0.11)
+        ("rust-emacs-module" ,rust-emacs-module-0.10)
+        ("rust-failure" ,rust-failure-0.1)
+        ("rust-failure-derive" ,rust-failure-derive-0.1)
+        ("rust-lazy-static" ,rust-lazy-static-1))))))
+
 (define-public rust-emacs-macros-0.17
   (package
     (name "rust-emacs-macros")
--
2.34.0

[bug#49946] [WIP PATCH v3 24/26] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
(emacs-tree-sitter-core): New variable.
---
 gnu/packages/tree-sitter.scm | 78 ++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c86244a300..893c02824d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -20,6 +20,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
+  #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system node)
   #:use-module (guix gexp)
@@ -620,3 +621,80 @@ (define (source-file? basename)
         "This package provides Typescript and TSX grammars for the Tree-sitter
 library.")
       (license license:expat))))
+
+;; Local package definition solely for building the native emacs module
+;; written in Rust.
+(define tree-sitter-emacs-module
+  (package
+    (name "tree-sitter-emacs-module")
+    (version "0.17.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0bl7709r4mhb5nmfcsiqj09ja8wn53x9nf5jrr5lq3n1lwz7qq66"))))
+    (build-system cargo-build-system)
+    (inputs
+     (list tree-sitter))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-emacs" ,rust-emacs-0.18)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-tree-sitter" ,rust-tree-sitter-for-emacs))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'chdir 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (add-after 'delete-cargo.lock 'do-not-fetch-from-github
+           (lambda _
+             (substitute* "Cargo.toml"
+               (("\\[patch.*") "")
+               (("git = .*") ""))))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out") "/lib")))
+               (install-file "target/release/libtsc_dyn.so" lib)))))))
+    (home-page #f)
+    (synopsis #f)
+    (description #f)
+    (license license:expat)))
+
+(define-public emacs-tree-sitter-core
+  (package
+    (name "emacs-tree-sitter-core")
+    (version (package-version tree-sitter-emacs-module))
+    (source (package-source tree-sitter-emacs-module))
+    (build-system emacs-build-system)
+    (native-inputs
+     (list tree-sitter-emacs-module))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'install 'install-module
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out")))
+                   (module (string-append (assoc-ref inputs "tree-sitter-emacs-module")
+                                          "/lib/libtsc_dyn.so")))
+               ;; Writing "LOCAL" in this file prevents the package from
+               ;; trying to download the module from the internet.
+               (call-with-output-file (string-append elpa "/DYN-VERSION")
+                 (lambda (port) (display "LOCAL" port)))
+               (substitute* "tsc-dyn-get.el"
+                 (("defcustom tsc-dyn-dir tsc--dir")
+                  (string-append "defcustom tsc-dyn-dir \"" elpa "\"")))
+               (copy-file module (string-append elpa "/tsc-dyn.so"))))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp, core library")
+    (description "This package provides core APIs of the Emacs binding for
+Tree-sitter, an incremental parsing system.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 25/26] gnu: Add emacs-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 893c02824d..b412f3d031 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -698,3 +698,30 @@ (module (string-append (assoc-ref inputs "tree-sitter-emacs-module")
     (description "This package provides core APIs of the Emacs binding for
 Tree-sitter, an incremental parsing system.")
     (license license:expat)))
+
+(define-public emacs-tree-sitter
+  (package
+    (name "emacs-tree-sitter")
+    (version (package-version emacs-tree-sitter-core))
+    (source (package-source emacs-tree-sitter-core))
+    (build-system emacs-build-system)
+    (propagated-inputs
+     (list emacs-tree-sitter-core))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "lisp"))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp")
+    (description "This package provides Emacs bindings for Tree-sitter, an
+incremental parsing library.  It aims to be the foundation for a new breed of
+Emacs packages that understand code structurally.  For example:
+
+@enumerate
+@item Faster, fine-grained code highlighting.
+@item More flexible code folding.
+@item Structural editing (like Paredit, or even better) for non-Lisp code.
+@item More informative indexing for imenu.
+@end enumerate")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 26/26] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter-langs): New variable.
---
 gnu/packages/tree-sitter.scm | 81 ++++++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index b412f3d031..00b91f5c91 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -725,3 +725,84 @@ (define-public emacs-tree-sitter
 @item More informative indexing for imenu.
 @end enumerate")
     (license license:expat)))
+
+(define-public emacs-tree-sitter-langs
+  (package
+    (name "emacs-tree-sitter-langs")
+    (version "0.11.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/tree-sitter-langs")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0r3yw25czg1vcxk9015yd8p1n0z427z2l4hizfjy8w8jjjg7cxxf"))))
+    (build-system emacs-build-system)
+    ;; FIXME: Support for some languages, such as ruby, is still missing from
+    ;; the bundle.
+    (inputs
+     (list tree-sitter-bash
+           tree-sitter-c
+           tree-sitter-cpp
+           tree-sitter-css
+           tree-sitter-go
+           tree-sitter-html
+           tree-sitter-java
+           tree-sitter-javascript
+           tree-sitter-json
+           tree-sitter-julia
+           tree-sitter-php
+           tree-sitter-python
+           tree-sitter-rust
+           tree-sitter-typescript))
+    (propagated-inputs
+     (list emacs-tree-sitter))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-before 'build 'fake-bundle
+           (lambda _
+             ;; The BUNDLE-VERSION file prevents the package from
+             ;; downloading libraries at load time.
+             (call-with-output-file "bin/BUNDLE-VERSION"
+               (lambda (port) (display ,version port)))))
+         (add-after 'install 'install-libs
+           (lambda* (#:key outputs inputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+               (mkdir-p (string-append elpa "/bin"))
+               ;; The BUNDLE-VERSION file prevents the package from
+               ;; downloading libraries at load time.
+               (call-with-output-file (string-append elpa "/bin/BUNDLE-VERSION")
+                 (lambda (port) (display ,version port)))
+
+               (define (install-lib lang)
+                 (let ((input (assoc-ref inputs (string-append "tree-sitter-" lang))))
+                   (symlink (string-append input "/lib/tree-sitter/" lang ".so")
+                            (string-append elpa "/bin/" lang ".so"))))
+               (for-each (lambda (input)
+                           (use-modules (ice-9 regex))
+                           (cond
+                            ((string-match "^tree-sitter-(.*)$" (car input))
+                             => (lambda (m)
+                                  (install-lib (match:substring m 1))))))
+                         inputs))))
+         (add-after 'install 'install-queries
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+                (copy-recursively "queries" (string-append elpa "/queries"))))))))
+    (home-page "https://ubolonton.github.io/emacs-tree-sitter/languages/")
+    (synopsis "Language support bundle for Tree-sitter")
+    (description "This package is a convenient language bundle for
+Tree-sitter.  For each supported language, this package provides:
+
+@enumerate
+@item Pre-compiled grammar binaries.
+@item An optional highlights.scm file that provides highlighting patterns.
+This is mainly intended for major modes that are not aware of tree-sitter.
+@item Optional query patterns for other minor modes that provide high-level
+functionalities on top of tree-sitter, such as code folding, evil text
+objects, ...etc.
+@end enumerate")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 00/31] gnu: Add tree-sitter for emacs.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 4449 bytes --]

Hi Guix!

Here's a v4 that's in a pretty good shape IMO. Here's a summary of the
changes compared to previous versions, there were lots actually! :-)

- Removed bundled icu4c headers from `tree-sitter', as pointed out on
  https://issues.guix.gnu.org/53523.

- Added a snippet that removes all files generated by the
  `tree-sitter generate' command, to be used in all grammar packages.

- For each grammar package, provide two outputs:

  * "out": With just the compiled grammar.

--8<---------------cut here---------------start------------->8---
/gnu/store/mb2hcaanm6ld8rx19p1ihk2fs1m7lcj8-tree-sitter-c-0.20.1
├── lib
│   └── tree-sitter
│       └── c.so
└── share
    └── doc
        └── tree-sitter-c-0.20.1
            └── LICENSE
--8<---------------cut here---------------end--------------->8---

  * "js": With the JS package output, so that it may be used as a
    dependency. For example C++ imports the C grammar, and Typescript
    imports JavaScript.

- Removed the dependency on node-gyp!

  By default, the `npm' build system builds JavaScript bindings for each
  grammar, so that node applications may use them as a library to parse
  a given language. We don't need this yet in Guix, we just want to
  generate C code from the JS grammar description and compile it.

  So when invoking `tree-sitter generate', we disable bindings
  generation and then we ignore all custom scripts when installing the
  JS package. This simplifies grammar packages a *lot*, given those
  custom script often try to download binaries and would have had to be
  patched.

- Grammars are compiled manually.

  Previous versions relied on doing `tree-sitter test' so that a grammar
  would be loaded and compiled by `tree-sitter' into its own cache. We
  would then install the library from that cache location, which can
  change across versions.

  This felt like a hack, looking at the source code, we're just
  compiling a couple of files at most, we can do that ourselve.

- I added support for a few extra languages, like ruby, C#, elixir and
  ocaml.

- Last but very much not least, tests for emacs-tree-sitter-langs are
  enabled!

  This is important because this package doesn't just bundle all the
  grammars, it also provides custom highlighting rules, which depend on
  the grammar and can easily be out-of-sync and break. Thanks to
  enabling these tests, I realized the php grammar need updating and
  didn't work for example.

Let me know if you have any feedback on this series! It's taking me
quite some time to follow it through, it's not the easiest thing to
package :-/.

I do wonder if I'm going about it the right way for Guix though, for
instance I'm wondering if I should rewrite the grammar packages using a
new custom build system, do people think it would be worth it? That
could be done as a follow-up of course.

Thanks,
Pierre

Pierre Langlois (31):
  gnu: Add rust-smallbitvec.
  gnu: Add rust-html-escape.
  gnu: Add rust-spin@0.7.
  gnu: Add tree-sitter.
  gnu: Add tree-sitter-cli.
  gnu: Add tree-sitter-c.
  gnu: Add tree-sitter-cpp.
  gnu: Add tree-sitter-bash.
  gnu: Add tree-sitter-css.
  gnu: Add tree-sitter-c-sharp.
  gnu: Add tree-sitter-elixir.
  gnu: Add tree-sitter-elm.
  gnu: Add tree-sitter-go.
  gnu: Add tree-sitter-html.
  gnu: Add tree-sitter-java.
  gnu: Add tree-sitter-javascript.
  gnu: Add tree-sitter-json.
  gnu: Add tree-sitter-julia.
  gnu: Add tree-sitter-ocaml.
  gnu: Add tree-sitter-php.
  gnu: Add tree-sitter-python.
  gnu: Add tree-sitter-ruby.
  gnu: Add tree-sitter-rust.
  gnu: Add tree-sitter-typescript.
  gnu: Add rust-tree-sitter-for-emacs.
  gnu: rust-emacs-module: Update to 0.18.
  gnu: rust-emacs-macros: Update to 0.17.
  gnu: rust-emacs: Update to 0.18.
  gnu: Add emacs-tree-sitter-core.
  gnu: Add emacs-tree-sitter.
  gnu: Add emacs-tree-sitter-langs.

 gnu/local.mk                                  |    2 +
 gnu/packages/crates-io.scm                    |  154 ++-
 .../rust-tree-sitter-text-provider-fix.patch  |   98 ++
 gnu/packages/tree-sitter.scm                  | 1088 +++++++++++++++++
 4 files changed, 1324 insertions(+), 18 deletions(-)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
 create mode 100644 gnu/packages/tree-sitter.scm

-- 
2.34.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v4 01/31] gnu: Add rust-smallbitvec.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-smallbitvec-2): New variable.
---
 gnu/packages/crates-io.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 6024c9c57a..291c289b5a 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -56109,6 +56109,28 @@ (define-public rust-sluice-0.4
         ("rust-futures-core-preview" ,rust-futures-core-preview-0.3)
         ("rust-futures-io-preview" ,rust-futures-io-preview-0.3))))))

+(define-public rust-smallbitvec-2
+  (package
+    (name "rust-smallbitvec")
+    (version "2.5.1")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "smallbitvec" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "0plrbldsjpwip3afbzd8fgrnvdhizcg5z4ncfqs4q6x4qjflzkkm"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-development-inputs
+       (("rust-bit-vec" ,rust-bit-vec-0.4)
+        ("rust-rand" ,rust-rand-0.4))))
+    (home-page "https://github.com/servo/smallbitvec")
+    (synopsis "A bit vector optimized for size and inline storage")
+    (description "This package provides a bit vector optimized for size and
+inline storage")
+    (license (list license:expat license:asl2.0))))
+
 (define-public rust-smallvec-1
   (package
     (name "rust-smallvec")
--
2.34.0

[bug#49946] [PATCH v4 02/31] gnu: Add rust-html-escape.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-html-escape-0.2): New variable.
---
 gnu/packages/crates-io.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 291c289b5a..986901740a 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -26943,6 +26943,30 @@ (define-public rust-htmlescape-0.3
      "This package provides a library for HTML entity encoding and decoding.")
     (license (list license:asl2.0 license:expat license:mpl2.0))))

+(define-public rust-html-escape-0.2
+  (package
+    (name "rust-html-escape")
+    (version "0.2.9")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "html-escape" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "1dxw9lpckrqzzqgbkw64ckbajr4b7xxnjdn8adgzqf2mm40shvl1"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-utf8-width" ,rust-utf8-width-0.1))
+       #:cargo-development-inputs
+       (("rust-bencher" ,rust-bencher-0.1))))
+    (home-page "https://magiclen.org/html-escape")
+    (synopsis "Library for encoding and escaping special characters in HTML")
+    (description
+     "This package provides a library for encoding and escaping special
+characters in HTML, decoding and unescaping HTML entities as well.")
+    (license license:expat)))
+
 (define-public rust-http-0.2
   (package
     (name "rust-http")
--
2.34.0

[bug#49946] [PATCH v4 03/31] gnu: Add rust-spin@0.7.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-spin-0.7): New variable.
---
 gnu/packages/crates-io.scm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 986901740a..a4bd79607c 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -56726,6 +56726,19 @@ (define-public rust-spin-0.9
 initializers are available.")
     (license license:expat)))

+(define-public rust-spin-0.7
+  (package
+    (inherit rust-spin-0.9)
+    (name "rust-spin")
+    (version "0.7.1")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "spin" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "0qjips9f6fsvkyd7wj3a4gzaqknn2q4kkb19957pl86im56pna0k"))))))
+
 (define-public rust-spin-0.5
   (package
     (inherit rust-spin-0.9)
--
2.34.0

[bug#49946] [PATCH v4 04/31] gnu: Add tree-sitter.

[Pierre Langlois]

* gnu/package/tree-sitter.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Register it.
---
 gnu/local.mk                 |  1 +
 gnu/packages/tree-sitter.scm | 83 ++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)
 create mode 100644 gnu/packages/tree-sitter.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index 433467ee39..97ab160c3c 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -584,6 +584,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/tmux.scm				\
   %D%/packages/toolkits.scm			\
   %D%/packages/tor.scm				\
+  %D%/packages/tree-sitter.scm			\
   %D%/packages/tv.scm				\
   %D%/packages/uglifyjs.scm			\
   %D%/packages/uml.scm				\
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
new file mode 100644
index 0000000000..10a3a6e57c
--- /dev/null
+++ b/gnu/packages/tree-sitter.scm
@@ -0,0 +1,83 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages tree-sitter)
+  #:use-module (guix gexp)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix git-download)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages icu4c))
+
+(define-public tree-sitter
+  (package
+    (name "tree-sitter")
+    (version "0.20.4")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1yldgdbf3l5l4ki52abdf81nwkcbvg219gwr3ydcjwfsg7hf7zhz"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  (delete-file-recursively "lib/src/unicode")))))
+    (inputs
+     (list icu4c))
+    (build-system gnu-build-system)
+    (arguments
+     (list
+      #:tests? #f ;No check target.
+      #:phases
+      #~(modify-phases %standard-phases
+          (delete 'configure)
+          (add-before 'build 'set-cc
+            (lambda _
+              (setenv "CC"
+                      #$(cc-for-target))))
+          (replace 'install
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out")) (lib (string-append
+                                                           out "/lib")))
+                (setenv "PREFIX" out)
+                (invoke "make" "install")))))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Incremental parsing system for programming tools")
+    (description
+     "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @code{libtree-sitter} runtime library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 05/31] gnu: Add tree-sitter-cli.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cli): New variable.
---
 gnu/packages/tree-sitter.scm | 66 ++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 10a3a6e57c..cbf8ecdc6c 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -19,10 +19,13 @@
 (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages crates-graphics)
+  #:use-module (gnu packages crates-io)
   #:use-module (gnu packages icu4c))

 (define-public tree-sitter
@@ -81,3 +84,66 @@ (define-public tree-sitter

 This package includes the @code{libtree-sitter} runtime library.")
     (license license:expat)))
+
+(define-public tree-sitter-cli
+  (package (inherit tree-sitter)
+    (name "tree-sitter-cli")
+    (build-system cargo-build-system)
+    (arguments
+     `(;; Running test requires downloading fixtures, see the
+       ;; script/fetch-fixtures script.
+       #:tests? #f
+       ;; FIXME: Installing the sources for the tree-sitter Rust bindings
+       ;; doesn't work out of the box due to tree-sitter having multiple
+       ;; Rust packages in the same repository (bindings and CLI).
+       #:install-source? #f
+       #:cargo-inputs
+       (("rust-ansi-term" ,rust-ansi-term-0.12)
+        ("rust-anyhow" ,rust-anyhow-1)
+        ("rust-atty" ,rust-atty-0.2)
+        ("rust-clap" ,rust-clap-2)
+        ("rust-difference" ,rust-difference-2)
+        ("rust-dirs" ,rust-dirs-3)
+        ("rust-html-escape" ,rust-html-escape-0.2)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-rustc-hash" ,rust-rustc-hash-1)
+        ("rust-serde" ,rust-serde-1)
+        ("rust-smallbitvec" ,rust-smallbitvec-2)
+        ("rust-spin" ,rust-spin-0.7)
+        ("rust-thiserror" ,rust-thiserror-1)
+        ("rust-tiny-http" ,rust-tiny-http-0.8)
+        ("rust-toml" ,rust-toml-0.5)
+        ("rust-walkdir" ,rust-walkdir-2)
+        ("rust-webbrowser" ,rust-webbrowser-0.5)
+        ("rust-which" ,rust-which-4))
+       #:cargo-development-inputs
+       (("rust-pretty-assertions" ,rust-pretty-assertions-0.7))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin"))
+                    (lib (string-append out "/lib")))
+               (mkdir-p bin)
+               (install-file "target/release/tree-sitter" bin)))))))
+    (synopsis "Incremental parsing system for programming tools")
+    (description "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @command{tree-sitter} command-line tool.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 07/31] gnu: Add tree-sitter-cpp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cpp): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 920c386fdf..e82c6e60a9 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -243,3 +243,27 @@ (define (source-file path)
     (description
       "This package provides a C grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-cpp
+  (package
+    (inherit tree-sitter-c)
+    (name "tree-sitter-cpp")
+    (version "0.20.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-cpp")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0hxcpdvyyig8njga1mxp4qcnbbnr1d0aiy27vahijwbh98b081nr"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (inputs
+     (list `(,tree-sitter-c "js")))
+    (home-page "https://github.com/tree-sitter/tree-sitter-cpp")
+    (synopsis "Tree-sitter C++ grammar")
+    (description
+     "This package provides a C++ grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 08/31] gnu: Add tree-sitter-bash.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-bash): New variable.
---
 gnu/packages/tree-sitter.scm | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e82c6e60a9..8f12559dde 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -267,3 +267,35 @@ (define-public tree-sitter-cpp
     (description
      "This package provides a C++ grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-bash
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-bash")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-bash")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "18c030bb65r50i6z37iy7jb9z9i8i36y7b08dbc9bchdifqsijs5"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+          (replace 'delete-dependencies
+           (lambda _
+             (delete-dependencies '("tree-sitter-cli"
+                                    "nan"
+                                    "node-gyp"
+                                    "prebuild"
+                                    "prebuild-install"))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-bash")
+    (synopsis "Tree-sitter Bash grammar")
+    (description
+      "This package provides a Bash grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 09/31] gnu: Add tree-sitter-css.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-css): New variable.
---
 gnu/packages/tree-sitter.scm | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 8f12559dde..f26732197e 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -299,3 +299,24 @@ (define-public tree-sitter-bash
     (description
       "This package provides a Bash grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-css
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-css")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-css")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "014jrlgi7zfza9g38hsr4vlbi8964i5p7iglaih6qmzaiml7bja2"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-css")
+    (synopsis "Tree-sitter CSS grammar")
+    (description
+      "This package provides a CSS grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 10/31] gnu: Add tree-sitter-c-sharp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-c-sharp): New variable.
---
 gnu/packages/tree-sitter.scm | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index f26732197e..5bb7e7549a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -268,6 +268,27 @@ (define-public tree-sitter-cpp
      "This package provides a C++ grammar for the Tree-sitter library.")
     (license license:expat)))

+(define-public tree-sitter-c-sharp
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-c-sharp")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-c-sharp")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "054fmpf47cwh59gbg00sc0nl237ba4rnxi73miz39yqzcs87055r"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-c-sharp")
+    (synopsis "Tree-sitter C# grammar")
+    (description
+      "This package provides a C# grammar for the Tree-sitter library.")
+    (license license:expat)))
+
 (define-public tree-sitter-bash
   (package (inherit tree-sitter-c)
     (name "tree-sitter-bash")
--
2.34.0

[bug#49946] [PATCH v4 11/31] gnu: Add tree-sitter-elixir.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-elixir): New variable.
---
 gnu/packages/tree-sitter.scm | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 5bb7e7549a..134d2768cf 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -341,3 +341,38 @@ (define-public tree-sitter-css
     (description
       "This package provides a CSS grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-elixir
+  (let ((commit "de20391afe5cb03ef1e8a8e43167e7b58cc52869")
+        (revision "1"))
+    (package (inherit tree-sitter-c)
+      (name "tree-sitter-elixir")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/elixir-lang/tree-sitter-elixir")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0zrkrwhw3g1vazsxcwrfd1fk4wvs9hdwmwp6073mfh370bz4140h"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+      (arguments
+       (substitute-keyword-arguments (package-arguments tree-sitter-c)
+         ((#:phases phases)
+          `(modify-phases ,phases
+             (replace 'delete-dependencies
+               (lambda _
+                 (delete-dependencies '("tree-sitter-cli"
+                                        "node-gyp"
+                                        "nan"
+                                        "prettier"))))))))
+      (home-page "https://elixir-lang.org/tree-sitter-elixir/")
+      (synopsis "Tree-sitter Elixir grammar")
+      (description
+       "This package provides a Elixir grammar for the Tree-sitter library.")
+      (license (list license:asl2.0
+                     ;; Files in tests/corpus are under MIT license.
+                     license:expat)))))
--
2.34.0

[bug#49946] [PATCH v4 12/31] gnu: Add tree-sitter-elm.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-elm): New variable.
---
 gnu/packages/tree-sitter.scm | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 134d2768cf..5ade030283 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -376,3 +376,36 @@ (define-public tree-sitter-elixir
       (license (list license:asl2.0
                      ;; Files in tests/corpus are under MIT license.
                      license:expat)))))
+
+(define-public tree-sitter-elm
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-elm")
+    (version "5.5.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/elm-tooling/tree-sitter-elm")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "10hbi4vyj4hjixqswdcbvzl60prldczz29mlp02if61wvwiwvqrw"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+          (replace 'delete-dependencies
+           (lambda _
+             (delete-dependencies '("tree-sitter-cli"
+                                    "nan"
+                                    "node-gyp"
+                                    "@asgerf/dts-tree-sitter"
+                                    "prebuild"
+                                    "prebuild-install"))))))))
+    (home-page "https://elm-tooling.github.io/tree-sitter-elm/")
+    (synopsis "Tree-sitter Elm grammar")
+    (description
+      "This package provides an Elm grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 13/31] gnu: Add tree-sitter-go.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-go): New variable.
---
 gnu/packages/tree-sitter.scm | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 5ade030283..c1e58d9c8e 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -409,3 +409,24 @@ (define-public tree-sitter-elm
     (description
       "This package provides an Elm grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-go
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-go")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-go")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0nxs47vd2fc2fr0qlxq496y852rwg39flhg334s7dlyq7d3lcx4x"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-go")
+    (synopsis "Tree-sitter Go grammar")
+    (description
+      "This package provides a Golang grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 14/31] gnu: Add tree-sitter-html.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-html): New variable.
---
 gnu/packages/tree-sitter.scm | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c1e58d9c8e..4bb9425eb1 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -430,3 +430,24 @@ (define-public tree-sitter-go
     (description
       "This package provides a Golang grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-html
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-html")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-html")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-html")
+    (synopsis "Tree-sitter HTML grammar")
+    (description
+      "This package provides a HTML grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 15/31] gnu: Add tree-sitter-java.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-java): New variable.
---
 gnu/packages/tree-sitter.scm | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 4bb9425eb1..d93f6a584a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -451,3 +451,24 @@ (define-public tree-sitter-html
     (description
       "This package provides a HTML grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-java
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-java")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-java")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "07zw9ygb45hnvlx9qlz7rlz8hc3byjy03d24v72i5iyhpiiwlhvl"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-java")
+    (synopsis "Tree-sitter Java grammar")
+    (description
+      "This package provides a Java grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 16/31] gnu: Add tree-sitter-javascript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-javascript): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d93f6a584a..976e4cf9ae 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -472,3 +472,25 @@ (define-public tree-sitter-java
     (description
       "This package provides a Java grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-javascript
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-javascript")
+    (version "0.20.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-javascript")
+                    (commit (string-append "rust-" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "175yrk382n2di0c2xn4gpv8y4n83x1lg4hqn04vabf0yqynlkq67"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
+    (synopsis "Tree-sitter Javascript grammar")
+    (description
+      "This package provides Javascript and JSX grammars for the Tree-sitter
+library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 17/31] gnu: Add tree-sitter-json.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-json): New variable.
---
 gnu/packages/tree-sitter.scm | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 976e4cf9ae..ea5a5a5156 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -494,3 +494,24 @@ (define-public tree-sitter-javascript
       "This package provides Javascript and JSX grammars for the Tree-sitter
 library.")
     (license license:expat)))
+
+(define-public tree-sitter-json
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-json")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-json")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "06pjh31bv9ja9hlnykk257a6zh8bsxg2fqa54al7qk1r4n9ksnff"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-json")
+    (synopsis "Tree-sitter JSON grammar")
+    (description
+      "This package provides a JSON grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 18/31] gnu: Add tree-sitter-julia.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-julia): New variable.
---
 gnu/packages/tree-sitter.scm | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index ea5a5a5156..081ec8caa0 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -515,3 +515,24 @@ (define-public tree-sitter-json
     (description
       "This package provides a JSON grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-julia
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-julia")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-julia")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1pbnmvhy2gq4vg1b0sjzmjm4s2gsgdjh7h01yj8qrrqbcl29c463"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-julia")
+    (synopsis "Tree-sitter Julia grammar")
+    (description
+      "This package provides a Julia grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 20/31] gnu: Add tree-sitter-php.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-php): New variable.
---
 gnu/packages/tree-sitter.scm | 38 ++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 904697154d..e64e248215 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -613,3 +613,41 @@ (define-public tree-sitter-ocaml
       (description
         "This package provides a OCaml grammar for the Tree-sitter library.")
       (license license:expat))))
+
+(define-public tree-sitter-php
+  (let ((commit "435fa00006c0d1515c37fbb4dd6a9de284af75ab")
+        (revision "1"))
+    (package (inherit tree-sitter-c)
+      (name "tree-sitter-php")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-php")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "05k4h58gi616gv41r0qqdb1x4rs8y94vghn2r10yczisgzq4vbad"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (delete-file "src/grammar.json")
+                    (delete-file "src/node-types.json")
+                    (delete-file "src/parser.c")
+                    (delete-file-recursively "src/tree_sitter")
+                    #t))))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+          (replace 'delete-dependencies
+           (lambda _
+             (delete-dependencies '("tree-sitter-cli"
+                                    "nan"
+                                    "shelljs"))))))))
+      (home-page "https://github.com/tree-sitter/tree-sitter-php")
+      (synopsis "Tree-sitter PHP grammar")
+      (description
+        "This package provides a PHP grammar for the Tree-sitter library.")
+      (license license:expat))))
--
2.34.0

[bug#49946] [PATCH v4 21/31] gnu: Add tree-sitter-python.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-python): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e64e248215..cecce9af50 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -651,3 +651,26 @@ (define-public tree-sitter-php
       (description
         "This package provides a PHP grammar for the Tree-sitter library.")
       (license license:expat))))
+
+(define-public tree-sitter-python
+  (let ((commit "ed0fe62e55dc617ed9dec8817ebf771aa7cf3c42")
+        (revision "1"))
+    (package (inherit tree-sitter-c)
+      (name "tree-sitter-python")
+      (version (git-version "0.19.1" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-python")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0wrfpg84mc3pzcrdi6n5fqwijkqr1nj5sqfnayb502krvqpjilal"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+      (home-page "https://github.com/tree-sitter/tree-sitter-python")
+      (synopsis "Tree-sitter Python grammar")
+      (description
+       "This package provides a Python grammar for the Tree-sitter library.")
+      (license license:expat))))
--
2.34.0

[bug#49946] [PATCH v4 22/31] gnu: Add tree-sitter-ruby.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-ruby): New variable.
---
 gnu/packages/tree-sitter.scm | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index cecce9af50..6fc9a0c91a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -674,3 +674,35 @@ (define-public tree-sitter-python
       (description
        "This package provides a Python grammar for the Tree-sitter library.")
       (license license:expat))))
+
+(define-public tree-sitter-ruby
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-ruby")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-ruby")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0m3h4928rbs300wcb6776h9r88hi32rybbhcaf6rdympl5nzi83v"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments tree-sitter-c)
+      ((#:phases phases)
+       `(modify-phases ,phases
+          (replace 'delete-dependencies
+           (lambda _
+             (delete-dependencies '("tree-sitter-cli"
+                                    "nan"
+                                    "node-gyp"
+                                    "prebuild"
+                                    "prebuild-install"))))))))
+    (home-page "https://github.com/tree-sitter/tree-sitter-ruby")
+    (synopsis "Tree-sitter Ruby grammar")
+    (description
+      "This package provides a Ruby grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 23/31] gnu: Add tree-sitter-rust.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-rust): New variable.
---
 gnu/packages/tree-sitter.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 6fc9a0c91a..f4df5c5335 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -25,6 +25,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages algebra)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages icu4c))
@@ -706,3 +707,27 @@ (define-public tree-sitter-ruby
     (description
       "This package provides a Ruby grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-rust
+  (package (inherit tree-sitter-c)
+    (name "tree-sitter-rust")
+    (version "0.20.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-rust")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "174j5pxwf80c4xniri39l3a6bb7nq96g2s8hh5sgv4i7xvbpfsmg"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (native-inputs
+     (modify-inputs (package-native-inputs tree-sitter-c)
+       (prepend bc)))
+    (home-page "https://github.com/tree-sitter/tree-sitter-rust")
+    (synopsis "Tree-sitter Rust grammar")
+    (description
+      "This package provides a Rust grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 24/31] gnu: Add tree-sitter-typescript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-typescript): New variable.
---
 gnu/packages/tree-sitter.scm | 69 ++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index f4df5c5335..309b38386d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -731,3 +731,72 @@ (define-public tree-sitter-rust
     (description
       "This package provides a Rust grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-typescript
+  (let ((commit "111b07762e86efab9a918b7c721f720c37e76b0a")
+        (revision "1"))
+    (package (inherit tree-sitter-c)
+      (name "tree-sitter-typescript")
+      (version (git-version "0.20.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-typescript")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32 "1364bm3wbqqzvp03cvpx1w89bbqynb1v62i46gy7f6rhib9bf63a"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (delete-file "binding.gyp")
+                    (delete-file-recursively "bindings")
+                    (for-each
+                     (lambda (lang)
+                       (with-directory-excursion lang
+                         (delete-file "src/grammar.json")
+                         (delete-file "src/node-types.json")
+                         (delete-file "src/parser.c")
+                         (delete-file-recursively "src/tree_sitter")))
+                     '("typescript" "tsx"))
+                    #t))))
+      (inputs
+       (list `(,tree-sitter-javascript "js")))
+      (arguments
+       (substitute-keyword-arguments (package-arguments tree-sitter-c)
+        ((#:phases phases)
+         #~(modify-phases #$phases
+             (replace 'build
+               (lambda _
+                 (for-each (lambda (dir)
+                             (with-directory-excursion dir
+                               (invoke "tree-sitter" "generate" "--no-bindings")))
+                           '("typescript" "tsx"))))
+             (replace 'check
+               (lambda _
+                 (for-each (lambda (dir)
+                             (with-directory-excursion dir
+                               (invoke "tree-sitter" "test")))
+                           '("typescript" "tsx"))))
+             (replace 'install
+               (lambda _
+                 (let ((lib (string-append #$output "/lib/tree-sitter/")))
+                   (mkdir-p lib)
+                   (for-each
+                    (lambda (lang)
+                      (invoke #$(cxx-for-target)
+                              "-shared"
+                              "-fPIC"
+                              "-fno-exceptions"
+                              "-O2"
+                              "-g"
+                              "-o" (string-append lib "/" lang ".so")
+                              "-xc" (string-append lang "/src/scanner.c")
+                              "-xc" (string-append lang "/src/parser.c")))
+                    '("typescript" "tsx")))))))))
+      (home-page "https://github.com/tree-sitter/tree-sitter-typescript")
+      (synopsis "Tree-sitter Typescript grammar")
+      (description
+        "This package provides Typescript and TSX grammars for the Tree-sitter
+library.")
+      (license license:expat))))
--
2.34.0

[bug#49946] [PATCH v4 25/31] gnu: Add rust-tree-sitter-for-emacs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter-for-emacs): New variable.
* gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
---
 gnu/local.mk                                  |  1 +
 .../rust-tree-sitter-text-provider-fix.patch  | 98 +++++++++++++++++++
 gnu/packages/tree-sitter.scm                  | 45 +++++++++
 3 files changed, 144 insertions(+)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 97ab160c3c..0c113956d3 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1775,6 +1775,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/rust-nettle-sys-disable-vendor.patch	 \
   %D%/packages/patches/rust-openssl-sys-no-vendor.patch	\
   %D%/packages/patches/rust-shell2batch-lint-fix.patch		\
+  %D%/packages/patches/rust-tree-sitter-text-provider-fix.patch	\
   %D%/packages/patches/rust-wl-clipboard-rs-newer-wl.patch      \
   %D%/packages/patches/sbc-fix-build-non-x86.patch		\
   %D%/packages/patches/sbcl-aserve-add-HTML-5-elements.patch	\
diff --git a/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
new file mode 100644
index 0000000000..beda2d8391
--- /dev/null
+++ b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
@@ -0,0 +1,98 @@
+From 475b822f47bdc58d832533448b6f6d9818554f37 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Tu=E1=BA=A5n-Anh=20Nguy=E1=BB=85n?= <ubolonton@gmail.com>
+Date: Sun, 25 Jul 2021 13:11:52 +0700
+Subject: [PATCH] Allow TextProvider's iterators to generate owned text
+
+---
+ binding_rust/lib.rs | 33 ++++++++++++++++++---------------
+ 1 file changed, 18 insertions(+), 15 deletions(-)
+
+diff --git a/lib/binding_rust/lib.rs b/lib/binding_rust/lib.rs
+index 068a60285..79c3991ed 100644
+--- a/binding_rust/lib.rs
++++ b/binding_rust/lib.rs
+@@ -8,6 +8,7 @@ pub mod allocations;
+ use std::os::unix::io::AsRawFd;
+
+ use std::{
++    borrow::Cow,
+     char, error,
+     ffi::CStr,
+     fmt, hash, iter,
+@@ -162,7 +163,8 @@ pub struct QueryCaptures<'a, 'tree: 'a, T: TextProvider<'a>> {
+ }
+
+ pub trait TextProvider<'a> {
+-    type I: Iterator<Item = &'a [u8]> + 'a;
++    type I: Iterator<Item = Cow<'a, [u8]>>;
++
+     fn text(&mut self, node: Node) -> Self::I;
+ }
+
+@@ -1799,19 +1801,19 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+         buffer2: &mut Vec<u8>,
+         text_provider: &mut impl TextProvider<'a>,
+     ) -> bool {
+-        fn get_text<'a, 'b: 'a, I: Iterator<Item = &'b [u8]>>(
++        fn get_text<'a, 'b: 'a, I: Iterator<Item = Cow<'b, [u8]>>>(
+             buffer: &'a mut Vec<u8>,
+             mut chunks: I,
+-        ) -> &'a [u8] {
+-            let first_chunk = chunks.next().unwrap_or(&[]);
++        ) -> Cow<'a, [u8]> {
++            let first_chunk = chunks.next().unwrap_or(Cow::Owned(vec![0u8; 0]));
+             if let Some(next_chunk) = chunks.next() {
+                 buffer.clear();
+-                buffer.extend_from_slice(first_chunk);
+-                buffer.extend_from_slice(next_chunk);
++                buffer.extend_from_slice(&first_chunk);
++                buffer.extend_from_slice(&next_chunk);
+                 for chunk in chunks {
+-                    buffer.extend_from_slice(chunk);
++                    buffer.extend_from_slice(&chunk);
+                 }
+-                buffer.as_slice()
++                Cow::Borrowed(buffer.as_slice())
+             } else {
+                 first_chunk
+             }
+@@ -1835,7 +1837,7 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+                 TextPredicate::CaptureMatchString(i, r, is_positive) => {
+                     let node = self.nodes_for_capture_index(*i).next().unwrap();
+                     let text = get_text(buffer1, text_provider.text(node));
+-                    r.is_match(text) == *is_positive
++                    r.is_match(&text) == *is_positive
+                 }
+             })
+     }
+@@ -1946,23 +1948,24 @@ impl<'cursor, 'tree> fmt::Debug for QueryMatch<'cursor, 'tree> {
+     }
+ }
+
+-impl<'a, F, I> TextProvider<'a> for F
++impl<'a, F, I, T> TextProvider<'a> for F
+ where
+     F: FnMut(Node) -> I,
+-    I: Iterator<Item = &'a [u8]> + 'a,
++    T: Into<Cow<'a, [u8]>>,
++    I: Iterator<Item = T>,
+ {
+-    type I = I;
++    type I = iter::Map<I, fn(T) -> Cow<'a, [u8]>>;
+
+     fn text(&mut self, node: Node) -> Self::I {
+-        (self)(node)
++        (self)(node).map(T::into)
+     }
+ }
+
+ impl<'a> TextProvider<'a> for &'a [u8] {
+-    type I = iter::Once<&'a [u8]>;
++    type I = iter::Once<Cow<'a, [u8]>>;
+
+     fn text(&mut self, node: Node) -> Self::I {
+-        iter::once(&self[node.byte_range()])
++        iter::once(Cow::Borrowed(&self[node.byte_range()]))
+     }
+ }
+
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 309b38386d..3c8b4588c4 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -22,9 +22,11 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system node)
+  #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages)
   #:use-module (gnu packages algebra)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
@@ -150,6 +152,49 @@ (define-public tree-sitter-cli
 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))

+;; We need to apply a patch in order to compile the rust bindings against the
+;; emacs tree-sitter module.
+;; See https://github.com/tree-sitter/tree-sitter/pull/1294
+(define-public rust-tree-sitter-for-emacs
+  (package
+    (name "rust-tree-sitter")
+    (version "0.20.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "tree-sitter" version))
+        (file-name (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1yg4p54hsfsxxknjq492i8b4rvibzpl2zdvr2bwvwakqgah05v33"))
+        (patches (search-patches "rust-tree-sitter-text-provider-fix.patch"))
+        (modules '((guix build utils)))
+        (snippet
+         '(begin
+            ;; Force
+            (delete-file-recursively "src")
+            (delete-file "binding_rust/build.rs")
+            (with-output-to-file "binding_rust/build.rs"
+              (lambda _
+                (format #t "fn main() {~@
+                        println!(\"cargo:rustc-link-lib=tree-sitter\");~@
+                        }~%")))
+            #f))))
+    (build-system cargo-build-system)
+    (inputs (list tree-sitter))
+    (arguments
+     `(#:tests? #f  ;; Running tests misinterprets comments as doc-tests.
+       #:cargo-inputs
+       (("rust-cc" ,rust-cc-1)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-regex" ,rust-regex-1)
+        ("rust-spin" ,rust-spin-0.7))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Rust bindings to the Tree-sitter parsing library")
+    (description "This package provides Rust bindings to the Tree-sitter
+parsing library.")
+    (license license:expat)))
+
 (define tree-sitter-delete-generated-files
   '(begin
      (delete-file "binding.gyp")
--
2.34.0

[bug#49946] [PATCH v4 26/31] gnu: rust-emacs-module: Update to 0.18.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-module-0.18): New variable.
(rust-emacs-module-0.10): Inherit from rust-emacs-module-0.18.
---
 gnu/packages/crates-io.scm | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index a4bd79607c..23444f1e7b 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -19100,10 +19100,10 @@ (define-public rust-emacs-macros-0.11
     (description "This package provides proc macros for Emacs modules.")
     (license license:bsd-3)))

-(define-public rust-emacs-module-0.10
+(define-public rust-emacs-module-0.18
   (package
     (name "rust-emacs-module")
-    (version "0.10.0")
+    (version "0.18.0")
     (source
       (origin
         (method url-fetch)
@@ -19112,11 +19112,11 @@ (define-public rust-emacs-module-0.10
          (string-append name "-" version ".tar.gz"))
         (sha256
          (base32
-          "1gf9lz735xbkyir53dyv362drfx3nin5an5cx39kd8q8kjjwix5g"))))
+          "1ypjyyv2ca3vza4sia91ckxamgfk63yd8frkvg3d4ph4fk4pn1mk"))))
     (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
-       (("rust-bindgen" ,rust-bindgen-0.48))))
+       (("rust-bindgen" ,rust-bindgen-0.59))))
     (inputs
      (list clang))
     (home-page "https://github.com/ubolonton/emacs-module-rs")
@@ -19125,6 +19125,23 @@ (define-public rust-emacs-module-0.10
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

+(define-public rust-emacs-module-0.10
+  (package (inherit rust-emacs-module-0.18)
+    (name "rust-emacs-module")
+    (version "0.10.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs_module" version))
+        (file-name
+         (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1gf9lz735xbkyir53dyv362drfx3nin5an5cx39kd8q8kjjwix5g"))))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-bindgen" ,rust-bindgen-0.48))))))
+
 (define-public rust-email-0.0.20
   (package
     (name "rust-email")
--
2.34.0

[bug#49946] [PATCH v4 27/31] gnu: rust-emacs-macros: Update to 0.17.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-macros-0.17): New variable.
(rust-emacs-macros-0.11): Inherit from rust-emacs-macros-0.17.
---
 gnu/packages/crates-io.scm | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 23444f1e7b..154ad963a5 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -19076,8 +19076,31 @@ (define-public rust-emacs-0.11
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

-(define-public rust-emacs-macros-0.11
+(define-public rust-emacs-macros-0.17
   (package
+    (name "rust-emacs-macros")
+    (version "0.17.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs-macros" version))
+        (file-name (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "0qg1dcn5acbirq617qq2fgg9adswif2dnr292s3qnq62wzgnyrb9"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-darling" ,rust-darling-0.10)
+        ("rust-quote" ,rust-quote-1)
+        ("rust-syn" ,rust-syn-1))))
+    (home-page "https://github.com/ubolonton/emacs-module-rs")
+    (synopsis "Proc macros for Emacs modules")
+    (description "This package provides proc macros for Emacs modules.")
+    (license license:bsd-3)))
+
+(define-public rust-emacs-macros-0.11
+  (package (inherit rust-emacs-macros-0.17)
     (name "rust-emacs-macros")
     (version "0.11.0")
     (source
@@ -19089,16 +19112,11 @@ (define-public rust-emacs-macros-0.11
         (sha256
          (base32
           "0390y8vafxdi334hhgrzvcqjq3n5ckcmvilqcfp8vajjq8irrly6"))))
-    (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
        (("rust-darling" ,rust-darling-0.9)
         ("rust-quote" ,rust-quote-0.6)
-        ("rust-syn" ,rust-syn-0.15))))
-    (home-page "https://github.com/ubolonton/emacs-module-rs")
-    (synopsis "Proc macros for Emacs modules")
-    (description "This package provides proc macros for Emacs modules.")
-    (license license:bsd-3)))
+        ("rust-syn" ,rust-syn-0.15))))))

 (define-public rust-emacs-module-0.18
   (package
--
2.34.0

[bug#49946] [PATCH v4 28/31] gnu: rust-emacs: Update to 0.18.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-emacs-0.18): New variable.
(rust-emacs-0.11): Inherit from rust-emacs-0.18.
---
 gnu/packages/crates-io.scm | 38 +++++++++++++++++++++++++++++++-------
 1 file changed, 31 insertions(+), 7 deletions(-)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 154ad963a5..6c0376c01a 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -19045,10 +19045,10 @@ (define-public rust-elf-0.0.10
      "This package provides a pure-Rust library for parsing ELF files.")
     (license (list license:expat license:asl2.0))))

-(define-public rust-emacs-0.11
+(define-public rust-emacs-0.18
   (package
     (name "rust-emacs")
-    (version "0.11.0")
+    (version "0.18.0")
     (source
       (origin
         (method url-fetch)
@@ -19057,16 +19057,18 @@ (define-public rust-emacs-0.11
          (string-append name "-" version ".tar.gz"))
         (sha256
          (base32
-          "1xn6nr8m3p24irlmv3i7c9n95yb10qc6aikwqv99qlmhbky5x0z7"))))
+          "0r860i73b2680i2fhdl2l1wwvvmf2zksncpckgkksdcx310ak5v7"))))
     (build-system cargo-build-system)
     (arguments
      `(#:cargo-inputs
-       (("rust-ctor" ,rust-ctor-0.1)
-        ("rust-emacs-macros" ,rust-emacs-macros-0.11)
-        ("rust-emacs-module" ,rust-emacs-module-0.10)
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-ctor" ,rust-ctor-0.1)
+        ("rust-emacs-macros" ,rust-emacs-macros-0.17)
+        ("rust-emacs-module" ,rust-emacs-module-0.18)
         ("rust-failure" ,rust-failure-0.1)
         ("rust-failure-derive" ,rust-failure-derive-0.1)
-        ("rust-lazy-static" ,rust-lazy-static-1))))
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-thiserror" ,rust-thiserror-1))))
     (inputs
      (list clang))
     (home-page "https://github.com/ubolonton/emacs-module-rs")
@@ -19076,6 +19078,28 @@ (define-public rust-emacs-0.11
 Emacs' support for dynamic modules.")
     (license license:bsd-3)))

+(define-public rust-emacs-0.11
+  (package (inherit rust-emacs-0.18)
+    (name "rust-emacs")
+    (version "0.11.0")
+    (source
+      (origin
+        (method url-fetch)
+        (uri (crate-uri "emacs" version))
+        (file-name
+         (string-append name "-" version ".tar.gz"))
+        (sha256
+         (base32
+          "1xn6nr8m3p24irlmv3i7c9n95yb10qc6aikwqv99qlmhbky5x0z7"))))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-ctor" ,rust-ctor-0.1)
+        ("rust-emacs-macros" ,rust-emacs-macros-0.11)
+        ("rust-emacs-module" ,rust-emacs-module-0.10)
+        ("rust-failure" ,rust-failure-0.1)
+        ("rust-failure-derive" ,rust-failure-derive-0.1)
+        ("rust-lazy-static" ,rust-lazy-static-1))))))
+
 (define-public rust-emacs-macros-0.17
   (package
     (name "rust-emacs-macros")
--
2.34.0

[bug#49946] [PATCH v4 29/31] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
(emacs-tree-sitter-core): New variable.
---
 gnu/packages/tree-sitter.scm | 78 ++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 3c8b4588c4..e2af7cff5a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -20,6 +20,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
+  #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system node)
   #:use-module (guix download)
@@ -845,3 +846,80 @@ (define-public tree-sitter-typescript
         "This package provides Typescript and TSX grammars for the Tree-sitter
 library.")
       (license license:expat))))
+
+;; Local package definition solely for building the native emacs module
+;; written in Rust.
+(define tree-sitter-emacs-module
+  (package
+    (name "tree-sitter-emacs-module")
+    (version "0.18.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1sdvz827v436qijs6xafakkfw2d16bvp8frymd818rppjc7a9dif"))))
+    (build-system cargo-build-system)
+    (inputs
+     (list tree-sitter))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-emacs" ,rust-emacs-0.18)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-tree-sitter" ,rust-tree-sitter-for-emacs))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'chdir 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (add-after 'delete-cargo.lock 'do-not-fetch-from-github
+           (lambda _
+             (substitute* "Cargo.toml"
+               (("\\[patch.*") "")
+               (("git = .*") ""))))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out") "/lib")))
+               (install-file "target/release/libtsc_dyn.so" lib)))))))
+    (home-page #f)
+    (synopsis #f)
+    (description #f)
+    (license license:expat)))
+
+(define-public emacs-tree-sitter-core
+  (package
+    (name "emacs-tree-sitter-core")
+    (version (package-version tree-sitter-emacs-module))
+    (source (package-source tree-sitter-emacs-module))
+    (build-system emacs-build-system)
+    (native-inputs
+     (list tree-sitter-emacs-module))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'install 'install-module
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out")))
+                   (module (string-append (assoc-ref inputs "tree-sitter-emacs-module")
+                                          "/lib/libtsc_dyn.so")))
+               ;; Writing the version number in this file prevents the package
+               ;; from trying to download the module from the internet.
+               (call-with-output-file (string-append elpa "/DYN-VERSION")
+                 (lambda (port) (display ,version port)))
+               (substitute* "tsc-dyn-get.el"
+                 (("defcustom tsc-dyn-dir tsc--dir")
+                  (string-append "defcustom tsc-dyn-dir \"" elpa "\"")))
+               (copy-file module (string-append elpa "/tsc-dyn.so"))))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp, core library")
+    (description "This package provides core APIs of the Emacs binding for
+Tree-sitter, an incremental parsing system.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 30/31] gnu: Add emacs-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e2af7cff5a..9a427bed7f 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -923,3 +923,30 @@ (module (string-append (assoc-ref inputs "tree-sitter-emacs-module")
     (description "This package provides core APIs of the Emacs binding for
 Tree-sitter, an incremental parsing system.")
     (license license:expat)))
+
+(define-public emacs-tree-sitter
+  (package
+    (name "emacs-tree-sitter")
+    (version (package-version emacs-tree-sitter-core))
+    (source (package-source emacs-tree-sitter-core))
+    (build-system emacs-build-system)
+    (propagated-inputs
+     (list emacs-tree-sitter-core))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "lisp"))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp")
+    (description "This package provides Emacs bindings for Tree-sitter, an
+incremental parsing library.  It aims to be the foundation for a new breed of
+Emacs packages that understand code structurally.  For example:
+
+@enumerate
+@item Faster, fine-grained code highlighting.
+@item More flexible code folding.
+@item Structural editing (like Paredit, or even better) for non-Lisp code.
+@item More informative indexing for imenu.
+@end enumerate")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v4 31/31] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm
(tree-sitter-langs-grammar-bundle, emacs-tree-sitter-langs): New variables.
---
 gnu/packages/tree-sitter.scm | 136 +++++++++++++++++++++++++++++++++++
 1 file changed, 136 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 9a427bed7f..d1341a2ddd 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -23,6 +23,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system node)
+  #:use-module (guix build-system trivial)
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
@@ -950,3 +951,138 @@ (define-public emacs-tree-sitter
 @item More informative indexing for imenu.
 @end enumerate")
     (license license:expat)))
+
+(define tree-sitter-langs-grammar-bundle
+  (package
+    (name "tree-sitter-langs-grammar-bundle")
+    (source #f)
+    (version "0.11.3")
+    (build-system trivial-build-system)
+    (inputs
+     ;; FIXME: Support for some languages is still left to package.
+     (list tree-sitter-bash
+           tree-sitter-c
+           tree-sitter-c-sharp
+           tree-sitter-cpp
+           tree-sitter-css
+           tree-sitter-elixir
+           tree-sitter-elm
+           tree-sitter-go
+           tree-sitter-html
+           tree-sitter-java
+           tree-sitter-javascript
+           tree-sitter-json
+           tree-sitter-julia
+           tree-sitter-ocaml
+           tree-sitter-php
+           tree-sitter-python
+           tree-sitter-rust
+           tree-sitter-ruby
+           tree-sitter-typescript))
+    (arguments
+     (list #:builder
+           (with-imported-modules '((guix build union)
+                                    (guix build utils))
+             #~(begin
+                 (use-modules (ice-9 match)
+                              (guix build union)
+                              (guix build utils))
+                 (union-build
+                  #$output
+                  (filter directory-exists?
+                          (map (match-lambda
+                                 ((name directory)
+                                  (string-append directory "/lib/tree-sitter")))
+                               '#$(package-inputs this-package))))
+                  ;; The BUNDLE-VERSION file prevents emacs-tree-sitter-langs
+                  ;; from downloading libraries at load time.
+                  (call-with-output-file (string-append #$output "/BUNDLE-VERSION")
+                    (lambda (port) (display #$version port)))))))
+    (synopsis #f)
+    (description #f)
+    (home-page #f)
+    (license #f)))
+
+(define-public emacs-tree-sitter-langs
+  (package
+    (name "emacs-tree-sitter-langs")
+    (version "0.11.3")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/tree-sitter-langs")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1sgpsjd3037pbvwk7lgw70gb4j0fyqikv4lwxwa6l5dwnqvqxgq6"))))
+    (build-system emacs-build-system)
+    (inputs
+     (list tree-sitter-langs-grammar-bundle))
+    (propagated-inputs
+     (list emacs-tree-sitter))
+    (arguments
+     (list
+      #:tests? #t
+      #:test-command ''("script/test")
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'remove-cask
+            (lambda _
+              (substitute* "script/test"
+                (("cask") ""))))
+          (add-before 'check 'bundle-for-testing
+            (lambda* (#:key inputs #:allow-other-keys)
+              (delete-file-recursively "bin")
+              (symlink #$tree-sitter-langs-grammar-bundle "bin")))
+          (add-before 'check 'patch-tree-sitter-require-test
+            (lambda _
+              (use-modules (ice-9 regex))
+              ;; This test needs a git repositories with submodules for
+              ;; each languages in order to map all repositories.  We patch
+              ;; the mapping function with one that invokes the tests for each
+              ;; packaged language.
+              (let ((supported-languages
+                     (map (lambda (lib)
+                            (match:substring
+                             (string-match "(.*)\\.so$" (basename lib))
+                             1))
+                          (find-files "bin" "\\.so$"))))
+                (substitute* "tree-sitter-langs-tests.el"
+                  (("tree-sitter-langs--map-repos")
+                   (call-with-output-string
+                     (lambda (port)
+                       (write `(lambda (fn)
+                                 (dolist (lang ',supported-languages)
+                                         (funcall fn lang)))
+                              port))))))))
+          ;; Tests for queries will fail given those languages are not
+          ;; packages yet.
+          (add-before 'check 'remove-unused-highlight-queries
+            (lambda _
+              (delete-file-recursively "queries/hcl")
+              (delete-file-recursively "queries/pgn")))
+          (add-before 'install 'install-bundle
+            (lambda _
+              (let ((elpa (elpa-directory #$output)))
+                (mkdir-p elpa)
+                (symlink #$tree-sitter-langs-grammar-bundle
+                         (string-append elpa "/bin")))))
+          (add-after 'install 'install-queries
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+                (copy-recursively "queries" (string-append elpa "/queries"))))))))
+    (home-page "https://ubolonton.github.io/emacs-tree-sitter/languages/")
+    (synopsis "Language support bundle for Tree-sitter")
+    (description "This package is a convenient language bundle for
+Tree-sitter.  For each supported language, this package provides:
+
+@enumerate
+@item Pre-compiled grammar binaries.
+@item An optional highlights.scm file that provides highlighting patterns.
+This is mainly intended for major modes that are not aware of tree-sitter.
+@item Optional query patterns for other minor modes that provide high-level
+functionalities on top of tree-sitter, such as code folding, evil text
+objects, ...etc.
+@end enumerate")
+    (license license:expat)))
--
2.34.0

[bug#49946] [WIP PATCH v3 00/26] gnu: Add tree-sitter for emacs (and neovim as well?).

[Luis Henrique Gomes Higino]

Hi Guix, Hi Pierre,

I apologize for my delay, the last few weeks were quite busy :-/.

Pierre Langlois <pierre.langlois@gmx.com> writes:

> I'm sharing it early as a WIP first, as I just noticed the other 
> day
> that Luis also contributed a patches for tree-sitter for 
> neovim[0]. The
> series I have here is quite a lot bigger though so I'm happy to 
> rebase
> it on top of yours, Luis :-). The package I had for tree-sitter 
> looks
> basically the same, I would then move it to a new file with all
> tree-sitter related packages, if you're happy with that.

That's OK by me!

> So I'm wondering, are these tree-sitter grammar packages also 
> useful for
> the neovim package for tree-sitter?

I'm not exactly sure about this. Currently, most Neovim users 
install their grammars through the installer included in 
https://github.com/nvim-treesitter/nvim-treesitter. That ensures 
the user gets the version that is ABI compatible with 
nvim-treesitter. That makes me uncertain if we could use the same 
grammar packages for Neovim and Emacs, at least for now.

> I do wonder if I'm going about it the right way for Guix though, 
> for
> instance I'm wondering if I should rewrite the grammar packages 
> using a
> new custom build system, do people think it would be worth it? 
> That
> could be done as a follow-up of course.

I'm not a very experienced packager myself, but from what I saw in 
your patch, it seems rather unnecessary, as the grammar packages 
are very simple.

-- 
Luis H. Higino

[bug#49946] [WIP PATCH v3 00/26] gnu: Add tree-sitter for emacs (and neovim as well?).

[Pierre Langlois]

[-- Attachment #1.1: Type: text/plain, Size: 3952 bytes --]

Hi Luis,

Luis Henrique Gomes Higino <luishenriquegh2701@gmail.com> writes:

> Hi Guix, Hi Pierre,
>
> I apologize for my delay, the last few weeks were quite busy :-/.

No worries! :-).

>
> Pierre Langlois <pierre.langlois@gmx.com> writes:
>
>> I'm sharing it early as a WIP first, as I just noticed the other day
>> that Luis also contributed a patches for tree-sitter for neovim[0]. The
>> series I have here is quite a lot bigger though so I'm happy to rebase
>> it on top of yours, Luis :-). The package I had for tree-sitter looks
>> basically the same, I would then move it to a new file with all
>> tree-sitter related packages, if you're happy with that.
>
> That's OK by me!
>
>> So I'm wondering, are these tree-sitter grammar packages also useful for
>> the neovim package for tree-sitter?
>
> I'm not exactly sure about this. Currently, most Neovim users install their
> grammars through the installer included in
> https://github.com/nvim-treesitter/nvim-treesitter. That ensures 
> the user gets the version that is ABI compatible with nvim-treesitter. That
> makes me uncertain if we could use the same grammar packages for Neovim and
> Emacs, at least for now.

I see, I suppose for the moment we're not packaging nvim-treesitter so
we don't have to concern ourselves with how language grammars are
delivered to users.

In the future it might be cool to package it, out of curiosity I took a
look at it and it:

  - [0]: Clones the repo with `git'.
  - [1]: Run `npm install' and `tree-sitter generate'.
  - [2]: Compiles the generated grammar with gcc.

[0]: https://github.com/nvim-treesitter/nvim-treesitter/blob/90485c890503f973271db1ae02ddba6d6fd46397/lua/nvim-treesitter/install.lua#L378
[1]: https://github.com/nvim-treesitter/nvim-treesitter/blob/90485c890503f973271db1ae02ddba6d6fd46397/lua/nvim-treesitter/install.lua#L228
[2]: https://github.com/nvim-treesitter/nvim-treesitter/blob/3aac7f9db9ee1973152426c097216e0071dd2293/lua/nvim-treesitter/shell_command_selectors.lua#L96

The only concern I can think of is that when it runs `npm install', it's
likely that it downloads a `tree-sitter' binary, rather than use
Guix's tree-sitter-cli package.  In my series I had to patch
package.json files for each grammar to remove `tree-sitter-cli' as a
dependency:

https://github.com/tree-sitter/tree-sitter-c/blob/e348e8ec5efd3aac020020e4af53d2ff18f393a9/package.json#L20

Otherwise it would pull in the official tree-sitter-cli node package,
which then downloads a `tree-sitter' binary :-/

https://github.com/tree-sitter/tree-sitter/blob/master/cli/npm/package.json

Anyways, I don't think that's something we should concern ourselves with
for now.  If we ever want to package nvim-treesitter, I think we might
have to patch it so it uses Guix-provided grammars instead of building
them locally.  For emacs that wasn't so difficult to do, but the real
issue is making sure the grammar is compabible, and luckily the emacs
package had tests we could run for that.  We would also need to run
tests if we're to package nvim-treesitter.

>
>> I do wonder if I'm going about it the right way for Guix though, for
>> instance I'm wondering if I should rewrite the grammar packages using a
>> new custom build system, do people think it would be worth it? That
>> could be done as a follow-up of course.
>
> I'm not a very experienced packager myself, but from what I saw in your patch,
> it seems rather unnecessary, as the grammar packages are very simple.

That was also my impression :-).  Since then I did play with building
one to see what it would look like, also as an escuse to go and learn
how they work.  I'm still on the fence about it though, I've attached
what it could look like, if anybody else is also unsure.  I think I'd
prefer to work on it as a potential follow-up after an initial set of
patches get merged.

Thanks,
Pierre


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Attachment #2: 0001-wip-build-system-for-tree-sitter-grammars.patch --]
[-- Type: text/x-patch, Size: 36180 bytes --]

From a38e3e66788129bbd441f9d28e450dba8a7438d1 Mon Sep 17 00:00:00 2001
From: Pierre Langlois <pierre.langlois@gmx.com>
Date: Wed, 23 Feb 2022 20:38:51 +0000
Subject: [PATCH] wip build-system for tree-sitter grammars.

---
 Makefile.am                             |   2 +
 gnu/packages/tree-sitter.scm            | 312 ++++++++----------------
 guix/build-system/tree-sitter.scm       | 109 +++++++++
 guix/build/tree-sitter-build-system.scm | 124 ++++++++++
 4 files changed, 342 insertions(+), 205 deletions(-)
 create mode 100644 guix/build-system/tree-sitter.scm
 create mode 100644 guix/build/tree-sitter-build-system.scm

diff --git a/Makefile.am b/Makefile.am
index 8850c4562c..c5da931041 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -167,6 +167,7 @@ MODULES =					\
   guix/build-system/ruby.scm			\
   guix/build-system/scons.scm			\
   guix/build-system/texlive.scm			\
+  guix/build-system/tree-sitter.scm		\
   guix/build-system/trivial.scm			\
   guix/ftp-client.scm				\
   guix/http-client.scm				\
@@ -220,6 +221,7 @@ MODULES =					\
   guix/build/ruby-build-system.scm		\
   guix/build/scons-build-system.scm		\
   guix/build/texlive-build-system.scm		\
+  guix/build/tree-sitter-build-system.scm	\
   guix/build/waf-build-system.scm		\
   guix/build/haskell-build-system.scm		\
   guix/build/julia-build-system.scm		\
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 0186518b23..725ab16910 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -23,6 +23,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system node)
+  #:use-module (guix build-system tree-sitter)
   #:use-module (guix build-system trivial)
   #:use-module (guix download)
   #:use-module (guix git-download)
@@ -224,70 +225,7 @@ (define-public tree-sitter-c
                 "0454jziys33i4kbwnvi9xcck0fzya792ghy32ahgk1hhv96xga9w"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
-    (build-system node-build-system)
-    (native-inputs
-     (list tree-sitter tree-sitter-cli))
-    (outputs '("out" "js"))
-    (arguments
-     (list
-      #:phases
-      (with-imported-modules '((guix build json))
-        #~(modify-phases %standard-phases
-            (add-after 'patch-dependencies 'delete-dependencies
-              (lambda _
-                (delete-dependencies '("tree-sitter-cli"
-                                       "nan"
-                                       "node-gyp"))))
-            (replace 'build
-              (lambda _
-                (invoke "tree-sitter" "generate" "--no-bindings")))
-            (replace 'check
-              (lambda _
-                (invoke "tree-sitter" "test")))
-            (replace 'install
-              (lambda _
-                (use-modules (guix build json)
-                             (ice-9 regex))
-                (let* ((name (assoc-ref
-                              (call-with-input-file "package.json" read-json)
-                              "name"))
-                       (lang (cond
-                              ((string-match "^(@.*/)?tree-sitter-(.*)$" name)
-                               => (lambda (m)
-                                    (match:substring m 2)))
-                              (else #f)))
-                       (lib (string-append #$output "/lib/tree-sitter")))
-                  (mkdir-p lib)
-                  (define (source-file path)
-                    (if (file-exists? path)
-                        path
-                        #f))
-                  (apply invoke
-                         `(,#$(cxx-for-target)
-                           "-shared"
-                           "-fPIC"
-                           "-fno-exceptions"
-                           "-O2"
-                           "-g"
-                           "-o" ,(string-append lib "/" lang ".so")
-                           ,@(cond
-                              ((source-file "src/scanner.c")
-                               => (lambda (file) (list "-xc" "-std=c99" file)))
-                              ((source-file "src/scanner.cc")
-                               => (lambda (file) (list file)))
-                              (else '()))
-                           "-xc" "src/parser.c")))))
-            (add-after 'install 'install-js-module
-              (lambda* (#:key inputs #:allow-other-keys)
-                (invoke (search-input-file inputs "/bin/npm")
-                        "--prefix" #$output:js
-                        "--global"
-                        "--offline"
-                        "--loglevel" "info"
-                        "--production"
-                        ;; Skip scripts to prevent building bindings via GYP.
-                        "--ignore-scripts"
-                        "install" "../package.tgz")))))))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-c")
     (synopsis "Tree-sitter C grammar")
     (description
@@ -296,7 +234,6 @@ (define (source-file path)
 
 (define-public tree-sitter-cpp
   (package
-    (inherit tree-sitter-c)
     (name "tree-sitter-cpp")
     (version "0.20.0")
     (source (origin
@@ -311,7 +248,8 @@ (define-public tree-sitter-cpp
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
     (inputs
-     (list `(,tree-sitter-c "js")))
+     (list tree-sitter-c))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-cpp")
     (synopsis "Tree-sitter C++ grammar")
     (description
@@ -319,7 +257,7 @@ (define-public tree-sitter-cpp
     (license license:expat)))
 
 (define-public tree-sitter-c-sharp
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-c-sharp")
     (version "0.19.1")
     (source (origin
@@ -333,6 +271,7 @@ (define-public tree-sitter-c-sharp
                 "054fmpf47cwh59gbg00sc0nl237ba4rnxi73miz39yqzcs87055r"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-c-sharp")
     (synopsis "Tree-sitter C# grammar")
     (description
@@ -340,7 +279,7 @@ (define-public tree-sitter-c-sharp
     (license license:expat)))
 
 (define-public tree-sitter-bash
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-bash")
     (version "0.19.0")
     (source (origin
@@ -354,17 +293,7 @@ (define-public tree-sitter-bash
                 "18c030bb65r50i6z37iy7jb9z9i8i36y7b08dbc9bchdifqsijs5"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
-    (arguments
-     (substitute-keyword-arguments (package-arguments tree-sitter-c)
-      ((#:phases phases)
-       `(modify-phases ,phases
-          (replace 'delete-dependencies
-           (lambda _
-             (delete-dependencies '("tree-sitter-cli"
-                                    "nan"
-                                    "node-gyp"
-                                    "prebuild"
-                                    "prebuild-install"))))))))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-bash")
     (synopsis "Tree-sitter Bash grammar")
     (description
@@ -372,7 +301,7 @@ (define-public tree-sitter-bash
     (license license:expat)))
 
 (define-public tree-sitter-css
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-css")
     (version "0.19.0")
     (source (origin
@@ -386,6 +315,7 @@ (define-public tree-sitter-css
                 "014jrlgi7zfza9g38hsr4vlbi8964i5p7iglaih6qmzaiml7bja2"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-css")
     (synopsis "Tree-sitter CSS grammar")
     (description
@@ -395,7 +325,7 @@ (define-public tree-sitter-css
 (define-public tree-sitter-elixir
   (let ((commit "de20391afe5cb03ef1e8a8e43167e7b58cc52869")
         (revision "1"))
-    (package (inherit tree-sitter-c)
+    (package
       (name "tree-sitter-elixir")
       (version (git-version "0.19.0" revision commit))
       (source (origin
@@ -409,16 +339,7 @@ (define-public tree-sitter-elixir
                   "0zrkrwhw3g1vazsxcwrfd1fk4wvs9hdwmwp6073mfh370bz4140h"))
                 (modules '((guix build utils)))
                 (snippet tree-sitter-delete-generated-files)))
-      (arguments
-       (substitute-keyword-arguments (package-arguments tree-sitter-c)
-         ((#:phases phases)
-          `(modify-phases ,phases
-             (replace 'delete-dependencies
-               (lambda _
-                 (delete-dependencies '("tree-sitter-cli"
-                                        "node-gyp"
-                                        "nan"
-                                        "prettier"))))))))
+      (build-system tree-sitter-build-system)
       (home-page "https://elixir-lang.org/tree-sitter-elixir/")
       (synopsis "Tree-sitter Elixir grammar")
       (description
@@ -428,7 +349,7 @@ (define-public tree-sitter-elixir
                      license:expat)))))
 
 (define-public tree-sitter-elm
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-elm")
     (version "5.5.1")
     (source (origin
@@ -442,18 +363,7 @@ (define-public tree-sitter-elm
                 "10hbi4vyj4hjixqswdcbvzl60prldczz29mlp02if61wvwiwvqrw"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
-    (arguments
-     (substitute-keyword-arguments (package-arguments tree-sitter-c)
-      ((#:phases phases)
-       `(modify-phases ,phases
-          (replace 'delete-dependencies
-           (lambda _
-             (delete-dependencies '("tree-sitter-cli"
-                                    "nan"
-                                    "node-gyp"
-                                    "@asgerf/dts-tree-sitter"
-                                    "prebuild"
-                                    "prebuild-install"))))))))
+    (build-system tree-sitter-build-system)
     (home-page "https://elm-tooling.github.io/tree-sitter-elm/")
     (synopsis "Tree-sitter Elm grammar")
     (description
@@ -461,7 +371,7 @@ (define-public tree-sitter-elm
     (license license:expat)))
 
 (define-public tree-sitter-go
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-go")
     (version "0.19.1")
     (source (origin
@@ -475,6 +385,7 @@ (define-public tree-sitter-go
                 "0nxs47vd2fc2fr0qlxq496y852rwg39flhg334s7dlyq7d3lcx4x"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-go")
     (synopsis "Tree-sitter Go grammar")
     (description
@@ -482,7 +393,7 @@ (define-public tree-sitter-go
     (license license:expat)))
 
 (define-public tree-sitter-html
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-html")
     (version "0.19.0")
     (source (origin
@@ -496,6 +407,7 @@ (define-public tree-sitter-html
                 "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-html")
     (synopsis "Tree-sitter HTML grammar")
     (description
@@ -503,7 +415,7 @@ (define-public tree-sitter-html
     (license license:expat)))
 
 (define-public tree-sitter-java
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-java")
     (version "0.19.1")
     (source (origin
@@ -517,6 +429,7 @@ (define-public tree-sitter-java
                 "07zw9ygb45hnvlx9qlz7rlz8hc3byjy03d24v72i5iyhpiiwlhvl"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-java")
     (synopsis "Tree-sitter Java grammar")
     (description
@@ -524,7 +437,7 @@ (define-public tree-sitter-java
     (license license:expat)))
 
 (define-public tree-sitter-javascript
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-javascript")
     (version "0.20.0")
     (source (origin
@@ -538,6 +451,7 @@ (define-public tree-sitter-javascript
                 "175yrk382n2di0c2xn4gpv8y4n83x1lg4hqn04vabf0yqynlkq67"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
     (synopsis "Tree-sitter Javascript grammar")
     (description
@@ -546,7 +460,7 @@ (define-public tree-sitter-javascript
     (license license:expat)))
 
 (define-public tree-sitter-json
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-json")
     (version "0.19.0")
     (source (origin
@@ -560,6 +474,7 @@ (define-public tree-sitter-json
                 "06pjh31bv9ja9hlnykk257a6zh8bsxg2fqa54al7qk1r4n9ksnff"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-json")
     (synopsis "Tree-sitter JSON grammar")
     (description
@@ -567,7 +482,7 @@ (define-public tree-sitter-json
     (license license:expat)))
 
 (define-public tree-sitter-julia
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-julia")
     (version "0.19.0")
     (source (origin
@@ -581,6 +496,7 @@ (define-public tree-sitter-julia
                 "1pbnmvhy2gq4vg1b0sjzmjm4s2gsgdjh7h01yj8qrrqbcl29c463"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-julia")
     (synopsis "Tree-sitter Julia grammar")
     (description
@@ -590,7 +506,7 @@ (define-public tree-sitter-julia
 (define-public tree-sitter-ocaml
   (let ((commit "0348562f385bc2bd67ecf181425e1afd6d454192")
         (revision "1"))
-    (package (inherit tree-sitter-c)
+    (package
       (name "tree-sitter-ocaml")
       (version (git-version "0.19.0" revision commit))
       (source (origin
@@ -617,43 +533,44 @@ (define-public tree-sitter-ocaml
                      '("ocaml" "interface"))
                     #t))))
       (arguments
-       (substitute-keyword-arguments (package-arguments tree-sitter-c)
-        ((#:phases phases)
-         #~(modify-phases #$phases
-             (replace 'build
-               (lambda _
-                 (for-each (lambda (dir)
-                             (with-directory-excursion dir
-                               (invoke "tree-sitter" "generate" "--no-bindings")))
-                           '("ocaml" "interface"))))
-             (replace 'check
-               (lambda _
-                 (for-each (lambda (dir)
-                             (with-directory-excursion dir
-                               (invoke "tree-sitter" "test")))
-                           '("ocaml" "interface"))))
-             (replace 'install
-               (lambda _
-                 (let ((lib (string-append #$output "/lib/tree-sitter/")))
-                   (mkdir-p lib)
-                   (invoke #$(cxx-for-target)
-                           "-shared"
-                           "-fPIC"
-                           "-fno-exceptions"
-                           "-O2"
-                           "-g"
-                           "-o" (string-append lib "/ocaml.so")
-                           "ocaml/src/scanner.cc"
-                           "-xc" "ocaml/src/parser.c")
-                   (invoke #$(cxx-for-target)
-                           "-shared"
-                           "-fPIC"
-                           "-fno-exceptions"
-                           "-O2"
-                           "-g"
-                           "-o" (string-append lib "/ocaml-interface.so")
-                           "interface/src/scanner.cc"
-                           "-xc" "interface/src/parser.c"))))))))
+       (list
+        #:phases
+        #~(modify-phases %standard-phases
+            (replace 'build
+              (lambda _
+                (for-each (lambda (dir)
+                            (with-directory-excursion dir
+                              (invoke "tree-sitter" "generate" "--no-bindings")))
+                          '("ocaml" "interface"))))
+            (replace 'check
+              (lambda _
+                (for-each (lambda (dir)
+                            (with-directory-excursion dir
+                              (invoke "tree-sitter" "test")))
+                          '("ocaml" "interface"))))
+            (replace 'install
+              (lambda _
+                (let ((lib (string-append #$output "/lib/tree-sitter/")))
+                  (mkdir-p lib)
+                  (invoke #$(cxx-for-target)
+                          "-shared"
+                          "-fPIC"
+                          "-fno-exceptions"
+                          "-O2"
+                          "-g"
+                          "-o" (string-append lib "/ocaml.so")
+                          "ocaml/src/scanner.cc"
+                          "-xc" "ocaml/src/parser.c")
+                  (invoke #$(cxx-for-target)
+                          "-shared"
+                          "-fPIC"
+                          "-fno-exceptions"
+                          "-O2"
+                          "-g"
+                          "-o" (string-append lib "/ocaml-interface.so")
+                          "interface/src/scanner.cc"
+                          "-xc" "interface/src/parser.c")))))))
+      (build-system tree-sitter-build-system)
       (home-page "https://github.com/tree-sitter/tree-sitter-ocaml")
       (synopsis "Tree-sitter OCaml grammar")
       (description
@@ -663,7 +580,7 @@ (define-public tree-sitter-ocaml
 (define-public tree-sitter-php
   (let ((commit "435fa00006c0d1515c37fbb4dd6a9de284af75ab")
         (revision "1"))
-    (package (inherit tree-sitter-c)
+    (package
       (name "tree-sitter-php")
       (version (git-version "0.19.0" revision commit))
       (source (origin
@@ -683,15 +600,7 @@ (define-public tree-sitter-php
                     (delete-file "src/parser.c")
                     (delete-file-recursively "src/tree_sitter")
                     #t))))
-    (arguments
-     (substitute-keyword-arguments (package-arguments tree-sitter-c)
-      ((#:phases phases)
-       `(modify-phases ,phases
-          (replace 'delete-dependencies
-           (lambda _
-             (delete-dependencies '("tree-sitter-cli"
-                                    "nan"
-                                    "shelljs"))))))))
+      (build-system tree-sitter-build-system)
       (home-page "https://github.com/tree-sitter/tree-sitter-php")
       (synopsis "Tree-sitter PHP grammar")
       (description
@@ -701,7 +610,7 @@ (define-public tree-sitter-php
 (define-public tree-sitter-python
   (let ((commit "ed0fe62e55dc617ed9dec8817ebf771aa7cf3c42")
         (revision "1"))
-    (package (inherit tree-sitter-c)
+    (package
       (name "tree-sitter-python")
       (version (git-version "0.19.1" revision commit))
       (source (origin
@@ -715,6 +624,7 @@ (define-public tree-sitter-python
                   "0wrfpg84mc3pzcrdi6n5fqwijkqr1nj5sqfnayb502krvqpjilal"))
                 (modules '((guix build utils)))
                 (snippet tree-sitter-delete-generated-files)))
+      (build-system tree-sitter-build-system)
       (home-page "https://github.com/tree-sitter/tree-sitter-python")
       (synopsis "Tree-sitter Python grammar")
       (description
@@ -722,7 +632,7 @@ (define-public tree-sitter-python
       (license license:expat))))
 
 (define-public tree-sitter-ruby
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-ruby")
     (version "0.19.0")
     (source (origin
@@ -736,17 +646,7 @@ (define-public tree-sitter-ruby
                 "0m3h4928rbs300wcb6776h9r88hi32rybbhcaf6rdympl5nzi83v"))
               (modules '((guix build utils)))
               (snippet tree-sitter-delete-generated-files)))
-    (arguments
-     (substitute-keyword-arguments (package-arguments tree-sitter-c)
-      ((#:phases phases)
-       `(modify-phases ,phases
-          (replace 'delete-dependencies
-           (lambda _
-             (delete-dependencies '("tree-sitter-cli"
-                                    "nan"
-                                    "node-gyp"
-                                    "prebuild"
-                                    "prebuild-install"))))))))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-ruby")
     (synopsis "Tree-sitter Ruby grammar")
     (description
@@ -754,7 +654,7 @@ (define-public tree-sitter-ruby
     (license license:expat)))
 
 (define-public tree-sitter-rust
-  (package (inherit tree-sitter-c)
+  (package
     (name "tree-sitter-rust")
     (version "0.20.1")
     (source (origin
@@ -771,6 +671,7 @@ (define-public tree-sitter-rust
     (native-inputs
      (modify-inputs (package-native-inputs tree-sitter-c)
        (prepend bc)))
+    (build-system tree-sitter-build-system)
     (home-page "https://github.com/tree-sitter/tree-sitter-rust")
     (synopsis "Tree-sitter Rust grammar")
     (description
@@ -780,7 +681,7 @@ (define-public tree-sitter-rust
 (define-public tree-sitter-typescript
   (let ((commit "111b07762e86efab9a918b7c721f720c37e76b0a")
         (revision "1"))
-    (package (inherit tree-sitter-c)
+    (package
       (name "tree-sitter-typescript")
       (version (git-version "0.20.0" revision commit))
       (source (origin
@@ -806,39 +707,40 @@ (define-public tree-sitter-typescript
                      '("typescript" "tsx"))
                     #t))))
       (inputs
-       (list `(,tree-sitter-javascript "js")))
+       (list tree-sitter-javascript))
       (arguments
-       (substitute-keyword-arguments (package-arguments tree-sitter-c)
-        ((#:phases phases)
-         #~(modify-phases #$phases
-             (replace 'build
-               (lambda _
-                 (for-each (lambda (dir)
-                             (with-directory-excursion dir
-                               (invoke "tree-sitter" "generate" "--no-bindings")))
-                           '("typescript" "tsx"))))
-             (replace 'check
-               (lambda _
-                 (for-each (lambda (dir)
-                             (with-directory-excursion dir
-                               (invoke "tree-sitter" "test")))
-                           '("typescript" "tsx"))))
-             (replace 'install
-               (lambda _
-                 (let ((lib (string-append #$output "/lib/tree-sitter/")))
-                   (mkdir-p lib)
-                   (for-each
-                    (lambda (lang)
-                      (invoke #$(cxx-for-target)
-                              "-shared"
-                              "-fPIC"
-                              "-fno-exceptions"
-                              "-O2"
-                              "-g"
-                              "-o" (string-append lib "/" lang ".so")
-                              "-xc" (string-append lang "/src/scanner.c")
-                              "-xc" (string-append lang "/src/parser.c")))
-                    '("typescript" "tsx")))))))))
+       (list
+        #:phases
+        #~(modify-phases %standard-phases
+            (replace 'build
+              (lambda _
+                (for-each (lambda (dir)
+                            (with-directory-excursion dir
+                              (invoke "tree-sitter" "generate" "--no-bindings")))
+                          '("typescript" "tsx"))))
+            (replace 'check
+              (lambda _
+                (for-each (lambda (dir)
+                            (with-directory-excursion dir
+                              (invoke "tree-sitter" "test")))
+                          '("typescript" "tsx"))))
+            (replace 'install
+              (lambda _
+                (let ((lib (string-append #$output "/lib/tree-sitter/")))
+                  (mkdir-p lib)
+                  (for-each
+                   (lambda (lang)
+                     (invoke #$(cxx-for-target)
+                             "-shared"
+                             "-fPIC"
+                             "-fno-exceptions"
+                             "-O2"
+                             "-g"
+                             "-o" (string-append lib "/" lang ".so")
+                             "-xc" (string-append lang "/src/scanner.c")
+                             "-xc" (string-append lang "/src/parser.c")))
+                   '("typescript" "tsx"))))))))
+      (build-system tree-sitter-build-system)
       (home-page "https://github.com/tree-sitter/tree-sitter-typescript")
       (synopsis "Tree-sitter Typescript grammar")
       (description
diff --git a/guix/build-system/tree-sitter.scm b/guix/build-system/tree-sitter.scm
new file mode 100644
index 0000000000..bfccea4007
--- /dev/null
+++ b/guix/build-system/tree-sitter.scm
@@ -0,0 +1,109 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build-system tree-sitter)
+  #:use-module (guix store)
+  #:use-module (guix utils)
+  #:use-module (guix packages)
+  #:use-module (guix gexp)
+  #:use-module (guix monads)
+  #:use-module (guix search-paths)
+  #:use-module (guix build-system)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system node)
+  #:use-module (ice-9 match)
+  #:export (%tree-sitter-build-system-modules
+            tree-sitter-build
+            tree-sitter-build-system))
+
+(define %tree-sitter-build-system-modules
+  ;; Build-side modules imported by default.
+  `((guix build tree-sitter-build-system)
+    ;; (guix build json)
+    ,@%node-build-system-modules))
+
+(define* (lower name
+                #:key source inputs native-inputs outputs system target
+                #:allow-other-keys
+                #:rest arguments)
+  ""
+  (define private-keywords
+    '(#:target #:inputs #:native-inputs #:outputs))
+
+  (and (not target)                    ;XXX: no cross-compilation
+       (bag
+         (name name)
+         (system system)
+         (host-inputs `(,@(if source
+                              `(("source" ,source))
+                              '())
+                        ,@(map (match-lambda
+                                 ((name package)
+                                  `(,name ,package "js")))
+                               inputs)
+                        ;; Keep the standard inputs of 'gnu-build-system'.
+                        ,@(standard-packages)))
+         (build-inputs `(("node" ,(module-ref (resolve-interface '(gnu packages node))
+                                              'node-lts))
+                         ("tree-sitter" ,(module-ref (resolve-interface '(gnu packages tree-sitter))
+                                                     'tree-sitter))
+                         ("tree-sitter-cli" ,(module-ref (resolve-interface '(gnu packages tree-sitter))
+                                                         'tree-sitter-cli))
+                         ,@native-inputs))
+         (outputs (match outputs
+                    (("out") (cons "js" outputs))
+                    (_ outputs)))
+         (build tree-sitter-build)
+         (arguments (strip-keyword-arguments private-keywords arguments)))))
+
+(define* (tree-sitter-build name inputs
+                            #:key
+                            source
+                            (phases '%standard-phases)
+                            (outputs '("out" "js"))
+                            (search-paths '())
+                            (system (%current-system))
+                            (guile #f)
+                            (imported-modules %tree-sitter-build-system-modules)
+                            (modules '((guix build utils)
+                                       (guix build tree-sitter-build-system))))
+  (define builder
+    (with-imported-modules imported-modules
+      #~(begin
+          (use-modules #$@(sexp->gexp modules))
+          (tree-sitter-build #:name #$name
+                             #:source #+source
+                             #:system #$system
+                             #:phases #$phases
+                             #:outputs #$(outputs->gexp outputs)
+                             #:search-paths '#$(sexp->gexp
+                                                (map search-path-specification->sexp
+                                                     search-paths))
+                             #:inputs #$(input-tuples->gexp inputs)))))
+
+  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+                                                  system #:graft? #f)))
+    (gexp->derivation name builder
+                      #:system system
+                      #:guile-for-build guile)))
+
+(define tree-sitter-build-system
+  (build-system
+    (name 'tree-sitter)
+    (description "")
+    (lower lower)))
diff --git a/guix/build/tree-sitter-build-system.scm b/guix/build/tree-sitter-build-system.scm
new file mode 100644
index 0000000000..bf1d2e363d
--- /dev/null
+++ b/guix/build/tree-sitter-build-system.scm
@@ -0,0 +1,124 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build tree-sitter-build-system)
+  #:use-module ((guix build node-build-system) #:prefix node:)
+  #:use-module (guix build json)
+  #:use-module (guix build utils)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module (srfi srfi-1)
+  #:export (%standard-phases
+            tree-sitter-build))
+
+(define (tree-sitter-node-inputs inputs)
+  ""
+  (alist-delete "node"
+    (alist-delete "source"
+      (filter (match-lambda
+                ((label . directory)
+                 (directory-exists? (string-append directory
+                                                   "/lib/node_modules")))
+                (_ #f))
+              inputs))))
+
+;;;
+;;; Phases.
+;;;
+
+(define* (adjust-dependencies #:key inputs #:allow-other-keys)
+  ""
+  (node:with-atomic-json-file-replacement "package.json"
+    (match-lambda
+      (('@ . pkg-meta-alist)
+       (cons '@ (map (match-lambda
+                       (("dependencies" @ . deps)
+                        '("dependencies" @))
+                       (("devDependencies" @ . deps)
+                        `("devDependencies" @
+                          ,@(map
+                             (lambda (input)
+                               `(,(car input) . "latest"))
+                             (tree-sitter-node-inputs inputs))))
+                       (other other))
+                     pkg-meta-alist))))))
+
+(define (build . _)
+  (invoke "tree-sitter" "generate" "--no-bindings")
+  #t)
+
+(define (check . _)
+  (invoke "tree-sitter" "test")
+  #t)
+
+(define* (install #:key outputs #:allow-other-keys)
+  (use-modules (guix build json)
+               (ice-9 regex))
+  (let* ((name (assoc-ref
+                (call-with-input-file "package.json" read-json)
+                "name"))
+         (lang (cond
+                ((string-match "^(@.*/)?tree-sitter-(.*)$" name)
+                 => (lambda (m)
+                      (match:substring m 2)))
+                (else #f)))
+         (lib (string-append (assoc-ref outputs "out")
+                             "/lib/tree-sitter")))
+    (mkdir-p lib)
+    (define (source-file path)
+      (if (file-exists? path)
+          path
+          #f))
+    (apply invoke
+           `("g++"
+             "-shared"
+             "-fPIC"
+             "-fno-exceptions"
+             "-O2"
+             "-g"
+             "-o" ,(string-append lib "/" lang ".so")
+             ,@(cond
+                ((source-file "src/scanner.c")
+                 => (lambda (file) (list "-xc" "-std=c99" file)))
+                ((source-file "src/scanner.cc")
+                 => (lambda (file) (list file)))
+                (else '()))
+             "-xc" "src/parser.c"))))
+
+(define* (install-js #:key inputs outputs #:allow-other-keys)
+  (invoke (search-input-file inputs "/bin/npm")
+          "--prefix" (assoc-ref outputs "js")
+          "--global"
+          "--offline"
+          "--loglevel" "info"
+          "--production"
+          ;; Skip scripts to prevent building bindings via GYP.
+          "--ignore-scripts"
+          "install" "../package.tgz"))
+
+(define %standard-phases
+  (modify-phases node:%standard-phases
+    (add-before 'patch-dependencies 'adjust-dependencies adjust-dependencies)
+    (replace 'build build)
+    (replace 'check check)
+    (replace 'install install)
+    (add-after 'install 'install-js install-js)))
+
+(define* (tree-sitter-build #:key inputs (phases %standard-phases)
+                            #:allow-other-keys #:rest args)
+  (apply node:node-build #:inputs inputs #:phases phases args))
-- 
2.34.0

[bug#49946] [WIP PATCH v3 00/26] gnu: Add tree-sitter for emacs (and neovim as well?).

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 143 bytes --]

Pierre Langlois schreef op wo 09-03-2022 om 13:46 [+0000]:
> +           `("g++"

Wouldn't the cross-compiler be necessary here?

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [WIP PATCH v3 00/26] gnu: Add tree-sitter for emacs (and neovim as well?).

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

Hi Maxime,

Thanks for taking a look!

Maxime Devos <maximedevos@telenet.be> writes:

> [[PGP Signed Part:Undecided]]
> Pierre Langlois schreef op wo 09-03-2022 om 13:46 [+0000]:
>> +           `("g++"
>
> Wouldn't the cross-compiler be necessary here?

Yeah it should, I've not yet implemented cross-compiling support in this
build-system yet, but I'm happy to do it!

Before continuing to work on this build-system though, do you have any
opinions about adding it to begin with?  As opposed to the current
approach of defining a base grammar package (see tree-sitter-c) and have
the other grammar packages inherit from it.  I'm wondering if it's worth
adding the build-system, when I don't think we should expect the number
of grammars we would package to grow too much.  But then again, the
package definitions do look quite a bit nicer with the build-system, so
I'm a bit on the fence about this :-).

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [WIP PATCH v3 00/26] gnu: Add tree-sitter for emacs (and neovim as well?).

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

Pierre Langlois schreef op do 10-03-2022 om 11:17 [+0000]:
> Before continuing to work on this build-system though, do you have
> any
> opinions about adding it to begin with?  As opposed to the current
> approach of defining a base grammar package (see tree-sitter-c) and
> have
> the other grammar packages inherit from it.  I'm wondering if it's
> worth
> adding the build-system, when I don't think we should expect the
> number
> of grammars we would package to grow too much.  But then again, the
> package definitions do look quite a bit nicer with the build-system,
> so
> I'm a bit on the fence about this :-).

Build systems only cost a module or two, there are quite a few tree
sitter packages (19 or so?) and the tree sitter packages have a lot
in common (custom test phases, install phases), so I wouldn't
hesitate to define a custom build system for tree-sitter stuff.

Even better would be to unify things a bit more, e.g. it looks like
tree-sitter-ocaml needs a custom 'install', 'build' and 'check' phase,
but they looks almost the same as the other phases for other treesitter
packages, so maybe 'tree-sitter-build-system' can be taught to mostly
automatically determine the exact invocation?  

Also, 'use-modules' only really works on the top-level, I recommend
#:modules instead, there are some problems with using it in expression
context (I don't think they are documented anywhere though).

Greetings,
Maxime

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [WIP PATCH v3 00/26] gnu: Add tree-sitter for emacs (and neovim as well?).

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 2810 bytes --]


Maxime Devos <maximedevos@telenet.be> writes:

> [[PGP Signed Part:Undecided]]
> Pierre Langlois schreef op do 10-03-2022 om 11:17 [+0000]:
>> Before continuing to work on this build-system though, do you have
>> any
>> opinions about adding it to begin with?  As opposed to the current
>> approach of defining a base grammar package (see tree-sitter-c) and
>> have
>> the other grammar packages inherit from it.  I'm wondering if it's
>> worth
>> adding the build-system, when I don't think we should expect the
>> number
>> of grammars we would package to grow too much.  But then again, the
>> package definitions do look quite a bit nicer with the build-system,
>> so
>> I'm a bit on the fence about this :-).
>
> Build systems only cost a module or two, there are quite a few tree
> sitter packages (19 or so?) and the tree sitter packages have a lot
> in common (custom test phases, install phases), so I wouldn't
> hesitate to define a custom build system for tree-sitter stuff.

Sounds good, I'll continue in that direction for v5.

>
> Even better would be to unify things a bit more, e.g. it looks like
> tree-sitter-ocaml needs a custom 'install', 'build' and 'check' phase,
> but they looks almost the same as the other phases for other treesitter
> packages, so maybe 'tree-sitter-build-system' can be taught to mostly
> automatically determine the exact invocation?  

I'll see what can be done, sadly so far I couldn't find a way to do the
right thing automatically for ocaml and typescript.  These are packages
that provide two grammars, and I didn't see a way to "discover" that
there are multiple languages supported.  However maybe they could be
split into multiple package definitions, if I can find a way to do that,
maybe by passing new #:keyword parameters.

Another idea could be to call back into the build-system phase, for
example something like:

--8<---------------cut here---------------start------------->8---
(replace 'build
  (lambda _
    (for-each (lambda (dir)
                (with-directory-excursion dir
                  ((assoc-ref tree-sitter:%standard-phases 'build))))
              '("ocaml" "interface"))))
--8<---------------cut here---------------end--------------->8---

I don't know if that works yet, we could do that if other nicer
approaches don't work.

>
> Also, 'use-modules' only really works on the top-level, I recommend
> #:modules instead, there are some problems with using it in expression
> context (I don't think they are documented anywhere though).

Ah, I see I had (use-modules (guix build json) (ice-9 regex)) in the
install phase for no reason indeed, those modules were already imported,
that's what you meant right?

Thanks for your input!  I'll work on a v5.
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [WIP PATCH v3 00/26] gnu: Add tree-sitter for emacs (and neovim as well?).

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

Pierre Langlois schreef op do 10-03-2022 om 13:19 [+0000]:
> I'll see what can be done, sadly so far I couldn't find a way to do
> the right thing automatically for ocaml and typescript.  These are
> packages that provide two grammars, and I didn't see a way to
> "discover" that there are multiple languages supported.  [...] maybe
> by passing new #:keyword parameters.
> 
> Another idea could be to call back into the build-system phase, for
> example something like:
> 
> --8<---------------cut here---------------start------------->8---
> (replace 'build
>   (lambda _
>     (for-each (lambda (dir)
>                 (with-directory-excursion dir
>                   ((assoc-ref tree-sitter:%standard-phases 'build))))
>               '("ocaml" "interface"))))

I would go for a keyword argument (#:languages 'auto by default,
or an explicit list like #:languages '("ocaml" "interface") for when
autodetection does not work), since it seems more declarative and
conciser than overriding the build phase.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [PATCH v5 00/27] gnu: Add tree-sitter for emacs.

[Pierre Langlois]

Hi Guix!

Here's v5 of my tree-sitter series for emacs, highlights:

  * New tree-sitter build-system for building grammars, based on the node
    build-system.
  * Rebased on top of master, moving the new tree-sitter package in a new
    module.
  * Dynamically link to libtree-sitter.so from rust packages instead of
    bundling it.

I think it's in a pretty good shape now, WDYT?

Thanks,
Pierre

Pierre Langlois (27):
  gnu: tree-sitter: Move to its own module.
  gnu: Add tree-sitter-cli.
  build-system: Add tree-sitter-build-system.
  gnu: Add tree-sitter-c.
  gnu: Add tree-sitter-cpp.
  gnu: Add tree-sitter-bash.
  gnu: Add tree-sitter-css.
  gnu: Add tree-sitter-c-sharp.
  gnu: Add tree-sitter-elixir.
  gnu: Add tree-sitter-elm.
  gnu: Add tree-sitter-go.
  gnu: Add tree-sitter-html.
  gnu: Add tree-sitter-java.
  gnu: Add tree-sitter-javascript.
  gnu: Add tree-sitter-json.
  gnu: Add tree-sitter-julia.
  gnu: Add tree-sitter-ocaml.
  gnu: Add tree-sitter-php.
  gnu: Add tree-sitter-python.
  gnu: Add tree-sitter-ruby.
  gnu: Add tree-sitter-rust.
  gnu: Add tree-sitter-typescript.
  gnu: Add rust-tree-sitter.
  gnu: Add rust-tree-sitter-for-emacs.
  gnu: Add emacs-tree-sitter-core.
  gnu: Add emacs-tree-sitter.
  gnu: Add emacs-tree-sitter-langs.

 Makefile.am                                   |   2 +
 doc/guix.texi                                 |  21 +-
 gnu/local.mk                                  |   2 +
 .../rust-tree-sitter-text-provider-fix.patch  |  98 ++
 gnu/packages/text-editors.scm                 |  52 -
 gnu/packages/tree-sitter.scm                  | 932 ++++++++++++++++++
 gnu/packages/vim.scm                          |   1 +
 guix/build-system/tree-sitter.scm             | 190 ++++
 guix/build/tree-sitter-build-system.scm       | 153 +++
 9 files changed, 1398 insertions(+), 53 deletions(-)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
 create mode 100644 gnu/packages/tree-sitter.scm
 create mode 100644 guix/build-system/tree-sitter.scm
 create mode 100644 guix/build/tree-sitter-build-system.scm

--
2.34.0

[bug#49946] [PATCH v5 01/27] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

* gnu/packages/text-editors.scm (tree-sitter): Move to ...
* gnu/packages/tree-sitter.scm: ... here, a new module.
* gnu/packages/vim.scm: Use (gnu packages tree-sitter).
* gnu/local.mk (GNU_SYSTEM_MODULES): Register tree-sitter.scm new module.
---
 gnu/local.mk                  |  1 +
 gnu/packages/text-editors.scm | 52 -----------------------
 gnu/packages/tree-sitter.scm  | 78 +++++++++++++++++++++++++++++++++++
 gnu/packages/vim.scm          |  1 +
 4 files changed, 80 insertions(+), 52 deletions(-)
 create mode 100644 gnu/packages/tree-sitter.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index a704161abc..03972441e6 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -587,6 +587,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/tmux.scm				\
   %D%/packages/toolkits.scm			\
   %D%/packages/tor.scm				\
+  %D%/packages/tree-sitter.scm			\
   %D%/packages/tv.scm				\
   %D%/packages/uglifyjs.scm			\
   %D%/packages/uml.scm				\
diff --git a/gnu/packages/text-editors.scm b/gnu/packages/text-editors.scm
index 7cbad042c9..ddbfef3e77 100644
--- a/gnu/packages/text-editors.scm
+++ b/gnu/packages/text-editors.scm
@@ -63,7 +63,6 @@ (define-module (gnu packages text-editors)
   #:use-module (gnu packages gtk)
   #:use-module (gnu packages guile)
   #:use-module (gnu packages haskell-xyz)
-  #:use-module (gnu packages icu4c)
   #:use-module (gnu packages image)
   #:use-module (gnu packages libbsd)
   #:use-module (gnu packages libreoffice)
@@ -1167,54 +1166,3 @@ (define-public virtaal
 provided by the Translate Toolkit, including XLIFF and PO.")
     (home-page "https://virtaal.translatehouse.org/")
     (license license:gpl2+)))
-
-(define-public tree-sitter
-  (package
-    (name "tree-sitter")
-    (version "0.20.6")
-    (source (origin
-              (method git-fetch)
-              (uri (git-reference
-                    (url "https://github.com/tree-sitter/tree-sitter")
-                    (commit (string-append "v" version))))
-              (file-name (git-file-name name version))
-              (sha256
-               (base32
-                "1z20518snyg0zp75qgs5bxmzjqws4dd19vnp6sya494za3qp5b6d"))
-              (modules '((guix build utils)))
-              (snippet '(begin
-                          ;; Remove bundled ICU parts
-                          (delete-file-recursively "lib/src/unicode")
-                          #t))))
-    (build-system gnu-build-system)
-    (inputs (list icu4c))
-    (arguments
-     (list #:phases
-           '(modify-phases %standard-phases
-              (delete 'configure))
-           #:tests? #f ; there are no tests for the runtime library
-           #:make-flags
-           #~(list (string-append "PREFIX="
-                                  #$output)
-                   (string-append "CC="
-                                  #$(cc-for-target)))))
-    (home-page "https://tree-sitter.github.io/tree-sitter/")
-    (synopsis "Incremental parsing system for programming tools")
-    (description
-     "Tree-sitter is a parser generator tool and an incremental parsing
-library.  It can build a concrete syntax tree for a source file and efficiently
-update the syntax tree as the source file is edited.
-
-Tree-sitter aims to be:
-
-@itemize
-@item General enough to parse any programming language
-@item Fast enough to parse on every keystroke in a text editor
-@item Robust enough to provide useful results even in the presence of syntax errors
-@item Dependency-free so that the runtime library (which is written in pure C)
-can be embedded in any application
-@end itemize
-
-This package includes the @code{libtree-sitter} runtime library.
-")
-    (license license:expat)))
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
new file mode 100644
index 0000000000..7116a10ed7
--- /dev/null
+++ b/gnu/packages/tree-sitter.scm
@@ -0,0 +1,78 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Luis Henrique Gomes Higino <luishenriquegh2701@gmail.com>
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages tree-sitter)
+  #:use-module (guix gexp)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix git-download)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages icu4c))
+
+(define-public tree-sitter
+  (package
+    (name "tree-sitter")
+    (version "0.20.6")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1z20518snyg0zp75qgs5bxmzjqws4dd19vnp6sya494za3qp5b6d"))
+              (modules '((guix build utils)))
+              (snippet '(begin
+                          ;; Remove bundled ICU parts
+                          (delete-file-recursively "lib/src/unicode")
+                          #t))))
+    (build-system gnu-build-system)
+    (inputs (list icu4c))
+    (arguments
+     (list #:phases
+           '(modify-phases %standard-phases
+              (delete 'configure))
+           #:tests? #f ; there are no tests for the runtime library
+           #:make-flags
+           #~(list (string-append "PREFIX="
+                                  #$output)
+                   (string-append "CC="
+                                  #$(cc-for-target)))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Incremental parsing system for programming tools")
+    (description
+     "Tree-sitter is a parser generator tool and an incremental parsing
+library.  It can build a concrete syntax tree for a source file and efficiently
+update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@itemize
+@item General enough to parse any programming language
+@item Fast enough to parse on every keystroke in a text editor
+@item Robust enough to provide useful results even in the presence of syntax errors
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application
+@end itemize
+
+This package includes the @code{libtree-sitter} runtime library.
+")
+    (license license:expat)))
diff --git a/gnu/packages/vim.scm b/gnu/packages/vim.scm
index bc5e36c6f2..f2fc39586e 100644
--- a/gnu/packages/vim.scm
+++ b/gnu/packages/vim.scm
@@ -71,6 +71,7 @@ (define-module (gnu packages vim)
   #:use-module (gnu packages tcl)
   #:use-module (gnu packages text-editors)
   #:use-module (gnu packages terminals)
+  #:use-module (gnu packages tree-sitter)
   #:use-module (gnu packages xdisorg)
   #:use-module (gnu packages xorg))

--
2.34.0

[bug#49946] [PATCH v5 02/27] gnu: Add tree-sitter-cli.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cli): New variable.
---
 gnu/packages/tree-sitter.scm | 79 ++++++++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 7116a10ed7..18c9fe0844 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -20,10 +20,13 @@
 (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages crates-graphics)
+  #:use-module (gnu packages crates-io)
   #:use-module (gnu packages icu4c))

 (define-public tree-sitter
@@ -76,3 +79,79 @@ (define-public tree-sitter
 This package includes the @code{libtree-sitter} runtime library.
 ")
     (license license:expat)))
+
+(define-public tree-sitter-cli
+  (package (inherit tree-sitter)
+    (name "tree-sitter-cli")
+    (source (origin
+              (inherit (package-source tree-sitter))
+              (snippet
+               '(begin
+                  ;; Remove the runtime library code and dynamically link to
+                  ;; it instead.
+                  (delete-file-recursively "lib/src")
+                  (delete-file "lib/binding_rust/build.rs")
+                  (with-output-to-file "lib/binding_rust/build.rs"
+                    (lambda _
+                      (format #t "fn main() {~@
+                              println!(\"cargo:rustc-link-lib=tree-sitter\");~@
+                              }~%")))
+                  #t))))
+    (build-system cargo-build-system)
+    (inputs (list tree-sitter))
+    (arguments
+     `(;; Running test requires downloading fixtures, see the
+       ;; script/fetch-fixtures script.
+       #:tests? #f
+       ;; We're only packaging the CLI program so we do not need to install
+       ;; sources.
+       #:install-source? #f
+       #:cargo-inputs
+       (("rust-ansi-term" ,rust-ansi-term-0.12)
+        ("rust-anyhow" ,rust-anyhow-1)
+        ("rust-atty" ,rust-atty-0.2)
+        ("rust-clap" ,rust-clap-2)
+        ("rust-difference" ,rust-difference-2)
+        ("rust-html-escape" ,rust-html-escape-0.2)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-rand" ,rust-rand-0.8)
+        ("rust-rustc-hash" ,rust-rustc-hash-1)
+        ("rust-semver" ,rust-semver-1)
+        ("rust-smallbitvec" ,rust-smallbitvec-2)
+        ("rust-thiserror" ,rust-thiserror-1)
+        ("rust-tiny-http" ,rust-tiny-http-0.8)
+        ("rust-toml" ,rust-toml-0.5)
+        ("rust-walkdir" ,rust-walkdir-2)
+        ("rust-webbrowser" ,rust-webbrowser-0.5)
+        ("rust-which" ,rust-which-4))
+       #:cargo-development-inputs
+       (("rust-pretty-assertions" ,rust-pretty-assertions-0.7))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin"))
+                    (lib (string-append out "/lib")))
+               (mkdir-p bin)
+               (install-file "target/release/tree-sitter" bin)))))))
+    (synopsis "Incremental parsing system for programming tools")
+    (description "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @command{tree-sitter} command-line tool.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 04/27] gnu: Add tree-sitter-c.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-c): New variable.
---
 gnu/packages/tree-sitter.scm | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 18c9fe0844..3f07b2d91c 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -22,6 +22,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
+  #:use-module (guix build-system tree-sitter)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
@@ -155,3 +156,35 @@ (define-public tree-sitter-cli

 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))
+
+(define tree-sitter-delete-generated-files
+  '(begin
+     (delete-file "binding.gyp")
+     (delete-file-recursively "bindings")
+     (delete-file "src/grammar.json")
+     (delete-file "src/node-types.json")
+     (delete-file "src/parser.c")
+     (delete-file-recursively "src/tree_sitter")
+     #t))
+
+(define-public tree-sitter-c
+  (package
+    (name "tree-sitter-c")
+    (version "0.20.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-c")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0454jziys33i4kbwnvi9xcck0fzya792ghy32ahgk1hhv96xga9w"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-c")
+    (synopsis "Tree-sitter C grammar")
+    (description
+     "This package provides a C grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 05/27] gnu: Add tree-sitter-cpp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cpp): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 3f07b2d91c..5e25d48c05 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -188,3 +188,26 @@ (define-public tree-sitter-c
     (description
      "This package provides a C grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-cpp
+  (package
+    (name "tree-sitter-cpp")
+    (version "0.20.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-cpp")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0hxcpdvyyig8njga1mxp4qcnbbnr1d0aiy27vahijwbh98b081nr"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (inputs (list tree-sitter-c))
+    (home-page "https://github.com/tree-sitter/tree-sitter-cpp")
+    (synopsis "Tree-sitter C++ grammar")
+    (description
+     "This package provides a C++ grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 06/27] gnu: Add tree-sitter-bash.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-bash): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 5e25d48c05..084071fe3d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -211,3 +211,25 @@ (define-public tree-sitter-cpp
     (description
      "This package provides a C++ grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-bash
+  (package
+    (name "tree-sitter-bash")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-bash")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "18c030bb65r50i6z37iy7jb9z9i8i36y7b08dbc9bchdifqsijs5"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-bash")
+    (synopsis "Tree-sitter Bash grammar")
+    (description
+     "This package provides a Bash grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 07/27] gnu: Add tree-sitter-css.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-css): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 084071fe3d..d2f8326be0 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -233,3 +233,25 @@ (define-public tree-sitter-bash
     (description
      "This package provides a Bash grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-css
+  (package
+    (name "tree-sitter-css")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-css")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "014jrlgi7zfza9g38hsr4vlbi8964i5p7iglaih6qmzaiml7bja2"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-css")
+    (synopsis "Tree-sitter CSS grammar")
+    (description
+     "This package provides a CSS grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 08/27] gnu: Add tree-sitter-c-sharp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-c-sharp): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d2f8326be0..de90ac2093 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -212,6 +212,28 @@ (define-public tree-sitter-cpp
      "This package provides a C++ grammar for the Tree-sitter library.")
     (license license:expat)))

+(define-public tree-sitter-c-sharp
+  (package
+    (name "tree-sitter-c-sharp")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-c-sharp")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "054fmpf47cwh59gbg00sc0nl237ba4rnxi73miz39yqzcs87055r"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-c-sharp")
+    (synopsis "Tree-sitter C# grammar")
+    (description
+     "This package provides a C# grammar for the Tree-sitter library.")
+    (license license:expat)))
+
 (define-public tree-sitter-bash
   (package
     (name "tree-sitter-bash")
--
2.34.0

[bug#49946] [PATCH v5 09/27] gnu: Add tree-sitter-elixir.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-elixir): New variable.
---
 gnu/packages/tree-sitter.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index de90ac2093..18ea00f7cf 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -277,3 +277,29 @@ (define-public tree-sitter-css
     (description
      "This package provides a CSS grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-elixir
+  (let ((commit "de20391afe5cb03ef1e8a8e43167e7b58cc52869")
+        (revision "1"))
+    (package
+      (name "tree-sitter-elixir")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/elixir-lang/tree-sitter-elixir")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0zrkrwhw3g1vazsxcwrfd1fk4wvs9hdwmwp6073mfh370bz4140h"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+      (build-system tree-sitter-build-system)
+      (home-page "https://elixir-lang.org/tree-sitter-elixir/")
+      (synopsis "Tree-sitter Elixir grammar")
+      (description
+       "This package provides a Elixir grammar for the Tree-sitter library.")
+      (license (list license:asl2.0
+                     ;; Files in tests/corpus are under MIT license.
+                     license:expat)))))
--
2.34.0

[bug#49946] [PATCH v5 10/27] gnu: Add tree-sitter-elm.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-elm): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 18ea00f7cf..0aafaf7c1b 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -303,3 +303,25 @@ (define-public tree-sitter-elixir
       (license (list license:asl2.0
                      ;; Files in tests/corpus are under MIT license.
                      license:expat)))))
+
+(define-public tree-sitter-elm
+  (package
+    (name "tree-sitter-elm")
+    (version "5.5.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/elm-tooling/tree-sitter-elm")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "10hbi4vyj4hjixqswdcbvzl60prldczz29mlp02if61wvwiwvqrw"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://elm-tooling.github.io/tree-sitter-elm/")
+    (synopsis "Tree-sitter Elm grammar")
+    (description
+     "This package provides an Elm grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 11/27] gnu: Add tree-sitter-go.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-go): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 0aafaf7c1b..585b44b9f9 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -325,3 +325,25 @@ (define-public tree-sitter-elm
     (description
      "This package provides an Elm grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-go
+  (package
+    (name "tree-sitter-go")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-go")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0nxs47vd2fc2fr0qlxq496y852rwg39flhg334s7dlyq7d3lcx4x"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-go")
+    (synopsis "Tree-sitter Go grammar")
+    (description
+     "This package provides a Golang grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 12/27] gnu: Add tree-sitter-html.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-html): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 585b44b9f9..15f832f8ca 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -347,3 +347,25 @@ (define-public tree-sitter-go
     (description
      "This package provides a Golang grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-html
+  (package
+    (name "tree-sitter-html")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-html")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-html")
+    (synopsis "Tree-sitter HTML grammar")
+    (description
+     "This package provides a HTML grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 13/27] gnu: Add tree-sitter-java.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-java): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 15f832f8ca..010a7c3751 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -369,3 +369,25 @@ (define-public tree-sitter-html
     (description
      "This package provides a HTML grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-java
+  (package
+    (name "tree-sitter-java")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-java")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "07zw9ygb45hnvlx9qlz7rlz8hc3byjy03d24v72i5iyhpiiwlhvl"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-java")
+    (synopsis "Tree-sitter Java grammar")
+    (description
+     "This package provides a Java grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 14/27] gnu: Add tree-sitter-javascript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-javascript): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 010a7c3751..30220a5a8c 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -391,3 +391,26 @@ (define-public tree-sitter-java
     (description
      "This package provides a Java grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-javascript
+  (package
+    (name "tree-sitter-javascript")
+    (version "0.20.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-javascript")
+                    (commit (string-append "rust-" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "175yrk382n2di0c2xn4gpv8y4n83x1lg4hqn04vabf0yqynlkq67"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
+    (synopsis "Tree-sitter Javascript grammar")
+    (description
+     "This package provides Javascript and JSX grammars for the Tree-sitter
+library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 15/27] gnu: Add tree-sitter-json.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-json): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 30220a5a8c..d43e16af09 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -414,3 +414,25 @@ (define-public tree-sitter-javascript
      "This package provides Javascript and JSX grammars for the Tree-sitter
 library.")
     (license license:expat)))
+
+(define-public tree-sitter-json
+  (package
+    (name "tree-sitter-json")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-json")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "06pjh31bv9ja9hlnykk257a6zh8bsxg2fqa54al7qk1r4n9ksnff"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-json")
+    (synopsis "Tree-sitter JSON grammar")
+    (description
+     "This package provides a JSON grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 16/27] gnu: Add tree-sitter-julia.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-julia): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d43e16af09..a535d8bc4a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -436,3 +436,25 @@ (define-public tree-sitter-json
     (description
      "This package provides a JSON grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-julia
+  (package
+    (name "tree-sitter-julia")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-julia")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1pbnmvhy2gq4vg1b0sjzmjm4s2gsgdjh7h01yj8qrrqbcl29c463"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-julia")
+    (synopsis "Tree-sitter Julia grammar")
+    (description
+     "This package provides a Julia grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 17/27] gnu: Add tree-sitter-ocaml.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-ocaml): New variable.
---
 gnu/packages/tree-sitter.scm | 39 ++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index a535d8bc4a..ff98044cc1 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -458,3 +458,42 @@ (define-public tree-sitter-julia
     (description
      "This package provides a Julia grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-ocaml
+  (let ((commit "0348562f385bc2bd67ecf181425e1afd6d454192")
+        (revision "1"))
+    (package
+      (name "tree-sitter-ocaml")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-ocaml")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0iqmwcz3c2ai4gyx4xli1rhn6hi6a0f60dn20f8jas9ham9dc2df"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (delete-file "binding.gyp")
+                    (delete-file-recursively "bindings")
+                    (for-each
+                     (lambda (lang)
+                       (with-directory-excursion lang
+                         (delete-file "src/grammar.json")
+                         (delete-file "src/node-types.json")
+                         (delete-file "src/parser.c")
+                         (delete-file-recursively "src/tree_sitter")))
+                     '("ocaml" "interface"))
+                    #t))))
+      (build-system tree-sitter-build-system)
+      (arguments
+       (list
+        #:grammar-directories '("ocaml" "interface")))
+      (home-page "https://github.com/tree-sitter/tree-sitter-ocaml")
+      (synopsis "Tree-sitter OCaml grammar")
+      (description
+       "This package provides a OCaml grammar for the Tree-sitter library.")
+      (license license:expat))))
--
2.34.0

[bug#49946] [PATCH v5 18/27] gnu: Add tree-sitter-php.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-php): New variable.
---
 gnu/packages/tree-sitter.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index ff98044cc1..654445f71d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -497,3 +497,33 @@ (define-public tree-sitter-ocaml
       (description
        "This package provides a OCaml grammar for the Tree-sitter library.")
       (license license:expat))))
+
+(define-public tree-sitter-php
+  (let ((commit "435fa00006c0d1515c37fbb4dd6a9de284af75ab")
+        (revision "1"))
+    (package
+      (name "tree-sitter-php")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-php")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "05k4h58gi616gv41r0qqdb1x4rs8y94vghn2r10yczisgzq4vbad"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (delete-file "src/grammar.json")
+                    (delete-file "src/node-types.json")
+                    (delete-file "src/parser.c")
+                    (delete-file-recursively "src/tree_sitter")
+                    #t))))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-php")
+    (synopsis "Tree-sitter PHP grammar")
+    (description
+     "This package provides a PHP grammar for the Tree-sitter library.")
+    (license license:expat))))
--
2.34.0

[bug#49946] [PATCH v5 19/27] gnu: Add tree-sitter-python.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-python): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 654445f71d..c43cafea2d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -527,3 +527,27 @@ (define-public tree-sitter-php
     (description
      "This package provides a PHP grammar for the Tree-sitter library.")
     (license license:expat))))
+
+(define-public tree-sitter-python
+  (let ((commit "ed0fe62e55dc617ed9dec8817ebf771aa7cf3c42")
+        (revision "1"))
+    (package
+      (name "tree-sitter-python")
+      (version (git-version "0.19.1" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-python")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0wrfpg84mc3pzcrdi6n5fqwijkqr1nj5sqfnayb502krvqpjilal"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-python")
+    (synopsis "Tree-sitter Python grammar")
+    (description
+     "This package provides a Python grammar for the Tree-sitter library.")
+    (license license:expat))))
--
2.34.0

[bug#49946] [PATCH v5 20/27] gnu: Add tree-sitter-ruby.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-ruby): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c43cafea2d..c7b1998253 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -551,3 +551,25 @@ (define-public tree-sitter-python
     (description
      "This package provides a Python grammar for the Tree-sitter library.")
     (license license:expat))))
+
+(define-public tree-sitter-ruby
+  (package
+    (name "tree-sitter-ruby")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-ruby")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0m3h4928rbs300wcb6776h9r88hi32rybbhcaf6rdympl5nzi83v"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-ruby")
+    (synopsis "Tree-sitter Ruby grammar")
+    (description
+     "This package provides a Ruby grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 21/27] gnu: Add tree-sitter-rust.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-rust): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c7b1998253..c37183e131 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -573,3 +573,25 @@ (define-public tree-sitter-ruby
     (description
      "This package provides a Ruby grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-rust
+  (package
+    (name "tree-sitter-rust")
+    (version "0.20.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-rust")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "174j5pxwf80c4xniri39l3a6bb7nq96g2s8hh5sgv4i7xvbpfsmg"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-rust")
+    (synopsis "Tree-sitter Rust grammar")
+    (description
+     "This package provides a Rust grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 22/27] gnu: Add tree-sitter-typescript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-typescript): New variable.
---
 gnu/packages/tree-sitter.scm | 40 ++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c37183e131..756dc0d540 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -595,3 +595,43 @@ (define-public tree-sitter-rust
     (description
      "This package provides a Rust grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-typescript
+  (let ((commit "111b07762e86efab9a918b7c721f720c37e76b0a")
+        (revision "1"))
+    (package
+      (name "tree-sitter-typescript")
+      (version (git-version "0.20.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-typescript")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32 "1364bm3wbqqzvp03cvpx1w89bbqynb1v62i46gy7f6rhib9bf63a"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (delete-file "binding.gyp")
+                    (delete-file-recursively "bindings")
+                    (for-each
+                     (lambda (lang)
+                       (with-directory-excursion lang
+                         (delete-file "src/grammar.json")
+                         (delete-file "src/node-types.json")
+                         (delete-file "src/parser.c")
+                         (delete-file-recursively "src/tree_sitter")))
+                     '("typescript" "tsx"))
+                    #t))))
+      (build-system tree-sitter-build-system)
+      (inputs (list tree-sitter-javascript))
+      (arguments
+       (list
+        #:grammar-directories '("typescript" "tsx")))
+      (home-page "https://github.com/tree-sitter/tree-sitter-typescript")
+      (synopsis "Tree-sitter Typescript grammar")
+      (description
+       "This package provides Typescript and TSX grammars for the Tree-sitter
+library.")
+      (license license:expat))))
--
2.34.0

[bug#49946] [PATCH v5 23/27] gnu: Add rust-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 39 ++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 756dc0d540..389cb944b9 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -23,6 +23,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
+  #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
@@ -157,6 +158,44 @@ (define-public tree-sitter-cli
 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))

+(define-public rust-tree-sitter
+  (package
+    (name "rust-tree-sitter")
+    (version "0.20.6")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "tree-sitter" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "0wcvxgnvj7ga1y7xa7wm0pmabkfj8936ifg8jacd4201cj0vgcq9"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Remove the runtime library code and dynamically link to
+                  ;; it instead.
+                  (delete-file-recursively "src")
+                  (delete-file "binding_rust/build.rs")
+                  (with-output-to-file "binding_rust/build.rs"
+                    (lambda _
+                      (format #t "fn main() {~@
+                              println!(\"cargo:rustc-link-lib=tree-sitter\");~@
+                              }~%")))
+                  #t))))
+    (build-system cargo-build-system)
+    (inputs (list tree-sitter))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-cc" ,rust-cc-1)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-regex" ,rust-regex-1)
+        ("rust-spin" ,rust-spin-0.7))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Rust bindings to the Tree-sitter parsing library")
+    (description "This package provides Rust bindings to the Tree-sitter
+parsing library.")
+    (license license:expat)))
+
 (define tree-sitter-delete-generated-files
   '(begin
      (delete-file "binding.gyp")
--
2.34.0

[bug#49946] [PATCH v5 24/27] gnu: Add rust-tree-sitter-for-emacs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter-for-emacs): New variable.
* gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
---
 gnu/local.mk                                  |  1 +
 .../rust-tree-sitter-text-provider-fix.patch  | 98 +++++++++++++++++++
 gnu/packages/tree-sitter.scm                  | 14 +++
 3 files changed, 113 insertions(+)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 03972441e6..aa353a6f68 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1780,6 +1780,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/rust-nettle-sys-disable-vendor.patch	 \
   %D%/packages/patches/rust-openssl-sys-no-vendor.patch	\
   %D%/packages/patches/rust-shell2batch-lint-fix.patch		\
+  %D%/packages/patches/rust-tree-sitter-text-provider-fix.patch	\
   %D%/packages/patches/rust-wl-clipboard-rs-newer-wl.patch      \
   %D%/packages/patches/sbc-fix-build-non-x86.patch		\
   %D%/packages/patches/sbcl-aserve-add-HTML-5-elements.patch	\
diff --git a/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
new file mode 100644
index 0000000000..215e7fc18b
--- /dev/null
+++ b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
@@ -0,0 +1,98 @@
+From e3576b180488b8231e1fc0ca130748577579d129 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Tu=E1=BA=A5n-Anh=20Nguy=E1=BB=85n?= <ubolonton@gmail.com>
+Date: Sun, 25 Jul 2021 13:11:52 +0700
+Subject: [PATCH] Allow TextProvider's iterators to generate owned text
+
+---
+ binding_rust/lib.rs | 33 ++++++++++++++++++---------------
+ 1 file changed, 18 insertions(+), 15 deletions(-)
+
+diff --git a/lib/binding_rust/lib.rs b/lib/binding_rust/lib.rs
+index e88a411c..cf214d92 100644
+--- a/binding_rust/lib.rs
++++ b/binding_rust/lib.rs
+@@ -5,6 +5,7 @@ mod util;
+ use std::os::unix::io::AsRawFd;
+
+ use std::{
++    borrow::Cow,
+     char, error,
+     ffi::CStr,
+     fmt, hash, iter,
+@@ -183,7 +184,8 @@ pub struct QueryCaptures<'a, 'tree: 'a, T: TextProvider<'a>> {
+ }
+
+ pub trait TextProvider<'a> {
+-    type I: Iterator<Item = &'a [u8]> + 'a;
++    type I: Iterator<Item = Cow<'a, [u8]>>;
++
+     fn text(&mut self, node: Node) -> Self::I;
+ }
+
+@@ -1840,19 +1842,19 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+         buffer2: &mut Vec<u8>,
+         text_provider: &mut impl TextProvider<'a>,
+     ) -> bool {
+-        fn get_text<'a, 'b: 'a, I: Iterator<Item = &'b [u8]>>(
++        fn get_text<'a, 'b: 'a, I: Iterator<Item = Cow<'b, [u8]>>>(
+             buffer: &'a mut Vec<u8>,
+             mut chunks: I,
+-        ) -> &'a [u8] {
+-            let first_chunk = chunks.next().unwrap_or(&[]);
++        ) -> Cow<'a, [u8]> {
++            let first_chunk = chunks.next().unwrap_or(Cow::Owned(vec![0u8; 0]));
+             if let Some(next_chunk) = chunks.next() {
+                 buffer.clear();
+-                buffer.extend_from_slice(first_chunk);
+-                buffer.extend_from_slice(next_chunk);
++                buffer.extend_from_slice(&first_chunk);
++                buffer.extend_from_slice(&next_chunk);
+                 for chunk in chunks {
+-                    buffer.extend_from_slice(chunk);
++                    buffer.extend_from_slice(&chunk);
+                 }
+-                buffer.as_slice()
++                Cow::Borrowed(buffer.as_slice())
+             } else {
+                 first_chunk
+             }
+@@ -1888,7 +1890,7 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+                     match node {
+                         Some(node) => {
+                             let text = get_text(buffer1, text_provider.text(node));
+-                            r.is_match(text) == *is_positive
++                            r.is_match(&text) == *is_positive
+                         }
+                         None => true,
+                     }
+@@ -2002,23 +2004,24 @@ impl<'cursor, 'tree> fmt::Debug for QueryMatch<'cursor, 'tree> {
+     }
+ }
+
+-impl<'a, F, I> TextProvider<'a> for F
++impl<'a, F, I, T> TextProvider<'a> for F
+ where
+     F: FnMut(Node) -> I,
+-    I: Iterator<Item = &'a [u8]> + 'a,
++    T: Into<Cow<'a, [u8]>>,
++    I: Iterator<Item = T>,
+ {
+-    type I = I;
++    type I = iter::Map<I, fn(T) -> Cow<'a, [u8]>>;
+
+     fn text(&mut self, node: Node) -> Self::I {
+-        (self)(node)
++        (self)(node).map(T::into)
+     }
+ }
+
+ impl<'a> TextProvider<'a> for &'a [u8] {
+-    type I = iter::Once<&'a [u8]>;
++    type I = iter::Once<Cow<'a, [u8]>>;
+
+     fn text(&mut self, node: Node) -> Self::I {
+-        iter::once(&self[node.byte_range()])
++        iter::once(Cow::Borrowed(&self[node.byte_range()]))
+     }
+ }
+
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 389cb944b9..5209876431 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -27,6 +27,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages icu4c))
@@ -196,6 +197,19 @@ (define-public rust-tree-sitter
 parsing library.")
     (license license:expat)))

+;; We need to apply a patch in order to compile the rust bindings against the
+;; emacs tree-sitter module.
+;; See https://github.com/tree-sitter/tree-sitter/pull/1294
+(define-public rust-tree-sitter-for-emacs
+  (package (inherit rust-tree-sitter)
+    (source (origin
+              (inherit (package-source rust-tree-sitter))
+              (patches (search-patches
+                        "rust-tree-sitter-text-provider-fix.patch"))))
+    ;; Do not show this package in the UI as it's only meant to be used for
+    ;; emacs's tree-sitter module.
+    (properties '((hidden? . #t)))))
+
 (define tree-sitter-delete-generated-files
   '(begin
      (delete-file "binding.gyp")
--
2.34.0

[bug#49946] [PATCH v5 25/27] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
(emacs-tree-sitter-core): New variable.
---
 gnu/packages/tree-sitter.scm | 78 ++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 5209876431..fb3d13b3c7 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -21,6 +21,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
+  #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
   #:use-module (guix download)
@@ -688,3 +689,80 @@ (define-public tree-sitter-typescript
        "This package provides Typescript and TSX grammars for the Tree-sitter
 library.")
       (license license:expat))))
+
+;; Local package definition solely for building the native emacs module
+;; written in Rust.
+(define tree-sitter-emacs-module
+  (package
+    (name "tree-sitter-emacs-module")
+    (version "0.18.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1sdvz827v436qijs6xafakkfw2d16bvp8frymd818rppjc7a9dif"))))
+    (build-system cargo-build-system)
+    (inputs
+     (list tree-sitter))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-emacs" ,rust-emacs-0.18)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-tree-sitter" ,rust-tree-sitter-for-emacs))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'chdir 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (add-after 'delete-cargo.lock 'do-not-fetch-from-github
+           (lambda _
+             (substitute* "Cargo.toml"
+               (("\\[patch.*") "")
+               (("git = .*") ""))))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out") "/lib")))
+               (install-file "target/release/libtsc_dyn.so" lib)))))))
+    (home-page #f)
+    (synopsis #f)
+    (description #f)
+    (license license:expat)))
+
+(define-public emacs-tree-sitter-core
+  (package
+    (name "emacs-tree-sitter-core")
+    (version (package-version tree-sitter-emacs-module))
+    (source (package-source tree-sitter-emacs-module))
+    (build-system emacs-build-system)
+    (native-inputs
+     (list tree-sitter-emacs-module))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'install 'install-module
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out")))
+                   (module (string-append (assoc-ref inputs "tree-sitter-emacs-module")
+                                          "/lib/libtsc_dyn.so")))
+               ;; Writing the version number in this file prevents the package
+               ;; from trying to download the module from the internet.
+               (call-with-output-file (string-append elpa "/DYN-VERSION")
+                 (lambda (port) (display ,version port)))
+               (substitute* "tsc-dyn-get.el"
+                 (("defcustom tsc-dyn-dir tsc--dir")
+                  (string-append "defcustom tsc-dyn-dir \"" elpa "\"")))
+               (copy-file module (string-append elpa "/tsc-dyn.so"))))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp, core library")
+    (description "This package provides core APIs of the Emacs binding for
+Tree-sitter, an incremental parsing system.")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 26/27] gnu: Add emacs-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index fb3d13b3c7..97a20494ec 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -766,3 +766,30 @@ (module (string-append (assoc-ref inputs "tree-sitter-emacs-module")
     (description "This package provides core APIs of the Emacs binding for
 Tree-sitter, an incremental parsing system.")
     (license license:expat)))
+
+(define-public emacs-tree-sitter
+  (package
+    (name "emacs-tree-sitter")
+    (version (package-version emacs-tree-sitter-core))
+    (source (package-source emacs-tree-sitter-core))
+    (build-system emacs-build-system)
+    (propagated-inputs
+     (list emacs-tree-sitter-core))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "lisp"))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp")
+    (description "This package provides Emacs bindings for Tree-sitter, an
+incremental parsing library.  It aims to be the foundation for a new breed of
+Emacs packages that understand code structurally.  For example:
+
+@enumerate
+@item Faster, fine-grained code highlighting.
+@item More flexible code folding.
+@item Structural editing (like Paredit, or even better) for non-Lisp code.
+@item More informative indexing for imenu.
+@end enumerate")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 27/27] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (make-emacs-tree-sitter-langs-grammar-bundle):
New procedure.
(emacs-tree-sitter-langs): New variable.
---
 gnu/packages/tree-sitter.scm | 137 +++++++++++++++++++++++++++++++++++
 1 file changed, 137 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 97a20494ec..ae0478c3e4 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -24,6 +24,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
+  #:use-module (guix build-system trivial)
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
@@ -793,3 +794,139 @@ (define-public emacs-tree-sitter
 @item More informative indexing for imenu.
 @end enumerate")
     (license license:expat)))
+
+(define (make-emacs-tree-sitter-langs-grammar-bundle version)
+  (package
+    (name "emacs-tree-sitter-langs-grammar-bundle")
+    (source #f)
+    (version version)
+    (build-system trivial-build-system)
+    (inputs
+     ;; FIXME: Support for some languages is still left to package.
+     (list tree-sitter-bash
+           tree-sitter-c
+           tree-sitter-c-sharp
+           tree-sitter-cpp
+           tree-sitter-css
+           tree-sitter-elixir
+           tree-sitter-elm
+           tree-sitter-go
+           tree-sitter-html
+           tree-sitter-java
+           tree-sitter-javascript
+           tree-sitter-json
+           tree-sitter-julia
+           tree-sitter-ocaml
+           tree-sitter-php
+           tree-sitter-python
+           tree-sitter-rust
+           tree-sitter-ruby
+           tree-sitter-typescript))
+    (arguments
+     (list #:builder
+           (with-imported-modules '((guix build union)
+                                    (guix build utils))
+             #~(begin
+                 (use-modules (ice-9 match)
+                              (guix build union)
+                              (guix build utils))
+                 (union-build
+                  #$output
+                  (filter directory-exists?
+                          (map (match-lambda
+                                 ((name directory)
+                                  (string-append directory "/lib/tree-sitter")))
+                               '#$(package-inputs this-package))))
+                  ;; The BUNDLE-VERSION file prevents emacs-tree-sitter-langs
+                  ;; from downloading libraries at load time.
+                  (call-with-output-file (string-append #$output "/BUNDLE-VERSION")
+                    (lambda (port) (display #$version port)))))))
+    (synopsis #f)
+    (description #f)
+    (home-page #f)
+    (license #f)))
+
+(define-public emacs-tree-sitter-langs
+  (package
+    (name "emacs-tree-sitter-langs")
+    (version "0.11.6")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/tree-sitter-langs")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0jyzpiyl3b88vwfzkda7ak77kl0b2nybvsz0j9cndl3pf95d62g0"))))
+    (build-system emacs-build-system)
+    (inputs
+     (list (make-emacs-tree-sitter-langs-grammar-bundle version)))
+    (propagated-inputs
+     (list emacs-tree-sitter))
+    (arguments
+     (list
+      #:tests? #t
+      #:test-command ''("script/test")
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'remove-cask
+            (lambda _
+              (substitute* "script/test"
+                (("cask") ""))))
+          (add-before 'check 'bundle-for-testing
+            (lambda* (#:key inputs #:allow-other-keys)
+              (delete-file-recursively "bin")
+              (symlink #$(make-emacs-tree-sitter-langs-grammar-bundle version)
+                       "bin")))
+          (add-before 'check 'patch-tree-sitter-require-test
+            (lambda _
+              (use-modules (ice-9 regex))
+              ;; This test needs a git repositories with submodules for
+              ;; each languages in order to map all repositories.  We patch
+              ;; the mapping function with one that invokes the tests for each
+              ;; packaged language.
+              (let ((supported-languages
+                     (map (lambda (lib)
+                            (match:substring
+                             (string-match "(.*)\\.so$" (basename lib))
+                             1))
+                          (find-files "bin" "\\.so$"))))
+                (substitute* "tree-sitter-langs-tests.el"
+                  (("tree-sitter-langs--map-repos")
+                   (call-with-output-string
+                     (lambda (port)
+                       (write `(lambda (fn)
+                                 (dolist (lang ',supported-languages)
+                                         (funcall fn lang)))
+                              port))))))))
+          ;; Tests for queries will fail given those languages are not
+          ;; packages yet.
+          (add-before 'check 'remove-unused-highlight-queries
+            (lambda _
+              (delete-file-recursively "queries/hcl")
+              (delete-file-recursively "queries/pgn")))
+          (add-before 'install 'install-bundle
+            (lambda _
+              (let ((elpa (elpa-directory #$output)))
+                (mkdir-p elpa)
+                (symlink #$(make-emacs-tree-sitter-langs-grammar-bundle version)
+                         (string-append elpa "/bin")))))
+          (add-after 'install 'install-queries
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+                (copy-recursively "queries" (string-append elpa "/queries"))))))))
+    (home-page "https://ubolonton.github.io/emacs-tree-sitter/languages/")
+    (synopsis "Language support bundle for Tree-sitter")
+    (description "This package is a convenient language bundle for
+Tree-sitter.  For each supported language, this package provides:
+
+@enumerate
+@item Pre-compiled grammar binaries.
+@item An optional highlights.scm file that provides highlighting patterns.
+This is mainly intended for major modes that are not aware of tree-sitter.
+@item Optional query patterns for other minor modes that provide high-level
+functionalities on top of tree-sitter, such as code folding, evil text
+objects, ...etc.
+@end enumerate")
+    (license license:expat)))
--
2.34.0

[bug#49946] [PATCH v5 27/27] gnu: Add emacs-tree-sitter-langs.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 534 bytes --]

Pierre Langlois schreef op di 29-03-2022 om 20:43 [+0100]:
> +                  ;; The BUNDLE-VERSION file prevents emacs-tree-sitter-langs
> +                  ;; from downloading libraries at load time.

WDYT of patching emacs-tree-sitter-langs to not download, such that it
doesn't download and run non-Guix libraries behind the user's back?

Also, why do we need a bundle at all, would simply installing emacs-
tree-sitter, and, e.g., tree-sitter-java, just work?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [PATCH v5 01/27] gnu: tree-sitter: Move to its own module.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 195 bytes --]

Pierre Langlois schreef op di 29-03-2022 om 20:42 [+0100]:
> -(define-public tree-sitter
> -  (package

Is my local guix too old, or is this package not yet in guix?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [PATCH v5 01/27] gnu: tree-sitter: Move to its own module.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 297 bytes --]

Pierre Langlois schreef op di 29-03-2022 om 20:42 [+0100]:
> -@item Dependency-free so that the runtime library (which is written in pure C)

It probably depends on glibc library, so strictly speaking this is
probably not technically correct ... but maybe good enough?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [PATCH v5 01/27] gnu: tree-sitter: Move to its own module.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 285 bytes --]

Maxime Devos schreef op di 29-03-2022 om 21:57 [+0200]:
> Pierre Langlois schreef op di 29-03-2022 om 20:42 [+0100]:
> > -(define-public tree-sitter
> > -  (package
> 
> Is my local guix too old, or is this package not yet in guix?

Nevermind, looks like my Guix was too old.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] State of emacs-tree-sitter

[Mekeor Melire]

Hello,

what's the current state of this set of patches?

Pierre, do you happen to have a channel or so where I could try out the
patch-series?

[bug#49946] State of emacs-tree-sitter

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

Hi,

Mekeor Melire <mekeor@posteo.de> writes:

> Hello,
>
> what's the current state of this set of patches?

I'm still working on it, sorry I've not had too much time for it these
days.

>
> Pierre, do you happen to have a channel or so where I could try out the
> patch-series?

I don't have a channel, however I just pushed the series rebased on top
of master on gitlab here: https://gitlab.com/planglois/guix/-/commits/emacs-tree-sitter

So if you'd like to test, you could use this instead of applying
patches:

  guix pull --url=https://gitlab.com/planglois/guix --branch=emacs-tree-sitter

I suppose I could move to a channel, but that's a little bit more work,
and I'd delete it immediately after we have something upstream. 

Let me know how testing goes!

Pierre


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v5 27/27] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

[-- Attachment #1.1: Type: text/plain, Size: 2930 bytes --]

Hi Maxime, sorry for the late reply!

Maxime Devos <maximedevos@telenet.be> writes:

> [[PGP Signed Part:Undecided]]
> Pierre Langlois schreef op di 29-03-2022 om 20:43 [+0100]:
>> +                  ;; The BUNDLE-VERSION file prevents emacs-tree-sitter-langs
>> +                  ;; from downloading libraries at load time.
>
> WDYT of patching emacs-tree-sitter-langs to not download, such that it
> doesn't download and run non-Guix libraries behind the user's back?

The way the current patchset works, by providing a compatible bundle, we
already prevent emacs-tree-sitter-langs from downloading binaries by
default.  I agree we could go further though, and entirely remove the
code that downloads binaries.

However I'm not sure about it.  Essentially, there is a
tree-sitter-langs-build.el file [0] that can either download binaries or
fetch sources and compile locally.  So a user could decide to opt-out of
using Guix binaries and instead use an alternative method.

I 100% agree that by default we should make the package use Guix
libraries, this way it also /just works/.  However, if somebody wants to
do things differently, I'm not sure we want to get in the way.  In the
end, this isn't so different from letting people use an alternative
package manager if they like.  We'd rather people used Guix of course :-).

I don't have a really strong opinion about this though, so if you still
prefer to delete the downloading code, I've attached an example patch
that entirely replace it with a bare-bones implementation as an example.

Let me know what you think!

[0]: https://github.com/emacs-tree-sitter/tree-sitter-langs/blob/master/tree-sitter-langs-build.el

> Also, why do we need a bundle at all, would simply installing emacs-
> tree-sitter, and, e.g., tree-sitter-java, just work?

Yeah, having a bundle is fundamentally how this package works AFAICT.

I think the main reason is that in order to do highlighting effectively
in emacs, it's not enough to install the tree-sitter runtime and a
tree-sitter-<lang> grammar.  You also want to "configure" how the
highlighting is done by providing custom "queries" expressions.

And this package provides queries for every language that it supports:
https://github.com/emacs-tree-sitter/tree-sitter-langs/tree/master/queries

All that being said, I believe that long-term the idea is that upstream
language-specific packages would eventually gain support for tree-sitter
and then this bundle "glue" package will no longer be necessary.
Especially if one day emacs proper gains native support for tree-sitter
(I think I saw some discussions about that on emacs-devel last year).

But given this package is quite useful though, I'd be surprised if it
goes away soon.

Hope this makes sense!

Thanks,
Pierre

>
> Greetings,
> Maxime.
>
> [[End of PGP Signed Part]]


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-wip.patch --]
[-- Type: text/x-patch, Size: 5034 bytes --]

From 7ad62ccef2446011dfbdfb2dbe8cc58f46fb05d8 Mon Sep 17 00:00:00 2001
From: Pierre Langlois <pierre.langlois@gmx.com>
Date: Sat, 2 Apr 2022 19:22:52 +0100
Subject: [PATCH] wip

---
 gnu/packages/tree-sitter.scm | 49 ++++++++++++++++++++++--------------
 1 file changed, 30 insertions(+), 19 deletions(-)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 7e14ebd1e3..d6892db6e3 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -795,11 +795,11 @@ (define-public emacs-tree-sitter
 @end enumerate")
     (license license:expat)))

-(define (make-emacs-tree-sitter-langs-grammar-bundle version)
+(define emacs-tree-sitter-langs-grammar-bundle
   (package
     (name "emacs-tree-sitter-langs-grammar-bundle")
     (source #f)
-    (version version)
+    (version (package-version tree-sitter))
     (build-system trivial-build-system)
     (inputs
      ;; FIXME: Support for some languages is still left to package.
@@ -836,11 +836,7 @@ (define (make-emacs-tree-sitter-langs-grammar-bundle version)
                           (map (match-lambda
                                  ((name directory)
                                   (string-append directory "/lib/tree-sitter")))
-                               '#$(package-inputs this-package))))
-                  ;; The BUNDLE-VERSION file prevents emacs-tree-sitter-langs
-                  ;; from downloading libraries at load time.
-                  (call-with-output-file (string-append #$output "/BUNDLE-VERSION")
-                    (lambda (port) (display #$version port)))))))
+                               '#$(package-inputs this-package))))))))
     (synopsis #f)
     (description #f)
     (home-page #f)
@@ -861,7 +857,7 @@ (define-public emacs-tree-sitter-langs
                 "1p2zbb6ac7wi6x6zpbczcmpkb2p45md2csd2bj43d8s56ckzw5mp"))))
     (build-system emacs-build-system)
     (inputs
-     (list (make-emacs-tree-sitter-langs-grammar-bundle version)))
+     (list emacs-tree-sitter-langs-grammar-bundle))
     (propagated-inputs
      (list emacs-tree-sitter))
     (arguments
@@ -870,15 +866,36 @@ (define-public emacs-tree-sitter-langs
       #:test-command ''("script/test")
       #:phases
       #~(modify-phases %standard-phases
+          (add-after 'unpack 'disable-downloader
+            (lambda _
+              (call-with-output-file "tree-sitter-langs-build.el"
+                (lambda (port)
+                  (let ((on-load-message
+                         (string-append
+                          "tree-sitter-langs: Grammar bundle already installed "
+                          "via Guix.  Installing external grammars via this "
+                          "function isn't supported, if a language you need is "
+                          "missing please report a bug at bug-guix@gnu.org.")))
+                    (format
+                     port
+                     ";;;###autoload
+                      (defun tree-sitter-langs-install-grammars
+                             (&optional skip-if-installed version os
+                                        keep-bundle)
+                        (interactive)
+                        (message \"~a\"))
+                      (defconst tree-sitter-langs--queries-dir
+                        (file-name-as-directory
+                          (concat (file-name-directory (locate-library \"tree-sitter-langs.el\"))
+                                  \"queries\")))
+                      (defun tree-sitter-langs--bin-dir () \"~a\")
+                      (provide 'tree-sitter-langs-build)"
+                     on-load-message
+                     #$emacs-tree-sitter-langs-grammar-bundle))))))
           (add-after 'unpack 'remove-cask
             (lambda _
               (substitute* "script/test"
                 (("cask") ""))))
-          (add-before 'check 'bundle-for-testing
-            (lambda* (#:key inputs #:allow-other-keys)
-              (delete-file-recursively "bin")
-              (symlink #$(make-emacs-tree-sitter-langs-grammar-bundle version)
-                       "bin")))
           (add-before 'check 'patch-tree-sitter-require-test
             (lambda _
               (use-modules (ice-9 regex))
@@ -906,12 +923,6 @@ (define-public emacs-tree-sitter-langs
             (lambda _
               (delete-file-recursively "queries/hcl")
               (delete-file-recursively "queries/pgn")))
-          (add-before 'install 'install-bundle
-            (lambda _
-              (let ((elpa (elpa-directory #$output)))
-                (mkdir-p elpa)
-                (symlink #$(make-emacs-tree-sitter-langs-grammar-bundle version)
-                         (string-append elpa "/bin")))))
           (add-after 'install 'install-queries
             (lambda* (#:key outputs #:allow-other-keys)
               (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
--
2.36.0

[bug#49946] [PATCH v5 01/27] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]


Maxime Devos <maximedevos@telenet.be> writes:

> [[PGP Signed Part:Undecided]]
> Pierre Langlois schreef op di 29-03-2022 om 20:42 [+0100]:
>> -@item Dependency-free so that the runtime library (which is written in pure C)
>
> It probably depends on glibc library, so strictly speaking this is
> probably not technically correct ... but maybe good enough?

Oh actually, looking at a generated parser.c file, that statement appears to be
true, I was also surprised.  The generated C code looks like a huge
table of rules, so it's more data than code really. For example:
https://raw.githubusercontent.com/tree-sitter/tree-sitter-c/master/src/parser.c

I suppose it was easier to generate C code and compile it, rather than
define a custom binary format.

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v5 27/27] gnu: Add emacs-tree-sitter-langs.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 756 bytes --]

Pierre Langlois schreef op zo 15-05-2022 om 13:20 [+0100]:
> I 100% agree that by default we should make the package use Guix
> libraries, this way it also /just works/.  However, if somebody wants
> to
> do things differently, I'm not sure we want to get in the way.  In
> the
> end, this isn't so different from letting people use an alternative
> package manager if they like.  We'd rather people used Guix of course
> :-).

OK, my proposal to prevent any accidents (what if we misinterpreted
BUNDLE-VERSION / the semantics change in a future version, and now
binaries are downloaded by default), not to stop people from choosing
to do.  Seems a bit risky though (see the bit about future changes in
semantics).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [PATCH v5 27/27] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 1365 bytes --]


Maxime Devos <maximedevos@telenet.be> writes:

> [[PGP Signed Part:Undecided]]
> Pierre Langlois schreef op zo 15-05-2022 om 13:20 [+0100]:
>> I 100% agree that by default we should make the package use Guix
>> libraries, this way it also /just works/.  However, if somebody wants
>> to
>> do things differently, I'm not sure we want to get in the way.  In
>> the
>> end, this isn't so different from letting people use an alternative
>> package manager if they like.  We'd rather people used Guix of course
>> :-).
>
> OK, my proposal to prevent any accidents (what if we misinterpreted
> BUNDLE-VERSION / the semantics change in a future version, and now
> binaries are downloaded by default), not to stop people from choosing
> to do.  Seems a bit risky though (see the bit about future changes in
> semantics).

Ah I see, that's a good point.  I /think/ today we should be able to
catch such things thanks to the tests, if the package decides to
download binaries during testing it'll fail.  But maybe it could, for
some reason, use Guix binaries during testing and then decide to
download its own when running outside of the build environment.

Now I'm conlicted :-), maybe rewriting the build.el file is the safest
option, and maybe if we get bug reports from people who would like to
use their own grammars we could revisit?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v5 25/27] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 4184 bytes --]


Pierre Langlois <pierre.langlois@gmx.com> writes:

> * gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
> (emacs-tree-sitter-core): New variable.
> ---
>  gnu/packages/tree-sitter.scm | 78 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 78 insertions(+)
>
> diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
> index 5209876431..fb3d13b3c7 100644
> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm
> @@ -21,6 +21,7 @@ (define-module (gnu packages tree-sitter)
>    #:use-module (guix gexp)
>    #:use-module ((guix licenses) #:prefix license:)
>    #:use-module (guix build-system cargo)
> +  #:use-module (guix build-system emacs)
>    #:use-module (guix build-system gnu)
>    #:use-module (guix build-system tree-sitter)
>    #:use-module (guix download)
> @@ -688,3 +689,80 @@ (define-public tree-sitter-typescript
>         "This package provides Typescript and TSX grammars for the Tree-sitter
>  library.")
>        (license license:expat))))
> +
> +;; Local package definition solely for building the native emacs module
> +;; written in Rust.
> +(define tree-sitter-emacs-module
> +  (package
> +    (name "tree-sitter-emacs-module")
> +    (version "0.18.0")
> +    (source (origin
> +              (method git-fetch)
> +              (uri (git-reference
> +                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
> +                    (commit version)))
> +              (file-name (git-file-name name version))
> +              (sha256
> +               (base32
> +                "1sdvz827v436qijs6xafakkfw2d16bvp8frymd818rppjc7a9dif"))))
> +    (build-system cargo-build-system)
> +    (inputs
> +     (list tree-sitter))
> +    (arguments
> +     `(#:cargo-inputs
> +       (("rust-anyhow" ,rust-anyhow-1)
> +        ("rust-emacs" ,rust-emacs-0.18)
> +        ("rust-libloading" ,rust-libloading-0.7)
> +        ("rust-once-cell" ,rust-once-cell-1)
> +        ("rust-tree-sitter" ,rust-tree-sitter-for-emacs))
> +       #:phases
> +       (modify-phases %standard-phases
> +         (add-after 'unpack 'chdir
> +           (lambda _ (chdir "core")))
> +         (add-after 'chdir 'delete-cargo.lock
> +           (lambda _ (delete-file "Cargo.lock")))
> +         (add-after 'delete-cargo.lock 'do-not-fetch-from-github
> +           (lambda _
> +             (substitute* "Cargo.toml"
> +               (("\\[patch.*") "")
> +               (("git = .*") ""))))
> +         (replace 'install
> +           (lambda* (#:key outputs #:allow-other-keys)
> +             (let ((lib (string-append (assoc-ref outputs "out") "/lib")))
> +               (install-file "target/release/libtsc_dyn.so" lib)))))))
> +    (home-page #f)
> +    (synopsis #f)
> +    (description #f)
> +    (license license:expat)))
> +
> +(define-public emacs-tree-sitter-core
> +  (package
> +    (name "emacs-tree-sitter-core")
> +    (version (package-version tree-sitter-emacs-module))
> +    (source (package-source tree-sitter-emacs-module))
> +    (build-system emacs-build-system)
> +    (native-inputs
> +     (list tree-sitter-emacs-module))
> +    (arguments
> +     `(#:phases
> +       (modify-phases %standard-phases
> +         (add-after 'unpack 'chdir
> +           (lambda _ (chdir "core")))
> +         (add-after 'install 'install-module
> +           (lambda* (#:key inputs outputs #:allow-other-keys)
> +             (let ((elpa (elpa-directory (assoc-ref outputs "out")))
> +                   (module (string-append (assoc-ref inputs "tree-sitter-emacs-module")
> +                                          "/lib/libtsc_dyn.so")))
> +               ;; Writing the version number in this file prevents the package
> +               ;; from trying to download the module from the internet.
> +               (call-with-output-file (string-append elpa "/DYN-VERSION")
> +                 (lambda (port) (display ,version port)))
> +               (substitute* "tsc-dyn-get.el"
> +                 (("defcustom tsc-dyn-dir tsc--dir")
> +                  (string-append "defcustom tsc-dyn-dir \"" elpa "\"")))

note to self: We can use emacs-subsitute-variables here.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v5 27/27] gnu: Add emacs-tree-sitter-langs.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

Pierre Langlois schreef op zo 15-05-2022 om 15:55 [+0100]:
> [...]
> 
> Ah I see, that's a good point.  I /think/ today we should be able to
> catch such things thanks to the tests, if the package decides to
> download binaries during testing it'll fail.  But maybe it could, for
> some reason, use Guix binaries during testing and then decide to
> download its own when running outside of the build environment.
> 
> Now I'm conlicted :-), maybe rewriting the build.el file is the safest
> option, and maybe if we get bug reports from people who would like to
> use their own grammars we could revisit?

Seems like an option to me (I don't think we actually know yet if there
will be interest in downloading the grammars from upstream or not among
potential Guix users).  Though FWIW, they can already use their own
grammars with "--with-source" transformations (to change an existing
grammar) or by packaging them (for new grammars) and the like (albeit
with Guix instead of Emacs).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#49946] [PATCH v6 00/27] gnu: Add tree-sitter for emacs.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

Hi Guix!

Here's a v6 of the tree-sitter for emacs series.

Changes are:

- Rebased on master.

- Updated typescript grammar to 0.20.1.

- Remove code that downloads binaries from emacs-tree-sitter and
  emacs-tree-sitter-langs. It's no longer possible to change the source
  of grammars used by the package, and will now only use Guix-provided
  ones.

  Given this behaviour is different from upstream, we also emit a
  message when grammars are loaded:

--8<---------------cut here---------------start------------->8---
  tree-sitter-langs: Grammar bundle already installed via Guix.
  Installing external grammars via this function isn't supported, if a
  language you need is missing please report a bug at bug-guix@gnu.org.
--8<---------------cut here---------------end--------------->8---

I also pushed this series to a branch on
https://gitlab.com/planglois/guix/-/tree/emacs-tree-sitter in case
that's useful for testing. 

Thanks!
Pierre

Pierre Langlois (27):
  gnu: tree-sitter: Move to its own module.
  gnu: Add tree-sitter-cli.
  build-system: Add tree-sitter-build-system.
  gnu: Add tree-sitter-c.
  gnu: Add tree-sitter-cpp.
  gnu: Add tree-sitter-bash.
  gnu: Add tree-sitter-css.
  gnu: Add tree-sitter-c-sharp.
  gnu: Add tree-sitter-elixir.
  gnu: Add tree-sitter-elm.
  gnu: Add tree-sitter-go.
  gnu: Add tree-sitter-html.
  gnu: Add tree-sitter-java.
  gnu: Add tree-sitter-javascript.
  gnu: Add tree-sitter-json.
  gnu: Add tree-sitter-julia.
  gnu: Add tree-sitter-ocaml.
  gnu: Add tree-sitter-php.
  gnu: Add tree-sitter-python.
  gnu: Add tree-sitter-ruby.
  gnu: Add tree-sitter-rust.
  gnu: Add tree-sitter-typescript.
  gnu: Add rust-tree-sitter.
  gnu: Add rust-tree-sitter-for-emacs.
  gnu: Add emacs-tree-sitter-core.
  gnu: Add emacs-tree-sitter.
  gnu: Add emacs-tree-sitter-langs.

 Makefile.am                                   |   2 +
 doc/guix.texi                                 |  21 +-
 gnu/local.mk                                  |   2 +
 .../rust-tree-sitter-text-provider-fix.patch  |  98 ++
 gnu/packages/text-editors.scm                 |  52 -
 gnu/packages/tree-sitter.scm                  | 947 ++++++++++++++++++
 gnu/packages/vim.scm                          |   1 +
 guix/build-system/tree-sitter.scm             | 190 ++++
 guix/build/tree-sitter-build-system.scm       | 153 +++
 9 files changed, 1413 insertions(+), 53 deletions(-)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
 create mode 100644 gnu/packages/tree-sitter.scm
 create mode 100644 guix/build-system/tree-sitter.scm
 create mode 100644 guix/build/tree-sitter-build-system.scm

-- 
2.36.0

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v6 01/27] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

* gnu/packages/text-editors.scm (tree-sitter): Move to ...
* gnu/packages/tree-sitter.scm: ... here, a new module.
* gnu/packages/vim.scm: Use (gnu packages tree-sitter).
* gnu/local.mk (GNU_SYSTEM_MODULES): Register tree-sitter.scm new module.
---
 gnu/local.mk                  |  1 +
 gnu/packages/text-editors.scm | 52 -----------------------
 gnu/packages/tree-sitter.scm  | 78 +++++++++++++++++++++++++++++++++++
 gnu/packages/vim.scm          |  1 +
 4 files changed, 80 insertions(+), 52 deletions(-)
 create mode 100644 gnu/packages/tree-sitter.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index cfb499915a..429a0b577b 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -589,6 +589,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/tmux.scm				\
   %D%/packages/toolkits.scm			\
   %D%/packages/tor.scm				\
+  %D%/packages/tree-sitter.scm			\
   %D%/packages/tv.scm				\
   %D%/packages/uglifyjs.scm			\
   %D%/packages/uml.scm				\
diff --git a/gnu/packages/text-editors.scm b/gnu/packages/text-editors.scm
index 7cbad042c9..ddbfef3e77 100644
--- a/gnu/packages/text-editors.scm
+++ b/gnu/packages/text-editors.scm
@@ -63,7 +63,6 @@ (define-module (gnu packages text-editors)
   #:use-module (gnu packages gtk)
   #:use-module (gnu packages guile)
   #:use-module (gnu packages haskell-xyz)
-  #:use-module (gnu packages icu4c)
   #:use-module (gnu packages image)
   #:use-module (gnu packages libbsd)
   #:use-module (gnu packages libreoffice)
@@ -1167,54 +1166,3 @@ (define-public virtaal
 provided by the Translate Toolkit, including XLIFF and PO.")
     (home-page "https://virtaal.translatehouse.org/")
     (license license:gpl2+)))
-
-(define-public tree-sitter
-  (package
-    (name "tree-sitter")
-    (version "0.20.6")
-    (source (origin
-              (method git-fetch)
-              (uri (git-reference
-                    (url "https://github.com/tree-sitter/tree-sitter")
-                    (commit (string-append "v" version))))
-              (file-name (git-file-name name version))
-              (sha256
-               (base32
-                "1z20518snyg0zp75qgs5bxmzjqws4dd19vnp6sya494za3qp5b6d"))
-              (modules '((guix build utils)))
-              (snippet '(begin
-                          ;; Remove bundled ICU parts
-                          (delete-file-recursively "lib/src/unicode")
-                          #t))))
-    (build-system gnu-build-system)
-    (inputs (list icu4c))
-    (arguments
-     (list #:phases
-           '(modify-phases %standard-phases
-              (delete 'configure))
-           #:tests? #f ; there are no tests for the runtime library
-           #:make-flags
-           #~(list (string-append "PREFIX="
-                                  #$output)
-                   (string-append "CC="
-                                  #$(cc-for-target)))))
-    (home-page "https://tree-sitter.github.io/tree-sitter/")
-    (synopsis "Incremental parsing system for programming tools")
-    (description
-     "Tree-sitter is a parser generator tool and an incremental parsing
-library.  It can build a concrete syntax tree for a source file and efficiently
-update the syntax tree as the source file is edited.
-
-Tree-sitter aims to be:
-
-@itemize
-@item General enough to parse any programming language
-@item Fast enough to parse on every keystroke in a text editor
-@item Robust enough to provide useful results even in the presence of syntax errors
-@item Dependency-free so that the runtime library (which is written in pure C)
-can be embedded in any application
-@end itemize
-
-This package includes the @code{libtree-sitter} runtime library.
-")
-    (license license:expat)))
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
new file mode 100644
index 0000000000..7116a10ed7
--- /dev/null
+++ b/gnu/packages/tree-sitter.scm
@@ -0,0 +1,78 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Luis Henrique Gomes Higino <luishenriquegh2701@gmail.com>
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages tree-sitter)
+  #:use-module (guix gexp)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix git-download)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages icu4c))
+
+(define-public tree-sitter
+  (package
+    (name "tree-sitter")
+    (version "0.20.6")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1z20518snyg0zp75qgs5bxmzjqws4dd19vnp6sya494za3qp5b6d"))
+              (modules '((guix build utils)))
+              (snippet '(begin
+                          ;; Remove bundled ICU parts
+                          (delete-file-recursively "lib/src/unicode")
+                          #t))))
+    (build-system gnu-build-system)
+    (inputs (list icu4c))
+    (arguments
+     (list #:phases
+           '(modify-phases %standard-phases
+              (delete 'configure))
+           #:tests? #f ; there are no tests for the runtime library
+           #:make-flags
+           #~(list (string-append "PREFIX="
+                                  #$output)
+                   (string-append "CC="
+                                  #$(cc-for-target)))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Incremental parsing system for programming tools")
+    (description
+     "Tree-sitter is a parser generator tool and an incremental parsing
+library.  It can build a concrete syntax tree for a source file and efficiently
+update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@itemize
+@item General enough to parse any programming language
+@item Fast enough to parse on every keystroke in a text editor
+@item Robust enough to provide useful results even in the presence of syntax errors
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application
+@end itemize
+
+This package includes the @code{libtree-sitter} runtime library.
+")
+    (license license:expat)))
diff --git a/gnu/packages/vim.scm b/gnu/packages/vim.scm
index bb459933b0..c7f7697168 100644
--- a/gnu/packages/vim.scm
+++ b/gnu/packages/vim.scm
@@ -71,6 +71,7 @@ (define-module (gnu packages vim)
   #:use-module (gnu packages tcl)
   #:use-module (gnu packages text-editors)
   #:use-module (gnu packages terminals)
+  #:use-module (gnu packages tree-sitter)
   #:use-module (gnu packages xdisorg)
   #:use-module (gnu packages xorg))

--
2.36.0

[bug#49946] [PATCH v6 02/27] gnu: Add tree-sitter-cli.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cli): New variable.
---
 gnu/packages/tree-sitter.scm | 79 ++++++++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 7116a10ed7..18c9fe0844 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -20,10 +20,13 @@
 (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages crates-graphics)
+  #:use-module (gnu packages crates-io)
   #:use-module (gnu packages icu4c))

 (define-public tree-sitter
@@ -76,3 +79,79 @@ (define-public tree-sitter
 This package includes the @code{libtree-sitter} runtime library.
 ")
     (license license:expat)))
+
+(define-public tree-sitter-cli
+  (package (inherit tree-sitter)
+    (name "tree-sitter-cli")
+    (source (origin
+              (inherit (package-source tree-sitter))
+              (snippet
+               '(begin
+                  ;; Remove the runtime library code and dynamically link to
+                  ;; it instead.
+                  (delete-file-recursively "lib/src")
+                  (delete-file "lib/binding_rust/build.rs")
+                  (with-output-to-file "lib/binding_rust/build.rs"
+                    (lambda _
+                      (format #t "fn main() {~@
+                              println!(\"cargo:rustc-link-lib=tree-sitter\");~@
+                              }~%")))
+                  #t))))
+    (build-system cargo-build-system)
+    (inputs (list tree-sitter))
+    (arguments
+     `(;; Running test requires downloading fixtures, see the
+       ;; script/fetch-fixtures script.
+       #:tests? #f
+       ;; We're only packaging the CLI program so we do not need to install
+       ;; sources.
+       #:install-source? #f
+       #:cargo-inputs
+       (("rust-ansi-term" ,rust-ansi-term-0.12)
+        ("rust-anyhow" ,rust-anyhow-1)
+        ("rust-atty" ,rust-atty-0.2)
+        ("rust-clap" ,rust-clap-2)
+        ("rust-difference" ,rust-difference-2)
+        ("rust-html-escape" ,rust-html-escape-0.2)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-rand" ,rust-rand-0.8)
+        ("rust-rustc-hash" ,rust-rustc-hash-1)
+        ("rust-semver" ,rust-semver-1)
+        ("rust-smallbitvec" ,rust-smallbitvec-2)
+        ("rust-thiserror" ,rust-thiserror-1)
+        ("rust-tiny-http" ,rust-tiny-http-0.8)
+        ("rust-toml" ,rust-toml-0.5)
+        ("rust-walkdir" ,rust-walkdir-2)
+        ("rust-webbrowser" ,rust-webbrowser-0.5)
+        ("rust-which" ,rust-which-4))
+       #:cargo-development-inputs
+       (("rust-pretty-assertions" ,rust-pretty-assertions-0.7))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin"))
+                    (lib (string-append out "/lib")))
+               (mkdir-p bin)
+               (install-file "target/release/tree-sitter" bin)))))))
+    (synopsis "Incremental parsing system for programming tools")
+    (description "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @command{tree-sitter} command-line tool.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 04/27] gnu: Add tree-sitter-c.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-c): New variable.
---
 gnu/packages/tree-sitter.scm | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 18c9fe0844..3f07b2d91c 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -22,6 +22,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
+  #:use-module (guix build-system tree-sitter)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
@@ -155,3 +156,35 @@ (define-public tree-sitter-cli

 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))
+
+(define tree-sitter-delete-generated-files
+  '(begin
+     (delete-file "binding.gyp")
+     (delete-file-recursively "bindings")
+     (delete-file "src/grammar.json")
+     (delete-file "src/node-types.json")
+     (delete-file "src/parser.c")
+     (delete-file-recursively "src/tree_sitter")
+     #t))
+
+(define-public tree-sitter-c
+  (package
+    (name "tree-sitter-c")
+    (version "0.20.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-c")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0454jziys33i4kbwnvi9xcck0fzya792ghy32ahgk1hhv96xga9w"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-c")
+    (synopsis "Tree-sitter C grammar")
+    (description
+     "This package provides a C grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 05/27] gnu: Add tree-sitter-cpp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cpp): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 3f07b2d91c..5e25d48c05 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -188,3 +188,26 @@ (define-public tree-sitter-c
     (description
      "This package provides a C grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-cpp
+  (package
+    (name "tree-sitter-cpp")
+    (version "0.20.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-cpp")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0hxcpdvyyig8njga1mxp4qcnbbnr1d0aiy27vahijwbh98b081nr"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (inputs (list tree-sitter-c))
+    (home-page "https://github.com/tree-sitter/tree-sitter-cpp")
+    (synopsis "Tree-sitter C++ grammar")
+    (description
+     "This package provides a C++ grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 06/27] gnu: Add tree-sitter-bash.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-bash): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 5e25d48c05..084071fe3d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -211,3 +211,25 @@ (define-public tree-sitter-cpp
     (description
      "This package provides a C++ grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-bash
+  (package
+    (name "tree-sitter-bash")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-bash")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "18c030bb65r50i6z37iy7jb9z9i8i36y7b08dbc9bchdifqsijs5"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-bash")
+    (synopsis "Tree-sitter Bash grammar")
+    (description
+     "This package provides a Bash grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 07/27] gnu: Add tree-sitter-css.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-css): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 084071fe3d..d2f8326be0 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -233,3 +233,25 @@ (define-public tree-sitter-bash
     (description
      "This package provides a Bash grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-css
+  (package
+    (name "tree-sitter-css")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-css")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "014jrlgi7zfza9g38hsr4vlbi8964i5p7iglaih6qmzaiml7bja2"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-css")
+    (synopsis "Tree-sitter CSS grammar")
+    (description
+     "This package provides a CSS grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 08/27] gnu: Add tree-sitter-c-sharp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-c-sharp): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d2f8326be0..de90ac2093 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -212,6 +212,28 @@ (define-public tree-sitter-cpp
      "This package provides a C++ grammar for the Tree-sitter library.")
     (license license:expat)))

+(define-public tree-sitter-c-sharp
+  (package
+    (name "tree-sitter-c-sharp")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-c-sharp")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "054fmpf47cwh59gbg00sc0nl237ba4rnxi73miz39yqzcs87055r"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-c-sharp")
+    (synopsis "Tree-sitter C# grammar")
+    (description
+     "This package provides a C# grammar for the Tree-sitter library.")
+    (license license:expat)))
+
 (define-public tree-sitter-bash
   (package
     (name "tree-sitter-bash")
--
2.36.0

[bug#49946] [PATCH v6 09/27] gnu: Add tree-sitter-elixir.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-elixir): New variable.
---
 gnu/packages/tree-sitter.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index de90ac2093..18ea00f7cf 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -277,3 +277,29 @@ (define-public tree-sitter-css
     (description
      "This package provides a CSS grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-elixir
+  (let ((commit "de20391afe5cb03ef1e8a8e43167e7b58cc52869")
+        (revision "1"))
+    (package
+      (name "tree-sitter-elixir")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/elixir-lang/tree-sitter-elixir")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0zrkrwhw3g1vazsxcwrfd1fk4wvs9hdwmwp6073mfh370bz4140h"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+      (build-system tree-sitter-build-system)
+      (home-page "https://elixir-lang.org/tree-sitter-elixir/")
+      (synopsis "Tree-sitter Elixir grammar")
+      (description
+       "This package provides a Elixir grammar for the Tree-sitter library.")
+      (license (list license:asl2.0
+                     ;; Files in tests/corpus are under MIT license.
+                     license:expat)))))
--
2.36.0

[bug#49946] [PATCH v6 10/27] gnu: Add tree-sitter-elm.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-elm): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 18ea00f7cf..0aafaf7c1b 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -303,3 +303,25 @@ (define-public tree-sitter-elixir
       (license (list license:asl2.0
                      ;; Files in tests/corpus are under MIT license.
                      license:expat)))))
+
+(define-public tree-sitter-elm
+  (package
+    (name "tree-sitter-elm")
+    (version "5.5.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/elm-tooling/tree-sitter-elm")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "10hbi4vyj4hjixqswdcbvzl60prldczz29mlp02if61wvwiwvqrw"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://elm-tooling.github.io/tree-sitter-elm/")
+    (synopsis "Tree-sitter Elm grammar")
+    (description
+     "This package provides an Elm grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 11/27] gnu: Add tree-sitter-go.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-go): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 0aafaf7c1b..585b44b9f9 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -325,3 +325,25 @@ (define-public tree-sitter-elm
     (description
      "This package provides an Elm grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-go
+  (package
+    (name "tree-sitter-go")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-go")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0nxs47vd2fc2fr0qlxq496y852rwg39flhg334s7dlyq7d3lcx4x"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-go")
+    (synopsis "Tree-sitter Go grammar")
+    (description
+     "This package provides a Golang grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 12/27] gnu: Add tree-sitter-html.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-html): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 585b44b9f9..15f832f8ca 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -347,3 +347,25 @@ (define-public tree-sitter-go
     (description
      "This package provides a Golang grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-html
+  (package
+    (name "tree-sitter-html")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-html")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-html")
+    (synopsis "Tree-sitter HTML grammar")
+    (description
+     "This package provides a HTML grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 13/27] gnu: Add tree-sitter-java.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-java): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 15f832f8ca..010a7c3751 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -369,3 +369,25 @@ (define-public tree-sitter-html
     (description
      "This package provides a HTML grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-java
+  (package
+    (name "tree-sitter-java")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-java")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "07zw9ygb45hnvlx9qlz7rlz8hc3byjy03d24v72i5iyhpiiwlhvl"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-java")
+    (synopsis "Tree-sitter Java grammar")
+    (description
+     "This package provides a Java grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 14/27] gnu: Add tree-sitter-javascript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-javascript): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 010a7c3751..30220a5a8c 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -391,3 +391,26 @@ (define-public tree-sitter-java
     (description
      "This package provides a Java grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-javascript
+  (package
+    (name "tree-sitter-javascript")
+    (version "0.20.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-javascript")
+                    (commit (string-append "rust-" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "175yrk382n2di0c2xn4gpv8y4n83x1lg4hqn04vabf0yqynlkq67"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
+    (synopsis "Tree-sitter Javascript grammar")
+    (description
+     "This package provides Javascript and JSX grammars for the Tree-sitter
+library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 15/27] gnu: Add tree-sitter-json.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-json): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 30220a5a8c..d43e16af09 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -414,3 +414,25 @@ (define-public tree-sitter-javascript
      "This package provides Javascript and JSX grammars for the Tree-sitter
 library.")
     (license license:expat)))
+
+(define-public tree-sitter-json
+  (package
+    (name "tree-sitter-json")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-json")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "06pjh31bv9ja9hlnykk257a6zh8bsxg2fqa54al7qk1r4n9ksnff"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-json")
+    (synopsis "Tree-sitter JSON grammar")
+    (description
+     "This package provides a JSON grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 16/27] gnu: Add tree-sitter-julia.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-julia): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d43e16af09..a535d8bc4a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -436,3 +436,25 @@ (define-public tree-sitter-json
     (description
      "This package provides a JSON grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-julia
+  (package
+    (name "tree-sitter-julia")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-julia")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1pbnmvhy2gq4vg1b0sjzmjm4s2gsgdjh7h01yj8qrrqbcl29c463"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-julia")
+    (synopsis "Tree-sitter Julia grammar")
+    (description
+     "This package provides a Julia grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 17/27] gnu: Add tree-sitter-ocaml.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-ocaml): New variable.
---
 gnu/packages/tree-sitter.scm | 39 ++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index a535d8bc4a..ff98044cc1 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -458,3 +458,42 @@ (define-public tree-sitter-julia
     (description
      "This package provides a Julia grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-ocaml
+  (let ((commit "0348562f385bc2bd67ecf181425e1afd6d454192")
+        (revision "1"))
+    (package
+      (name "tree-sitter-ocaml")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-ocaml")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0iqmwcz3c2ai4gyx4xli1rhn6hi6a0f60dn20f8jas9ham9dc2df"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (delete-file "binding.gyp")
+                    (delete-file-recursively "bindings")
+                    (for-each
+                     (lambda (lang)
+                       (with-directory-excursion lang
+                         (delete-file "src/grammar.json")
+                         (delete-file "src/node-types.json")
+                         (delete-file "src/parser.c")
+                         (delete-file-recursively "src/tree_sitter")))
+                     '("ocaml" "interface"))
+                    #t))))
+      (build-system tree-sitter-build-system)
+      (arguments
+       (list
+        #:grammar-directories '("ocaml" "interface")))
+      (home-page "https://github.com/tree-sitter/tree-sitter-ocaml")
+      (synopsis "Tree-sitter OCaml grammar")
+      (description
+       "This package provides a OCaml grammar for the Tree-sitter library.")
+      (license license:expat))))
--
2.36.0

[bug#49946] [PATCH v6 18/27] gnu: Add tree-sitter-php.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-php): New variable.
---
 gnu/packages/tree-sitter.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index ff98044cc1..654445f71d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -497,3 +497,33 @@ (define-public tree-sitter-ocaml
       (description
        "This package provides a OCaml grammar for the Tree-sitter library.")
       (license license:expat))))
+
+(define-public tree-sitter-php
+  (let ((commit "435fa00006c0d1515c37fbb4dd6a9de284af75ab")
+        (revision "1"))
+    (package
+      (name "tree-sitter-php")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-php")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "05k4h58gi616gv41r0qqdb1x4rs8y94vghn2r10yczisgzq4vbad"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (delete-file "src/grammar.json")
+                    (delete-file "src/node-types.json")
+                    (delete-file "src/parser.c")
+                    (delete-file-recursively "src/tree_sitter")
+                    #t))))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-php")
+    (synopsis "Tree-sitter PHP grammar")
+    (description
+     "This package provides a PHP grammar for the Tree-sitter library.")
+    (license license:expat))))
--
2.36.0

[bug#49946] [PATCH v6 19/27] gnu: Add tree-sitter-python.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-python): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 654445f71d..c43cafea2d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -527,3 +527,27 @@ (define-public tree-sitter-php
     (description
      "This package provides a PHP grammar for the Tree-sitter library.")
     (license license:expat))))
+
+(define-public tree-sitter-python
+  (let ((commit "ed0fe62e55dc617ed9dec8817ebf771aa7cf3c42")
+        (revision "1"))
+    (package
+      (name "tree-sitter-python")
+      (version (git-version "0.19.1" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-python")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0wrfpg84mc3pzcrdi6n5fqwijkqr1nj5sqfnayb502krvqpjilal"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-python")
+    (synopsis "Tree-sitter Python grammar")
+    (description
+     "This package provides a Python grammar for the Tree-sitter library.")
+    (license license:expat))))
--
2.36.0

[bug#49946] [PATCH v6 20/27] gnu: Add tree-sitter-ruby.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-ruby): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c43cafea2d..c7b1998253 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -551,3 +551,25 @@ (define-public tree-sitter-python
     (description
      "This package provides a Python grammar for the Tree-sitter library.")
     (license license:expat))))
+
+(define-public tree-sitter-ruby
+  (package
+    (name "tree-sitter-ruby")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-ruby")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0m3h4928rbs300wcb6776h9r88hi32rybbhcaf6rdympl5nzi83v"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-ruby")
+    (synopsis "Tree-sitter Ruby grammar")
+    (description
+     "This package provides a Ruby grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 21/27] gnu: Add tree-sitter-rust.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-rust): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c7b1998253..c37183e131 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -573,3 +573,25 @@ (define-public tree-sitter-ruby
     (description
      "This package provides a Ruby grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-rust
+  (package
+    (name "tree-sitter-rust")
+    (version "0.20.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-rust")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "174j5pxwf80c4xniri39l3a6bb7nq96g2s8hh5sgv4i7xvbpfsmg"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-rust")
+    (synopsis "Tree-sitter Rust grammar")
+    (description
+     "This package provides a Rust grammar for the Tree-sitter library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 22/27] gnu: Add tree-sitter-typescript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-typescript): New variable.
---
 gnu/packages/tree-sitter.scm | 38 ++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c37183e131..04747f47e5 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -595,3 +595,41 @@ (define-public tree-sitter-rust
     (description
      "This package provides a Rust grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-typescript
+  (package
+    (name "tree-sitter-typescript")
+    (version "0.20.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-typescript")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32 "07fl9d968lal0aqj4f0n16p3n94cjkgfp54wynfr8gbdkjss5v5x"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  (delete-file "binding.gyp")
+                  (delete-file-recursively "bindings")
+                  (for-each
+                   (lambda (lang)
+                     (with-directory-excursion lang
+                       (delete-file "src/grammar.json")
+                       (delete-file "src/node-types.json")
+                       (delete-file "src/parser.c")
+                       (delete-file-recursively "src/tree_sitter")))
+                   '("typescript" "tsx"))
+                  #t))))
+    (build-system tree-sitter-build-system)
+    (inputs (list tree-sitter-javascript))
+    (arguments
+     (list
+      #:grammar-directories '("typescript" "tsx")))
+    (home-page "https://github.com/tree-sitter/tree-sitter-typescript")
+    (synopsis "Tree-sitter Typescript grammar")
+    (description
+     "This package provides Typescript and TSX grammars for the Tree-sitter
+library.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 23/27] gnu: Add rust-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 39 ++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 04747f47e5..58ecc445a6 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -23,6 +23,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
+  #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
@@ -157,6 +158,44 @@ (define-public tree-sitter-cli
 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))

+(define-public rust-tree-sitter
+  (package
+    (name "rust-tree-sitter")
+    (version "0.20.6")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "tree-sitter" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "0wcvxgnvj7ga1y7xa7wm0pmabkfj8936ifg8jacd4201cj0vgcq9"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Remove the runtime library code and dynamically link to
+                  ;; it instead.
+                  (delete-file-recursively "src")
+                  (delete-file "binding_rust/build.rs")
+                  (with-output-to-file "binding_rust/build.rs"
+                    (lambda _
+                      (format #t "fn main() {~@
+                              println!(\"cargo:rustc-link-lib=tree-sitter\");~@
+                              }~%")))
+                  #t))))
+    (build-system cargo-build-system)
+    (inputs (list tree-sitter))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-cc" ,rust-cc-1)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-regex" ,rust-regex-1)
+        ("rust-spin" ,rust-spin-0.7))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Rust bindings to the Tree-sitter parsing library")
+    (description "This package provides Rust bindings to the Tree-sitter
+parsing library.")
+    (license license:expat)))
+
 (define tree-sitter-delete-generated-files
   '(begin
      (delete-file "binding.gyp")
--
2.36.0

[bug#49946] [PATCH v6 24/27] gnu: Add rust-tree-sitter-for-emacs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter-for-emacs): New variable.
* gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
---
 gnu/local.mk                                  |  1 +
 .../rust-tree-sitter-text-provider-fix.patch  | 98 +++++++++++++++++++
 gnu/packages/tree-sitter.scm                  | 14 +++
 3 files changed, 113 insertions(+)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 429a0b577b..7681eda58d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1787,6 +1787,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/rust-nettle-sys-disable-vendor.patch	 \
   %D%/packages/patches/rust-openssl-sys-no-vendor.patch	\
   %D%/packages/patches/rust-shell2batch-lint-fix.patch		\
+  %D%/packages/patches/rust-tree-sitter-text-provider-fix.patch	\
   %D%/packages/patches/rust-wl-clipboard-rs-newer-wl.patch      \
   %D%/packages/patches/sbc-fix-build-non-x86.patch		\
   %D%/packages/patches/sbcl-aserve-add-HTML-5-elements.patch	\
diff --git a/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
new file mode 100644
index 0000000000..215e7fc18b
--- /dev/null
+++ b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
@@ -0,0 +1,98 @@
+From e3576b180488b8231e1fc0ca130748577579d129 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Tu=E1=BA=A5n-Anh=20Nguy=E1=BB=85n?= <ubolonton@gmail.com>
+Date: Sun, 25 Jul 2021 13:11:52 +0700
+Subject: [PATCH] Allow TextProvider's iterators to generate owned text
+
+---
+ binding_rust/lib.rs | 33 ++++++++++++++++++---------------
+ 1 file changed, 18 insertions(+), 15 deletions(-)
+
+diff --git a/lib/binding_rust/lib.rs b/lib/binding_rust/lib.rs
+index e88a411c..cf214d92 100644
+--- a/binding_rust/lib.rs
++++ b/binding_rust/lib.rs
+@@ -5,6 +5,7 @@ mod util;
+ use std::os::unix::io::AsRawFd;
+
+ use std::{
++    borrow::Cow,
+     char, error,
+     ffi::CStr,
+     fmt, hash, iter,
+@@ -183,7 +184,8 @@ pub struct QueryCaptures<'a, 'tree: 'a, T: TextProvider<'a>> {
+ }
+
+ pub trait TextProvider<'a> {
+-    type I: Iterator<Item = &'a [u8]> + 'a;
++    type I: Iterator<Item = Cow<'a, [u8]>>;
++
+     fn text(&mut self, node: Node) -> Self::I;
+ }
+
+@@ -1840,19 +1842,19 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+         buffer2: &mut Vec<u8>,
+         text_provider: &mut impl TextProvider<'a>,
+     ) -> bool {
+-        fn get_text<'a, 'b: 'a, I: Iterator<Item = &'b [u8]>>(
++        fn get_text<'a, 'b: 'a, I: Iterator<Item = Cow<'b, [u8]>>>(
+             buffer: &'a mut Vec<u8>,
+             mut chunks: I,
+-        ) -> &'a [u8] {
+-            let first_chunk = chunks.next().unwrap_or(&[]);
++        ) -> Cow<'a, [u8]> {
++            let first_chunk = chunks.next().unwrap_or(Cow::Owned(vec![0u8; 0]));
+             if let Some(next_chunk) = chunks.next() {
+                 buffer.clear();
+-                buffer.extend_from_slice(first_chunk);
+-                buffer.extend_from_slice(next_chunk);
++                buffer.extend_from_slice(&first_chunk);
++                buffer.extend_from_slice(&next_chunk);
+                 for chunk in chunks {
+-                    buffer.extend_from_slice(chunk);
++                    buffer.extend_from_slice(&chunk);
+                 }
+-                buffer.as_slice()
++                Cow::Borrowed(buffer.as_slice())
+             } else {
+                 first_chunk
+             }
+@@ -1888,7 +1890,7 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+                     match node {
+                         Some(node) => {
+                             let text = get_text(buffer1, text_provider.text(node));
+-                            r.is_match(text) == *is_positive
++                            r.is_match(&text) == *is_positive
+                         }
+                         None => true,
+                     }
+@@ -2002,23 +2004,24 @@ impl<'cursor, 'tree> fmt::Debug for QueryMatch<'cursor, 'tree> {
+     }
+ }
+
+-impl<'a, F, I> TextProvider<'a> for F
++impl<'a, F, I, T> TextProvider<'a> for F
+ where
+     F: FnMut(Node) -> I,
+-    I: Iterator<Item = &'a [u8]> + 'a,
++    T: Into<Cow<'a, [u8]>>,
++    I: Iterator<Item = T>,
+ {
+-    type I = I;
++    type I = iter::Map<I, fn(T) -> Cow<'a, [u8]>>;
+
+     fn text(&mut self, node: Node) -> Self::I {
+-        (self)(node)
++        (self)(node).map(T::into)
+     }
+ }
+
+ impl<'a> TextProvider<'a> for &'a [u8] {
+-    type I = iter::Once<&'a [u8]>;
++    type I = iter::Once<Cow<'a, [u8]>>;
+
+     fn text(&mut self, node: Node) -> Self::I {
+-        iter::once(&self[node.byte_range()])
++        iter::once(Cow::Borrowed(&self[node.byte_range()]))
+     }
+ }
+
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 58ecc445a6..4d1d3b2177 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -27,6 +27,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages icu4c))
@@ -196,6 +197,19 @@ (define-public rust-tree-sitter
 parsing library.")
     (license license:expat)))

+;; We need to apply a patch in order to compile the rust bindings against the
+;; emacs tree-sitter module.
+;; See https://github.com/tree-sitter/tree-sitter/pull/1294
+(define-public rust-tree-sitter-for-emacs
+  (package (inherit rust-tree-sitter)
+    (source (origin
+              (inherit (package-source rust-tree-sitter))
+              (patches (search-patches
+                        "rust-tree-sitter-text-provider-fix.patch"))))
+    ;; Do not show this package in the UI as it's only meant to be used for
+    ;; emacs's tree-sitter module.
+    (properties '((hidden? . #t)))))
+
 (define tree-sitter-delete-generated-files
   '(begin
      (delete-file "binding.gyp")
--
2.36.0

[bug#49946] [PATCH v6 25/27] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
(emacs-tree-sitter-core): New variable.
---
 gnu/packages/tree-sitter.scm | 84 ++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 4d1d3b2177..3e01474285 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -21,6 +21,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
+  #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
   #:use-module (guix download)
@@ -686,3 +687,86 @@ (define-public tree-sitter-typescript
      "This package provides Typescript and TSX grammars for the Tree-sitter
 library.")
     (license license:expat)))
+
+;; Local package definition solely for building the native emacs module
+;; written in Rust.
+(define tree-sitter-emacs-module
+  (package
+    (name "tree-sitter-emacs-module")
+    (version "0.18.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1sdvz827v436qijs6xafakkfw2d16bvp8frymd818rppjc7a9dif"))))
+    (build-system cargo-build-system)
+    (inputs
+     (list tree-sitter))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-emacs" ,rust-emacs-0.18)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-tree-sitter" ,rust-tree-sitter-for-emacs))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'chdir 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (add-after 'delete-cargo.lock 'do-not-fetch-from-github
+           (lambda _
+             (substitute* "Cargo.toml"
+               (("\\[patch.*") "")
+               (("git = .*") ""))))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out") "/lib")))
+               (mkdir-p lib)
+               (copy-file "target/release/libtsc_dyn.so"
+                          (string-append lib "/tsc-dyn.so"))))))))
+    (home-page #f)
+    (synopsis #f)
+    (description #f)
+    (license license:expat)))
+
+(define-public emacs-tree-sitter-core
+  (package
+    (name "emacs-tree-sitter-core")
+    (version (package-version tree-sitter-emacs-module))
+    (source (package-source tree-sitter-emacs-module))
+    (build-system emacs-build-system)
+    (native-inputs
+     (list tree-sitter-emacs-module))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-before 'install 'find-tsc-dyn
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out")))
+                   (module (search-input-file inputs "/lib/tsc-dyn.so")))
+               ;; Install the tsc-dyn module in site-lisp and the current
+               ;; directory for test to pass.
+               (install-file module elpa)
+               (copy-file module "tsc-dyn.so")
+               ;; We replace the tsc-dyn-get.el file with an empty stub to
+               ;; prevent the code from downloading the module.
+               (call-with-output-file "tsc-dyn-get.el"
+                 (lambda (port)
+                   (for-each
+                    (lambda (sexp) (write sexp port))
+                    '((defun tsc-dyn-get-ensure (requested)
+                        nil)
+                      (provide 'tsc-dyn-get)))))))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp, core library")
+    (description "This package provides core APIs of the Emacs binding for
+Tree-sitter, an incremental parsing system.")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 26/27] gnu: Add emacs-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 3e01474285..cb3f326367 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -770,3 +770,30 @@ (module (search-input-file inputs "/lib/tsc-dyn.so")))
     (description "This package provides core APIs of the Emacs binding for
 Tree-sitter, an incremental parsing system.")
     (license license:expat)))
+
+(define-public emacs-tree-sitter
+  (package
+    (name "emacs-tree-sitter")
+    (version (package-version emacs-tree-sitter-core))
+    (source (package-source emacs-tree-sitter-core))
+    (build-system emacs-build-system)
+    (propagated-inputs
+     (list emacs-tree-sitter-core))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "lisp"))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp")
+    (description "This package provides Emacs bindings for Tree-sitter, an
+incremental parsing library.  It aims to be the foundation for a new breed of
+Emacs packages that understand code structurally.  For example:
+
+@enumerate
+@item Faster, fine-grained code highlighting.
+@item More flexible code folding.
+@item Structural editing (like Paredit, or even better) for non-Lisp code.
+@item More informative indexing for imenu.
+@end enumerate")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 27/27] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (make-emacs-tree-sitter-langs-grammar-bundle):
New procedure.
(emacs-tree-sitter-langs): New variable.
---
 gnu/packages/tree-sitter.scm | 148 +++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index cb3f326367..b80d912bfc 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -24,6 +24,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
+  #:use-module (guix build-system trivial)
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
@@ -797,3 +798,150 @@ (define-public emacs-tree-sitter
 @item More informative indexing for imenu.
 @end enumerate")
     (license license:expat)))
+
+(define emacs-tree-sitter-langs-grammar-bundle
+  (package
+    (name "emacs-tree-sitter-langs-grammar-bundle")
+    (source #f)
+    (version (package-version tree-sitter))
+    (build-system trivial-build-system)
+    (inputs
+     ;; FIXME: Support for some languages is still left to package.
+     (list tree-sitter-bash
+           tree-sitter-c
+           tree-sitter-c-sharp
+           tree-sitter-cpp
+           tree-sitter-css
+           tree-sitter-elixir
+           tree-sitter-elm
+           tree-sitter-go
+           tree-sitter-html
+           tree-sitter-java
+           tree-sitter-javascript
+           tree-sitter-json
+           tree-sitter-julia
+           tree-sitter-ocaml
+           tree-sitter-php
+           tree-sitter-python
+           tree-sitter-rust
+           tree-sitter-ruby
+           tree-sitter-typescript))
+    (arguments
+     (list #:builder
+           (with-imported-modules '((guix build union)
+                                    (guix build utils))
+             #~(begin
+                 (use-modules (ice-9 match)
+                              (guix build union)
+                              (guix build utils))
+                 (union-build
+                  #$output
+                  (filter directory-exists?
+                          (map (match-lambda
+                                 ((name directory)
+                                  (string-append directory "/lib/tree-sitter")))
+                               '#$(package-inputs this-package))))))))
+    (synopsis #f)
+    (description #f)
+    (home-page #f)
+    (license #f)))
+
+(define-public emacs-tree-sitter-langs
+  (package
+    (name "emacs-tree-sitter-langs")
+    (version "0.12.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/tree-sitter-langs")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1p2zbb6ac7wi6x6zpbczcmpkb2p45md2csd2bj43d8s56ckzw5mp"))))
+    (build-system emacs-build-system)
+    (inputs
+     (list emacs-tree-sitter-langs-grammar-bundle))
+    (propagated-inputs
+     (list emacs-tree-sitter))
+    (arguments
+     (list
+      #:tests? #t
+      #:test-command ''("script/test")
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'disable-downloader
+            (lambda _
+              (call-with-output-file "tree-sitter-langs-build.el"
+                (lambda (port)
+                  (let ((on-load-message
+                         (string-append
+                          "tree-sitter-langs: Grammar bundle already installed "
+                          "via Guix.  Installing external grammars via this "
+                          "function isn't supported, if a language you need is "
+                          "missing please report a bug at bug-guix@gnu.org.")))
+                    (format
+                     port
+                     ";;;###autoload
+                      (defun tree-sitter-langs-install-grammars
+                             (&optional skip-if-installed version os
+                                        keep-bundle)
+                        (interactive)
+                        (message \"~a\"))
+                      (defconst tree-sitter-langs--queries-dir
+                        (file-name-as-directory
+                          (concat (file-name-directory (locate-library \"tree-sitter-langs.el\"))
+                                  \"queries\")))
+                      (defun tree-sitter-langs--bin-dir () \"~a\")
+                      (provide 'tree-sitter-langs-build)"
+                     on-load-message
+                     #$emacs-tree-sitter-langs-grammar-bundle))))))
+          (add-after 'unpack 'remove-cask
+            (lambda _
+              (substitute* "script/test"
+                (("cask") ""))))
+          (add-before 'check 'patch-tree-sitter-require-test
+            (lambda _
+              (use-modules (ice-9 regex))
+              ;; This test needs a git repositories with submodules for
+              ;; each languages in order to map all repositories.  We patch
+              ;; the mapping function with one that invokes the tests for each
+              ;; packaged language.
+              (let ((supported-languages
+                     (map (lambda (lib)
+                            (match:substring
+                             (string-match "(.*)\\.so$" (basename lib))
+                             1))
+                          (find-files "bin" "\\.so$"))))
+                (substitute* "tree-sitter-langs-tests.el"
+                  (("tree-sitter-langs--map-repos")
+                   (call-with-output-string
+                     (lambda (port)
+                       (write `(lambda (fn)
+                                 (dolist (lang ',supported-languages)
+                                         (funcall fn lang)))
+                              port))))))))
+          ;; Tests for queries will fail given those languages are not
+          ;; packages yet.
+          (add-before 'check 'remove-unused-highlight-queries
+            (lambda _
+              (delete-file-recursively "queries/hcl")
+              (delete-file-recursively "queries/pgn")))
+          (add-after 'install 'install-queries
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+                (copy-recursively "queries" (string-append elpa "/queries"))))))))
+    (home-page "https://ubolonton.github.io/emacs-tree-sitter/languages/")
+    (synopsis "Language support bundle for Tree-sitter")
+    (description "This package is a convenient language bundle for
+Tree-sitter.  For each supported language, this package provides:
+
+@enumerate
+@item Pre-compiled grammar binaries.
+@item An optional highlights.scm file that provides highlighting patterns.
+This is mainly intended for major modes that are not aware of tree-sitter.
+@item Optional query patterns for other minor modes that provide high-level
+functionalities on top of tree-sitter, such as code folding, evil text
+objects, ...etc.
+@end enumerate")
+    (license license:expat)))
--
2.36.0

[bug#49946] [PATCH v6 03/27] build-system: Add tree-sitter-build-system.

[Pierre Langlois]

[-- Attachment #1.1: Type: text/plain, Size: 320 bytes --]

I'm not sure why, but when I look at either
https://mail.gnu.org/archive/html/guix-patches/ or https://issues.guix.gnu.org/49946
this patch always seems to be missing. I wonder why, I used `git
send-email' and didn't see any errors.

I'm sending it as an attachment in case people are not receiving it.

Thanks,
Pierre


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Attachment #2: 0003-build-system-Add-tree-sitter-build-system.patch --]
[-- Type: text/x-patch, Size: 19141 bytes --]

From f16fb42db676303292de5cd9063d0f21f0baceeb Mon Sep 17 00:00:00 2001
From: Pierre Langlois <pierre.langlois@gmx.com>
Date: Tue, 29 Mar 2022 20:13:34 +0100
Subject: [PATCH v6 03/27] build-system: Add tree-sitter-build-system.

* guix/build-system/tree-sitter.scm: New module.
* guix/build/tree-sitter-build-system.scm: Likewise.
* Makefile.am (MODULES): Add them.
* doc/guix.texi: Document it.
---
 Makefile.am                             |   2 +
 doc/guix.texi                           |  21 ++-
 guix/build-system/tree-sitter.scm       | 190 ++++++++++++++++++++++++
 guix/build/tree-sitter-build-system.scm | 153 +++++++++++++++++++
 4 files changed, 365 insertions(+), 1 deletion(-)
 create mode 100644 guix/build-system/tree-sitter.scm
 create mode 100644 guix/build/tree-sitter-build-system.scm

diff --git a/Makefile.am b/Makefile.am
index 85a22be99c..de5cdbd328 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -168,6 +168,7 @@ MODULES =					\
   guix/build-system/ruby.scm			\
   guix/build-system/scons.scm			\
   guix/build-system/texlive.scm			\
+  guix/build-system/tree-sitter.scm		\
   guix/build-system/trivial.scm			\
   guix/ftp-client.scm				\
   guix/http-client.scm				\
@@ -221,6 +222,7 @@ MODULES =					\
   guix/build/ruby-build-system.scm		\
   guix/build/scons-build-system.scm		\
   guix/build/texlive-build-system.scm		\
+  guix/build/tree-sitter-build-system.scm	\
   guix/build/waf-build-system.scm		\
   guix/build/haskell-build-system.scm		\
   guix/build/julia-build-system.scm		\
diff --git a/doc/guix.texi b/doc/guix.texi
index e1fb50a634..1783d74551 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -79,7 +79,7 @@ Copyright @copyright{} 2020 Jack Hill@*
 Copyright @copyright{} 2020 Naga Malleswari@*
 Copyright @copyright{} 2020, 2021 Brice Waegeneire@*
 Copyright @copyright{} 2020 R Veera Kumar@*
-Copyright @copyright{} 2020, 2021 Pierre Langlois@*
+Copyright @copyright{} 2020, 2021, 2022 Pierre Langlois@*
 Copyright @copyright{} 2020 pinoaffe@*
 Copyright @copyright{} 2020 André Batista@*
 Copyright @copyright{} 2020, 2021 Alexandru-Sergiu Marton@*
@@ -9281,6 +9281,25 @@ be specified with the @code{#:node} parameter which defaults to
 @code{node}.
 @end defvr
 
+@defvr {Scheme Variable} tree-sitter-build-system
+
+This variable is exported by @code{(guix build-system tree-sitter)}.  It
+implements procedures to compile grammars for the
+@url{https://tree-sitter.github.io/tree-sitter/, Tree-sitter} parsing
+library.  It essentially runs @code{tree-sitter generate} to translate
+@code{grammar.js} grammars to JSON and then to C.  Which it then
+compiles to native code.
+
+Tree-sitter packages may support multiple grammars, so this build system
+supports a @code{#:grammar-directories} keyword to specify a list of
+locations where a @code{grammar.js} file may be found.
+
+Grammars sometimes depend on each other, such as C++ depending on C and
+TypeScript depending on JavaScript.  You may use inputs to declare such
+dependencies.
+
+@end defvr
+
 Lastly, for packages that do not need anything as sophisticated, a
 ``trivial'' build system is provided.  It is trivial in the sense that
 it provides basically no support: it does not pull any implicit inputs,
diff --git a/guix/build-system/tree-sitter.scm b/guix/build-system/tree-sitter.scm
new file mode 100644
index 0000000000..aeb96e3ef5
--- /dev/null
+++ b/guix/build-system/tree-sitter.scm
@@ -0,0 +1,190 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build-system tree-sitter)
+  #:use-module (guix store)
+  #:use-module (guix utils)
+  #:use-module (guix packages)
+  #:use-module (guix gexp)
+  #:use-module (guix monads)
+  #:use-module (guix search-paths)
+  #:use-module (guix build-system)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system node)
+  #:use-module (ice-9 match)
+  #:export (%tree-sitter-build-system-modules
+            tree-sitter-build
+            tree-sitter-build-system))
+
+(define %tree-sitter-build-system-modules
+  ;; Build-side modules imported by default.
+  `((guix build tree-sitter-build-system)
+    ,@%node-build-system-modules))
+
+(define* (lower name
+                #:key source inputs native-inputs outputs system target
+                #:allow-other-keys
+                #:rest arguments)
+  "Return a bag for NAME from the given arguments."
+  (define private-keywords
+    `(#:inputs #:native-inputs #:outputs ,@(if target
+                                               '()
+                                               '(#:target))))
+  (define node
+    (module-ref (resolve-interface '(gnu packages node))
+                'node-lts))
+  (define tree-sitter
+    (module-ref (resolve-interface '(gnu packages tree-sitter))
+                'tree-sitter))
+  (define tree-sitter-cli
+    (module-ref (resolve-interface '(gnu packages tree-sitter))
+                'tree-sitter-cli))
+  ;; Grammars depend on each other via JS modules, which we package into a
+  ;; dedicated js output.
+  (define grammar-inputs
+    (map (match-lambda
+           ((name package)
+            `(,name ,package "js")))
+         inputs))
+  (bag
+    (name name)
+    (system system) (target target)
+    (build-inputs `(,@(if source
+                          `(("source" ,source))
+                          '())
+                    ("node" ,node)
+                    ("tree-sitter-cli" ,tree-sitter-cli)
+                    ,@native-inputs
+                    ,@(if target '() grammar-inputs)
+                    ;; Keep the standard inputs of 'gnu-build-system'.
+                    ,@(if target
+                          (standard-cross-packages target 'host)
+                          '())
+                    ,@(standard-packages)))
+    (host-inputs `(("tree-sitter" ,tree-sitter)
+                   ,@(if target grammar-inputs '())))
+    ;; Keep the standard inputs of 'gnu-buid-system'.
+    (target-inputs (if target
+                       (standard-cross-packages target 'target)
+                       '()))
+    ;; XXX: this is a hack to get around issue #41569.
+    (outputs (match outputs
+               (("out") (cons "js" outputs))
+               (_ outputs)))
+    (build (if target tree-sitter-cross-build tree-sitter-build))
+    (arguments (strip-keyword-arguments private-keywords arguments))))
+
+(define* (tree-sitter-build name inputs
+                            #:key
+                            source
+                            (phases '%standard-phases)
+                            (grammar-directories '("."))
+                            (tests? #t)
+                            (outputs '("out" "js"))
+                            (search-paths '())
+                            (system (%current-system))
+                            (guile #f)
+                            (imported-modules %tree-sitter-build-system-modules)
+                            (modules '((guix build utils)
+                                       (guix build tree-sitter-build-system))))
+  (define builder
+    (with-imported-modules imported-modules
+      #~(begin
+          (use-modules #$@(sexp->gexp modules))
+          (tree-sitter-build #:name #$name
+                             #:source #+source
+                             #:system #$system
+                             #:phases #$phases
+                             #:tests? #$tests?
+                             #:grammar-directories '#$grammar-directories
+                             #:outputs #$(outputs->gexp outputs)
+                             #:search-paths '#$(sexp->gexp
+                                                (map search-path-specification->sexp
+                                                     search-paths))
+                             #:inputs #$(input-tuples->gexp inputs)))))
+
+  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+                                                  system #:graft? #f)))
+    (gexp->derivation name builder
+                      #:system system
+                      #:guile-for-build guile)))
+
+(define* (tree-sitter-cross-build name
+                                  #:key
+                                  target
+                                  build-inputs target-inputs host-inputs
+                                  guile source
+                                  (phases '%standard-phases)
+                                  (grammar-directories '("."))
+                                  (tests? #t)
+                                  (outputs '("out" "js"))
+                                  (search-paths '())
+                                  (native-search-paths '())
+                                  (system (%current-system))
+                                  (build (nix-system->gnu-triplet system))
+                                  (imported-modules %tree-sitter-build-system-modules)
+                                  (modules '((guix build utils)
+                                             (guix build tree-sitter-build-system))))
+  (define builder
+    (with-imported-modules imported-modules
+      #~(begin
+          (use-modules #$@(sexp->gexp modules))
+
+          (define %build-host-inputs
+            #+(input-tuples->gexp build-inputs))
+
+          (define %build-target-inputs
+            (append #$(input-tuples->gexp host-inputs)
+                    #+(input-tuples->gexp target-inputs)))
+
+          (define %build-inputs
+            (append %build-host-inputs %build-target-inputs))
+
+          (tree-sitter-build #:name #$name
+                             #:source #+source
+                             #:system #$system
+                             #:build #$build
+                             #:target #$target
+                             #:phases #$phases
+                             #:tests? #$tests?
+                             #:grammar-directories '#$grammar-directories
+                             #:outputs #$(outputs->gexp outputs)
+                             #:inputs %build-target-inputs
+                             #:native-inputs %build-host-inputs
+                             #:search-paths '#$(sexp->gexp
+                                                (map search-path-specification->sexp
+                                                     search-paths))
+                             #:native-search-paths '#$(sexp->gexp
+                                                       (map
+                                                        search-path-specification->sexp
+                                                        native-search-paths))))))
+
+  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+                                                  system #:graft? #f)))
+    (gexp->derivation name builder
+                      #:system system
+                      #:target target
+                      #:guile-for-build guile)))
+
+(define tree-sitter-build-system
+  (build-system
+    (name 'tree-sitter)
+    (description "The Tree-sitter grammar build system")
+    (lower lower)))
+
+;;; tree-sitter.scm ends here
diff --git a/guix/build/tree-sitter-build-system.scm b/guix/build/tree-sitter-build-system.scm
new file mode 100644
index 0000000000..ad59f56dfe
--- /dev/null
+++ b/guix/build/tree-sitter-build-system.scm
@@ -0,0 +1,153 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build tree-sitter-build-system)
+  #:use-module ((guix build node-build-system) #:prefix node:)
+  #:use-module (guix build json)
+  #:use-module (guix build utils)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module (srfi srfi-1)
+  #:export (%standard-phases
+            tree-sitter-build))
+
+;; Commentary:
+;;
+;; Build procedures for tree-sitter grammar packages.  This is the
+;; builder-side code, which builds on top fo the node build-system.
+;;
+;; Tree-sitter grammars are written in JavaScript and compiled to a native
+;; shared object.  The `tree-sitter generate' command invokes `node' in order
+;; to evaluate the grammar.js into a grammar.json file, which is then
+;; translated into C code.  We then compile the C code ourselve.  Packages
+;; also sometimes add extra manually written C/C++ code.
+;;
+;; In order to support grammars depending on each other, such as C and C++,
+;; JavaScript and TypeScript, this build-system installs the source of the
+;; node module in a dedicated "js" output.
+;;
+;; Code:
+
+(define* (patch-dependencies #:key inputs #:allow-other-keys)
+  "Rewrite dependencies in 'package.json'.  We remove all runtime dependencies
+and replace development dependencies with tree-sitter grammar node modules."
+
+  (define (rewrite package.json)
+    (map (match-lambda
+           (("dependencies" @ . _)
+            '("dependencies" @))
+           (("devDependencies" @ . _)
+            `("devDependencies" @
+              ,@(filter-map (match-lambda
+                              ((key . directory)
+                               (let ((node-module
+                                      (string-append directory
+                                                     "/lib/node_modules/"
+                                                     key)))
+                                 (and (directory-exists? node-module)
+                                      `(,key . ,node-module)))))
+                            (alist-delete "node" inputs))))
+           (other other))
+         package.json))
+
+  (node:with-atomic-json-file-replacement "package.json"
+    (match-lambda
+      (('@ . package.json)
+       (cons '@ (rewrite package.json))))))
+
+;; FIXME: The node build-system's configure phase does not support
+;; cross-compiling so we re-define it.
+(define* (configure #:key native-inputs inputs #:allow-other-keys)
+  (invoke (search-input-file (or native-inputs inputs) "/bin/npm")
+          "--offline" "--ignore-scripts" "install"))
+
+(define* (build #:key grammar-directories #:allow-other-keys)
+  (for-each (lambda (dir)
+              (with-directory-excursion dir
+                ;; Avoid generating binding code for other languages, we do
+                ;; not support this use-case yet and it relies on running
+                ;; `node-gyp' to build native addons.
+                (invoke "tree-sitter" "generate" "--no-bindings")))
+            grammar-directories))
+
+(define* (check #:key grammar-directories tests? #:allow-other-keys)
+  (when tests?
+    (for-each (lambda (dir)
+                (with-directory-excursion dir
+                  (invoke "tree-sitter" "test")))
+              grammar-directories)))
+
+(define* (install #:key target grammar-directories outputs #:allow-other-keys)
+  (let ((lib (string-append (assoc-ref outputs "out")
+                            "/lib/tree-sitter")))
+    (mkdir-p lib)
+    (define (compile-language dir)
+      (with-directory-excursion dir
+        (let ((lang (assoc-ref (call-with-input-file "src/grammar.json"
+                                 read-json)
+                               "name"))
+              (source-file (lambda (path)
+                             (if (file-exists? path)
+                                 path
+                                 #f))))
+          (apply invoke
+                 `(,(if target
+                        (string-append target "-g++")
+                        "g++")
+                   "-shared"
+                   "-fPIC"
+                   "-fno-exceptions"
+                   "-O2"
+                   "-g"
+                   "-o" ,(string-append lib "/" lang ".so")
+                   ;; An additional `scanner.{c,cc}' file is sometimes
+                   ;; provided.
+                   ,@(cond
+                      ((source-file "src/scanner.c")
+                       => (lambda (file) (list "-xc" "-std=c99" file)))
+                      ((source-file "src/scanner.cc")
+                       => (lambda (file) (list file)))
+                      (else '()))
+                   "-xc" "src/parser.c")))))
+    (for-each compile-language grammar-directories)))
+
+(define* (install-js #:key native-inputs inputs outputs #:allow-other-keys)
+  (invoke (search-input-file (or native-inputs inputs) "/bin/npm")
+          "--prefix" (assoc-ref outputs "js")
+          "--global"
+          "--offline"
+          "--loglevel" "info"
+          "--production"
+          ;; Skip scripts to prevent building bindings via GYP.
+          "--ignore-scripts"
+          "install" "../package.tgz"))
+
+(define %standard-phases
+  (modify-phases node:%standard-phases
+    (replace 'patch-dependencies patch-dependencies)
+    (replace 'configure configure)
+    (replace 'build build)
+    (replace 'check check)
+    (replace 'install install)
+    (add-after 'install 'install-js install-js)))
+
+(define* (tree-sitter-build #:key inputs (phases %standard-phases)
+                            #:allow-other-keys #:rest args)
+  (apply node:node-build #:inputs inputs #:phases phases args))
+
+;;; tree-sitter-build-system.scm ends here
-- 
2.36.0

[bug#49946] [PATCH v7 00/32] gnu: Add tree-sitter for emacs.

[Pierre Langlois]

Hi Guix!

With the recent news of tree-sitter support making it into emacs, I thought
it'd be a good idea to make sure this series is rebased and up-to-date,
hopefully nobody's working on this while not being aware of it.

This does not use the new native emacs support for tree-sitter, so the emacs
packages it introduces could eventually become obsolete. However the grammars
and the build system should be re-usable by emacs natively too.

On top of rebasing, I also:

  - Updated to 0.20.7.
  - Re-introduced a couple of rust crates needed by tree-sitter-cli.
  - Added support for R and Haskell.
  - Updated some of the grammars to make the emacs-tree-sitter-langs tests
    happy.

Thanks,
Pierre

Pierre Langlois (32):
  gnu: tree-sitter: Move to its own module.
  gnu: tree-sitter: Update to 0.20.7.
  gnu: Add rust-html-escape.
  gnu: Add rust-smallbitvec-2.
  gnu: Add tree-sitter-cli.
  build-system: Add tree-sitter-build-system.
  gnu: Add tree-sitter-c.
  gnu: Add tree-sitter-cpp.
  gnu: Add tree-sitter-bash.
  gnu: Add tree-sitter-css.
  gnu: Add tree-sitter-c-sharp.
  gnu: Add tree-sitter-elixir.
  gnu: Add tree-sitter-elm.
  gnu: Add tree-sitter-go.
  gnu: Add tree-sitter-haskell.
  gnu: Add tree-sitter-html.
  gnu: Add tree-sitter-java.
  gnu: Add tree-sitter-javascript.
  gnu: Add tree-sitter-json.
  gnu: Add tree-sitter-julia.
  gnu: Add tree-sitter-ocaml.
  gnu: Add tree-sitter-php.
  gnu: Add tree-sitter-python.
  gnu: Add tree-sitter-r.
  gnu: Add tree-sitter-ruby.
  gnu: Add tree-sitter-rust.
  gnu: Add tree-sitter-typescript.
  gnu: Add rust-tree-sitter.
  gnu: Add rust-tree-sitter-for-emacs.
  gnu: Add emacs-tree-sitter-core.
  gnu: Add emacs-tree-sitter.
  gnu: Add emacs-tree-sitter-langs.

 Makefile.am                                   |    2 +
 doc/guix.texi                                 |   21 +-
 gnu/local.mk                                  |    2 +
 gnu/packages/crates-io.scm                    |   46 +
 .../rust-tree-sitter-text-provider-fix.patch  |   98 ++
 gnu/packages/text-editors.scm                 |   52 -
 gnu/packages/tree-sitter.scm                  | 1001 +++++++++++++++++
 gnu/packages/vim.scm                          |    1 +
 guix/build-system/tree-sitter.scm             |  190 ++++
 guix/build/tree-sitter-build-system.scm       |  153 +++
 10 files changed, 1513 insertions(+), 53 deletions(-)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
 create mode 100644 gnu/packages/tree-sitter.scm
 create mode 100644 guix/build-system/tree-sitter.scm
 create mode 100644 guix/build/tree-sitter-build-system.scm

-- 
2.38.1

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

* gnu/packages/text-editors.scm (tree-sitter): Move to ...
* gnu/packages/tree-sitter.scm: ... here, a new module.
* gnu/packages/vim.scm: Use (gnu packages tree-sitter).
* gnu/local.mk (GNU_SYSTEM_MODULES): Register tree-sitter.scm new module.
---
 gnu/local.mk                  |  1 +
 gnu/packages/text-editors.scm | 52 -----------------------
 gnu/packages/tree-sitter.scm  | 78 +++++++++++++++++++++++++++++++++++
 gnu/packages/vim.scm          |  1 +
 4 files changed, 80 insertions(+), 52 deletions(-)
 create mode 100644 gnu/packages/tree-sitter.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index 7278c50e4f..7d11f4bb27 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -603,6 +603,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/tmux.scm				\
   %D%/packages/toolkits.scm			\
   %D%/packages/tor.scm				\
+  %D%/packages/tree-sitter.scm			\
   %D%/packages/tv.scm				\
   %D%/packages/uglifyjs.scm			\
   %D%/packages/uml.scm				\
diff --git a/gnu/packages/text-editors.scm b/gnu/packages/text-editors.scm
index ed77113726..57178e7b87 100644
--- a/gnu/packages/text-editors.scm
+++ b/gnu/packages/text-editors.scm
@@ -70,7 +70,6 @@ (define-module (gnu packages text-editors)
   #:use-module (gnu packages gtk)
   #:use-module (gnu packages guile)
   #:use-module (gnu packages haskell-xyz)
-  #:use-module (gnu packages icu4c)
   #:use-module (gnu packages image)
   #:use-module (gnu packages libbsd)
   #:use-module (gnu packages libreoffice)
@@ -1219,57 +1218,6 @@ (define-public edlin
 FreeDOS as a functional clone of the old MS-DOS program edlin.")
     (license license:gpl2+)))
 
-(define-public tree-sitter
-  (package
-    (name "tree-sitter")
-    (version "0.20.6")
-    (source (origin
-              (method git-fetch)
-              (uri (git-reference
-                    (url "https://github.com/tree-sitter/tree-sitter")
-                    (commit (string-append "v" version))))
-              (file-name (git-file-name name version))
-              (sha256
-               (base32
-                "1z20518snyg0zp75qgs5bxmzjqws4dd19vnp6sya494za3qp5b6d"))
-              (modules '((guix build utils)))
-              (snippet '(begin
-                          ;; Remove bundled ICU parts
-                          (delete-file-recursively "lib/src/unicode")
-                          #t))))
-    (build-system gnu-build-system)
-    (inputs (list icu4c))
-    (arguments
-     (list #:phases
-           '(modify-phases %standard-phases
-              (delete 'configure))
-           #:tests? #f ; there are no tests for the runtime library
-           #:make-flags
-           #~(list (string-append "PREFIX="
-                                  #$output)
-                   (string-append "CC="
-                                  #$(cc-for-target)))))
-    (home-page "https://tree-sitter.github.io/tree-sitter/")
-    (synopsis "Incremental parsing system for programming tools")
-    (description
-     "Tree-sitter is a parser generator tool and an incremental parsing
-library.  It can build a concrete syntax tree for a source file and efficiently
-update the syntax tree as the source file is edited.
-
-Tree-sitter aims to be:
-
-@itemize
-@item General enough to parse any programming language
-@item Fast enough to parse on every keystroke in a text editor
-@item Robust enough to provide useful results even in the presence of syntax errors
-@item Dependency-free so that the runtime library (which is written in pure C)
-can be embedded in any application
-@end itemize
-
-This package includes the @code{libtree-sitter} runtime library.
-")
-    (license license:expat)))
-
 (define-public mle
   (package
     (name "mle")
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
new file mode 100644
index 0000000000..7116a10ed7
--- /dev/null
+++ b/gnu/packages/tree-sitter.scm
@@ -0,0 +1,78 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Luis Henrique Gomes Higino <luishenriquegh2701@gmail.com>
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages tree-sitter)
+  #:use-module (guix gexp)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix git-download)
+  #:use-module (guix packages)
+  #:use-module (guix utils)
+  #:use-module (gnu packages icu4c))
+
+(define-public tree-sitter
+  (package
+    (name "tree-sitter")
+    (version "0.20.6")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1z20518snyg0zp75qgs5bxmzjqws4dd19vnp6sya494za3qp5b6d"))
+              (modules '((guix build utils)))
+              (snippet '(begin
+                          ;; Remove bundled ICU parts
+                          (delete-file-recursively "lib/src/unicode")
+                          #t))))
+    (build-system gnu-build-system)
+    (inputs (list icu4c))
+    (arguments
+     (list #:phases
+           '(modify-phases %standard-phases
+              (delete 'configure))
+           #:tests? #f ; there are no tests for the runtime library
+           #:make-flags
+           #~(list (string-append "PREFIX="
+                                  #$output)
+                   (string-append "CC="
+                                  #$(cc-for-target)))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Incremental parsing system for programming tools")
+    (description
+     "Tree-sitter is a parser generator tool and an incremental parsing
+library.  It can build a concrete syntax tree for a source file and efficiently
+update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@itemize
+@item General enough to parse any programming language
+@item Fast enough to parse on every keystroke in a text editor
+@item Robust enough to provide useful results even in the presence of syntax errors
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application
+@end itemize
+
+This package includes the @code{libtree-sitter} runtime library.
+")
+    (license license:expat)))
diff --git a/gnu/packages/vim.scm b/gnu/packages/vim.scm
index 957f00a92e..0c44c3c114 100644
--- a/gnu/packages/vim.scm
+++ b/gnu/packages/vim.scm
@@ -72,6 +72,7 @@ (define-module (gnu packages vim)
   #:use-module (gnu packages tcl)
   #:use-module (gnu packages text-editors)
   #:use-module (gnu packages terminals)
+  #:use-module (gnu packages tree-sitter)
   #:use-module (gnu packages xdisorg)
   #:use-module (gnu packages xorg))
 
-- 
2.38.1

[bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter): Update to 0.20.7.
---
 gnu/packages/tree-sitter.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 7116a10ed7..8ef4e87e41 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -29,7 +29,7 @@ (define-module (gnu packages tree-sitter)
 (define-public tree-sitter
   (package
     (name "tree-sitter")
-    (version "0.20.6")
+    (version "0.20.7")
     (source (origin
               (method git-fetch)
               (uri (git-reference
@@ -38,7 +38,7 @@ (define-public tree-sitter
               (file-name (git-file-name name version))
               (sha256
                (base32
-                "1z20518snyg0zp75qgs5bxmzjqws4dd19vnp6sya494za3qp5b6d"))
+                "1nv2a2hr22w8ix71b6rkkxv9rfvhvwlmyql0g6lva9qzj4vy50p4"))
               (modules '((guix build utils)))
               (snippet '(begin
                           ;; Remove bundled ICU parts
-- 
2.38.1

[bug#49946] [PATCH v7 03/32] gnu: Add rust-html-escape.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-html-escape): New variable.
---
 gnu/packages/crates-io.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index 792779a28f..cb3d83de5d 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -25913,6 +25913,30 @@ (define-public rust-htmlescape-0.3
      "This package provides a library for HTML entity encoding and decoding.")
     (license (list license:asl2.0 license:expat license:mpl2.0))))
 
+(define-public rust-html-escape-0.2
+  (package
+    (name "rust-html-escape")
+    (version "0.2.12")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "html-escape" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "01f2v3c6j2rk5h2lhdbh62j07cm1fvzqw4vplj2sms83jpx5qc8m"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-inputs
+       (("rust-utf8-width" ,rust-utf8-width-0.1))
+       #:cargo-development-inputs
+       (("rust-bencher" ,rust-bencher-0.1))))
+    (home-page "https://magiclen.org/html-escape")
+    (synopsis "Library for encoding and escaping special characters in HTML")
+    (description
+     "This package provides a library for encoding and escaping special
+characters in HTML, decoding and unescaping HTML entities as well.")
+    (license license:expat)))
+
 (define-public rust-hts-sys-2
   (package
     (name "rust-hts-sys")
-- 
2.38.1

[bug#49946] [PATCH v7 04/32] gnu: Add rust-smallbitvec-2.

[Pierre Langlois]

* gnu/packages/crates-io.scm (rust-smallbitvec-2): New variable.
---
 gnu/packages/crates-io.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/crates-io.scm b/gnu/packages/crates-io.scm
index cb3d83de5d..9ca52587eb 100644
--- a/gnu/packages/crates-io.scm
+++ b/gnu/packages/crates-io.scm
@@ -53163,6 +53163,28 @@ (define-public rust-sluice-0.5
 I/O programming.")
     (license license:expat)))
 
+(define-public rust-smallbitvec-2
+  (package
+    (name "rust-smallbitvec")
+    (version "2.5.1")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "smallbitvec" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "0plrbldsjpwip3afbzd8fgrnvdhizcg5z4ncfqs4q6x4qjflzkkm"))))
+    (build-system cargo-build-system)
+    (arguments
+     `(#:cargo-development-inputs
+       (("rust-bit-vec" ,rust-bit-vec-0.4)
+        ("rust-rand" ,rust-rand-0.4))))
+    (home-page "https://github.com/servo/smallbitvec")
+    (synopsis "Bit vector optimized for size and inline storage")
+    (description "This package provides a bit vector optimized for size and
+inline storage.")
+    (license (list license:expat license:asl2.0))))
+
 (define-public rust-smallvec-1
   (package
     (name "rust-smallvec")
-- 
2.38.1

[bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cli): New variable.
---
 gnu/packages/tree-sitter.scm | 80 ++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 8ef4e87e41..30bf17b79b 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -20,10 +20,13 @@
 (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages crates-graphics)
+  #:use-module (gnu packages crates-io)
   #:use-module (gnu packages icu4c))
 
 (define-public tree-sitter
@@ -76,3 +79,80 @@ (define-public tree-sitter
 This package includes the @code{libtree-sitter} runtime library.
 ")
     (license license:expat)))
+
+(define-public tree-sitter-cli
+  (package (inherit tree-sitter)
+    (name "tree-sitter-cli")
+    (source (origin
+              (inherit (package-source tree-sitter))
+              (snippet
+               '(begin
+                  ;; Remove the runtime library code and dynamically link to
+                  ;; it instead.
+                  (delete-file-recursively "lib/src")
+                  (delete-file "lib/binding_rust/build.rs")
+                  (with-output-to-file "lib/binding_rust/build.rs"
+                    (lambda _
+                      (format #t "fn main() {~@
+                              println!(\"cargo:rustc-link-lib=tree-sitter\");~@
+                              }~%")))
+                  #t))))
+    (build-system cargo-build-system)
+    (inputs (list tree-sitter))
+    (arguments
+     `(;; Running test requires downloading fixtures, see the
+       ;; script/fetch-fixtures script.
+       #:tests? #f
+       ;; We're only packaging the CLI program so we do not need to install
+       ;; sources.
+       #:install-source? #f
+       #:cargo-inputs
+       (("rust-ansi-term" ,rust-ansi-term-0.12)
+        ("rust-anyhow" ,rust-anyhow-1)
+        ("rust-atty" ,rust-atty-0.2)
+        ("rust-clap" ,rust-clap-2)
+        ("rust-difference" ,rust-difference-2)
+        ("rust-dirs" ,rust-dirs-3)
+        ("rust-html-escape" ,rust-html-escape-0.2)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-rand" ,rust-rand-0.8)
+        ("rust-rustc-hash" ,rust-rustc-hash-1)
+        ("rust-semver" ,rust-semver-1)
+        ("rust-smallbitvec" ,rust-smallbitvec-2)
+        ("rust-thiserror" ,rust-thiserror-1)
+        ("rust-tiny-http" ,rust-tiny-http-0.8)
+        ("rust-toml" ,rust-toml-0.5)
+        ("rust-walkdir" ,rust-walkdir-2)
+        ("rust-webbrowser" ,rust-webbrowser-0.5)
+        ("rust-which" ,rust-which-4))
+       #:cargo-development-inputs
+       (("rust-pretty-assertions" ,rust-pretty-assertions-0.7))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (bin (string-append out "/bin"))
+                    (lib (string-append out "/lib")))
+               (mkdir-p bin)
+               (install-file "target/release/tree-sitter" bin)))))))
+    (synopsis "Incremental parsing system for programming tools")
+    (description "Tree-sitter is a parser generator tool and an incremental
+parsing library.  It can build a concrete syntax tree for a source file and
+efficiently update the syntax tree as the source file is edited.
+
+Tree-sitter aims to be:
+
+@enumerate
+@item General enough to parse any programming language.
+@item Fast enough to parse on every keystroke in a text editor.
+@item Robust enough to provide useful results even in the presence of syntax
+errors.
+@item Dependency-free so that the runtime library (which is written in pure C)
+can be embedded in any application.
+@end enumerate
+
+This package includes the @command{tree-sitter} command-line tool.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 07/32] gnu: Add tree-sitter-c.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-c): New variable.
---
 gnu/packages/tree-sitter.scm | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 30bf17b79b..4f75b0251a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -22,6 +22,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
+  #:use-module (guix build-system tree-sitter)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
@@ -156,3 +157,35 @@ (define-public tree-sitter-cli
 
 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))
+
+(define tree-sitter-delete-generated-files
+  '(begin
+     (delete-file "binding.gyp")
+     (delete-file-recursively "bindings")
+     (delete-file "src/grammar.json")
+     (delete-file "src/node-types.json")
+     (delete-file "src/parser.c")
+     (delete-file-recursively "src/tree_sitter")
+     #t))
+
+(define-public tree-sitter-c
+  (package
+    (name "tree-sitter-c")
+    (version "0.20.2")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-c")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1w03r4l773ki4iq2xxsc2pqxf3pjsbybq3xq4glmnsihgylibn8v"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-c")
+    (synopsis "Tree-sitter C grammar")
+    (description
+     "This package provides a C grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 08/32] gnu: Add tree-sitter-cpp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-cpp): New variable.
---
 gnu/packages/tree-sitter.scm | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 4f75b0251a..63dc990475 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -189,3 +189,28 @@ (define-public tree-sitter-c
     (description
      "This package provides a C grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-cpp
+  (let ((commit "5ead1e26c6ab71919db0f1880c46a278a93bc5ea")
+        (revision "1"))
+    (package
+      (name "tree-sitter-cpp")
+      (version (git-version "0.20.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-cpp")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "1572qhfw1jjkm1q6c110lnnj2n384a97fgn645c5q9ikciv8kac7"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+      (build-system tree-sitter-build-system)
+      (inputs (list tree-sitter-c))
+      (home-page "https://github.com/tree-sitter/tree-sitter-cpp")
+      (synopsis "Tree-sitter C++ grammar")
+      (description
+       "This package provides a C++ grammar for the Tree-sitter library.")
+      (license license:expat))))
-- 
2.38.1

[bug#49946] [PATCH v7 09/32] gnu: Add tree-sitter-bash.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-bash): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 63dc990475..f583dba24d 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -214,3 +214,25 @@ (define-public tree-sitter-cpp
       (description
        "This package provides a C++ grammar for the Tree-sitter library.")
       (license license:expat))))
+
+(define-public tree-sitter-bash
+  (package
+    (name "tree-sitter-bash")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-bash")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "18c030bb65r50i6z37iy7jb9z9i8i36y7b08dbc9bchdifqsijs5"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-bash")
+    (synopsis "Tree-sitter Bash grammar")
+    (description
+     "This package provides a Bash grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 10/32] gnu: Add tree-sitter-css.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-css): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index f583dba24d..e3ec418419 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -236,3 +236,25 @@ (define-public tree-sitter-bash
     (description
      "This package provides a Bash grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-css
+  (package
+    (name "tree-sitter-css")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-css")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "014jrlgi7zfza9g38hsr4vlbi8964i5p7iglaih6qmzaiml7bja2"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-css")
+    (synopsis "Tree-sitter CSS grammar")
+    (description
+     "This package provides a CSS grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 11/32] gnu: Add tree-sitter-c-sharp.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-c-sharp): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index e3ec418419..0f65e50440 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -215,6 +215,28 @@ (define-public tree-sitter-cpp
        "This package provides a C++ grammar for the Tree-sitter library.")
       (license license:expat))))
 
+(define-public tree-sitter-c-sharp
+  (package
+    (name "tree-sitter-c-sharp")
+    (version "0.19.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-c-sharp")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "054fmpf47cwh59gbg00sc0nl237ba4rnxi73miz39yqzcs87055r"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-c-sharp")
+    (synopsis "Tree-sitter C# grammar")
+    (description
+     "This package provides a C# grammar for the Tree-sitter library.")
+    (license license:expat)))
+
 (define-public tree-sitter-bash
   (package
     (name "tree-sitter-bash")
-- 
2.38.1

[bug#49946] [PATCH v7 12/32] gnu: Add tree-sitter-elixir.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-elixir): New variable.
---
 gnu/packages/tree-sitter.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 0f65e50440..3ffb91a1a7 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -280,3 +280,29 @@ (define-public tree-sitter-css
     (description
      "This package provides a CSS grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-elixir
+  (let ((commit "de20391afe5cb03ef1e8a8e43167e7b58cc52869")
+        (revision "1"))
+    (package
+      (name "tree-sitter-elixir")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/elixir-lang/tree-sitter-elixir")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0zrkrwhw3g1vazsxcwrfd1fk4wvs9hdwmwp6073mfh370bz4140h"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+      (build-system tree-sitter-build-system)
+      (home-page "https://elixir-lang.org/tree-sitter-elixir/")
+      (synopsis "Tree-sitter Elixir grammar")
+      (description
+       "This package provides a Elixir grammar for the Tree-sitter library.")
+      (license (list license:asl2.0
+                     ;; Files in tests/corpus are under MIT license.
+                     license:expat)))))
-- 
2.38.1

[bug#49946] [PATCH v7 13/32] gnu: Add tree-sitter-elm.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-elm): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 3ffb91a1a7..11548f526b 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -306,3 +306,25 @@ (define-public tree-sitter-elixir
       (license (list license:asl2.0
                      ;; Files in tests/corpus are under MIT license.
                      license:expat)))))
+
+(define-public tree-sitter-elm
+  (package
+    (name "tree-sitter-elm")
+    (version "5.6.3")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/elm-tooling/tree-sitter-elm")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0b5jpj8bnil1ylisyc4w48j8a30dyf3zylhidj73mlrb8rf7xm2s"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://elm-tooling.github.io/tree-sitter-elm/")
+    (synopsis "Tree-sitter Elm grammar")
+    (description
+     "This package provides an Elm grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 14/32] gnu: Add tree-sitter-go.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-go): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 11548f526b..61d6eba777 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -328,3 +328,27 @@ (define-public tree-sitter-elm
     (description
      "This package provides an Elm grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-go
+  (let ((commit "05900faa3cdb5d2d8c8bd5e77ee698487e0a8611")
+        (revision "1"))
+    (package
+      (name "tree-sitter-go")
+      (version (git-version "0.19.1" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-go")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "1qymkdi4qcnj8ywmsanb6pdl9zd71cbm6kzl87zk241h7dhkkkvz"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+      (build-system tree-sitter-build-system)
+      (home-page "https://github.com/tree-sitter/tree-sitter-go")
+      (synopsis "Tree-sitter Go grammar")
+      (description
+       "This package provides a Golang grammar for the Tree-sitter library.")
+      (license license:expat))))
-- 
2.38.1

[bug#49946] [PATCH v7 15/32] gnu: Add tree-sitter-haskell.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-haskell): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 61d6eba777..668788e582 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -352,3 +352,27 @@ (define-public tree-sitter-go
       (description
        "This package provides a Golang grammar for the Tree-sitter library.")
       (license license:expat))))
+
+(define-public tree-sitter-haskell
+  (let ((commit "e30bdfd53eb28c73f26a68b77d436fd2140af167")
+        (revision "1"))
+    (package
+     (name "tree-sitter-haskell")
+     (version (git-version "0.14.0" revision commit))
+     (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-haskell")
+                    (commit commit)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1ylgs6lv1dyh3wxf756fii070r32hzbgddrfjbi2v369vmvg337p"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+     (build-system tree-sitter-build-system)
+     (home-page "https://github.com/tree-sitter/tree-sitter-haskell")
+     (synopsis "Tree-sitter Haskell grammar")
+     (description
+      "This package provides a Haskell grammar for the Tree-sitter library.")
+     (license license:expat))))
-- 
2.38.1

[bug#49946] [PATCH v7 16/32] gnu: Add tree-sitter-html.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-html): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 668788e582..f8edddccc5 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -376,3 +376,25 @@ (define-public tree-sitter-haskell
      (description
       "This package provides a Haskell grammar for the Tree-sitter library.")
      (license license:expat))))
+
+(define-public tree-sitter-html
+  (package
+    (name "tree-sitter-html")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-html")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-html")
+    (synopsis "Tree-sitter HTML grammar")
+    (description
+     "This package provides a HTML grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 17/32] gnu: Add tree-sitter-java.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-java): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index f8edddccc5..1782fa8e19 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -398,3 +398,25 @@ (define-public tree-sitter-html
     (description
      "This package provides a HTML grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-java
+  (package
+    (name "tree-sitter-java")
+    (version "0.20.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-java")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1i9zfgqibinz3rkx6yws1wk49iys32x901dki65qihbxcmcfh341"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-java")
+    (synopsis "Tree-sitter Java grammar")
+    (description
+     "This package provides a Java grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 18/32] gnu: Add tree-sitter-javascript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-javascript): New variable.
---
 gnu/packages/tree-sitter.scm | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 1782fa8e19..99fdd2de2c 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -420,3 +420,26 @@ (define-public tree-sitter-java
     (description
      "This package provides a Java grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-javascript
+  (package
+    (name "tree-sitter-javascript")
+    (version "0.20.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-javascript")
+                    (commit (string-append "rust-" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "175yrk382n2di0c2xn4gpv8y4n83x1lg4hqn04vabf0yqynlkq67"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-javascript")
+    (synopsis "Tree-sitter Javascript grammar")
+    (description
+     "This package provides Javascript and JSX grammars for the Tree-sitter
+library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 19/32] gnu: Add tree-sitter-json.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-json): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 99fdd2de2c..df9c8f7f4c 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -443,3 +443,25 @@ (define-public tree-sitter-javascript
      "This package provides Javascript and JSX grammars for the Tree-sitter
 library.")
     (license license:expat)))
+
+(define-public tree-sitter-json
+  (package
+    (name "tree-sitter-json")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-json")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "06pjh31bv9ja9hlnykk257a6zh8bsxg2fqa54al7qk1r4n9ksnff"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-json")
+    (synopsis "Tree-sitter JSON grammar")
+    (description
+     "This package provides a JSON grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 20/32] gnu: Add tree-sitter-julia.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-julia): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index df9c8f7f4c..0f9f9097a6 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -465,3 +465,25 @@ (define-public tree-sitter-json
     (description
      "This package provides a JSON grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-julia
+  (package
+    (name "tree-sitter-julia")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-julia")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1pbnmvhy2gq4vg1b0sjzmjm4s2gsgdjh7h01yj8qrrqbcl29c463"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-julia")
+    (synopsis "Tree-sitter Julia grammar")
+    (description
+     "This package provides a Julia grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 21/32] gnu: Add tree-sitter-ocaml.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-ocaml): New variable.
---
 gnu/packages/tree-sitter.scm | 39 ++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 0f9f9097a6..244eb2e8ec 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -487,3 +487,42 @@ (define-public tree-sitter-julia
     (description
      "This package provides a Julia grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-ocaml
+  (let ((commit "0348562f385bc2bd67ecf181425e1afd6d454192")
+        (revision "1"))
+    (package
+      (name "tree-sitter-ocaml")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-ocaml")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0iqmwcz3c2ai4gyx4xli1rhn6hi6a0f60dn20f8jas9ham9dc2df"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (delete-file "binding.gyp")
+                    (delete-file-recursively "bindings")
+                    (for-each
+                     (lambda (lang)
+                       (with-directory-excursion lang
+                         (delete-file "src/grammar.json")
+                         (delete-file "src/node-types.json")
+                         (delete-file "src/parser.c")
+                         (delete-file-recursively "src/tree_sitter")))
+                     '("ocaml" "interface"))
+                    #t))))
+      (build-system tree-sitter-build-system)
+      (arguments
+       (list
+        #:grammar-directories '("ocaml" "interface")))
+      (home-page "https://github.com/tree-sitter/tree-sitter-ocaml")
+      (synopsis "Tree-sitter OCaml grammar")
+      (description
+       "This package provides a OCaml grammar for the Tree-sitter library.")
+      (license license:expat))))
-- 
2.38.1

[bug#49946] [PATCH v7 22/32] gnu: Add tree-sitter-php.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-php): New variable.
---
 gnu/packages/tree-sitter.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 244eb2e8ec..0a1bd4739a 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -526,3 +526,33 @@ (define-public tree-sitter-ocaml
       (description
        "This package provides a OCaml grammar for the Tree-sitter library.")
       (license license:expat))))
+
+(define-public tree-sitter-php
+  (let ((commit "435fa00006c0d1515c37fbb4dd6a9de284af75ab")
+        (revision "1"))
+    (package
+      (name "tree-sitter-php")
+      (version (git-version "0.19.0" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-php")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "05k4h58gi616gv41r0qqdb1x4rs8y94vghn2r10yczisgzq4vbad"))
+                (modules '((guix build utils)))
+                (snippet
+                 '(begin
+                    (delete-file "src/grammar.json")
+                    (delete-file "src/node-types.json")
+                    (delete-file "src/parser.c")
+                    (delete-file-recursively "src/tree_sitter")
+                    #t))))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-php")
+    (synopsis "Tree-sitter PHP grammar")
+    (description
+     "This package provides a PHP grammar for the Tree-sitter library.")
+    (license license:expat))))
-- 
2.38.1

[bug#49946] [PATCH v7 23/32] gnu: Add tree-sitter-python.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-python): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 0a1bd4739a..c6256cd427 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -556,3 +556,27 @@ (define-public tree-sitter-php
     (description
      "This package provides a PHP grammar for the Tree-sitter library.")
     (license license:expat))))
+
+(define-public tree-sitter-python
+  (let ((commit "ed0fe62e55dc617ed9dec8817ebf771aa7cf3c42")
+        (revision "1"))
+    (package
+      (name "tree-sitter-python")
+      (version (git-version "0.19.1" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/tree-sitter/tree-sitter-python")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "0wrfpg84mc3pzcrdi6n5fqwijkqr1nj5sqfnayb502krvqpjilal"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-python")
+    (synopsis "Tree-sitter Python grammar")
+    (description
+     "This package provides a Python grammar for the Tree-sitter library.")
+    (license license:expat))))
-- 
2.38.1

[bug#49946] [PATCH v7 24/32] gnu: Add tree-sitter-r.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-r): New variable.
---
 gnu/packages/tree-sitter.scm | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index c6256cd427..7d00ee209f 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -580,3 +580,27 @@ (define-public tree-sitter-python
     (description
      "This package provides a Python grammar for the Tree-sitter library.")
     (license license:expat))))
+
+(define-public tree-sitter-r
+  (let ((commit "80efda55672d1293aa738f956c7ae384ecdc31b4")
+        (revision "1"))
+    (package
+      (name "tree-sitter-r")
+      (version (git-version "0.0.1" revision commit))
+      (source (origin
+                (method git-fetch)
+                (uri (git-reference
+                      (url "https://github.com/r-lib/tree-sitter-r")
+                      (commit commit)))
+                (file-name (git-file-name name version))
+                (sha256
+                 (base32
+                  "1n7yxi2wf9xj8snw0b85a5w40vhf7x1pwirnwfk78ilr6hhz4ix9"))
+                (modules '((guix build utils)))
+                (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/r-lib/tree-sitter-r")
+    (synopsis "Tree-sitter R grammar")
+    (description
+     "This package provides a R grammar for the Tree-sitter library.")
+    (license license:expat))))
-- 
2.38.1

[bug#49946] [PATCH v7 25/32] gnu: Add tree-sitter-ruby.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-ruby): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 7d00ee209f..19e5308818 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -604,3 +604,25 @@ (define-public tree-sitter-r
     (description
      "This package provides a R grammar for the Tree-sitter library.")
     (license license:expat))))
+
+(define-public tree-sitter-ruby
+  (package
+    (name "tree-sitter-ruby")
+    (version "0.19.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-ruby")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "0m3h4928rbs300wcb6776h9r88hi32rybbhcaf6rdympl5nzi83v"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-ruby")
+    (synopsis "Tree-sitter Ruby grammar")
+    (description
+     "This package provides a Ruby grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 26/32] gnu: Add tree-sitter-rust.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-rust): New variable.
---
 gnu/packages/tree-sitter.scm | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 19e5308818..54f2affff2 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -626,3 +626,25 @@ (define-public tree-sitter-ruby
     (description
      "This package provides a Ruby grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-rust
+  (package
+    (name "tree-sitter-rust")
+    (version "0.20.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-rust")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "174j5pxwf80c4xniri39l3a6bb7nq96g2s8hh5sgv4i7xvbpfsmg"))
+              (modules '((guix build utils)))
+              (snippet tree-sitter-delete-generated-files)))
+    (build-system tree-sitter-build-system)
+    (home-page "https://github.com/tree-sitter/tree-sitter-rust")
+    (synopsis "Tree-sitter Rust grammar")
+    (description
+     "This package provides a Rust grammar for the Tree-sitter library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 27/32] gnu: Add tree-sitter-typescript.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-typescript): New variable.
---
 gnu/packages/tree-sitter.scm | 38 ++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 54f2affff2..7cf9a41105 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -648,3 +648,41 @@ (define-public tree-sitter-rust
     (description
      "This package provides a Rust grammar for the Tree-sitter library.")
     (license license:expat)))
+
+(define-public tree-sitter-typescript
+  (package
+    (name "tree-sitter-typescript")
+    (version "0.20.1")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/tree-sitter/tree-sitter-typescript")
+                    (commit (string-append "v" version))))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32 "07fl9d968lal0aqj4f0n16p3n94cjkgfp54wynfr8gbdkjss5v5x"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  (delete-file "binding.gyp")
+                  (delete-file-recursively "bindings")
+                  (for-each
+                   (lambda (lang)
+                     (with-directory-excursion lang
+                       (delete-file "src/grammar.json")
+                       (delete-file "src/node-types.json")
+                       (delete-file "src/parser.c")
+                       (delete-file-recursively "src/tree_sitter")))
+                   '("typescript" "tsx"))
+                  #t))))
+    (build-system tree-sitter-build-system)
+    (inputs (list tree-sitter-javascript))
+    (arguments
+     (list
+      #:grammar-directories '("typescript" "tsx")))
+    (home-page "https://github.com/tree-sitter/tree-sitter-typescript")
+    (synopsis "Tree-sitter Typescript grammar")
+    (description
+     "This package provides Typescript and TSX grammars for the Tree-sitter
+library.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 28/32] gnu: Add rust-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 38 ++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 7cf9a41105..ca92fccfd6 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -23,6 +23,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
+  #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
@@ -158,6 +159,43 @@ (define-public tree-sitter-cli
 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))
 
+(define-public rust-tree-sitter
+  (package
+    (name "rust-tree-sitter")
+    (version "0.20.7")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "tree-sitter" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "183niy9malzr4dm81swcgl05xkwqamim6ck0gw5xb6k78nprz6jl"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Remove the runtime library code and dynamically link to
+                  ;; it instead.
+                  (delete-file-recursively "src")
+                  (delete-file "binding_rust/build.rs")
+                  (with-output-to-file "binding_rust/build.rs"
+                    (lambda _
+                      (format #t "fn main() {~@
+                              println!(\"cargo:rustc-link-lib=tree-sitter\");~@
+                              }~%")))
+                  #t))))
+    (build-system cargo-build-system)
+    (inputs (list tree-sitter))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-cc" ,rust-cc-1)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-regex" ,rust-regex-1))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Rust bindings to the Tree-sitter parsing library")
+    (description "This package provides Rust bindings to the Tree-sitter
+parsing library.")
+    (license license:expat)))
+
 (define tree-sitter-delete-generated-files
   '(begin
      (delete-file "binding.gyp")
-- 
2.38.1

[bug#49946] [PATCH v7 29/32] gnu: Add rust-tree-sitter-for-emacs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter-for-emacs): New variable.
* gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
---
 gnu/local.mk                                  |  1 +
 .../rust-tree-sitter-text-provider-fix.patch  | 98 +++++++++++++++++++
 gnu/packages/tree-sitter.scm                  | 14 +++
 3 files changed, 113 insertions(+)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 7d11f4bb27..961d90bee9 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1832,6 +1832,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/rust-nettle-disable-vendor.patch		 \
   %D%/packages/patches/rust-nettle-sys-disable-vendor.patch	 \
   %D%/packages/patches/rust-openssl-sys-no-vendor.patch	\
+  %D%/packages/patches/rust-tree-sitter-text-provider-fix.patch	\
   %D%/packages/patches/rust-wl-clipboard-rs-newer-wl.patch      \
   %D%/packages/patches/sbc-fix-build-non-x86.patch		\
   %D%/packages/patches/sbcl-aserve-add-HTML-5-elements.patch	\
diff --git a/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
new file mode 100644
index 0000000000..215e7fc18b
--- /dev/null
+++ b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
@@ -0,0 +1,98 @@
+From e3576b180488b8231e1fc0ca130748577579d129 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Tu=E1=BA=A5n-Anh=20Nguy=E1=BB=85n?= <ubolonton@gmail.com>
+Date: Sun, 25 Jul 2021 13:11:52 +0700
+Subject: [PATCH] Allow TextProvider's iterators to generate owned text
+
+---
+ binding_rust/lib.rs | 33 ++++++++++++++++++---------------
+ 1 file changed, 18 insertions(+), 15 deletions(-)
+
+diff --git a/lib/binding_rust/lib.rs b/lib/binding_rust/lib.rs
+index e88a411c..cf214d92 100644
+--- a/binding_rust/lib.rs
++++ b/binding_rust/lib.rs
+@@ -5,6 +5,7 @@ mod util;
+ use std::os::unix::io::AsRawFd;
+ 
+ use std::{
++    borrow::Cow,
+     char, error,
+     ffi::CStr,
+     fmt, hash, iter,
+@@ -183,7 +184,8 @@ pub struct QueryCaptures<'a, 'tree: 'a, T: TextProvider<'a>> {
+ }
+ 
+ pub trait TextProvider<'a> {
+-    type I: Iterator<Item = &'a [u8]> + 'a;
++    type I: Iterator<Item = Cow<'a, [u8]>>;
++
+     fn text(&mut self, node: Node) -> Self::I;
+ }
+ 
+@@ -1840,19 +1842,19 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+         buffer2: &mut Vec<u8>,
+         text_provider: &mut impl TextProvider<'a>,
+     ) -> bool {
+-        fn get_text<'a, 'b: 'a, I: Iterator<Item = &'b [u8]>>(
++        fn get_text<'a, 'b: 'a, I: Iterator<Item = Cow<'b, [u8]>>>(
+             buffer: &'a mut Vec<u8>,
+             mut chunks: I,
+-        ) -> &'a [u8] {
+-            let first_chunk = chunks.next().unwrap_or(&[]);
++        ) -> Cow<'a, [u8]> {
++            let first_chunk = chunks.next().unwrap_or(Cow::Owned(vec![0u8; 0]));
+             if let Some(next_chunk) = chunks.next() {
+                 buffer.clear();
+-                buffer.extend_from_slice(first_chunk);
+-                buffer.extend_from_slice(next_chunk);
++                buffer.extend_from_slice(&first_chunk);
++                buffer.extend_from_slice(&next_chunk);
+                 for chunk in chunks {
+-                    buffer.extend_from_slice(chunk);
++                    buffer.extend_from_slice(&chunk);
+                 }
+-                buffer.as_slice()
++                Cow::Borrowed(buffer.as_slice())
+             } else {
+                 first_chunk
+             }
+@@ -1888,7 +1890,7 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+                     match node {
+                         Some(node) => {
+                             let text = get_text(buffer1, text_provider.text(node));
+-                            r.is_match(text) == *is_positive
++                            r.is_match(&text) == *is_positive
+                         }
+                         None => true,
+                     }
+@@ -2002,23 +2004,24 @@ impl<'cursor, 'tree> fmt::Debug for QueryMatch<'cursor, 'tree> {
+     }
+ }
+ 
+-impl<'a, F, I> TextProvider<'a> for F
++impl<'a, F, I, T> TextProvider<'a> for F
+ where
+     F: FnMut(Node) -> I,
+-    I: Iterator<Item = &'a [u8]> + 'a,
++    T: Into<Cow<'a, [u8]>>,
++    I: Iterator<Item = T>,
+ {
+-    type I = I;
++    type I = iter::Map<I, fn(T) -> Cow<'a, [u8]>>;
+ 
+     fn text(&mut self, node: Node) -> Self::I {
+-        (self)(node)
++        (self)(node).map(T::into)
+     }
+ }
+ 
+ impl<'a> TextProvider<'a> for &'a [u8] {
+-    type I = iter::Once<&'a [u8]>;
++    type I = iter::Once<Cow<'a, [u8]>>;
+ 
+     fn text(&mut self, node: Node) -> Self::I {
+-        iter::once(&self[node.byte_range()])
++        iter::once(Cow::Borrowed(&self[node.byte_range()]))
+     }
+ }
+ 
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index ca92fccfd6..d0214f3a12 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -27,6 +27,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix git-download)
   #:use-module (guix packages)
   #:use-module (guix utils)
+  #:use-module (gnu packages)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages icu4c))
@@ -196,6 +197,19 @@ (define-public rust-tree-sitter
 parsing library.")
     (license license:expat)))
 
+;; We need to apply a patch in order to compile the rust bindings against the
+;; emacs tree-sitter module.
+;; See https://github.com/tree-sitter/tree-sitter/pull/1294
+(define-public rust-tree-sitter-for-emacs
+  (package (inherit rust-tree-sitter)
+    (source (origin
+              (inherit (package-source rust-tree-sitter))
+              (patches (search-patches
+                        "rust-tree-sitter-text-provider-fix.patch"))))
+    ;; Do not show this package in the UI as it's only meant to be used for
+    ;; emacs's tree-sitter module.
+    (properties '((hidden? . #t)))))
+
 (define tree-sitter-delete-generated-files
   '(begin
      (delete-file "binding.gyp")
-- 
2.38.1

[bug#49946] [PATCH v7 30/32] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
(emacs-tree-sitter-core): New variable.
---
 gnu/packages/tree-sitter.scm | 84 ++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d0214f3a12..d23fa2fbd8 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -21,6 +21,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix gexp)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix build-system cargo)
+  #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
   #:use-module (guix download)
@@ -738,3 +739,86 @@ (define-public tree-sitter-typescript
      "This package provides Typescript and TSX grammars for the Tree-sitter
 library.")
     (license license:expat)))
+
+;; Local package definition solely for building the native emacs module
+;; written in Rust.
+(define tree-sitter-emacs-module
+  (package
+    (name "tree-sitter-emacs-module")
+    (version "0.18.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1sdvz827v436qijs6xafakkfw2d16bvp8frymd818rppjc7a9dif"))))
+    (build-system cargo-build-system)
+    (inputs
+     (list tree-sitter))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-anyhow" ,rust-anyhow-1)
+        ("rust-emacs" ,rust-emacs-0.18)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-tree-sitter" ,rust-tree-sitter-for-emacs))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-after 'chdir 'delete-cargo.lock
+           (lambda _ (delete-file "Cargo.lock")))
+         (add-after 'delete-cargo.lock 'do-not-fetch-from-github
+           (lambda _
+             (substitute* "Cargo.toml"
+               (("\\[patch.*") "")
+               (("git = .*") ""))))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((lib (string-append (assoc-ref outputs "out") "/lib")))
+               (mkdir-p lib)
+               (copy-file "target/release/libtsc_dyn.so"
+                          (string-append lib "/tsc-dyn.so"))))))))
+    (home-page #f)
+    (synopsis #f)
+    (description #f)
+    (license license:expat)))
+
+(define-public emacs-tree-sitter-core
+  (package
+    (name "emacs-tree-sitter-core")
+    (version (package-version tree-sitter-emacs-module))
+    (source (package-source tree-sitter-emacs-module))
+    (build-system emacs-build-system)
+    (native-inputs
+     (list tree-sitter-emacs-module))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "core")))
+         (add-before 'install 'find-tsc-dyn
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let ((elpa (elpa-directory (assoc-ref outputs "out")))
+                   (module (search-input-file inputs "/lib/tsc-dyn.so")))
+               ;; Install the tsc-dyn module in site-lisp and the current
+               ;; directory for test to pass.
+               (install-file module elpa)
+               (copy-file module "tsc-dyn.so")
+               ;; We replace the tsc-dyn-get.el file with an empty stub to
+               ;; prevent the code from downloading the module.
+               (call-with-output-file "tsc-dyn-get.el"
+                 (lambda (port)
+                   (for-each
+                    (lambda (sexp) (write sexp port))
+                    '((defun tsc-dyn-get-ensure (requested)
+                        nil)
+                      (provide 'tsc-dyn-get)))))))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp, core library")
+    (description "This package provides core APIs of the Emacs binding for
+Tree-sitter, an incremental parsing system.")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 31/32] gnu: Add emacs-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index d23fa2fbd8..26b0206575 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -822,3 +822,30 @@ (module (search-input-file inputs "/lib/tsc-dyn.so")))
     (description "This package provides core APIs of the Emacs binding for
 Tree-sitter, an incremental parsing system.")
     (license license:expat)))
+
+(define-public emacs-tree-sitter
+  (package
+    (name "emacs-tree-sitter")
+    (version (package-version emacs-tree-sitter-core))
+    (source (package-source emacs-tree-sitter-core))
+    (build-system emacs-build-system)
+    (propagated-inputs
+     (list emacs-tree-sitter-core))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'chdir
+           (lambda _ (chdir "lisp"))))))
+    (home-page "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+    (synopsis "Tree-sitter bindings for Emacs Lisp")
+    (description "This package provides Emacs bindings for Tree-sitter, an
+incremental parsing library.  It aims to be the foundation for a new breed of
+Emacs packages that understand code structurally.  For example:
+
+@enumerate
+@item Faster, fine-grained code highlighting.
+@item More flexible code folding.
+@item Structural editing (like Paredit, or even better) for non-Lisp code.
+@item More informative indexing for imenu.
+@end enumerate")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 32/32] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (make-emacs-tree-sitter-langs-grammar-bundle):
New procedure.
(emacs-tree-sitter-langs): New variable.
---
 gnu/packages/tree-sitter.scm | 150 +++++++++++++++++++++++++++++++++++
 1 file changed, 150 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 26b0206575..02d515bdc4 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -24,6 +24,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
+  #:use-module (guix build-system trivial)
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix packages)
@@ -849,3 +850,152 @@ (define-public emacs-tree-sitter
 @item More informative indexing for imenu.
 @end enumerate")
     (license license:expat)))
+
+(define emacs-tree-sitter-langs-grammar-bundle
+  (package
+    (name "emacs-tree-sitter-langs-grammar-bundle")
+    (source #f)
+    (version (package-version tree-sitter))
+    (build-system trivial-build-system)
+    (inputs
+     ;; FIXME: Support for some languages is still left to package.
+     (list tree-sitter-bash
+           tree-sitter-c
+           tree-sitter-c-sharp
+           tree-sitter-cpp
+           tree-sitter-css
+           tree-sitter-elixir
+           tree-sitter-elm
+           tree-sitter-go
+           tree-sitter-haskell
+           tree-sitter-html
+           tree-sitter-java
+           tree-sitter-javascript
+           tree-sitter-json
+           tree-sitter-julia
+           tree-sitter-ocaml
+           tree-sitter-php
+           tree-sitter-python
+           tree-sitter-r
+           tree-sitter-rust
+           tree-sitter-ruby
+           tree-sitter-typescript))
+    (arguments
+     (list #:builder
+           (with-imported-modules '((guix build union)
+                                    (guix build utils))
+             #~(begin
+                 (use-modules (ice-9 match)
+                              (guix build union)
+                              (guix build utils))
+                 (union-build
+                  #$output
+                  (filter directory-exists?
+                          (map (match-lambda
+                                 ((name directory)
+                                  (string-append directory "/lib/tree-sitter")))
+                               '#$(package-inputs this-package))))))))
+    (synopsis #f)
+    (description #f)
+    (home-page #f)
+    (license #f)))
+
+(define-public emacs-tree-sitter-langs
+  (package
+    (name "emacs-tree-sitter-langs")
+    (version "0.12.6")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/tree-sitter-langs")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1qqf5naf4a9zm0am1rn9yn2b2s6q3h9fapsl5d546vbsyvak5q40"))))
+    (build-system emacs-build-system)
+    (inputs
+     (list emacs-tree-sitter-langs-grammar-bundle))
+    (propagated-inputs
+     (list emacs-tree-sitter))
+    (arguments
+     (list
+      #:tests? #t
+      #:test-command ''("script/test")
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'disable-downloader
+            (lambda _
+              (call-with-output-file "tree-sitter-langs-build.el"
+                (lambda (port)
+                  (let ((on-load-message
+                         (string-append
+                          "tree-sitter-langs: Grammar bundle already installed "
+                          "via Guix.  Installing external grammars via this "
+                          "function isn't supported, if a language you need is "
+                          "missing please report a bug at bug-guix@gnu.org.")))
+                    (format
+                     port
+                     ";;;###autoload
+                      (defun tree-sitter-langs-install-grammars
+                             (&optional skip-if-installed version os
+                                        keep-bundle)
+                        (interactive)
+                        (message \"~a\"))
+                      (defconst tree-sitter-langs--queries-dir
+                        (file-name-as-directory
+                          (concat (file-name-directory (locate-library \"tree-sitter-langs.el\"))
+                                  \"queries\")))
+                      (defun tree-sitter-langs--bin-dir () \"~a\")
+                      (provide 'tree-sitter-langs-build)"
+                     on-load-message
+                     #$emacs-tree-sitter-langs-grammar-bundle))))))
+          (add-after 'unpack 'remove-cask
+            (lambda _
+              (substitute* "script/test"
+                (("cask") ""))))
+          (add-before 'check 'patch-tree-sitter-require-test
+            (lambda _
+              (use-modules (ice-9 regex))
+              ;; This test needs a git repositories with submodules for
+              ;; each languages in order to map all repositories.  We patch
+              ;; the mapping function with one that invokes the tests for each
+              ;; packaged language.
+              (let ((supported-languages
+                     (map (lambda (lib)
+                            (match:substring
+                             (string-match "(.*)\\.so$" (basename lib))
+                             1))
+                          (find-files "bin" "\\.so$"))))
+                (substitute* "tree-sitter-langs-tests.el"
+                  (("tree-sitter-langs--map-repos")
+                   (call-with-output-string
+                     (lambda (port)
+                       (write `(lambda (fn)
+                                 (dolist (lang ',supported-languages)
+                                         (funcall fn lang)))
+                              port))))))))
+          ;; Tests for queries will fail given those languages are not
+          ;; packages yet.
+          (add-before 'check 'remove-unused-highlight-queries
+            (lambda _
+              (delete-file-recursively "queries/hcl")
+              (delete-file-recursively "queries/pgn")))
+          (add-after 'install 'install-queries
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
+                (copy-recursively "queries" (string-append elpa "/queries"))))))))
+    (home-page "https://ubolonton.github.io/emacs-tree-sitter/languages/")
+    (synopsis "Language support bundle for Tree-sitter")
+    (description "This package is a convenient language bundle for
+Tree-sitter.  For each supported language, this package provides:
+
+@enumerate
+@item Pre-compiled grammar binaries.
+@item An optional highlights.scm file that provides highlighting patterns.
+This is mainly intended for major modes that are not aware of tree-sitter.
+@item Optional query patterns for other minor modes that provide high-level
+functionalities on top of tree-sitter, such as code folding, evil text
+objects, ...etc.
+@end enumerate")
+    (license license:expat)))
-- 
2.38.1

[bug#49946] [PATCH v7 06/32] build-system: Add tree-sitter-build-system.

[Pierre Langlois]

[-- Attachment #1.1: Type: text/plain, Size: 77 bytes --]

It appears this email is making it through, so here it is as an
attachment:


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Attachment #2: 0006-build-system-Add-tree-sitter-build-system.patch --]
[-- Type: text/x-patch, Size: 19142 bytes --]

From 4a3c1fff8460a03bfb7c1aada9863205cd6f22fd Mon Sep 17 00:00:00 2001
From: Pierre Langlois <pierre.langlois@gmx.com>
Date: Tue, 29 Mar 2022 20:13:34 +0100
Subject: [PATCH v7 06/32] build-system: Add tree-sitter-build-system.

* guix/build-system/tree-sitter.scm: New module.
* guix/build/tree-sitter-build-system.scm: Likewise.
* Makefile.am (MODULES): Add them.
* doc/guix.texi: Document it.
---
 Makefile.am                             |   2 +
 doc/guix.texi                           |  21 ++-
 guix/build-system/tree-sitter.scm       | 190 ++++++++++++++++++++++++
 guix/build/tree-sitter-build-system.scm | 153 +++++++++++++++++++
 4 files changed, 365 insertions(+), 1 deletion(-)
 create mode 100644 guix/build-system/tree-sitter.scm
 create mode 100644 guix/build/tree-sitter-build-system.scm

diff --git a/Makefile.am b/Makefile.am
index c3af23b68e..a16c4fcd7e 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -178,6 +178,7 @@ MODULES =					\
   guix/build-system/ruby.scm			\
   guix/build-system/scons.scm			\
   guix/build-system/texlive.scm			\
+  guix/build-system/tree-sitter.scm		\
   guix/build-system/trivial.scm			\
   guix/ftp-client.scm				\
   guix/http-client.scm				\
@@ -234,6 +235,7 @@ MODULES =					\
   guix/build/ruby-build-system.scm		\
   guix/build/scons-build-system.scm		\
   guix/build/texlive-build-system.scm		\
+  guix/build/tree-sitter-build-system.scm	\
   guix/build/waf-build-system.scm		\
   guix/build/haskell-build-system.scm		\
   guix/build/julia-build-system.scm		\
diff --git a/doc/guix.texi b/doc/guix.texi
index e547d469f4..4e997f7176 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -79,7 +79,7 @@ Copyright @copyright{} 2020 Jack Hill@*
 Copyright @copyright{} 2020 Naga Malleswari@*
 Copyright @copyright{} 2020, 2021 Brice Waegeneire@*
 Copyright @copyright{} 2020 R Veera Kumar@*
-Copyright @copyright{} 2020, 2021 Pierre Langlois@*
+Copyright @copyright{} 2020, 2021, 2022 Pierre Langlois@*
 Copyright @copyright{} 2020 pinoaffe@*
 Copyright @copyright{} 2020 André Batista@*
 Copyright @copyright{} 2020, 2021 Alexandru-Sergiu Marton@*
@@ -9732,6 +9732,25 @@ be specified with the @code{#:node} parameter which defaults to
 @code{node}.
 @end defvr
 
+@defvr {Scheme Variable} tree-sitter-build-system
+
+This variable is exported by @code{(guix build-system tree-sitter)}.  It
+implements procedures to compile grammars for the
+@url{https://tree-sitter.github.io/tree-sitter/, Tree-sitter} parsing
+library.  It essentially runs @code{tree-sitter generate} to translate
+@code{grammar.js} grammars to JSON and then to C.  Which it then
+compiles to native code.
+
+Tree-sitter packages may support multiple grammars, so this build system
+supports a @code{#:grammar-directories} keyword to specify a list of
+locations where a @code{grammar.js} file may be found.
+
+Grammars sometimes depend on each other, such as C++ depending on C and
+TypeScript depending on JavaScript.  You may use inputs to declare such
+dependencies.
+
+@end defvr
+
 Lastly, for packages that do not need anything as sophisticated, a
 ``trivial'' build system is provided.  It is trivial in the sense that
 it provides basically no support: it does not pull any implicit inputs,
diff --git a/guix/build-system/tree-sitter.scm b/guix/build-system/tree-sitter.scm
new file mode 100644
index 0000000000..aeb96e3ef5
--- /dev/null
+++ b/guix/build-system/tree-sitter.scm
@@ -0,0 +1,190 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build-system tree-sitter)
+  #:use-module (guix store)
+  #:use-module (guix utils)
+  #:use-module (guix packages)
+  #:use-module (guix gexp)
+  #:use-module (guix monads)
+  #:use-module (guix search-paths)
+  #:use-module (guix build-system)
+  #:use-module (guix build-system gnu)
+  #:use-module (guix build-system node)
+  #:use-module (ice-9 match)
+  #:export (%tree-sitter-build-system-modules
+            tree-sitter-build
+            tree-sitter-build-system))
+
+(define %tree-sitter-build-system-modules
+  ;; Build-side modules imported by default.
+  `((guix build tree-sitter-build-system)
+    ,@%node-build-system-modules))
+
+(define* (lower name
+                #:key source inputs native-inputs outputs system target
+                #:allow-other-keys
+                #:rest arguments)
+  "Return a bag for NAME from the given arguments."
+  (define private-keywords
+    `(#:inputs #:native-inputs #:outputs ,@(if target
+                                               '()
+                                               '(#:target))))
+  (define node
+    (module-ref (resolve-interface '(gnu packages node))
+                'node-lts))
+  (define tree-sitter
+    (module-ref (resolve-interface '(gnu packages tree-sitter))
+                'tree-sitter))
+  (define tree-sitter-cli
+    (module-ref (resolve-interface '(gnu packages tree-sitter))
+                'tree-sitter-cli))
+  ;; Grammars depend on each other via JS modules, which we package into a
+  ;; dedicated js output.
+  (define grammar-inputs
+    (map (match-lambda
+           ((name package)
+            `(,name ,package "js")))
+         inputs))
+  (bag
+    (name name)
+    (system system) (target target)
+    (build-inputs `(,@(if source
+                          `(("source" ,source))
+                          '())
+                    ("node" ,node)
+                    ("tree-sitter-cli" ,tree-sitter-cli)
+                    ,@native-inputs
+                    ,@(if target '() grammar-inputs)
+                    ;; Keep the standard inputs of 'gnu-build-system'.
+                    ,@(if target
+                          (standard-cross-packages target 'host)
+                          '())
+                    ,@(standard-packages)))
+    (host-inputs `(("tree-sitter" ,tree-sitter)
+                   ,@(if target grammar-inputs '())))
+    ;; Keep the standard inputs of 'gnu-buid-system'.
+    (target-inputs (if target
+                       (standard-cross-packages target 'target)
+                       '()))
+    ;; XXX: this is a hack to get around issue #41569.
+    (outputs (match outputs
+               (("out") (cons "js" outputs))
+               (_ outputs)))
+    (build (if target tree-sitter-cross-build tree-sitter-build))
+    (arguments (strip-keyword-arguments private-keywords arguments))))
+
+(define* (tree-sitter-build name inputs
+                            #:key
+                            source
+                            (phases '%standard-phases)
+                            (grammar-directories '("."))
+                            (tests? #t)
+                            (outputs '("out" "js"))
+                            (search-paths '())
+                            (system (%current-system))
+                            (guile #f)
+                            (imported-modules %tree-sitter-build-system-modules)
+                            (modules '((guix build utils)
+                                       (guix build tree-sitter-build-system))))
+  (define builder
+    (with-imported-modules imported-modules
+      #~(begin
+          (use-modules #$@(sexp->gexp modules))
+          (tree-sitter-build #:name #$name
+                             #:source #+source
+                             #:system #$system
+                             #:phases #$phases
+                             #:tests? #$tests?
+                             #:grammar-directories '#$grammar-directories
+                             #:outputs #$(outputs->gexp outputs)
+                             #:search-paths '#$(sexp->gexp
+                                                (map search-path-specification->sexp
+                                                     search-paths))
+                             #:inputs #$(input-tuples->gexp inputs)))))
+
+  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+                                                  system #:graft? #f)))
+    (gexp->derivation name builder
+                      #:system system
+                      #:guile-for-build guile)))
+
+(define* (tree-sitter-cross-build name
+                                  #:key
+                                  target
+                                  build-inputs target-inputs host-inputs
+                                  guile source
+                                  (phases '%standard-phases)
+                                  (grammar-directories '("."))
+                                  (tests? #t)
+                                  (outputs '("out" "js"))
+                                  (search-paths '())
+                                  (native-search-paths '())
+                                  (system (%current-system))
+                                  (build (nix-system->gnu-triplet system))
+                                  (imported-modules %tree-sitter-build-system-modules)
+                                  (modules '((guix build utils)
+                                             (guix build tree-sitter-build-system))))
+  (define builder
+    (with-imported-modules imported-modules
+      #~(begin
+          (use-modules #$@(sexp->gexp modules))
+
+          (define %build-host-inputs
+            #+(input-tuples->gexp build-inputs))
+
+          (define %build-target-inputs
+            (append #$(input-tuples->gexp host-inputs)
+                    #+(input-tuples->gexp target-inputs)))
+
+          (define %build-inputs
+            (append %build-host-inputs %build-target-inputs))
+
+          (tree-sitter-build #:name #$name
+                             #:source #+source
+                             #:system #$system
+                             #:build #$build
+                             #:target #$target
+                             #:phases #$phases
+                             #:tests? #$tests?
+                             #:grammar-directories '#$grammar-directories
+                             #:outputs #$(outputs->gexp outputs)
+                             #:inputs %build-target-inputs
+                             #:native-inputs %build-host-inputs
+                             #:search-paths '#$(sexp->gexp
+                                                (map search-path-specification->sexp
+                                                     search-paths))
+                             #:native-search-paths '#$(sexp->gexp
+                                                       (map
+                                                        search-path-specification->sexp
+                                                        native-search-paths))))))
+
+  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
+                                                  system #:graft? #f)))
+    (gexp->derivation name builder
+                      #:system system
+                      #:target target
+                      #:guile-for-build guile)))
+
+(define tree-sitter-build-system
+  (build-system
+    (name 'tree-sitter)
+    (description "The Tree-sitter grammar build system")
+    (lower lower)))
+
+;;; tree-sitter.scm ends here
diff --git a/guix/build/tree-sitter-build-system.scm b/guix/build/tree-sitter-build-system.scm
new file mode 100644
index 0000000000..574b0f2a1c
--- /dev/null
+++ b/guix/build/tree-sitter-build-system.scm
@@ -0,0 +1,153 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix build tree-sitter-build-system)
+  #:use-module ((guix build node-build-system) #:prefix node:)
+  #:use-module (guix build json)
+  #:use-module (guix build utils)
+  #:use-module (ice-9 match)
+  #:use-module (ice-9 regex)
+  #:use-module (srfi srfi-1)
+  #:export (%standard-phases
+            tree-sitter-build))
+
+;; Commentary:
+;;
+;; Build procedures for tree-sitter grammar packages.  This is the
+;; builder-side code, which builds on top fo the node build-system.
+;;
+;; Tree-sitter grammars are written in JavaScript and compiled to a native
+;; shared object.  The `tree-sitter generate' command invokes `node' in order
+;; to evaluate the grammar.js into a grammar.json file, which is then
+;; translated into C code.  We then compile the C code ourselves.  Packages
+;; also sometimes add extra manually written C/C++ code.
+;;
+;; In order to support grammars depending on each other, such as C and C++,
+;; JavaScript and TypeScript, this build-system installs the source of the
+;; node module in a dedicated "js" output.
+;;
+;; Code:
+
+(define* (patch-dependencies #:key inputs #:allow-other-keys)
+  "Rewrite dependencies in 'package.json'.  We remove all runtime dependencies
+and replace development dependencies with tree-sitter grammar node modules."
+
+  (define (rewrite package.json)
+    (map (match-lambda
+           (("dependencies" @ . _)
+            '("dependencies" @))
+           (("devDependencies" @ . _)
+            `("devDependencies" @
+              ,@(filter-map (match-lambda
+                              ((key . directory)
+                               (let ((node-module
+                                      (string-append directory
+                                                     "/lib/node_modules/"
+                                                     key)))
+                                 (and (directory-exists? node-module)
+                                      `(,key . ,node-module)))))
+                            (alist-delete "node" inputs))))
+           (other other))
+         package.json))
+
+  (node:with-atomic-json-file-replacement "package.json"
+    (match-lambda
+      (('@ . package.json)
+       (cons '@ (rewrite package.json))))))
+
+;; FIXME: The node build-system's configure phase does not support
+;; cross-compiling so we re-define it.
+(define* (configure #:key native-inputs inputs #:allow-other-keys)
+  (invoke (search-input-file (or native-inputs inputs) "/bin/npm")
+          "--offline" "--ignore-scripts" "install"))
+
+(define* (build #:key grammar-directories #:allow-other-keys)
+  (for-each (lambda (dir)
+              (with-directory-excursion dir
+                ;; Avoid generating binding code for other languages, we do
+                ;; not support this use-case yet and it relies on running
+                ;; `node-gyp' to build native addons.
+                (invoke "tree-sitter" "generate" "--no-bindings")))
+            grammar-directories))
+
+(define* (check #:key grammar-directories tests? #:allow-other-keys)
+  (when tests?
+    (for-each (lambda (dir)
+                (with-directory-excursion dir
+                  (invoke "tree-sitter" "test")))
+              grammar-directories)))
+
+(define* (install #:key target grammar-directories outputs #:allow-other-keys)
+  (let ((lib (string-append (assoc-ref outputs "out")
+                            "/lib/tree-sitter")))
+    (mkdir-p lib)
+    (define (compile-language dir)
+      (with-directory-excursion dir
+        (let ((lang (assoc-ref (call-with-input-file "src/grammar.json"
+                                 read-json)
+                               "name"))
+              (source-file (lambda (path)
+                             (if (file-exists? path)
+                                 path
+                                 #f))))
+          (apply invoke
+                 `(,(if target
+                        (string-append target "-g++")
+                        "g++")
+                   "-shared"
+                   "-fPIC"
+                   "-fno-exceptions"
+                   "-O2"
+                   "-g"
+                   "-o" ,(string-append lib "/" lang ".so")
+                   ;; An additional `scanner.{c,cc}' file is sometimes
+                   ;; provided.
+                   ,@(cond
+                      ((source-file "src/scanner.c")
+                       => (lambda (file) (list "-xc" "-std=c99" file)))
+                      ((source-file "src/scanner.cc")
+                       => (lambda (file) (list file)))
+                      (else '()))
+                   "-xc" "src/parser.c")))))
+    (for-each compile-language grammar-directories)))
+
+(define* (install-js #:key native-inputs inputs outputs #:allow-other-keys)
+  (invoke (search-input-file (or native-inputs inputs) "/bin/npm")
+          "--prefix" (assoc-ref outputs "js")
+          "--global"
+          "--offline"
+          "--loglevel" "info"
+          "--production"
+          ;; Skip scripts to prevent building bindings via GYP.
+          "--ignore-scripts"
+          "install" "../package.tgz"))
+
+(define %standard-phases
+  (modify-phases node:%standard-phases
+    (replace 'patch-dependencies patch-dependencies)
+    (replace 'configure configure)
+    (replace 'build build)
+    (replace 'check check)
+    (replace 'install install)
+    (add-after 'install 'install-js install-js)))
+
+(define* (tree-sitter-build #:key inputs (phases %standard-phases)
+                            #:allow-other-keys #:rest args)
+  (apply node:node-build #:inputs inputs #:phases phases args))
+
+;;; tree-sitter-build-system.scm ends here
-- 
2.38.1

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[( via Guix-patches via]

Heya,

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/text-editors.scm (tree-sitter): Move to ...
> * gnu/packages/tree-sitter.scm: ... here, a new module.
> * gnu/packages/vim.scm: Use (gnu packages tree-sitter).
> * gnu/local.mk (GNU_SYSTEM_MODULES): Register tree-sitter.scm new module.

> --- /dev/null
> +++ b/gnu/packages/tree-sitter.scm

> @@ -0,0 +1,78 @@

> +(define-module (gnu packages tree-sitter)
> +  #:use-module (guix gexp)
> +  #:use-module ((guix licenses) #:prefix license:)
> +  #:use-module (guix build-system gnu)
> +  #:use-module (guix git-download)
> +  #:use-module (guix packages)
> +  #:use-module (guix utils)
> +  #:use-module (gnu packages icu4c))

Please sort these alphabetically.

> +(define-public tree-sitter
> +  (package
> +    (name "tree-sitter")
> +    (version "0.20.6")
> +    (source (origin
> +              (method git-fetch)
> +              (uri (git-reference
> +                    (url "https://github.com/tree-sitter/tree-sitter")
> +                    (commit (string-append "v" version))))
> +              (file-name (git-file-name name version))
> +              (sha256
> +               (base32
> +                "1z20518snyg0zp75qgs5bxmzjqws4dd19vnp6sya494za3qp5b6d"))
> +              (modules '((guix build utils)))
> +              (snippet '(begin

Please change it to use a gexp: ``(snippet #~(begin ...))''.

> +                          ;; Remove bundled ICU parts
> +                          (delete-file-recursively "lib/src/unicode")
> +                          #t))))

This #T is now unnecessary.

> +    (arguments
> +     (list #:phases
> +           '(modify-phases %standard-phases
> +              (delete 'configure))

Please use a gexp here, too.

> +           #:make-flags
> +           #~(list (string-append "PREFIX="
> +                                  #$output)
> +                   (string-append "CC="
> +                                  #$(cc-for-target)))))

Maybe modify the indentation here a little.

  (string-append "PREFIX=" #$output)
  (string-append "CC=" #$(cc-for-target))

> +
> +This package includes the @code{libtree-sitter} runtime library.
> +")

Please remove the newline at the end of the description.

    -- (

[bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter): Update to 0.20.7.

LGTM, but maybe you could use this commit to make the stylistic
changes I suggested in the last reply.

    -- (

[bug#49946] [PATCH v7 03/32] gnu: Add rust-html-escape.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/crates-io.scm (rust-html-escape): New variable.

> --- a/gnu/packages/crates-io.scm
> +++ b/gnu/packages/crates-io.scm

> @@ -25913,6 +25913,30 @@ (define-public rust-htmlescape-0.3

> +    (arguments
> +     `(#:cargo-inputs
> +       (("rust-utf8-width" ,rust-utf8-width-0.1))
> +       #:cargo-development-inputs
> +       (("rust-bencher" ,rust-bencher-0.1))))

Please use LIST here:

  (arguments
   (list #:cargo-inputs
         `(("rust-utf8-width" ,rust-utf8-width-0.1))
         #:cargo-development-inputs
        `(("rust-bencher" ,rust-bencher-0.1))))
  
> +    (description
> +     "This package provides a library for encoding and escaping special
> +characters in HTML, decoding and unescaping HTML entities as well.")

This isn't quite a full, grammatical sentence (I guess it was lifted from
the crates.io page?) How about this?

  (description
   "This package provides a library for encoding special characters
as HTML entities and vice versa.")

    -- (

[bug#49946] [PATCH v7 04/32] gnu: Add rust-smallbitvec-2.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/crates-io.scm (rust-smallbitvec-2): New variable.

> --- a/gnu/packages/crates-io.scm
> +++ b/gnu/packages/crates-io.scm

> @@ -53163,6 +53163,28 @@ (define-public rust-sluice-0.5

> +     `(#:cargo-development-inputs
> +       (("rust-bit-vec" ,rust-bit-vec-0.4)
> +        ("rust-rand" ,rust-rand-0.4))))

LIST here please.

    -- (

[bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-cli): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -76,3 +79,80 @@ (define-public tree-sitter
> +              (snippet
> +               '(begin

Please use a gexp here.

> +                  (with-output-to-file "lib/binding_rust/build.rs"
> +                    (lambda _
> +                      (format #t "fn main() {~@
> +                              println!(\"cargo:rustc-link-lib=tree-sitter\");~@
> +                              }~%")))
> +                  #t))))

The #T is redundant.

> +    (arguments
> +     `(;; Running test requires downloading fixtures, see the

Please use LIST.

> +       ;; script/fetch-fixtures script.
> +       #:tests? #f

Couldn't we fetch the fixtures ourselves with a secondary origin,
like I did here? <https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/golang.scm#n10561>

> +       #:phases
> +       (modify-phases %standard-phases
> +         (add-after 'unpack 'delete-cargo.lock
> +           (lambda _ (delete-file "Cargo.lock")))
> +         (replace 'install
> +           (lambda* (#:key outputs #:allow-other-keys)
> +             (let* ((out (assoc-ref outputs "out"))
> +                    (bin (string-append out "/bin"))
> +                    (lib (string-append out "/lib")))
> +               (mkdir-p bin)
> +               (install-file "target/release/tree-sitter" bin)))))))

Update this to modern Guix style, please:

  #~(modify-phases %standard-phases
      (add-after 'unpack 'delete-cargo-lock
        (lambda _
          (delete-file "Cargo.lock")))
      (replace 'install
        (lambda* (#:key outputs #:allow-other-keys)
          (let ((bin (string-append #$output "/bin")))
            (mkdir-p bin)
            (install-file "target/release/tree-sitter" bin)))))

    -- (

[bug#49946] [PATCH v7 07/32] gnu: Add tree-sitter-c.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-c): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -156,3 +157,35 @@ (define-public tree-sitter-cli

>  This package includes the @command{tree-sitter} command-line tool.")
>      (license license:expat)))
> +
> +(define tree-sitter-delete-generated-files
> +  '(begin
> +     (delete-file "binding.gyp")
> +     (delete-file-recursively "bindings")
> +     (delete-file "src/grammar.json")
> +     (delete-file "src/node-types.json")
> +     (delete-file "src/parser.c")
> +     (delete-file-recursively "src/tree_sitter")
> +     #t))

No need to outline this; just do

  (snippet
   #~(begin
       (for-each delete-file
                 (list "binding.gyp"
                       "src/grammar.json"
                       "src/node-types.json"
                       "src/parser.c"))
       (for-each delete-file-recursively
                 (list "bindings"
                       "src/tree_sitter"))))

Shouldn't some of this deletion (namely, that of pregenerated files)
be handled by tree-sitter, though?

> +    (build-system tree-sitter-build-system)

...I think you may have forgotten to send the patches for the
tree-sitter-build-system?

    -- (

[bug#49946] [PATCH v7 07/32] gnu: Add tree-sitter-c.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> +(define tree-sitter-delete-generated-files
> +  '(begin
> +     (delete-file "binding.gyp")
> +     (delete-file-recursively "bindings")
> +     (delete-file "src/grammar.json")
> +     (delete-file "src/node-types.json")
> +     (delete-file "src/parser.c")
> +     (delete-file-recursively "src/tree_sitter")
> +     #t))

Aah, I just realised that the other grammars use this... My point
about incorporating it into tree-sitter-build-system still stands,
though.

    -- (

[bug#49946] [PATCH v7 09/32] gnu: Add tree-sitter-bash.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-bash): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 10/32] gnu: Add tree-sitter-css.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-css): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 11/32] gnu: Add tree-sitter-c-sharp.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-c-sharp): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 12/32] gnu: Add tree-sitter-elixir.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-elixir): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 13/32] gnu: Add tree-sitter-elm.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-elm): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 14/32] gnu: Add tree-sitter-go.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-go): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -328,3 +328,27 @@ (define-public tree-sitter-elm

> +(define-public tree-sitter-go
> +  (let ((commit "05900faa3cdb5d2d8c8bd5e77ee698487e0a8611")
> +        (revision "1"))

Why are you using a commit? Please add an explanatory comment. And
start at (revision "0") :)

    -- (

[bug#49946] [PATCH v7 08/32] gnu: Add tree-sitter-cpp.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-cpp): New variable.

LGTM (apart from nonexistant tree-sitter-build-system :))

    -- (

[bug#49946] [PATCH v7 08/32] gnu: Add tree-sitter-cpp.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-cpp): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -189,3 +189,28 @@ (define-public tree-sitter-c

> +(define-public tree-sitter-cpp
> +  (let ((commit "5ead1e26c6ab71919db0f1880c46a278a93bc5ea")
> +        (revision "1"))

Please an an explanation of why you're using a commit, and start
at revision "0".

    -- (

[bug#49946] [PATCH v7 16/32] gnu: Add tree-sitter-html.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-html): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 17/32] gnu: Add tree-sitter-java.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-java): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 18/32] gnu: Add tree-sitter-javascript.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-javascript): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 19/32] gnu: Add tree-sitter-json.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-json): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 20/32] gnu: Add tree-sitter-julia.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-julia): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 15/32] gnu: Add tree-sitter-haskell.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-haskell): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -352,3 +352,27 @@ (define-public tree-sitter-go
> +(define-public tree-sitter-haskell
> +  (let ((commit "e30bdfd53eb28c73f26a68b77d436fd2140af167")
> +        (revision "1"))

Explanatory comment and start at revision 0.
> +    (package
> +     (name "tree-sitter-haskell")
       ^
Indentation is off; two spaces needed here...

> +     (source (origin
> +              (method git-fetch)
                ^
...and here.

    -- (

[bug#49946] [PATCH v7 21/32] gnu: Add tree-sitter-ocaml.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-ocaml): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -487,3 +487,42 @@ (define-public tree-sitter-julia

> +  (let ((commit "0348562f385bc2bd67ecf181425e1afd6d454192")
> +        (revision "1"))

Explain and start at 0.

> +                    (delete-file "binding.gyp")
> +                    (delete-file-recursively "bindings")
> +                    (for-each
> +                     (lambda (lang)
> +                       (with-directory-excursion lang
> +                         (delete-file "src/grammar.json")
> +                         (delete-file "src/node-types.json")
> +                         (delete-file "src/parser.c")
> +                         (delete-file-recursively "src/tree_sitter")))
> +                     '("ocaml" "interface"))

This stuff should definitely be handled by the
tree-sitter-build-system...

> +      (arguments
> +       (list
> +        #:grammar-directories '("ocaml" "interface")))

Unnecessary newline and use of a quoted list; please use

  (list #:grammar-directories (list "ocaml" "interface"))

    -- (

[bug#49946] [PATCH v7 22/32] gnu: Add tree-sitter-php.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-php): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -526,3 +526,33 @@ (define-public tree-sitter-ocaml

> +  (let ((commit "435fa00006c0d1515c37fbb4dd6a9de284af75ab")
> +        (revision "1"))

Explain and start at 0.

    -- (

[bug#49946] [PATCH v7 23/32] gnu: Add tree-sitter-python.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-python): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -556,3 +556,27 @@ (define-public tree-sitter-php

> +  (let ((commit "ed0fe62e55dc617ed9dec8817ebf771aa7cf3c42")
> +        (revision "1"))

Explain and start at 0.

    -- (

[bug#49946] [PATCH v7 24/32] gnu: Add tree-sitter-r.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-r): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -580,3 +580,27 @@ (define-public tree-sitter-python

> +(define-public tree-sitter-r
> +  (let ((commit "80efda55672d1293aa738f956c7ae384ecdc31b4")
> +        (revision "1"))

Explain and start at 0.

    -- (

[bug#49946] [PATCH v7 25/32] gnu: Add tree-sitter-ruby.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-ruby): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 26/32] gnu: Add tree-sitter-rust.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-rust): New variable.

LGTM

    -- (

[bug#49946] [PATCH v7 27/32] gnu: Add tree-sitter-typescript.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-typescript): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -648,3 +648,41 @@ (define-public tree-sitter-rust

> +    (arguments
> +     (list
> +      #:grammar-directories '("typescript" "tsx")))

Superfluous newline.

    -- (

[bug#49946] [PATCH v7 28/32] gnu: Add rust-tree-sitter.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (rust-tree-sitter): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -158,6 +159,43 @@ (define-public tree-sitter-cli

> +               '(begin
> +                  ;; Remove the runtime library code and dynamically link to
> +                  ;; it instead.
> +                  (delete-file-recursively "src")
> +                  (delete-file "binding_rust/build.rs")
> +                  (with-output-to-file "binding_rust/build.rs"
> +                    (lambda _
> +                      (format #t "fn main() {~@
> +                              println!(\"cargo:rustc-link-lib=tree-sitter\");~@
> +                              }~%")))
> +                  #t))))

Maybe you should actually extract *this* snippet, as you also use
it with tree-sitter-cli?

> +    (arguments
> +     `(#:cargo-inputs
> +       (("rust-cc" ,rust-cc-1)
> +        ("rust-lazy-static" ,rust-lazy-static-1)
> +        ("rust-regex" ,rust-regex-1))))

(list ...) please.

    -- (

[bug#49946] [PATCH v7 29/32] gnu: Add rust-tree-sitter-for-emacs.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (rust-tree-sitter-for-emacs): New variable.
> * gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Register it.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -196,6 +197,19 @@ (define-public rust-tree-sitter
> +;; We need to apply a patch in order to compile the rust bindings against the
> +;; emacs tree-sitter module.
> +;; See https://github.com/tree-sitter/tree-sitter/pull/1294
> +(define-public rust-tree-sitter-for-emacs
> +  (package (inherit rust-tree-sitter)
> +    (source (origin
> +              (inherit (package-source rust-tree-sitter))
> +              (patches (search-patches
> +                        "rust-tree-sitter-text-provider-fix.patch"))))
> +    ;; Do not show this package in the UI as it's only meant to be used for
> +    ;; emacs's tree-sitter module.
> +    (properties '((hidden? . #t)))))

You can use ``hidden-package'', which is a bit nicer than ``properties''
imo:

  (hidden-package
   (package (inherit rust-tree-sitter)
     ...))

    -- (

[bug#49946] [PATCH v7 30/32] gnu: Add emacs-tree-sitter-core.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
> (emacs-tree-sitter-core): New variable.

Please split these two into separate patches.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -738,3 +739,86 @@ (define-public tree-sitter-typescript

> +    (arguments
> +     `(#:cargo-inputs

  (list #:cargo-inputs
        ...)

> +       #:phases
> +       (modify-phases %standard-phases
> +         (add-after 'unpack 'chdir
> +           (lambda _ (chdir "core")))
> +         (add-after 'chdir 'delete-cargo.lock
> +           (lambda _ (delete-file "Cargo.lock")))
> +         (add-after 'delete-cargo.lock 'do-not-fetch-from-github
> +           (lambda _
> +             (substitute* "Cargo.toml"
> +               (("\\[patch.*") "")
> +               (("git = .*") ""))))
> +         (replace 'install
> +           (lambda* (#:key outputs #:allow-other-keys)
> +             (let ((lib (string-append (assoc-ref outputs "out") "/lib")))
> +               (mkdir-p lib)
> +               (copy-file "target/release/libtsc_dyn.so"
> +                          (string-append lib "/tsc-dyn.so"))))))))

Please use gexps and #$output here:

  (replace 'install
    (lambda _
      (let ((lib (string-append #$output "/lib")))
        (mkdir-p lib)
        (copy-file "target/release/libtsc_dyn.so"
                   (string-append lib "/tsc-dyn.so")))))

> +    (name "emacs-tree-sitter-core")
> +    (version (package-version tree-sitter-emacs-module))
> +    (source (package-source tree-sitter-emacs-module))

This ``version'' and ``source'' are unnecessary.

> +    (arguments
> +     `(#:phases

LIST, please.

> +       (modify-phases %standard-phases

Gexp this, please.

> +             (let ((elpa (elpa-directory (assoc-ref outputs "out")))

Please use #$output.

> +                   (module (search-input-file inputs "/lib/tsc-dyn.so")))

The leading / is redundant; this will do:

  (search-input-file inputs "lib/tsc-dyn.so")

    -- (

[bug#49946] [PATCH v7 30/32] gnu: Add emacs-tree-sitter-core.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> +(define-public emacs-tree-sitter-core
> +  (package
> +    (name "emacs-tree-sitter-core")
> +    (version (package-version tree-sitter-emacs-module))
> +    (source (package-source tree-sitter-emacs-module))

Never mind re ``source'' and ``version''; I just realised you didn't inherit from the
tree-sitter-emacs-module package :)

    -- (

[bug#49946] [PATCH v7 31/32] gnu: Add emacs-tree-sitter.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (emacs-tree-sitter): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -822,3 +822,30 @@ (module (search-input-file inputs "/lib/tsc-dyn.so")))

> +    (arguments
> +     `(#:phases

LIST please.

> +       (modify-phases %standard-phases

Aaaand gexp.

    -- (

[bug#49946] [PATCH v7 32/32] gnu: Add emacs-tree-sitter-langs.

[( via Guix-patches via]

On Fri Nov 25, 2022 at 1:21 AM GMT, Pierre Langlois wrote:
> * gnu/packages/tree-sitter.scm (make-emacs-tree-sitter-langs-grammar-bundle):
> New procedure.

Doesn't look like a procedure to me :) It probably should also be
a hidden-package, no?

> (emacs-tree-sitter-langs): New variable.

> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm

> @@ -849,3 +850,152 @@ (define-public emacs-tree-sitter

> +    (synopsis #f)
> +    (description #f)

Please add a description and synopsis.

> +     (list
> +      #:tests? #t

Superfluous newline; please use:

  (list #:tests? #t ...)

> +          (add-after 'install 'install-queries
> +            (lambda* (#:key outputs #:allow-other-keys)
> +              (let ((elpa (elpa-directory (assoc-ref outputs "out"))))
> +                (copy-recursively "queries" (string-append elpa "/queries"))))))))

Please use #$output here.

    -- (

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 185 bytes --]

Hi!

"(" <paren@disroot.org> writes:

> Heya,

Thanks for taking a look and spending time reviewing this, this is
great. I'll work on your suggestions over the weekend.

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] About migrating support from emacs-tree-sitter to treesit

[Luis Henrique Gomes Higino]

Hi guix,

I was taking a look at this patch series again and wondering if it 
could
be modified to support the new built in treesit module coming in
Emacs 29 instead of emacs-tree-sitter, as the latter recommends 
using
the native integration going forward [1]. Maybe we should leave 
this to when
Emacs 29 is released, but I think working on this in advance could
result on it being delivered earlier, which would be great. What 
do you think Pierre?

[1]: 
https://github.com/emacs-tree-sitter/elisp-tree-sitter/commit/d3eab879e9b0ccc9582113d81a93ad06f3c6bfb1

PS: I'm willing to help

Regards,
-- 
Luis H. Higino

[bug#49946] About migrating support from emacs-tree-sitter to treesit

[Luis Henrique Gomes Higino]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

Hi guix,

Luis Henrique Gomes Higino <luishenriquegh2701@gmail.com> writes:

> Hi guix,
>
> I was taking a look at this patch series again and wondering if 
> it
> could
> be modified to support the new built in treesit module coming in
> Emacs 29 instead of emacs-tree-sitter, as the latter recommends 
> using
> the native integration going forward [1]. Maybe we should leave 
> this
> to when
> Emacs 29 is released, but I think working on this in advance 
> could
> result on it being delivered earlier, which would be great. What 
> do
> you think Pierre?
>
> [1]:
> https://github.com/emacs-tree-sitter/elisp-tree-sitter/commit/d3eab879e9b0ccc9582113d81a93ad06f3c6bfb1
>
> PS: I'm willing to help
>
> Regards,

I played around this idea this weekend and I got `treesit' to use 
the Guix installed grammars :) !

First, I applied a patch that added tree-sitter as an input for 
emacs-next [1]. 

[1]: https://issues.guix.gnu.org/61139

Then, I did the edits below on top of Pierre's latest patch 
series:


[-- Attachment #2: 0001-Support-treesit-module-from-emacs.patch --]
[-- Type: text/plain, Size: 1668 bytes --]

From fb96ca9c6039d07856c8f3525919ed9036b39470 Mon Sep 17 00:00:00 2001
From: Luis Henrique Gomes Higino <luishenriquegh2701@gmail.com>
Date: Sun, 29 Jan 2023 20:24:04 -0300
Subject: [PATCH] Support treesit module from emacs

---
 gnu/packages/emacs.scm                  | 8 +++++++-
 guix/build/tree-sitter-build-system.scm | 2 +-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/emacs.scm b/gnu/packages/emacs.scm
index 951c1ad6df..193547ff42 100644
--- a/gnu/packages/emacs.scm
+++ b/gnu/packages/emacs.scm
@@ -409,7 +409,13 @@ (define-public emacs-next
          (prepend sqlite tree-sitter)))
       (native-inputs
        (modify-inputs (package-native-inputs emacs)
-         (prepend autoconf))))))
+         (prepend autoconf)))
+      (native-search-paths
+       (cons
+        (search-path-specification
+         (variable "TREE_SITTER_GRAMMARS")
+         (files '("lib/tree-sitter")))
+        (package-native-search-paths emacs))))))
 
 (define-public emacs-next-pgtk
   (package
diff --git a/guix/build/tree-sitter-build-system.scm b/guix/build/tree-sitter-build-system.scm
index 574b0f2a1c..ee1aa9976c 100644
--- a/guix/build/tree-sitter-build-system.scm
+++ b/guix/build/tree-sitter-build-system.scm
@@ -114,7 +114,7 @@ (define (compile-language dir)
                    "-fno-exceptions"
                    "-O2"
                    "-g"
-                   "-o" ,(string-append lib "/" lang ".so")
+                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
                    ;; An additional `scanner.{c,cc}' file is sometimes
                    ;; provided.
                    ,@(cond
-- 
2.39.1


[-- Attachment #3: Type: text/plain, Size: 1022 bytes --]


Then, I ran `./pre-inst-env guix shell emacs-next tree-sitter-c 
tree-sitter-cpp --container --preserve="^TERM$" -- emacs -Q -nw` 
and executed the following code inside that Emacs session:

--8<---------------cut here---------------start------------->8---
(defvar guix-treesit-grammar-paths
  (let ((grammars-path (if (getenv "TREE_SITTER_GRAMMARS")
                           (getenv "TREE_SITTER_GRAMMARS")
                         "")))
    (split-string grammars-path ":")))

(if (treesit-available-p)
    (setq treesit-extra-load-path guix-treesit-grammar-paths))
--8<---------------cut here---------------end--------------->8---

Doing these steps, Emacs' treesit was able to use the grammars 
installed by Guix in c-ts-mode. I don't know if a native 
search-path is the appropriate approach or how to include the 
above elisp (or something similar) in the emacs-next package, but 
I hope this helps Pierre or anyone else with more understanding to 
do the v8 of this patch series :).

Regards,
-- 
Luis H. Higino

[bug#49946] About migrating support from emacs-tree-sitter to treesit

[Pierre Langlois]

Hi Luis,

Luis Henrique Gomes Higino <luishenriquegh2701@gmail.com> writes:

> Hi guix,
>
> Luis Henrique Gomes Higino <luishenriquegh2701@gmail.com> writes:
>
>> Hi guix,
>>
>> I was taking a look at this patch series again and wondering if it
>> could
>> be modified to support the new built in treesit module coming in
>> Emacs 29 instead of emacs-tree-sitter, as the latter recommends using
>> the native integration going forward [1]. Maybe we should leave this
>> to when
>> Emacs 29 is released, but I think working on this in advance could
>> result on it being delivered earlier, which would be great. What do
>> you think Pierre?
>>
>> [1]:
>> https://github.com/emacs-tree-sitter/elisp-tree-sitter/commit/d3eab879e9b0ccc9582113d81a93ad06f3c6bfb1
>>
>> PS: I'm willing to help
>>
>> Regards,
>
> I played around this idea this weekend and I got `treesit' to use the Guix
> installed grammars :) !
>
> First, I applied a patch that added tree-sitter as an input for emacs-next
> [1]. [1]: https://issues.guix.gnu.org/61139
>
> Then, I did the edits below on top of Pierre's latest patch series:

This is super cool, thanks for looking into this! and sorry for replying
so late.  I agree that we should try and get these grammars integrated
into emacs as soon as we can, it's the end goal after all (by that I
mean that I'm not sure the Rust-based emacs module will live on after
emacs 29 is released).

But anyways, it's great that you got it working with minimal changes,
looks like there won't be any blockers :-).

Regarding the patch series, I do have a new version in the pipeline
(fixing style and improving testing) but I'm afraid I've had no time to
work on it in the past month and a half. I should have more time
starting from next weekend. 

Thanks!
Pierre

[bug#49946] [PATCH 00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

[zimoun]

Hi,

Sorry, I have totally overlooked this tree-sitter story. :-) And I have
not read all the patch versions and thread yet.


On Fri, 25 Nov 2022 at 01:57, Pierre Langlois <pierre.langlois@gmx.com> wrote:

> +(define* (install #:key target grammar-directories outputs #:allow-other-keys)
> +  (let ((lib (string-append (assoc-ref outputs "out")
> +                            "/lib/tree-sitter")))
> +    (mkdir-p lib)
> +    (define (compile-language dir)
> +      (with-directory-excursion dir
> +        (let ((lang (assoc-ref (call-with-input-file "src/grammar.json"
> +                                 read-json)
> +                               "name"))
> +              (source-file (lambda (path)
> +                             (if (file-exists? path)
> +                                 path
> +                                 #f))))
> +          (apply invoke
> +                 `(,(if target
> +                        (string-append target "-g++")
> +                        "g++")

Why is g++ required?  Is gcc not enough?  I thought from Tree-sitter
documentation that it was only using C files and not C++; but…

> +                   "-shared"
> +                   "-fPIC"
> +                   "-fno-exceptions"
> +                   "-O2"
> +                   "-g"
> +                   "-o" ,(string-append lib "/" lang ".so")
> +                   ;; An additional `scanner.{c,cc}' file is sometimes
> +                   ;; provided.
> +                   ,@(cond
> +                      ((source-file "src/scanner.c")
> +                       => (lambda (file) (list "-xc" "-std=c99" file)))
> +                      ((source-file "src/scanner.cc")

…apparently not.


Cheers,
simon

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

On 2022-11-25 10:18, Pierre Langlois wrote:

> Hi!
>
> "(" <paren@disroot.org> writes:
>
>> Heya,
>
> Thanks for taking a look and spending time reviewing this, this is
> great. I'll work on your suggestions over the weekend.
>
> Thanks,
> Pierre

Hi Pierre!

Thank you very much for all the work.

I applied tree-sitter and tree-sitter-cli patches, adjusted them,
addressed comments of unmatching-paren and succesfully built respective
packages.  I'll push them later today or tomorrow.  Just letting you
know to prevent duplication of effort.

Also, later I'll make a separate message on tree-sitter grammars and the
way I see they can be added to Guix and Emacs.

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[zimoun]

Hi,

On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:

> I applied tree-sitter and tree-sitter-cli patches,

Just to be sure to understand, you have only applied 02/32 and 05/32,
right?


[bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
id:20221125012142.22579-3-pierre.langlois@gmx.com
http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com

[bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
id:20221125012142.22579-6-pierre.langlois@gmx.com
http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com

Leaving out all the others, right?

Cheers,
simon

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 1902 bytes --]

On 2023-02-09 13:39, zimoun wrote:

> Hi,
>
> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>
>> I applied tree-sitter and tree-sitter-cli patches,
>
> Just to be sure to understand, you have only applied 02/32 and 05/32,
> right?
>
>
> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
> id:20221125012142.22579-3-pierre.langlois@gmx.com
> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>
> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
> id:20221125012142.22579-6-pierre.langlois@gmx.com
> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>
> Leaving out all the others, right?

Merged first 5 patches from 01 to 05, also added one more commit, which
addresses some things from reviews and one commit, which adds html
grammar.

The html grammar is added for the testing purposes.  It relies on
generated parser.c and scanner.c and we will need to repackage it using
grammar.js instead.  I'm not sure if a separate build system is needed
for this, I guess we can just rewrite tree-sitter-grammar function,
which generates packages as in example with tree-sitter-grammar-html:
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158

Anyway, I only skimmed tree-sitter-build-system source code, and plan to
read it carefully, evaluate and either introduce new build system or
just move all needed parts to tree-sitter-grammar function.  WDYT?
After we done with it we can package all other grammars.

According to emacs integration: I already have a working prototype on my
local setup, which uses builtin treesit package, it's similiar to what
Luis mentioned with search-paths.  We just need to figure out a few
minor details and I'll add it.

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH 00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

[Andrew Tropin]

[-- Attachment #1.1: Type: text/plain, Size: 1699 bytes --]

On 2021-08-08 23:29, Pierre Langlois wrote:

> 4. Finally, emacs support!
>
>   emacs-tree-sitter can be used for syntax highlighting, replacing
>   font-lock.  The tree-sitter runtime library is built as a Rust crate
>   and exposed to emacs using a module thanks to the rust-emacs crate.
>
>   Then emacs-tree-sitter-core provides just the core APIs as exposed by
>   the rust module, then emacs-tree-sitter integrates it with emacs'
>   syntax highlighting (and probably other things).
>
>   Finally, we bundle together the various language grammars into a
>   single emacs-tree-sitter-langs package so that they can get loaded
>   along with their respective major modes.  Note that there are grammars
>   left to support which I've not yet managed to package, but what we
>   have already is a good start I think :-). So far I've tested C++,
>   Rust, Python and JavaScript and JSON.
>
>   * gnu: Add rust-bindgen@0.56.
>   * gnu: Add rust-tree-sitter.
>   * gnu: rust-emacs-module: Update to 0.16.
>   * gnu: rust-emacs-macros: Update to 0.17.
>   * gnu: rust-emacs: Update to 0.17.
>   * gnu: Add emacs-tree-sitter-core.
>   * gnu: Add emacs-tree-sitter.
>   * gnu: Add emacs-tree-sitter-langs.

Hi Pierre!

Implemented a loading of tree-sitter grammars in Emacs using guix
native-search-paths and built-in treesit package.  I did it for emacs
package, but it will work only with Emacs 29 and greater (right now it's
emacs-next and similiar packages), until emacs package updated to
version 29 it won't take any effect on it except providing one more
environment variable in guix profile.

I tested it with emacs-next-pgtk, the patch is attached below, please
let me know what do you think!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-emacs-Add-TREE_SITTER_GRAMMAR_PATH-support.patch --]
[-- Type: text/x-patch, Size: 2554 bytes --]

From b5ecd4e3734e9dd0bc76ebe95cab9c43aa85a3fe Mon Sep 17 00:00:00 2001
From: Andrew Tropin <andrew@trop.in>
Date: Fri, 10 Feb 2023 12:32:12 +0400
Subject: [PATCH] gnu: emacs: Add TREE_SITTER_GRAMMAR_PATH support.

gnu/packages/emacs.scm (emacs)[native-search-paths]: Add a search-path for
tree-sitter grammars.
gnu/packages/aux-files/emacs/guix-emacs.el: Add directories from
TREE_SITTER_GRAMMAR_PATH to treesit-extra-load-path.
---
 gnu/packages/aux-files/emacs/guix-emacs.el |  7 +++++++
 gnu/packages/emacs.scm                     | 10 +++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/aux-files/emacs/guix-emacs.el b/gnu/packages/aux-files/emacs/guix-emacs.el
index 56dbcb8d67..708093267d 100644
--- a/gnu/packages/aux-files/emacs/guix-emacs.el
+++ b/gnu/packages/aux-files/emacs/guix-emacs.el
@@ -76,6 +76,13 @@ The files in the list do not have extensions (.el, .elc)."
                    (when (file-directory-p pkg-dir)
                      (package-load-descriptor pkg-dir)))))))))))
 
+;; If emacs built with tree-sitter, read the value of the environment variable
+;; to make tree-sitter grammars available in emacs out-of-the-box.
+(with-eval-after-load 'treesit
+  (when-let ((grammar-path (getenv "TREE_SITTER_GRAMMAR_PATH")))
+    (mapcar (lambda (x) (add-to-list 'treesit-extra-load-path x))
+            (split-string grammar-path ":"))))
+
 (provide 'guix-emacs)
 
 ;;; guix-emacs.el ends here
diff --git a/gnu/packages/emacs.scm b/gnu/packages/emacs.scm
index 4ce41deb88..a3cc9d2e22 100644
--- a/gnu/packages/emacs.scm
+++ b/gnu/packages/emacs.scm
@@ -367,7 +367,15 @@ (define* (emacs-byte-compile-directory dir)
             (files '("lib/emacs/native-site-lisp")))
            (search-path-specification
             (variable "INFOPATH")
-            (files '("share/info")))))
+            (files '("share/info")))
+           ;; tree-sitter support is not yet available in emacs 28, but this
+           ;; search path won't harm and also will be benifitial for
+           ;; emacs-next and other emacs-* packages, which have tree-sitter
+           ;; support enabled.  Please, remove this comment, when emacs
+           ;; package is updated to 29.
+           (search-path-specification
+            (variable "TREE_SITTER_GRAMMAR_PATH")
+            (files '("lib/tree-sitter")))))
 
     (home-page "https://www.gnu.org/software/emacs/")
     (synopsis "The extensible, customizable, self-documenting text editor")
-- 
2.39.1


[-- Attachment #1.3: Type: text/plain, Size: 37 bytes --]


-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 5444 bytes --]

On 2023-02-09 18:04, Andrew Tropin wrote:

> On 2023-02-09 13:39, zimoun wrote:
>
>> Hi,
>>
>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>
>>> I applied tree-sitter and tree-sitter-cli patches,
>>
>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>> right?
>>
>>
>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>
>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>
>> Leaving out all the others, right?
>
> Merged first 5 patches from 01 to 05, also added one more commit, which
> addresses some things from reviews and one commit, which adds html
> grammar.
>
> The html grammar is added for the testing purposes.  It relies on
> generated parser.c and scanner.c and we will need to repackage it using
> grammar.js instead.  I'm not sure if a separate build system is needed
> for this, I guess we can just rewrite tree-sitter-grammar function,
> which generates packages as in example with tree-sitter-grammar-html:
> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>
> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
> read it carefully, evaluate and either introduce new build system or
> just move all needed parts to tree-sitter-grammar function.  WDYT?
> After we done with it we can package all other grammars.

Ok, I realized that the proper build process for tree-sitter grammars is
a little harder than I expected, tree-sitter-build system make sense.  I
reviewed it, made a small change:

--8<---------------cut here---------------start------------->8---
@@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
 ;; Commentary:
 ;;
 ;; Build procedures for tree-sitter grammar packages.  This is the
-;; builder-side code, which builds on top fo the node build-system.
+;; builder-side code, which builds on top of the node build-system.
 ;;
 ;; Tree-sitter grammars are written in JavaScript and compiled to a native
 ;; shared object.  The `tree-sitter generate' command invokes `node' in order
@@ -114,7 +114,7 @@ (define (compile-language dir)
                    "-fno-exceptions"
                    "-O2"
                    "-g"
-                   "-o" ,(string-append lib "/" lang ".so")
+                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
                    ;; An additional `scanner.{c,cc}' file is sometimes
                    ;; provided.
                    ,@(cond
--8<---------------cut here---------------end--------------->8---

rewrote html grammar to use this build system and made it work with
built-in treesit package.  Also, tried examples of c and cpp grammars
from patches in this thread.

If you ok with it, I'll push the build system to master and update the
html grammar accordingly.

The final result will look like this:

--8<---------------cut here---------------start------------->8---
(define tree-sitter-delete-generated-files
  #~(begin
      (delete-file "binding.gyp")
      (delete-file-recursively "bindings")
      (delete-file "src/grammar.json")
      (delete-file "src/node-types.json")
      (delete-file "src/parser.c")
      (delete-file-recursively "src/tree_sitter")))

(define* (tree-sitter-grammar
          language language-for-synopsis version commit hash
          #:key
          (repository-url
           (format #f "https://github.com/tree-sitter/tree-sitter-~a" language))
          (inputs '()))
  (let ((synopsis (string-append language-for-synopsis
                                 " grammar for tree-sitter"))
        (name (string-append "tree-sitter-grammar-" language)))
    (package
      (name name)
      (version version)
      (home-page repository-url)
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url repository-url)
                      (commit commit)))
                (file-name (git-file-name name version))
                (sha256 (base32 hash))
                (modules '((guix build utils)))
                (snippet tree-sitter-delete-generated-files)))
      (build-system tree-sitter-build-system)
      (inputs inputs)
      (synopsis synopsis)
      (description (string-append synopsis "."))
      (license license:expat))))

(define-public tree-sitter-grammar-html
  (tree-sitter-grammar
   "html" "HTML"
   "0.19.0" "v0.19.0"
   "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
--8<---------------cut here---------------end--------------->8---

After that we can bring the rest of the grammars.

>
> According to emacs integration: I already have a working prototype on
> my local setup, which uses builtin treesit package, it's similiar to
> what Luis mentioned with search-paths.  We just need to figure out a
> few minor details and I'll add it.

I've sent a patch for emacs+tree-sitter integration in a different reply
in this thread.

Thank you very much for all the work, it's really valuable!

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 7287 bytes --]

Hi Andrew, thanks for pushing this along! It's great to see things
getting merged.

Andrew Tropin <andrew@trop.in> writes:

> [[PGP Signed Part:Undecided]]
> On 2023-02-09 18:04, Andrew Tropin wrote:
>
>> On 2023-02-09 13:39, zimoun wrote:
>>
>>> Hi,
>>>
>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>
>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>
>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>> right?
>>>
>>>
>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>
>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>
>>> Leaving out all the others, right?
>>
>> Merged first 5 patches from 01 to 05, also added one more commit, which
>> addresses some things from reviews and one commit, which adds html
>> grammar.
>>
>> The html grammar is added for the testing purposes.  It relies on
>> generated parser.c and scanner.c and we will need to repackage it using
>> grammar.js instead.  I'm not sure if a separate build system is needed
>> for this, I guess we can just rewrite tree-sitter-grammar function,
>> which generates packages as in example with tree-sitter-grammar-html:
>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>
>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>> read it carefully, evaluate and either introduce new build system or
>> just move all needed parts to tree-sitter-grammar function.  WDYT?
>> After we done with it we can package all other grammars.
>
> Ok, I realized that the proper build process for tree-sitter grammars is
> a little harder than I expected, tree-sitter-build system make sense.  I
> reviewed it, made a small change:

Ah great, I was going to comment to try and push for us to keep the
build system. I originally went with a template package and inheritance,
but Maxime suggested moving to a build-system which ended up making the
package definitions a *lot* nicer IMO (see previous discussion here
https://issues.guix.gnu.org/49946#144). It also allows us to deal with
grammars that depend on each other more nicely I think.

>
> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>  ;; Commentary:
>  ;;
>  ;; Build procedures for tree-sitter grammar packages.  This is the
> -;; builder-side code, which builds on top fo the node build-system.
> +;; builder-side code, which builds on top of the node build-system.
>  ;;
>  ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>  ;; shared object.  The `tree-sitter generate' command invokes `node' in order
> @@ -114,7 +114,7 @@ (define (compile-language dir)
>                     "-fno-exceptions"
>                     "-O2"
>                     "-g"
> -                   "-o" ,(string-append lib "/" lang ".so")
> +                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>                     ;; An additional `scanner.{c,cc}' file is sometimes
>                     ;; provided.
>                     ,@(cond
>
>
> rewrote html grammar to use this build system and made it work with
> built-in treesit package.  Also, tried examples of c and cpp grammars
> from patches in this thread.
>
> If you ok with it, I'll push the build system to master and update the
> html grammar accordingly.
>
> The final result will look like this:
>
> (define tree-sitter-delete-generated-files
>   #~(begin
>       (delete-file "binding.gyp")
>       (delete-file-recursively "bindings")
>       (delete-file "src/grammar.json")
>       (delete-file "src/node-types.json")
>       (delete-file "src/parser.c")
>       (delete-file-recursively "src/tree_sitter")))
>
> (define* (tree-sitter-grammar
>           language language-for-synopsis version commit hash
>           #:key
>           (repository-url
>            (format #f "https://github.com/tree-sitter/tree-sitter-~a" language))
>           (inputs '()))
>   (let ((synopsis (string-append language-for-synopsis
>                                  " grammar for tree-sitter"))
>         (name (string-append "tree-sitter-grammar-" language)))
>     (package
>       (name name)
>       (version version)
>       (home-page repository-url)
>       (source (origin
>                 (method git-fetch)
>                 (uri (git-reference
>                       (url repository-url)
>                       (commit commit)))
>                 (file-name (git-file-name name version))
>                 (sha256 (base32 hash))
>                 (modules '((guix build utils)))
>                 (snippet tree-sitter-delete-generated-files)))
>       (build-system tree-sitter-build-system)
>       (inputs inputs)
>       (synopsis synopsis)
>       (description (string-append synopsis "."))
>       (license license:expat))))
>
> (define-public tree-sitter-grammar-html
>   (tree-sitter-grammar
>    "html" "HTML"
>    "0.19.0" "v0.19.0"
>    "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
>
> After that we can bring the rest of the grammars.

I would suggest to rmeove the `tree-sitter-grammar' function, and keep
grammars as "regular" package records, even though it's a little bit
more verbose:

--8<---------------cut here---------------start------------->8---
(define-public tree-sitter-html
  (package
    (name "tree-sitter-html")
    (version "0.19.0")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/tree-sitter/tree-sitter-html")
                    (commit (string-append "v" version))))
              (file-name (git-file-name name version))
              (sha256
               (base32
                "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
              (modules '((guix build utils)))
              (snippet tree-sitter-delete-generated-files)))
    (build-system tree-sitter-build-system)
    (home-page "https://github.com/tree-sitter/tree-sitter-html")
    (synopsis "Tree-sitter HTML grammar")
    (description
     "This package provides a HTML grammar for the Tree-sitter library.")
    (license license:expat)))
--8<---------------cut here---------------end--------------->8---

This way, they look like any other package in Guix, which makes it
easier for us to apply automatic changes in the future if needed (for
example like how the input format could be automically updated for all
"simple" package definitions, but had to be manual whenever custom code
refactoring was done). Does this make sense?

>
>>
>> According to emacs integration: I already have a working prototype on
>> my local setup, which uses builtin treesit package, it's similiar to
>> what Luis mentioned with search-paths.  We just need to figure out a
>> few minor details and I'll add it.
>
> I've sent a patch for emacs+tree-sitter integration in a different reply
> in this thread.
>
> Thank you very much for all the work, it's really valuable!

:-)

Thanks,
Pierre


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH 00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 2103 bytes --]

Hi!

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> Sorry, I have totally overlooked this tree-sitter story. :-) And I have
> not read all the patch versions and thread yet.
>
>
> On Fri, 25 Nov 2022 at 01:57, Pierre Langlois <pierre.langlois@gmx.com> wrote:
>
>> +(define* (install #:key target grammar-directories outputs #:allow-other-keys)
>> +  (let ((lib (string-append (assoc-ref outputs "out")
>> +                            "/lib/tree-sitter")))
>> +    (mkdir-p lib)
>> +    (define (compile-language dir)
>> +      (with-directory-excursion dir
>> +        (let ((lang (assoc-ref (call-with-input-file "src/grammar.json"
>> +                                 read-json)
>> +                               "name"))
>> +              (source-file (lambda (path)
>> +                             (if (file-exists? path)
>> +                                 path
>> +                                 #f))))
>> +          (apply invoke
>> +                 `(,(if target
>> +                        (string-append target "-g++")
>> +                        "g++")
>
> Why is g++ required?  Is gcc not enough?  I thought from Tree-sitter
> documentation that it was only using C files and not C++; but…
>
>> +                   "-shared"
>> +                   "-fPIC"
>> +                   "-fno-exceptions"
>> +                   "-O2"
>> +                   "-g"
>> +                   "-o" ,(string-append lib "/" lang ".so")
>> +                   ;; An additional `scanner.{c,cc}' file is sometimes
>> +                   ;; provided.
>> +                   ,@(cond
>> +                      ((source-file "src/scanner.c")
>> +                       => (lambda (file) (list "-xc" "-std=c99" file)))
>> +                      ((source-file "src/scanner.cc")
>
> …apparently not.

Yeah this is a little strange. IIUC, the final code generated from the
grammar.js->grammar.json->grammar.c process is plain C, but grammar
 packages may also provide some custom hand-written code in terms of a
scanner.{c,cc} file.

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH 00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 5718 bytes --]

Hi,

Andrew Tropin <andrew@trop.in> writes:

> [[PGP Signed Part:Undecided]]
> On 2021-08-08 23:29, Pierre Langlois wrote:
>
>> 4. Finally, emacs support!
>>
>>   emacs-tree-sitter can be used for syntax highlighting, replacing
>>   font-lock.  The tree-sitter runtime library is built as a Rust crate
>>   and exposed to emacs using a module thanks to the rust-emacs crate.
>>
>>   Then emacs-tree-sitter-core provides just the core APIs as exposed by
>>   the rust module, then emacs-tree-sitter integrates it with emacs'
>>   syntax highlighting (and probably other things).
>>
>>   Finally, we bundle together the various language grammars into a
>>   single emacs-tree-sitter-langs package so that they can get loaded
>>   along with their respective major modes.  Note that there are grammars
>>   left to support which I've not yet managed to package, but what we
>>   have already is a good start I think :-). So far I've tested C++,
>>   Rust, Python and JavaScript and JSON.
>>
>>   * gnu: Add rust-bindgen@0.56.
>>   * gnu: Add rust-tree-sitter.
>>   * gnu: rust-emacs-module: Update to 0.16.
>>   * gnu: rust-emacs-macros: Update to 0.17.
>>   * gnu: rust-emacs: Update to 0.17.
>>   * gnu: Add emacs-tree-sitter-core.
>>   * gnu: Add emacs-tree-sitter.
>>   * gnu: Add emacs-tree-sitter-langs.
>
> Hi Pierre!
>
> Implemented a loading of tree-sitter grammars in Emacs using guix
> native-search-paths and built-in treesit package.  I did it for emacs
> package, but it will work only with Emacs 29 and greater (right now it's
> emacs-next and similiar packages), until emacs package updated to
> version 29 it won't take any effect on it except providing one more
> environment variable in guix profile.
>
> I tested it with emacs-next-pgtk, the patch is attached below, please
> let me know what do you think!

This is cool, thanks! I'm not familiar with how upstream emacs loads
grammars so I can't comment too much on the details. That being said, I
think it'd be good to get it to work with emacs 28 as well, I've been
using it for the past 6 months and it's working nicely. I'm happy to
take care of that part of the work though and let others focus on
emacs-next.

We have the following packages for emacs 28 support:

>>   * gnu: Add rust-bindgen@0.56.
>>   * gnu: Add rust-tree-sitter.
>>   * gnu: rust-emacs-module: Update to 0.16.
>>   * gnu: rust-emacs-macros: Update to 0.17.
>>   * gnu: rust-emacs: Update to 0.17.
>>   * gnu: Add emacs-tree-sitter-core.
>>   * gnu: Add emacs-tree-sitter.
>>   * gnu: Add emacs-tree-sitter-langs.

Most of those would eventually go away I think, except probably
emacs-tree-sitter-langs, which is really useful to provide basic
tree-sitter syntax highlighint support to many language modes that may
not need to be changed themselves.

For instance, there is a PR to eventually move over to using the core
emacs 29 treesit module: https://github.com/emacs-tree-sitter/tree-sitter-langs/pull/157

>
> From b5ecd4e3734e9dd0bc76ebe95cab9c43aa85a3fe Mon Sep 17 00:00:00 2001
> From: Andrew Tropin <andrew@trop.in>
> Date: Fri, 10 Feb 2023 12:32:12 +0400
> Subject: [PATCH] gnu: emacs: Add TREE_SITTER_GRAMMAR_PATH support.
>
> gnu/packages/emacs.scm (emacs)[native-search-paths]: Add a search-path for
> tree-sitter grammars.
> gnu/packages/aux-files/emacs/guix-emacs.el: Add directories from
> TREE_SITTER_GRAMMAR_PATH to treesit-extra-load-path.
> ---
>  gnu/packages/aux-files/emacs/guix-emacs.el |  7 +++++++
>  gnu/packages/emacs.scm                     | 10 +++++++++-
>  2 files changed, 16 insertions(+), 1 deletion(-)
>
> diff --git a/gnu/packages/aux-files/emacs/guix-emacs.el b/gnu/packages/aux-files/emacs/guix-emacs.el
> index 56dbcb8d67..708093267d 100644
> --- a/gnu/packages/aux-files/emacs/guix-emacs.el
> +++ b/gnu/packages/aux-files/emacs/guix-emacs.el
> @@ -76,6 +76,13 @@ The files in the list do not have extensions (.el, .elc)."
>                     (when (file-directory-p pkg-dir)
>                       (package-load-descriptor pkg-dir)))))))))))
>  
> +;; If emacs built with tree-sitter, read the value of the environment variable
> +;; to make tree-sitter grammars available in emacs out-of-the-box.
> +(with-eval-after-load 'treesit
> +  (when-let ((grammar-path (getenv "TREE_SITTER_GRAMMAR_PATH")))
> +    (mapcar (lambda (x) (add-to-list 'treesit-extra-load-path x))
> +            (split-string grammar-path ":"))))
> +
>  (provide 'guix-emacs)
>  
>  ;;; guix-emacs.el ends here
> diff --git a/gnu/packages/emacs.scm b/gnu/packages/emacs.scm
> index 4ce41deb88..a3cc9d2e22 100644
> --- a/gnu/packages/emacs.scm
> +++ b/gnu/packages/emacs.scm
> @@ -367,7 +367,15 @@ (define* (emacs-byte-compile-directory dir)
>              (files '("lib/emacs/native-site-lisp")))
>             (search-path-specification
>              (variable "INFOPATH")
> -            (files '("share/info")))))
> +            (files '("share/info")))
> +           ;; tree-sitter support is not yet available in emacs 28, but this
> +           ;; search path won't harm and also will be benifitial for
> +           ;; emacs-next and other emacs-* packages, which have tree-sitter
> +           ;; support enabled.  Please, remove this comment, when emacs
> +           ;; package is updated to 29.
> +           (search-path-specification
> +            (variable "TREE_SITTER_GRAMMAR_PATH")
> +            (files '("lib/tree-sitter")))))
>  
>      (home-page "https://www.gnu.org/software/emacs/")
>      (synopsis "The extensible, customizable, self-documenting text editor")
> -- 
> 2.39.1

LGTM!


Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 4162 bytes --]


Pierre Langlois <pierre.langlois@gmx.com> writes:

> [[PGP Signed Part:Undecided]]
> Hi Andrew, thanks for pushing this along! It's great to see things
> getting merged.
>
> Andrew Tropin <andrew@trop.in> writes:
>
>> [[PGP Signed Part:Undecided]]
>> On 2023-02-09 18:04, Andrew Tropin wrote:
>>
>>> On 2023-02-09 13:39, zimoun wrote:
>>>
>>>> Hi,
>>>>
>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>
>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>
>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>> right?
>>>>
>>>>
>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>
>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>
>>>> Leaving out all the others, right?
>>>
>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>> addresses some things from reviews and one commit, which adds html
>>> grammar.
>>>
>>> The html grammar is added for the testing purposes.  It relies on
>>> generated parser.c and scanner.c and we will need to repackage it using
>>> grammar.js instead.  I'm not sure if a separate build system is needed
>>> for this, I guess we can just rewrite tree-sitter-grammar function,
>>> which generates packages as in example with tree-sitter-grammar-html:
>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>>
>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>>> read it carefully, evaluate and either introduce new build system or
>>> just move all needed parts to tree-sitter-grammar function.  WDYT?
>>> After we done with it we can package all other grammars.
>>
>> Ok, I realized that the proper build process for tree-sitter grammars is
>> a little harder than I expected, tree-sitter-build system make sense.  I
>> reviewed it, made a small change:
>
> Ah great, I was going to comment to try and push for us to keep the
> build system. I originally went with a template package and inheritance,
> but Maxime suggested moving to a build-system which ended up making the
> package definitions a *lot* nicer IMO (see previous discussion here
> https://issues.guix.gnu.org/49946#144). It also allows us to deal with
> grammars that depend on each other more nicely I think.
>
>>
>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>>  ;; Commentary:
>>  ;;
>>  ;; Build procedures for tree-sitter grammar packages.  This is the
>> -;; builder-side code, which builds on top fo the node build-system.
>> +;; builder-side code, which builds on top of the node build-system.
>>  ;;
>>  ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>>  ;; shared object.  The `tree-sitter generate' command invokes `node' in order
>> @@ -114,7 +114,7 @@ (define (compile-language dir)
>>                     "-fno-exceptions"
>>                     "-O2"
>>                     "-g"
>> -                   "-o" ,(string-append lib "/" lang ".so")
>> +                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>>                     ;; An additional `scanner.{c,cc}' file is sometimes
>>                     ;; provided.
>>                     ,@(cond
>>
>>
>> rewrote html grammar to use this build system and made it work with
>> built-in treesit package.  Also, tried examples of c and cpp grammars
>> from patches in this thread.
>>
>> If you ok with it, I'll push the build system to master and update the
>> html grammar accordingly.

Oh, I forgot to say, this change to the build system LGTM! I'm really
happy to see it merged soon :-). The path change will probably break the
emacs-28-based tree-sitter support, but that's OK, it's better for the
build-system to be made to target emacs 29's builtin support. I'm sure I
can work around for emacs 28.

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

[-- Attachment #1.1: Type: text/plain, Size: 1367 bytes --]

Hi,

Andrew Tropin <andrew@trop.in> writes:

> [[PGP Signed Part:Undecided]]
> On 2023-02-09 13:39, zimoun wrote:
>
>> Hi,
>>
>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>
>>> I applied tree-sitter and tree-sitter-cli patches,
>>
>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>> right?
>>
>>
>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>
>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>
>> Leaving out all the others, right?
>
> Merged first 5 patches from 01 to 05, also added one more commit, which
> addresses some things from reviews and one commit, which adds html
> grammar.

I had also started rebasing addressing comments on the tree-sitter-cli
package, and had a few extra changes left that haven't been pushed.

I realised we could remove a now redundant comment, and while enabling
*some* tests, I realized we should directly depend on node and dot, so
that people may use the `tree-sitter' command, without needing them in
the same profile.

What do you think of the following two patches?


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-Remove-tree-sitter-comment-from-emacs.scm-import.patch --]
[-- Type: text/x-patch, Size: 937 bytes --]

From b24b04015261cfea2143be63671877c9c2c8d458 Mon Sep 17 00:00:00 2001
From: Pierre Langlois <pierre.langlois@gmx.com>
Date: Fri, 10 Feb 2023 16:10:25 +0000
Subject: [PATCH 1/2] gnu: Remove tree-sitter comment from emacs.scm imports.

* gnu/packages/emacs.scm: Remove tree-sitter comment.
---
 gnu/packages/emacs.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/emacs.scm b/gnu/packages/emacs.scm
index 4ce41deb88..4791570d12 100644
--- a/gnu/packages/emacs.scm
+++ b/gnu/packages/emacs.scm
@@ -76,7 +76,7 @@ (define-module (gnu packages emacs)
   #:use-module (gnu packages sqlite)
   #:use-module (gnu packages texinfo)
   #:use-module (gnu packages tls)
-  #:use-module (gnu packages tree-sitter) ; for tree-sitter
+  #:use-module (gnu packages tree-sitter)
   #:use-module (gnu packages web)       ; for jansson
   #:use-module (gnu packages webkit)
   #:use-module (gnu packages xml)
-- 
2.39.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-gnu-tree-sitter-cli-Enable-some-tests-and-add-node-a.patch --]
[-- Type: text/x-patch, Size: 4251 bytes --]

From 8225d51edeb528c51d30ad0f225eb96be5586a37 Mon Sep 17 00:00:00 2001
From: Pierre Langlois <pierre.langlois@gmx.com>
Date: Tue, 29 Mar 2022 20:13:11 +0100
Subject: [PATCH 2/2] gnu: tree-sitter-cli: Enable some tests and add node and
 dot.

* gnu/packages/tree-sitter.scm (tree-sitter-cli)[inputs]: Add graphviz and
node-lts.
[arguments]<#:cargo-test-flags>: Skip tests that require downloading grammars.
<#:phases>: Add 'patch-node and 'patch-dot phases.  Tweak install phase.
---
 gnu/packages/tree-sitter.scm | 49 ++++++++++++++++++++++++++++--------
 1 file changed, 39 insertions(+), 10 deletions(-)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 2e44339ca2..4331ffdd97 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -24,7 +24,9 @@ (define-module (gnu packages tree-sitter)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
+  #:use-module (gnu packages graphviz)
   #:use-module (gnu packages icu4c)
+  #:use-module (gnu packages node)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix gexp)
@@ -96,13 +98,30 @@ (define-public tree-sitter-cli
                               println!(\"cargo:rustc-link-lib=tree-sitter\");~@
                               }~%")))))))
     (build-system cargo-build-system)
-    (inputs (list tree-sitter))
+    (inputs
+     (list tree-sitter graphviz node-lts))
     (arguments
      (list
-      ;; Running test requires downloading fixtures, see the
-      ;; script/fetch-fixtures script, which fetches grammars.  Maybe it make
-      ;; sence to run tests in the grammar's packages?
-      #:tests? #f
+      #:cargo-test-flags
+      ''("--release" "--"
+         ;; Skip tests which rely on downloading grammar fixtures.  It is
+         ;; difficult to support such tests given upstream does not encode
+         ;; which version of the grammars are expected.
+         ;; Instead, we do run some tests for each grammar in the tree-sitter
+         ;; build-system, by running `tree-sitter test'.  This isn't as
+         ;; complete as running all tests from tree-sitter-cli, but it's a
+         ;; good compromise compared to maintaining two different sets of
+         ;; grammars (Guix packages vs test fixtures).
+         "--skip=tests::corpus_test"
+         "--skip=tests::highlight_test"
+         "--skip=tests::node_test"
+         "--skip=tests::parser_test"
+         "--skip=tests::pathological_test"
+         "--skip=tests::query_test"
+         "--skip=tests::tags_test"
+         "--skip=tests::test_highlight_test"
+         "--skip=tests::test_tags_test"
+         "--skip=tests::tree_test")
       ;; We're only packaging the CLI program so we do not need to install
       ;; sources.
       #:install-source? #f
@@ -132,11 +151,21 @@ (define-public tree-sitter-cli
           (add-after 'unpack 'delete-cargo-lock
             (lambda _
               (delete-file "Cargo.lock")))
-          (replace 'install
-            (lambda* (#:key outputs #:allow-other-keys)
-              (let ((bin (string-append #$output "/bin")))
-                (mkdir-p bin)
-                (install-file "target/release/tree-sitter" bin)))))))
+          (add-after 'unpack 'patch-node
+            (lambda _
+              (substitute* "cli/src/generate/mod.rs"
+                (("Command::new\\(\"node\"\\)")
+                 (string-append "Command::new(\"" #$node-lts "/bin/node\")")))))
+          (add-after 'unpack 'patch-dot
+            (lambda _
+              (substitute* "cli/src/util.rs"
+                (("Command::new\\(\"dot\"\\)")
+                 (string-append "Command::new(\"" #$graphviz "/bin/dot\")")))))
+           (replace 'install
+             (lambda _
+               (let ((bin (string-append #$output "/bin")))
+                 (mkdir-p bin)
+                 (install-file "target/release/tree-sitter" bin)))))))
     (description "Tree-sitter is a parser generator tool and an incremental
 parsing library.  It can build a concrete syntax tree for a source file and
 efficiently update the syntax tree as the source file is edited.
-- 
2.39.1


[-- Attachment #4: Type: text/plain, Size: 16 bytes --]


Thanks,
Pierre

[bug#49946] [PATCH 00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

[Liliana Marie Prikler]

Am Freitag, dem 10.02.2023 um 13:21 +0400 schrieb Andrew Tropin:

> +           ;; tree-sitter support is not yet available in emacs 28,
> but this
> +           ;; search path won't harm and also will be benifitial for
> +           ;; emacs-next and other emacs-* packages, which have
> tree-sitter
> +           ;; support enabled.  Please, remove this comment, when
> emacs
> +           ;; package is updated to 29.
IIRC it's "beneficial".  Otherwise LGTM

[bug#49946] [PATCH 00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

On 2023-02-10 19:02, Liliana Marie Prikler wrote:

> Am Freitag, dem 10.02.2023 um 13:21 +0400 schrieb Andrew Tropin:
>
>> +           ;; tree-sitter support is not yet available in emacs 28,
>> but this
>> +           ;; search path won't harm and also will be benifitial for
>> +           ;; emacs-next and other emacs-* packages, which have
>> tree-sitter
>> +           ;; support enabled.  Please, remove this comment, when
>> emacs
>> +           ;; package is updated to 29.
> IIRC it's "beneficial".  Otherwise LGTM

Nice catch, thank you very much, pushed as
c77a3d4554175ee5320ccc713aa21b1ba3e6b8a3

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 8729 bytes --]

On 2023-02-10 15:48, Pierre Langlois wrote:

> Hi Andrew, thanks for pushing this along! It's great to see things
> getting merged.
>
> Andrew Tropin <andrew@trop.in> writes:
>
>> [[PGP Signed Part:Undecided]]
>> On 2023-02-09 18:04, Andrew Tropin wrote:
>>
>>> On 2023-02-09 13:39, zimoun wrote:
>>>
>>>> Hi,
>>>>
>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>
>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>
>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>> right?
>>>>
>>>>
>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>
>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>
>>>> Leaving out all the others, right?
>>>
>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>> addresses some things from reviews and one commit, which adds html
>>> grammar.
>>>
>>> The html grammar is added for the testing purposes.  It relies on
>>> generated parser.c and scanner.c and we will need to repackage it using
>>> grammar.js instead.  I'm not sure if a separate build system is needed
>>> for this, I guess we can just rewrite tree-sitter-grammar function,
>>> which generates packages as in example with tree-sitter-grammar-html:
>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>>
>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>>> read it carefully, evaluate and either introduce new build system or
>>> just move all needed parts to tree-sitter-grammar function.  WDYT?
>>> After we done with it we can package all other grammars.
>>
>> Ok, I realized that the proper build process for tree-sitter grammars is
>> a little harder than I expected, tree-sitter-build system make sense.  I
>> reviewed it, made a small change:
>
> Ah great, I was going to comment to try and push for us to keep the
> build system. I originally went with a template package and inheritance,
> but Maxime suggested moving to a build-system which ended up making the
> package definitions a *lot* nicer IMO (see previous discussion here
> https://issues.guix.gnu.org/49946#144). It also allows us to deal with
> grammars that depend on each other more nicely I think.
>
>>
>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>>  ;; Commentary:
>>  ;;
>>  ;; Build procedures for tree-sitter grammar packages.  This is the
>> -;; builder-side code, which builds on top fo the node build-system.
>> +;; builder-side code, which builds on top of the node build-system.
>>  ;;
>>  ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>>  ;; shared object.  The `tree-sitter generate' command invokes `node' in order
>> @@ -114,7 +114,7 @@ (define (compile-language dir)
>>                     "-fno-exceptions"
>>                     "-O2"
>>                     "-g"
>> -                   "-o" ,(string-append lib "/" lang ".so")
>> +                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>>                     ;; An additional `scanner.{c,cc}' file is sometimes
>>                     ;; provided.
>>                     ,@(cond
>>
>>
>> rewrote html grammar to use this build system and made it work with
>> built-in treesit package.  Also, tried examples of c and cpp grammars
>> from patches in this thread.
>>
>> If you ok with it, I'll push the build system to master and update the
>> html grammar accordingly.
>>
>> The final result will look like this:
>>
>> (define tree-sitter-delete-generated-files
>>   #~(begin
>>       (delete-file "binding.gyp")
>>       (delete-file-recursively "bindings")
>>       (delete-file "src/grammar.json")
>>       (delete-file "src/node-types.json")
>>       (delete-file "src/parser.c")
>>       (delete-file-recursively "src/tree_sitter")))
>>
>> (define* (tree-sitter-grammar
>>           language language-for-synopsis version commit hash
>>           #:key
>>           (repository-url
>>            (format #f "https://github.com/tree-sitter/tree-sitter-~a" language))
>>           (inputs '()))
>>   (let ((synopsis (string-append language-for-synopsis
>>                                  " grammar for tree-sitter"))
>>         (name (string-append "tree-sitter-grammar-" language)))
>>     (package
>>       (name name)
>>       (version version)
>>       (home-page repository-url)
>>       (source (origin
>>                 (method git-fetch)
>>                 (uri (git-reference
>>                       (url repository-url)
>>                       (commit commit)))
>>                 (file-name (git-file-name name version))
>>                 (sha256 (base32 hash))
>>                 (modules '((guix build utils)))
>>                 (snippet tree-sitter-delete-generated-files)))
>>       (build-system tree-sitter-build-system)
>>       (inputs inputs)
>>       (synopsis synopsis)
>>       (description (string-append synopsis "."))
>>       (license license:expat))))
>>
>> (define-public tree-sitter-grammar-html
>>   (tree-sitter-grammar
>>    "html" "HTML"
>>    "0.19.0" "v0.19.0"
>>    "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
>>
>> After that we can bring the rest of the grammars.
>
> I would suggest to rmeove the `tree-sitter-grammar' function, and keep
> grammars as "regular" package records, even though it's a little bit
> more verbose:
>
> --8<---------------cut here---------------start------------->8---
> (define-public tree-sitter-html
>   (package
>     (name "tree-sitter-html")

It seems tree-sitter-html mimics upstream package name and probably make
more sense than tree-sitter-grammar-html used by me.

>     (version "0.19.0")
>     (source (origin
>               (method git-fetch)
>               (uri (git-reference
>                     (url "https://github.com/tree-sitter/tree-sitter-html")
>                     (commit (string-append "v" version))))
>               (file-name (git-file-name name version))
>               (sha256
>                (base32
>                 "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
>               (modules '((guix build utils)))
>               (snippet tree-sitter-delete-generated-files)))
>     (build-system tree-sitter-build-system)
>     (home-page "https://github.com/tree-sitter/tree-sitter-html")
>     (synopsis "Tree-sitter HTML grammar")
>     (description
>      "This package provides a HTML grammar for the Tree-sitter library.")
>     (license license:expat)))
> --8<---------------cut here---------------end--------------->8---
>
> This way, they look like any other package in Guix, which makes it
> easier for us to apply automatic changes in the future if needed (for
> example like how the input format could be automically updated for all
> "simple" package definitions, but had to be manual whenever custom code
> refactoring was done). Does this make sense?

Make sense, but on the other hand we already have hunspell, aspell
dictionaries and probably a few more others, which are very similiar in
spirit and we already have to keep in mind their existence on such
automatic code updates.

It looks that the packages differ only in url for the source code, lang
name and sometimes in inputs.  Having template package function can make
management of shared parts more centralized, reduce possibility of
copy-paste mistakes, when the description wasn't updated and so on and
can reduce the amount of the code overall (which also reduces the change
of introducing an error).

I don't have a strong opinion on this topic, but leaning towards the
template function slightly more, however I'm completely ok with the
standalone package definitions as well.  WDYT?

>
>>
>>>
>>> According to emacs integration: I already have a working prototype on
>>> my local setup, which uses builtin treesit package, it's similiar to
>>> what Luis mentioned with search-paths.  We just need to figure out a
>>> few minor details and I'll add it.
>>
>> I've sent a patch for emacs+tree-sitter integration in a different reply
>> in this thread.
>>
>> Thank you very much for all the work, it's really valuable!
>
> :-)
>
> Thanks,
> Pierre
>

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 4587 bytes --]

On 2023-02-10 17:02, Pierre Langlois wrote:

> Pierre Langlois <pierre.langlois@gmx.com> writes:
>
>> [[PGP Signed Part:Undecided]]
>> Hi Andrew, thanks for pushing this along! It's great to see things
>> getting merged.
>>
>> Andrew Tropin <andrew@trop.in> writes:
>>
>>> [[PGP Signed Part:Undecided]]
>>> On 2023-02-09 18:04, Andrew Tropin wrote:
>>>
>>>> On 2023-02-09 13:39, zimoun wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>>
>>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>>
>>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>>> right?
>>>>>
>>>>>
>>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>
>>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>
>>>>> Leaving out all the others, right?
>>>>
>>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>>> addresses some things from reviews and one commit, which adds html
>>>> grammar.
>>>>
>>>> The html grammar is added for the testing purposes.  It relies on
>>>> generated parser.c and scanner.c and we will need to repackage it using
>>>> grammar.js instead.  I'm not sure if a separate build system is needed
>>>> for this, I guess we can just rewrite tree-sitter-grammar function,
>>>> which generates packages as in example with tree-sitter-grammar-html:
>>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>>>
>>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>>>> read it carefully, evaluate and either introduce new build system or
>>>> just move all needed parts to tree-sitter-grammar function.  WDYT?
>>>> After we done with it we can package all other grammars.
>>>
>>> Ok, I realized that the proper build process for tree-sitter grammars is
>>> a little harder than I expected, tree-sitter-build system make sense.  I
>>> reviewed it, made a small change:
>>
>> Ah great, I was going to comment to try and push for us to keep the
>> build system. I originally went with a template package and inheritance,
>> but Maxime suggested moving to a build-system which ended up making the
>> package definitions a *lot* nicer IMO (see previous discussion here
>> https://issues.guix.gnu.org/49946#144). It also allows us to deal with
>> grammars that depend on each other more nicely I think.
>>
>>>
>>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>>>  ;; Commentary:
>>>  ;;
>>>  ;; Build procedures for tree-sitter grammar packages.  This is the
>>> -;; builder-side code, which builds on top fo the node build-system.
>>> +;; builder-side code, which builds on top of the node build-system.
>>>  ;;
>>>  ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>>>  ;; shared object.  The `tree-sitter generate' command invokes `node' in order
>>> @@ -114,7 +114,7 @@ (define (compile-language dir)
>>>                     "-fno-exceptions"
>>>                     "-O2"
>>>                     "-g"
>>> -                   "-o" ,(string-append lib "/" lang ".so")
>>> +                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>>>                     ;; An additional `scanner.{c,cc}' file is sometimes
>>>                     ;; provided.
>>>                     ,@(cond
>>>
>>>
>>> rewrote html grammar to use this build system and made it work with
>>> built-in treesit package.  Also, tried examples of c and cpp grammars
>>> from patches in this thread.
>>>
>>> If you ok with it, I'll push the build system to master and update the
>>> html grammar accordingly.
>
> Oh, I forgot to say, this change to the build system LGTM! I'm really
> happy to see it merged soon :-). The path change will probably break the
> emacs-28-based tree-sitter support, but that's OK, it's better for the
> build-system to be made to target emacs 29's builtin support. I'm sure I
> can work around for emacs 28.

Actually, I think we can build grammars with both names, just providing
two .so files instead of one.  If you won't find a better workaround we
can go this way.

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 7215 bytes --]

On 2023-02-10 17:04, Pierre Langlois wrote:

> Hi,
>
> Andrew Tropin <andrew@trop.in> writes:
>
>> [[PGP Signed Part:Undecided]]
>> On 2023-02-09 13:39, zimoun wrote:
>>
>>> Hi,
>>>
>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>
>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>
>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>> right?
>>>
>>>
>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>
>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>
>>> Leaving out all the others, right?
>>
>> Merged first 5 patches from 01 to 05, also added one more commit, which
>> addresses some things from reviews and one commit, which adds html
>> grammar.
>
> I had also started rebasing addressing comments on the tree-sitter-cli
> package, and had a few extra changes left that haven't been pushed.
>
> I realised we could remove a now redundant comment, and while enabling
> *some* tests, I realized we should directly depend on node and dot, so
> that people may use the `tree-sitter' command, without needing them in
> the same profile.

Sounds very valid! :)

>
> What do you think of the following two patches?
>

I applied them locally and tested, LGTM.  Will push them a little later
with the rest of tree-sitter related changes.

> From b24b04015261cfea2143be63671877c9c2c8d458 Mon Sep 17 00:00:00 2001
> From: Pierre Langlois <pierre.langlois@gmx.com> Date: Fri, 10 Feb 2023
> 16:10:25 +0000 Subject: [PATCH 1/2] gnu: Remove tree-sitter comment
> from emacs.scm imports.
>
> * gnu/packages/emacs.scm: Remove tree-sitter comment.
> ---
>  gnu/packages/emacs.scm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/gnu/packages/emacs.scm b/gnu/packages/emacs.scm
> index 4ce41deb88..4791570d12 100644
> --- a/gnu/packages/emacs.scm
> +++ b/gnu/packages/emacs.scm
> @@ -76,7 +76,7 @@ (define-module (gnu packages emacs)
>    #:use-module (gnu packages sqlite)
>    #:use-module (gnu packages texinfo)
>    #:use-module (gnu packages tls)
> -  #:use-module (gnu packages tree-sitter) ; for tree-sitter
> +  #:use-module (gnu packages tree-sitter)
>    #:use-module (gnu packages web)       ; for jansson
>    #:use-module (gnu packages webkit)
>    #:use-module (gnu packages xml)
> -- 
> 2.39.1
>
> From 8225d51edeb528c51d30ad0f225eb96be5586a37 Mon Sep 17 00:00:00 2001
> From: Pierre Langlois <pierre.langlois@gmx.com>
> Date: Tue, 29 Mar 2022 20:13:11 +0100
> Subject: [PATCH 2/2] gnu: tree-sitter-cli: Enable some tests and add node and
>  dot.
>
> * gnu/packages/tree-sitter.scm (tree-sitter-cli)[inputs]: Add graphviz and
> node-lts.
> [arguments]<#:cargo-test-flags>: Skip tests that require downloading grammars.
> <#:phases>: Add 'patch-node and 'patch-dot phases.  Tweak install phase.
> ---
>  gnu/packages/tree-sitter.scm | 49 ++++++++++++++++++++++++++++--------
>  1 file changed, 39 insertions(+), 10 deletions(-)
>
> diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
> index 2e44339ca2..4331ffdd97 100644
> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm
> @@ -24,7 +24,9 @@ (define-module (gnu packages tree-sitter)
>    #:use-module ((guix licenses) #:prefix license:)
>    #:use-module (gnu packages crates-graphics)
>    #:use-module (gnu packages crates-io)
> +  #:use-module (gnu packages graphviz)
>    #:use-module (gnu packages icu4c)
> +  #:use-module (gnu packages node)
>    #:use-module (guix build-system cargo)
>    #:use-module (guix build-system gnu)
>    #:use-module (guix gexp)
> @@ -96,13 +98,30 @@ (define-public tree-sitter-cli
>                                println!(\"cargo:rustc-link-lib=tree-sitter\");~@
>                                }~%")))))))
>      (build-system cargo-build-system)
> -    (inputs (list tree-sitter))
> +    (inputs
> +     (list tree-sitter graphviz node-lts))
>      (arguments
>       (list
> -      ;; Running test requires downloading fixtures, see the
> -      ;; script/fetch-fixtures script, which fetches grammars.  Maybe it make
> -      ;; sence to run tests in the grammar's packages?
> -      #:tests? #f
> +      #:cargo-test-flags
> +      ''("--release" "--"
> +         ;; Skip tests which rely on downloading grammar fixtures.  It is
> +         ;; difficult to support such tests given upstream does not encode
> +         ;; which version of the grammars are expected.
> +         ;; Instead, we do run some tests for each grammar in the tree-sitter
> +         ;; build-system, by running `tree-sitter test'.  This isn't as
> +         ;; complete as running all tests from tree-sitter-cli, but it's a
> +         ;; good compromise compared to maintaining two different sets of
> +         ;; grammars (Guix packages vs test fixtures).
> +         "--skip=tests::corpus_test"
> +         "--skip=tests::highlight_test"
> +         "--skip=tests::node_test"
> +         "--skip=tests::parser_test"
> +         "--skip=tests::pathological_test"
> +         "--skip=tests::query_test"
> +         "--skip=tests::tags_test"
> +         "--skip=tests::test_highlight_test"
> +         "--skip=tests::test_tags_test"
> +         "--skip=tests::tree_test")
>        ;; We're only packaging the CLI program so we do not need to install
>        ;; sources.
>        #:install-source? #f
> @@ -132,11 +151,21 @@ (define-public tree-sitter-cli
>            (add-after 'unpack 'delete-cargo-lock
>              (lambda _
>                (delete-file "Cargo.lock")))
> -          (replace 'install
> -            (lambda* (#:key outputs #:allow-other-keys)
> -              (let ((bin (string-append #$output "/bin")))
> -                (mkdir-p bin)
> -                (install-file "target/release/tree-sitter" bin)))))))
> +          (add-after 'unpack 'patch-node
> +            (lambda _
> +              (substitute* "cli/src/generate/mod.rs"
> +                (("Command::new\\(\"node\"\\)")
> +                 (string-append "Command::new(\"" #$node-lts "/bin/node\")")))))
> +          (add-after 'unpack 'patch-dot
> +            (lambda _
> +              (substitute* "cli/src/util.rs"
> +                (("Command::new\\(\"dot\"\\)")
> +                 (string-append "Command::new(\"" #$graphviz "/bin/dot\")")))))
> +           (replace 'install
> +             (lambda _
> +               (let ((bin (string-append #$output "/bin")))
> +                 (mkdir-p bin)
> +                 (install-file "target/release/tree-sitter" bin)))))))
>      (description "Tree-sitter is a parser generator tool and an incremental
>  parsing library.  It can build a concrete syntax tree for a source file and
>  efficiently update the syntax tree as the source file is edited.

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH v7 06/32] build-system: Add tree-sitter-build-system.

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 20531 bytes --]

On 2022-11-25 01:57, Pierre Langlois wrote:

> It appears this email is making it through, so here it is as an
> attachment:
>
> From 4a3c1fff8460a03bfb7c1aada9863205cd6f22fd Mon Sep 17 00:00:00 2001
> From: Pierre Langlois <pierre.langlois@gmx.com>
> Date: Tue, 29 Mar 2022 20:13:34 +0100
> Subject: [PATCH v7 06/32] build-system: Add tree-sitter-build-system.
>
> * guix/build-system/tree-sitter.scm: New module.
> * guix/build/tree-sitter-build-system.scm: Likewise.
> * Makefile.am (MODULES): Add them.
> * doc/guix.texi: Document it.
> ---
>  Makefile.am                             |   2 +
>  doc/guix.texi                           |  21 ++-
>  guix/build-system/tree-sitter.scm       | 190 ++++++++++++++++++++++++
>  guix/build/tree-sitter-build-system.scm | 153 +++++++++++++++++++
>  4 files changed, 365 insertions(+), 1 deletion(-)
>  create mode 100644 guix/build-system/tree-sitter.scm
>  create mode 100644 guix/build/tree-sitter-build-system.scm
>
> diff --git a/Makefile.am b/Makefile.am
> index c3af23b68e..a16c4fcd7e 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -178,6 +178,7 @@ MODULES =					\
>    guix/build-system/ruby.scm			\
>    guix/build-system/scons.scm			\
>    guix/build-system/texlive.scm			\
> +  guix/build-system/tree-sitter.scm		\
>    guix/build-system/trivial.scm			\
>    guix/ftp-client.scm				\
>    guix/http-client.scm				\
> @@ -234,6 +235,7 @@ MODULES =					\
>    guix/build/ruby-build-system.scm		\
>    guix/build/scons-build-system.scm		\
>    guix/build/texlive-build-system.scm		\
> +  guix/build/tree-sitter-build-system.scm	\
>    guix/build/waf-build-system.scm		\
>    guix/build/haskell-build-system.scm		\
>    guix/build/julia-build-system.scm		\
> diff --git a/doc/guix.texi b/doc/guix.texi
> index e547d469f4..4e997f7176 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -79,7 +79,7 @@ Copyright @copyright{} 2020 Jack Hill@*
>  Copyright @copyright{} 2020 Naga Malleswari@*
>  Copyright @copyright{} 2020, 2021 Brice Waegeneire@*
>  Copyright @copyright{} 2020 R Veera Kumar@*
> -Copyright @copyright{} 2020, 2021 Pierre Langlois@*
> +Copyright @copyright{} 2020, 2021, 2022 Pierre Langlois@*
>  Copyright @copyright{} 2020 pinoaffe@*
>  Copyright @copyright{} 2020 André Batista@*
>  Copyright @copyright{} 2020, 2021 Alexandru-Sergiu Marton@*
> @@ -9732,6 +9732,25 @@ be specified with the @code{#:node} parameter which defaults to
>  @code{node}.
>  @end defvr
>  
> +@defvr {Scheme Variable} tree-sitter-build-system
> +
> +This variable is exported by @code{(guix build-system tree-sitter)}.  It
> +implements procedures to compile grammars for the
> +@url{https://tree-sitter.github.io/tree-sitter/, Tree-sitter} parsing
> +library.  It essentially runs @code{tree-sitter generate} to translate
> +@code{grammar.js} grammars to JSON and then to C.  Which it then
> +compiles to native code.
> +
> +Tree-sitter packages may support multiple grammars, so this build system
> +supports a @code{#:grammar-directories} keyword to specify a list of
> +locations where a @code{grammar.js} file may be found.
> +
> +Grammars sometimes depend on each other, such as C++ depending on C and
> +TypeScript depending on JavaScript.  You may use inputs to declare such
> +dependencies.
> +
> +@end defvr
> +
>  Lastly, for packages that do not need anything as sophisticated, a
>  ``trivial'' build system is provided.  It is trivial in the sense that
>  it provides basically no support: it does not pull any implicit inputs,
> diff --git a/guix/build-system/tree-sitter.scm b/guix/build-system/tree-sitter.scm
> new file mode 100644
> index 0000000000..aeb96e3ef5
> --- /dev/null
> +++ b/guix/build-system/tree-sitter.scm
> @@ -0,0 +1,190 @@
> +;;; GNU Guix --- Functional package management for GNU
> +;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
> +;;;
> +;;; This file is part of GNU Guix.
> +;;;
> +;;; GNU Guix is free software; you can redistribute it and/or modify it
> +;;; under the terms of the GNU General Public License as published by
> +;;; the Free Software Foundation; either version 3 of the License, or (at
> +;;; your option) any later version.
> +;;;
> +;;; GNU Guix is distributed in the hope that it will be useful, but
> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;;; GNU General Public License for more details.
> +;;;
> +;;; You should have received a copy of the GNU General Public License
> +;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
> +
> +(define-module (guix build-system tree-sitter)
> +  #:use-module (guix store)
> +  #:use-module (guix utils)
> +  #:use-module (guix packages)
> +  #:use-module (guix gexp)
> +  #:use-module (guix monads)
> +  #:use-module (guix search-paths)
> +  #:use-module (guix build-system)
> +  #:use-module (guix build-system gnu)
> +  #:use-module (guix build-system node)
> +  #:use-module (ice-9 match)
> +  #:export (%tree-sitter-build-system-modules
> +            tree-sitter-build
> +            tree-sitter-build-system))
> +
> +(define %tree-sitter-build-system-modules
> +  ;; Build-side modules imported by default.
> +  `((guix build tree-sitter-build-system)
> +    ,@%node-build-system-modules))
> +
> +(define* (lower name
> +                #:key source inputs native-inputs outputs system target
> +                #:allow-other-keys
> +                #:rest arguments)
> +  "Return a bag for NAME from the given arguments."
> +  (define private-keywords
> +    `(#:inputs #:native-inputs #:outputs ,@(if target
> +                                               '()
> +                                               '(#:target))))
> +  (define node
> +    (module-ref (resolve-interface '(gnu packages node))
> +                'node-lts))
> +  (define tree-sitter
> +    (module-ref (resolve-interface '(gnu packages tree-sitter))
> +                'tree-sitter))
> +  (define tree-sitter-cli
> +    (module-ref (resolve-interface '(gnu packages tree-sitter))
> +                'tree-sitter-cli))
> +  ;; Grammars depend on each other via JS modules, which we package into a
> +  ;; dedicated js output.
> +  (define grammar-inputs
> +    (map (match-lambda
> +           ((name package)
> +            `(,name ,package "js")))
> +         inputs))
> +  (bag
> +    (name name)
> +    (system system) (target target)
> +    (build-inputs `(,@(if source
> +                          `(("source" ,source))
> +                          '())
> +                    ("node" ,node)
> +                    ("tree-sitter-cli" ,tree-sitter-cli)
> +                    ,@native-inputs
> +                    ,@(if target '() grammar-inputs)
> +                    ;; Keep the standard inputs of 'gnu-build-system'.
> +                    ,@(if target
> +                          (standard-cross-packages target 'host)
> +                          '())
> +                    ,@(standard-packages)))
> +    (host-inputs `(("tree-sitter" ,tree-sitter)
> +                   ,@(if target grammar-inputs '())))
> +    ;; Keep the standard inputs of 'gnu-buid-system'.
> +    (target-inputs (if target
> +                       (standard-cross-packages target 'target)
> +                       '()))
> +    ;; XXX: this is a hack to get around issue #41569.
> +    (outputs (match outputs
> +               (("out") (cons "js" outputs))
> +               (_ outputs)))
> +    (build (if target tree-sitter-cross-build tree-sitter-build))
> +    (arguments (strip-keyword-arguments private-keywords arguments))))
> +
> +(define* (tree-sitter-build name inputs
> +                            #:key
> +                            source
> +                            (phases '%standard-phases)
> +                            (grammar-directories '("."))
> +                            (tests? #t)
> +                            (outputs '("out" "js"))
> +                            (search-paths '())
> +                            (system (%current-system))
> +                            (guile #f)
> +                            (imported-modules %tree-sitter-build-system-modules)
> +                            (modules '((guix build utils)
> +                                       (guix build tree-sitter-build-system))))
> +  (define builder
> +    (with-imported-modules imported-modules
> +      #~(begin
> +          (use-modules #$@(sexp->gexp modules))
> +          (tree-sitter-build #:name #$name
> +                             #:source #+source
> +                             #:system #$system
> +                             #:phases #$phases
> +                             #:tests? #$tests?
> +                             #:grammar-directories '#$grammar-directories
> +                             #:outputs #$(outputs->gexp outputs)
> +                             #:search-paths '#$(sexp->gexp
> +                                                (map search-path-specification->sexp
> +                                                     search-paths))
> +                             #:inputs #$(input-tuples->gexp inputs)))))
> +
> +  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
> +                                                  system #:graft? #f)))
> +    (gexp->derivation name builder
> +                      #:system system
> +                      #:guile-for-build guile)))
> +
> +(define* (tree-sitter-cross-build name
> +                                  #:key
> +                                  target
> +                                  build-inputs target-inputs host-inputs
> +                                  guile source
> +                                  (phases '%standard-phases)
> +                                  (grammar-directories '("."))
> +                                  (tests? #t)
> +                                  (outputs '("out" "js"))
> +                                  (search-paths '())
> +                                  (native-search-paths '())
> +                                  (system (%current-system))
> +                                  (build (nix-system->gnu-triplet system))
> +                                  (imported-modules %tree-sitter-build-system-modules)
> +                                  (modules '((guix build utils)
> +                                             (guix build tree-sitter-build-system))))
> +  (define builder
> +    (with-imported-modules imported-modules
> +      #~(begin
> +          (use-modules #$@(sexp->gexp modules))
> +
> +          (define %build-host-inputs
> +            #+(input-tuples->gexp build-inputs))
> +
> +          (define %build-target-inputs
> +            (append #$(input-tuples->gexp host-inputs)
> +                    #+(input-tuples->gexp target-inputs)))
> +
> +          (define %build-inputs
> +            (append %build-host-inputs %build-target-inputs))
> +
> +          (tree-sitter-build #:name #$name
> +                             #:source #+source
> +                             #:system #$system
> +                             #:build #$build
> +                             #:target #$target
> +                             #:phases #$phases
> +                             #:tests? #$tests?
> +                             #:grammar-directories '#$grammar-directories
> +                             #:outputs #$(outputs->gexp outputs)
> +                             #:inputs %build-target-inputs
> +                             #:native-inputs %build-host-inputs
> +                             #:search-paths '#$(sexp->gexp
> +                                                (map search-path-specification->sexp
> +                                                     search-paths))
> +                             #:native-search-paths '#$(sexp->gexp
> +                                                       (map
> +                                                        search-path-specification->sexp
> +                                                        native-search-paths))))))
> +
> +  (mlet %store-monad ((guile (package->derivation (or guile (default-guile))
> +                                                  system #:graft? #f)))
> +    (gexp->derivation name builder
> +                      #:system system
> +                      #:target target
> +                      #:guile-for-build guile)))
> +
> +(define tree-sitter-build-system
> +  (build-system
> +    (name 'tree-sitter)
> +    (description "The Tree-sitter grammar build system")
> +    (lower lower)))
> +
> +;;; tree-sitter.scm ends here
> diff --git a/guix/build/tree-sitter-build-system.scm b/guix/build/tree-sitter-build-system.scm
> new file mode 100644
> index 0000000000..574b0f2a1c
> --- /dev/null
> +++ b/guix/build/tree-sitter-build-system.scm
> @@ -0,0 +1,153 @@
> +;;; GNU Guix --- Functional package management for GNU
> +;;; Copyright © 2022 Pierre Langlois <pierre.langlois@gmx.com>
> +;;;
> +;;; This file is part of GNU Guix.
> +;;;
> +;;; GNU Guix is free software; you can redistribute it and/or modify it
> +;;; under the terms of the GNU General Public License as published by
> +;;; the Free Software Foundation; either version 3 of the License, or (at
> +;;; your option) any later version.
> +;;;
> +;;; GNU Guix is distributed in the hope that it will be useful, but
> +;;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;;; GNU General Public License for more details.
> +;;;
> +;;; You should have received a copy of the GNU General Public License
> +;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
> +
> +(define-module (guix build tree-sitter-build-system)
> +  #:use-module ((guix build node-build-system) #:prefix node:)
> +  #:use-module (guix build json)
> +  #:use-module (guix build utils)
> +  #:use-module (ice-9 match)
> +  #:use-module (ice-9 regex)
> +  #:use-module (srfi srfi-1)
> +  #:export (%standard-phases
> +            tree-sitter-build))
> +
> +;; Commentary:
> +;;
> +;; Build procedures for tree-sitter grammar packages.  This is the
> +;; builder-side code, which builds on top fo the node build-system.
> +;;
> +;; Tree-sitter grammars are written in JavaScript and compiled to a native
> +;; shared object.  The `tree-sitter generate' command invokes `node' in order
> +;; to evaluate the grammar.js into a grammar.json file, which is then
> +;; translated into C code.  We then compile the C code ourselves.  Packages
> +;; also sometimes add extra manually written C/C++ code.
> +;;
> +;; In order to support grammars depending on each other, such as C and C++,
> +;; JavaScript and TypeScript, this build-system installs the source of the
> +;; node module in a dedicated "js" output.
> +;;
> +;; Code:
> +
> +(define* (patch-dependencies #:key inputs #:allow-other-keys)
> +  "Rewrite dependencies in 'package.json'.  We remove all runtime dependencies
> +and replace development dependencies with tree-sitter grammar node modules."
> +
> +  (define (rewrite package.json)
> +    (map (match-lambda
> +           (("dependencies" @ . _)
> +            '("dependencies" @))
> +           (("devDependencies" @ . _)
> +            `("devDependencies" @
> +              ,@(filter-map (match-lambda
> +                              ((key . directory)
> +                               (let ((node-module
> +                                      (string-append directory
> +                                                     "/lib/node_modules/"
> +                                                     key)))
> +                                 (and (directory-exists? node-module)
> +                                      `(,key . ,node-module)))))
> +                            (alist-delete "node" inputs))))
> +           (other other))
> +         package.json))
> +
> +  (node:with-atomic-json-file-replacement "package.json"
> +    (match-lambda
> +      (('@ . package.json)
> +       (cons '@ (rewrite package.json))))))
> +
> +;; FIXME: The node build-system's configure phase does not support
> +;; cross-compiling so we re-define it.
> +(define* (configure #:key native-inputs inputs #:allow-other-keys)
> +  (invoke (search-input-file (or native-inputs inputs) "/bin/npm")
> +          "--offline" "--ignore-scripts" "install"))
> +
> +(define* (build #:key grammar-directories #:allow-other-keys)
> +  (for-each (lambda (dir)
> +              (with-directory-excursion dir
> +                ;; Avoid generating binding code for other languages, we do
> +                ;; not support this use-case yet and it relies on running
> +                ;; `node-gyp' to build native addons.
> +                (invoke "tree-sitter" "generate" "--no-bindings")))
> +            grammar-directories))
> +
> +(define* (check #:key grammar-directories tests? #:allow-other-keys)
> +  (when tests?
> +    (for-each (lambda (dir)
> +                (with-directory-excursion dir
> +                  (invoke "tree-sitter" "test")))
> +              grammar-directories)))
> +
> +(define* (install #:key target grammar-directories outputs #:allow-other-keys)
> +  (let ((lib (string-append (assoc-ref outputs "out")
> +                            "/lib/tree-sitter")))
> +    (mkdir-p lib)
> +    (define (compile-language dir)
> +      (with-directory-excursion dir
> +        (let ((lang (assoc-ref (call-with-input-file "src/grammar.json"
> +                                 read-json)
> +                               "name"))
> +              (source-file (lambda (path)
> +                             (if (file-exists? path)
> +                                 path
> +                                 #f))))
> +          (apply invoke
> +                 `(,(if target
> +                        (string-append target "-g++")
> +                        "g++")
> +                   "-shared"
> +                   "-fPIC"
> +                   "-fno-exceptions"
> +                   "-O2"
> +                   "-g"
> +                   "-o" ,(string-append lib "/" lang ".so")
> +                   ;; An additional `scanner.{c,cc}' file is sometimes
> +                   ;; provided.
> +                   ,@(cond
> +                      ((source-file "src/scanner.c")
> +                       => (lambda (file) (list "-xc" "-std=c99" file)))
> +                      ((source-file "src/scanner.cc")
> +                       => (lambda (file) (list file)))
> +                      (else '()))
> +                   "-xc" "src/parser.c")))))
> +    (for-each compile-language grammar-directories)))
> +
> +(define* (install-js #:key native-inputs inputs outputs #:allow-other-keys)
> +  (invoke (search-input-file (or native-inputs inputs) "/bin/npm")
> +          "--prefix" (assoc-ref outputs "js")
> +          "--global"
> +          "--offline"
> +          "--loglevel" "info"
> +          "--production"
> +          ;; Skip scripts to prevent building bindings via GYP.
> +          "--ignore-scripts"
> +          "install" "../package.tgz"))
> +
> +(define %standard-phases
> +  (modify-phases node:%standard-phases
> +    (replace 'patch-dependencies patch-dependencies)
> +    (replace 'configure configure)
> +    (replace 'build build)
> +    (replace 'check check)
> +    (replace 'install install)
> +    (add-after 'install 'install-js install-js)))
> +
> +(define* (tree-sitter-build #:key inputs (phases %standard-phases)
> +                            #:allow-other-keys #:rest args)
> +  (apply node:node-build #:inputs inputs #:phases phases args))
> +
> +;;; tree-sitter-build-system.scm ends here

Applied, slightly reformatted, pushed as
dbd4d2d0707b486f1e2c8659e94e1d3b15e4351e

Also, migrated tree-sitter-grammar fn to this build system.

I still try to get the taste of packaging grammars with
tree-sitter-grammar helper and just as plain packages and try to
understand how good/bad template function idea is in this case and if we
want to keep it or not.

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 9375 bytes --]

Hi,

Andrew Tropin <andrew@trop.in> writes:

> [[PGP Signed Part:Undecided]]
> On 2023-02-10 15:48, Pierre Langlois wrote:
>
>> Hi Andrew, thanks for pushing this along! It's great to see things
>> getting merged.
>>
>> Andrew Tropin <andrew@trop.in> writes:
>>
>>> [[PGP Signed Part:Undecided]]
>>> On 2023-02-09 18:04, Andrew Tropin wrote:
>>>
>>>> On 2023-02-09 13:39, zimoun wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>>
>>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>>
>>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>>> right?
>>>>>
>>>>>
>>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>
>>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>
>>>>> Leaving out all the others, right?
>>>>
>>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>>> addresses some things from reviews and one commit, which adds html
>>>> grammar.
>>>>
>>>> The html grammar is added for the testing purposes.  It relies on
>>>> generated parser.c and scanner.c and we will need to repackage it using
>>>> grammar.js instead.  I'm not sure if a separate build system is needed
>>>> for this, I guess we can just rewrite tree-sitter-grammar function,
>>>> which generates packages as in example with tree-sitter-grammar-html:
>>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>>>
>>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>>>> read it carefully, evaluate and either introduce new build system or
>>>> just move all needed parts to tree-sitter-grammar function.  WDYT?
>>>> After we done with it we can package all other grammars.
>>>
>>> Ok, I realized that the proper build process for tree-sitter grammars is
>>> a little harder than I expected, tree-sitter-build system make sense.  I
>>> reviewed it, made a small change:
>>
>> Ah great, I was going to comment to try and push for us to keep the
>> build system. I originally went with a template package and inheritance,
>> but Maxime suggested moving to a build-system which ended up making the
>> package definitions a *lot* nicer IMO (see previous discussion here
>> https://issues.guix.gnu.org/49946#144). It also allows us to deal with
>> grammars that depend on each other more nicely I think.
>>
>>>
>>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>>>  ;; Commentary:
>>>  ;;
>>>  ;; Build procedures for tree-sitter grammar packages.  This is the
>>> -;; builder-side code, which builds on top fo the node build-system.
>>> +;; builder-side code, which builds on top of the node build-system.
>>>  ;;
>>>  ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>>>  ;; shared object.  The `tree-sitter generate' command invokes `node' in order
>>> @@ -114,7 +114,7 @@ (define (compile-language dir)
>>>                     "-fno-exceptions"
>>>                     "-O2"
>>>                     "-g"
>>> -                   "-o" ,(string-append lib "/" lang ".so")
>>> +                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>>>                     ;; An additional `scanner.{c,cc}' file is sometimes
>>>                     ;; provided.
>>>                     ,@(cond
>>>
>>>
>>> rewrote html grammar to use this build system and made it work with
>>> built-in treesit package.  Also, tried examples of c and cpp grammars
>>> from patches in this thread.
>>>
>>> If you ok with it, I'll push the build system to master and update the
>>> html grammar accordingly.
>>>
>>> The final result will look like this:
>>>
>>> (define tree-sitter-delete-generated-files
>>>   #~(begin
>>>       (delete-file "binding.gyp")
>>>       (delete-file-recursively "bindings")
>>>       (delete-file "src/grammar.json")
>>>       (delete-file "src/node-types.json")
>>>       (delete-file "src/parser.c")
>>>       (delete-file-recursively "src/tree_sitter")))
>>>
>>> (define* (tree-sitter-grammar
>>>           language language-for-synopsis version commit hash
>>>           #:key
>>>           (repository-url
>>>            (format #f "https://github.com/tree-sitter/tree-sitter-~a" language))
>>>           (inputs '()))
>>>   (let ((synopsis (string-append language-for-synopsis
>>>                                  " grammar for tree-sitter"))
>>>         (name (string-append "tree-sitter-grammar-" language)))
>>>     (package
>>>       (name name)
>>>       (version version)
>>>       (home-page repository-url)
>>>       (source (origin
>>>                 (method git-fetch)
>>>                 (uri (git-reference
>>>                       (url repository-url)
>>>                       (commit commit)))
>>>                 (file-name (git-file-name name version))
>>>                 (sha256 (base32 hash))
>>>                 (modules '((guix build utils)))
>>>                 (snippet tree-sitter-delete-generated-files)))
>>>       (build-system tree-sitter-build-system)
>>>       (inputs inputs)
>>>       (synopsis synopsis)
>>>       (description (string-append synopsis "."))
>>>       (license license:expat))))
>>>
>>> (define-public tree-sitter-grammar-html
>>>   (tree-sitter-grammar
>>>    "html" "HTML"
>>>    "0.19.0" "v0.19.0"
>>>    "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
>>>
>>> After that we can bring the rest of the grammars.
>>
>> I would suggest to rmeove the `tree-sitter-grammar' function, and keep
>> grammars as "regular" package records, even though it's a little bit
>> more verbose:
>>
>> --8<---------------cut here---------------start------------->8---
>> (define-public tree-sitter-html
>>   (package
>>     (name "tree-sitter-html")
>
> It seems tree-sitter-html mimics upstream package name and probably make
> more sense than tree-sitter-grammar-html used by me.

Yeah, at some point I think I had named the packages with "grammar" as
well, but thought it was a bit of a mouthful.  I'm also thinking one day
we may build language bindings as part of the build system (Rust and
NodeJS I think ATM), so those packages could do more than ship the
grammar in the future (although we don't know if we'll ever really need
that).

>
>>     (version "0.19.0")
>>     (source (origin
>>               (method git-fetch)
>>               (uri (git-reference
>>                     (url "https://github.com/tree-sitter/tree-sitter-html")
>>                     (commit (string-append "v" version))))
>>               (file-name (git-file-name name version))
>>               (sha256
>>                (base32
>>                 "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
>>               (modules '((guix build utils)))
>>               (snippet tree-sitter-delete-generated-files)))
>>     (build-system tree-sitter-build-system)
>>     (home-page "https://github.com/tree-sitter/tree-sitter-html")
>>     (synopsis "Tree-sitter HTML grammar")
>>     (description
>>      "This package provides a HTML grammar for the Tree-sitter library.")
>>     (license license:expat)))
>> --8<---------------cut here---------------end--------------->8---
>>
>> This way, they look like any other package in Guix, which makes it
>> easier for us to apply automatic changes in the future if needed (for
>> example like how the input format could be automically updated for all
>> "simple" package definitions, but had to be manual whenever custom code
>> refactoring was done). Does this make sense?
>
> Make sense, but on the other hand we already have hunspell, aspell
> dictionaries and probably a few more others, which are very similiar in
> spirit and we already have to keep in mind their existence on such
> automatic code updates.
>
> It looks that the packages differ only in url for the source code, lang
> name and sometimes in inputs.  Having template package function can make
> management of shared parts more centralized, reduce possibility of
> copy-paste mistakes, when the description wasn't updated and so on and
> can reduce the amount of the code overall (which also reduces the change
> of introducing an error).
>
> I don't have a strong opinion on this topic, but leaning towards the
> template function slightly more, however I'm completely ok with the
> standalone package definitions as well.  WDYT?

I can think of both cost/benefits to the template so I don't have a
strong opinion either :-).

I do like the template to make sure people don't forget to delete
generated files, that's quite important as it seems upstream packages
often check-in the generated C code.  Although, we could probably assert
that with in the build-system phase?  I'll think about that.

On the other hand, I wonder how the template works for packages that
provide multiple grammars (see ocaml and typescript for example).  I
guess we could use the template for trivial packages, and standalone
definitions for more complex ones?  In general, if we keep the template
interface really simple, then I'm happy with it.

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 10653 bytes --]


Andrew Tropin <andrew@trop.in> writes:

> [[PGP Signed Part:Undecided]]
> On 2023-02-10 17:02, Pierre Langlois wrote:
>
>> Pierre Langlois <pierre.langlois@gmx.com> writes:
>>
>>> [[PGP Signed Part:Undecided]]
>>> Hi Andrew, thanks for pushing this along! It's great to see things
>>> getting merged.
>>>
>>> Andrew Tropin <andrew@trop.in> writes:
>>>
>>>> [[PGP Signed Part:Undecided]]
>>>> On 2023-02-09 18:04, Andrew Tropin wrote:
>>>>
>>>>> On 2023-02-09 13:39, zimoun wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>>>
>>>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>>>
>>>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>>>> right?
>>>>>>
>>>>>>
>>>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>>
>>>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>>
>>>>>> Leaving out all the others, right?
>>>>>
>>>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>>>> addresses some things from reviews and one commit, which adds html
>>>>> grammar.
>>>>>
>>>>> The html grammar is added for the testing purposes.  It relies on
>>>>> generated parser.c and scanner.c and we will need to repackage it using
>>>>> grammar.js instead.  I'm not sure if a separate build system is needed
>>>>> for this, I guess we can just rewrite tree-sitter-grammar function,
>>>>> which generates packages as in example with tree-sitter-grammar-html:
>>>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>>>>
>>>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>>>>> read it carefully, evaluate and either introduce new build system or
>>>>> just move all needed parts to tree-sitter-grammar function.  WDYT?
>>>>> After we done with it we can package all other grammars.
>>>>
>>>> Ok, I realized that the proper build process for tree-sitter grammars is
>>>> a little harder than I expected, tree-sitter-build system make sense.  I
>>>> reviewed it, made a small change:
>>>
>>> Ah great, I was going to comment to try and push for us to keep the
>>> build system. I originally went with a template package and inheritance,
>>> but Maxime suggested moving to a build-system which ended up making the
>>> package definitions a *lot* nicer IMO (see previous discussion here
>>> https://issues.guix.gnu.org/49946#144). It also allows us to deal with
>>> grammars that depend on each other more nicely I think.
>>>
>>>>
>>>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>>>>  ;; Commentary:
>>>>  ;;
>>>>  ;; Build procedures for tree-sitter grammar packages.  This is the
>>>> -;; builder-side code, which builds on top fo the node build-system.
>>>> +;; builder-side code, which builds on top of the node build-system.
>>>>  ;;
>>>>  ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>>>>  ;; shared object.  The `tree-sitter generate' command invokes `node' in order
>>>> @@ -114,7 +114,7 @@ (define (compile-language dir)
>>>>                     "-fno-exceptions"
>>>>                     "-O2"
>>>>                     "-g"
>>>> -                   "-o" ,(string-append lib "/" lang ".so")
>>>> +                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>>>>                     ;; An additional `scanner.{c,cc}' file is sometimes
>>>>                     ;; provided.
>>>>                     ,@(cond
>>>>
>>>>
>>>> rewrote html grammar to use this build system and made it work with
>>>> built-in treesit package.  Also, tried examples of c and cpp grammars
>>>> from patches in this thread.
>>>>
>>>> If you ok with it, I'll push the build system to master and update the
>>>> html grammar accordingly.
>>
>> Oh, I forgot to say, this change to the build system LGTM! I'm really
>> happy to see it merged soon :-). The path change will probably break the
>> emacs-28-based tree-sitter support, but that's OK, it's better for the
>> build-system to be made to target emacs 29's builtin support. I'm sure I
>> can work around for emacs 28.
>
> Actually, I think we can build grammars with both names, just providing
> two .so files instead of one.  If you won't find a better workaround we
> can go this way.

That's no problem, I was able to work-around it quite easily.  The
emacs-tree-sitter-langs package expects all the grammars supported to be
in a single directory, so we have to bundle them.  So we can quite
easily create symlinks from the bundle with the expected names,
something like:

--8<---------------cut here---------------start------------->8---
(define-public emacs-tree-sitter-langs-grammar-bundle
  (package
    (name "emacs-tree-sitter-langs-grammar-bundle")
    (source #f)
    (version (package-version tree-sitter))
    (build-system trivial-build-system)
    (inputs
     ;; FIXME: Support for some languages is still left to package.
     (list tree-sitter-bash
           tree-sitter-c
           tree-sitter-c-sharp
           tree-sitter-cpp
           tree-sitter-css
           tree-sitter-elixir
           tree-sitter-elm
           tree-sitter-go
           tree-sitter-haskell
           tree-sitter-html
           tree-sitter-java
           tree-sitter-javascript
           tree-sitter-json
           tree-sitter-julia
           tree-sitter-lua
           tree-sitter-ocaml
           tree-sitter-php
           tree-sitter-python
           tree-sitter-r
           tree-sitter-rust
           tree-sitter-ruby
           tree-sitter-typescript))
    (arguments
     (list #:builder
           (with-imported-modules '((guix build union)
                                    (guix build utils))
             #~(begin
                 (use-modules (ice-9 match)
                              (srfi srfi-1)
                              (guix build union)
                              (guix build utils))
                 (mkdir-p #$output)
                 (for-each
                  (lambda (lib)
                    (symlink lib
                             (string-append #$output "/"
                                            (substring (basename lib)
                                                       (string-length "libtree-sitter-")))))
                  (append-map (match-lambda
                                ((name directory)
                                 (find-files directory "libtree-sitter-.*\\.so$")))
                              '#$(package-inputs this-package)))))))
    (synopsis #f)
    (description #f)
    (home-page #f)
    (license #f)))
--8<---------------cut here---------------end--------------->8---

Giving us:

--8<---------------cut here---------------start------------->8---
/gnu/store/fr9h59wgn55ilfifvm5df7xzxfwh38pc-emacs-tree-sitter-langs-grammar-bundle-0.20.7
├── bash.so -> /gnu/store/9qzvcdlpryjl44klfq0i2liqf6lsp5jq-tree-sitter-bash-0.19.0/lib/tree-sitter/libtree-sitter-bash.so
├── cpp.so -> /gnu/store/my02kq2dr6h48nmrl2dnfnm5rggx837w-tree-sitter-cpp-0.20.0-1.5ead1e2/lib/tree-sitter/libtree-sitter-cpp.so
├── c_sharp.so -> /gnu/store/mp8jvbhx5xlgj5bxa52zhmk5x8i7md5h-tree-sitter-c-sharp-0.19.1/lib/tree-sitter/libtree-sitter-c_sharp.so
├── c.so -> /gnu/store/y9ln76yx68q495vx8bnqdy87x7k8ihy5-tree-sitter-c-0.20.2/lib/tree-sitter/libtree-sitter-c.so
├── css.so -> /gnu/store/fwji59sdwvqpqyb94m55qw1ak92cmys9-tree-sitter-css-0.19.0/lib/tree-sitter/libtree-sitter-css.so
├── elixir.so -> /gnu/store/f6ismnxf7hlh1sq87zzarw56avvrzsl9-tree-sitter-elixir-0.19.0-1.de20391/lib/tree-sitter/libtree-sitter-elixir.so
├── elm.so -> /gnu/store/56zjl2ljvba3z1j6zfmpfa8mq19a3g3a-tree-sitter-elm-5.6.3/lib/tree-sitter/libtree-sitter-elm.so
├── go.so -> /gnu/store/qfy5sw6za96wkszbi21adxbxzggvjz3g-tree-sitter-go-0.19.1-1.05900fa/lib/tree-sitter/libtree-sitter-go.so
├── haskell.so -> /gnu/store/cq3chmqcb8g1nf5mzb6yhzdd6x9gvg1v-tree-sitter-haskell-0.14.0-1.e30bdfd/lib/tree-sitter/libtree-sitter-haskell.so
├── html.so -> /gnu/store/5pqfyjrg3yyvxaxidk690ffls3yb7wbi-tree-sitter-html-0.19.0/lib/tree-sitter/libtree-sitter-html.so
├── javascript.so -> /gnu/store/28s09v3dfb1c9bdkq2791z9abxnih66p-tree-sitter-javascript-0.20.0/lib/tree-sitter/libtree-sitter-javascript.so
├── java.so -> /gnu/store/i7jlqf5hbv0rhlkp4qlyc2d5ndw69dx4-tree-sitter-java-0.20.1/lib/tree-sitter/libtree-sitter-java.so
├── json.so -> /gnu/store/5dv4r74p6gd9w2ncs6pjrhz5hbw47262-tree-sitter-json-0.19.0/lib/tree-sitter/libtree-sitter-json.so
├── julia.so -> /gnu/store/5k37g1sdsllgh64p0w6ggabsni6jqlkr-tree-sitter-julia-0.19.0/lib/tree-sitter/libtree-sitter-julia.so
├── lua.so -> /gnu/store/nqzn4a6kgb2rx4y44pxdm4sqf3pzcpz1-tree-sitter-lua-0.0.14/lib/tree-sitter/libtree-sitter-lua.so
├── ocaml_interface.so -> /gnu/store/3h7krcj3xxclirb8afxh65ipabw1821l-tree-sitter-ocaml-0.19.0-1.0348562/lib/tree-sitter/libtree-sitter-ocaml_interface.so
├── ocaml.so -> /gnu/store/3h7krcj3xxclirb8afxh65ipabw1821l-tree-sitter-ocaml-0.19.0-1.0348562/lib/tree-sitter/libtree-sitter-ocaml.so
├── php.so -> /gnu/store/ymxf5m8jhihbrag7v2pghgydj3byp7wh-tree-sitter-php-0.19.0-1.435fa00/lib/tree-sitter/libtree-sitter-php.so
├── python.so -> /gnu/store/ggmzicwfxb7gz1rr9lfkx8cak62bfw7v-tree-sitter-python-0.19.1-1.ed0fe62/lib/tree-sitter/libtree-sitter-python.so
├── r.so -> /gnu/store/y9dxbnbb5dyf0rq3kpar7ip4w3lq6sb9-tree-sitter-r-0.0.1-1.80efda5/lib/tree-sitter/libtree-sitter-r.so
├── ruby.so -> /gnu/store/ky8n30dw16ck6byaqnhbf9ib7xp7j0yw-tree-sitter-ruby-0.20.0/lib/tree-sitter/libtree-sitter-ruby.so
├── rust.so -> /gnu/store/25zdpwrgq1xibhv7xpg64i4g71xah6g1-tree-sitter-rust-0.20.1/lib/tree-sitter/libtree-sitter-rust.so
├── tsx.so -> /gnu/store/wqpcphz855yjrginwqrymd3xzzxb8k8l-tree-sitter-typescript-0.20.1/lib/tree-sitter/libtree-sitter-tsx.so
└── typescript.so -> /gnu/store/wqpcphz855yjrginwqrymd3xzzxb8k8l-tree-sitter-typescript-0.20.1/lib/tree-sitter/libtree-sitter-typescript.so
--8<---------------cut here---------------end--------------->8---

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 1746 bytes --]


Andrew Tropin <andrew@trop.in> writes:

> [[PGP Signed Part:Undecided]]
> On 2023-02-10 17:04, Pierre Langlois wrote:
>
>> Hi,
>>
>> Andrew Tropin <andrew@trop.in> writes:
>>
>>> [[PGP Signed Part:Undecided]]
>>> On 2023-02-09 13:39, zimoun wrote:
>>>
>>>> Hi,
>>>>
>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>
>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>
>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>> right?
>>>>
>>>>
>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>
>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>
>>>> Leaving out all the others, right?
>>>
>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>> addresses some things from reviews and one commit, which adds html
>>> grammar.
>>
>> I had also started rebasing addressing comments on the tree-sitter-cli
>> package, and had a few extra changes left that haven't been pushed.
>>
>> I realised we could remove a now redundant comment, and while enabling
>> *some* tests, I realized we should directly depend on node and dot, so
>> that people may use the `tree-sitter' command, without needing them in
>> the same profile.
>
> Sounds very valid! :)
>
>>
>> What do you think of the following two patches?
>>
>
> I applied them locally and tested, LGTM.  Will push them a little later
> with the rest of tree-sitter related changes.

Cool, thanks!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Andrew Tropin]

[-- Attachment #1: Type: text/plain, Size: 10587 bytes --]

On 2023-02-12 12:07, Pierre Langlois wrote:

> Hi,
>
> Andrew Tropin <andrew@trop.in> writes:
>
>> [[PGP Signed Part:Undecided]]
>> On 2023-02-10 15:48, Pierre Langlois wrote:
>>
>>> Hi Andrew, thanks for pushing this along! It's great to see things
>>> getting merged.
>>>
>>> Andrew Tropin <andrew@trop.in> writes:
>>>
>>>> [[PGP Signed Part:Undecided]]
>>>> On 2023-02-09 18:04, Andrew Tropin wrote:
>>>>
>>>>> On 2023-02-09 13:39, zimoun wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>>>
>>>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>>>
>>>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>>>> right?
>>>>>>
>>>>>>
>>>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>>
>>>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>>
>>>>>> Leaving out all the others, right?
>>>>>
>>>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>>>> addresses some things from reviews and one commit, which adds html
>>>>> grammar.
>>>>>
>>>>> The html grammar is added for the testing purposes.  It relies on
>>>>> generated parser.c and scanner.c and we will need to repackage it using
>>>>> grammar.js instead.  I'm not sure if a separate build system is needed
>>>>> for this, I guess we can just rewrite tree-sitter-grammar function,
>>>>> which generates packages as in example with tree-sitter-grammar-html:
>>>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>>>>
>>>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>>>>> read it carefully, evaluate and either introduce new build system or
>>>>> just move all needed parts to tree-sitter-grammar function.  WDYT?
>>>>> After we done with it we can package all other grammars.
>>>>
>>>> Ok, I realized that the proper build process for tree-sitter grammars is
>>>> a little harder than I expected, tree-sitter-build system make sense.  I
>>>> reviewed it, made a small change:
>>>
>>> Ah great, I was going to comment to try and push for us to keep the
>>> build system. I originally went with a template package and inheritance,
>>> but Maxime suggested moving to a build-system which ended up making the
>>> package definitions a *lot* nicer IMO (see previous discussion here
>>> https://issues.guix.gnu.org/49946#144). It also allows us to deal with
>>> grammars that depend on each other more nicely I think.
>>>
>>>>
>>>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>>>>  ;; Commentary:
>>>>  ;;
>>>>  ;; Build procedures for tree-sitter grammar packages.  This is the
>>>> -;; builder-side code, which builds on top fo the node build-system.
>>>> +;; builder-side code, which builds on top of the node build-system.
>>>>  ;;
>>>>  ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>>>>  ;; shared object.  The `tree-sitter generate' command invokes `node' in order
>>>> @@ -114,7 +114,7 @@ (define (compile-language dir)
>>>>                     "-fno-exceptions"
>>>>                     "-O2"
>>>>                     "-g"
>>>> -                   "-o" ,(string-append lib "/" lang ".so")
>>>> +                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>>>>                     ;; An additional `scanner.{c,cc}' file is sometimes
>>>>                     ;; provided.
>>>>                     ,@(cond
>>>>
>>>>
>>>> rewrote html grammar to use this build system and made it work with
>>>> built-in treesit package.  Also, tried examples of c and cpp grammars
>>>> from patches in this thread.
>>>>
>>>> If you ok with it, I'll push the build system to master and update the
>>>> html grammar accordingly.
>>>>
>>>> The final result will look like this:
>>>>
>>>> (define tree-sitter-delete-generated-files
>>>>   #~(begin
>>>>       (delete-file "binding.gyp")
>>>>       (delete-file-recursively "bindings")
>>>>       (delete-file "src/grammar.json")
>>>>       (delete-file "src/node-types.json")
>>>>       (delete-file "src/parser.c")
>>>>       (delete-file-recursively "src/tree_sitter")))
>>>>
>>>> (define* (tree-sitter-grammar
>>>>           language language-for-synopsis version commit hash
>>>>           #:key
>>>>           (repository-url
>>>>            (format #f "https://github.com/tree-sitter/tree-sitter-~a" language))
>>>>           (inputs '()))
>>>>   (let ((synopsis (string-append language-for-synopsis
>>>>                                  " grammar for tree-sitter"))
>>>>         (name (string-append "tree-sitter-grammar-" language)))
>>>>     (package
>>>>       (name name)
>>>>       (version version)
>>>>       (home-page repository-url)
>>>>       (source (origin
>>>>                 (method git-fetch)
>>>>                 (uri (git-reference
>>>>                       (url repository-url)
>>>>                       (commit commit)))
>>>>                 (file-name (git-file-name name version))
>>>>                 (sha256 (base32 hash))
>>>>                 (modules '((guix build utils)))
>>>>                 (snippet tree-sitter-delete-generated-files)))
>>>>       (build-system tree-sitter-build-system)
>>>>       (inputs inputs)
>>>>       (synopsis synopsis)
>>>>       (description (string-append synopsis "."))
>>>>       (license license:expat))))
>>>>
>>>> (define-public tree-sitter-grammar-html
>>>>   (tree-sitter-grammar
>>>>    "html" "HTML"
>>>>    "0.19.0" "v0.19.0"
>>>>    "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
>>>>
>>>> After that we can bring the rest of the grammars.
>>>
>>> I would suggest to rmeove the `tree-sitter-grammar' function, and keep
>>> grammars as "regular" package records, even though it's a little bit
>>> more verbose:
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> (define-public tree-sitter-html
>>>   (package
>>>     (name "tree-sitter-html")
>>
>> It seems tree-sitter-html mimics upstream package name and probably make
>> more sense than tree-sitter-grammar-html used by me.
>
> Yeah, at some point I think I had named the packages with "grammar" as
> well, but thought it was a bit of a mouthful.  I'm also thinking one day
> we may build language bindings as part of the build system (Rust and
> NodeJS I think ATM), so those packages could do more than ship the
> grammar in the future (although we don't know if we'll ever really need
> that).
>
>>
>>>     (version "0.19.0")
>>>     (source (origin
>>>               (method git-fetch)
>>>               (uri (git-reference
>>>                     (url "https://github.com/tree-sitter/tree-sitter-html")
>>>                     (commit (string-append "v" version))))
>>>               (file-name (git-file-name name version))
>>>               (sha256
>>>                (base32
>>>                 "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
>>>               (modules '((guix build utils)))
>>>               (snippet tree-sitter-delete-generated-files)))
>>>     (build-system tree-sitter-build-system)
>>>     (home-page "https://github.com/tree-sitter/tree-sitter-html")
>>>     (synopsis "Tree-sitter HTML grammar")
>>>     (description
>>>      "This package provides a HTML grammar for the Tree-sitter library.")
>>>     (license license:expat)))
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> This way, they look like any other package in Guix, which makes it
>>> easier for us to apply automatic changes in the future if needed (for
>>> example like how the input format could be automically updated for all
>>> "simple" package definitions, but had to be manual whenever custom code
>>> refactoring was done). Does this make sense?
>>
>> Make sense, but on the other hand we already have hunspell, aspell
>> dictionaries and probably a few more others, which are very similiar in
>> spirit and we already have to keep in mind their existence on such
>> automatic code updates.
>>
>> It looks that the packages differ only in url for the source code, lang
>> name and sometimes in inputs.  Having template package function can make
>> management of shared parts more centralized, reduce possibility of
>> copy-paste mistakes, when the description wasn't updated and so on and
>> can reduce the amount of the code overall (which also reduces the change
>> of introducing an error).
>>
>> I don't have a strong opinion on this topic, but leaning towards the
>> template function slightly more, however I'm completely ok with the
>> standalone package definitions as well.  WDYT?
>
> I can think of both cost/benefits to the template so I don't have a
> strong opinion either :-).
>
> I do like the template to make sure people don't forget to delete
> generated files, that's quite important as it seems upstream packages
> often check-in the generated C code.  Although, we could probably assert
> that with in the build-system phase?  I'll think about that.
>
> On the other hand, I wonder how the template works for packages that
> provide multiple grammars (see ocaml and typescript for example).  I
> guess we could use the template for trivial packages, and standalone
> definitions for more complex ones?  In general, if we keep the template
> interface really simple, then I'm happy with it.

Hi Pierre!

I spend two days trying grammars with and without helper function and
found hepler quite helpful to reduce boilerplate and errors from
copypaste, so I went the way with helper.  The logic inside is quite
trivial, the only downside I found so far is that in cases when
repository url constructed automatically I can't easily open the repo
url in the browser.

I packaged all the grammars from this thread and a few more on top of
it.  Updated them to usually latest versions, added some comments, when
needed.

If I forgot to reply on something or you have any comments/ideas, let me
know! :)

Kudos to Pierre and everyone, who helped with all the tree-sitter stuff.

-- 
Best regards,
Andrew Tropin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#49946] [PATCH v8 0/7] Add tree-sitter support for Emacs 28

[Pierre Langlois]

Hi Guix!

This is a rebase of the emacs-tree-sitter and emacs-tree-sitter-langs packages
on top of tree-sitter grammars that were merged. This brings tree-sitter to
Emacs 28, via a module.

When Emacs 29 is released with native support, some of this work may no longer
be required. However, the final emacs-tree-sitter-langs package is likely to
live on as a convenient way to add syntax highlighting support to major modes
that do not yet have native support. As it naturally will take time for
packages to transition over.

See this issue to track making emacs-tree-sitter-langs compatible with Emacs
29:  https://github.com/emacs-tree-sitter/tree-sitter-langs/pull/157

Anymore feedback on these last few patches? I'm happy to finally be close to
finishing this work!

Thanks,
Pierre

Pierre Langlois (7):
  gnu: Add tree-sitter-lua.
  gnu: Add tree-sitter-rust@0.20.2.
  gnu: Add rust-tree-sitter.
  gnu: Add rust-tree-sitter-for-emacs.
  gnu: Add emacs-tree-sitter-core.
  gnu: Add emacs-tree-sitter.
  gnu: Add emacs-tree-sitter-langs.

 gnu/local.mk                                  |   1 +
 .../rust-tree-sitter-text-provider-fix.patch  |  98 +++++
 gnu/packages/tree-sitter.scm                  | 378 ++++++++++++++++++
 3 files changed, 477 insertions(+)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch

-- 
2.39.1

[bug#49946] [PATCH v8 1/7] gnu: Add tree-sitter-lua.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-lua): New variable.
---
 gnu/packages/tree-sitter.scm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 8dda7737ca..395ae10d88 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -359,6 +359,13 @@ (define-public tree-sitter-julia
    "1pbnmvhy2gq4vg1b0sjzmjm4s2gsgdjh7h01yj8qrrqbcl29c463"
    "0.19.0"))
 
+(define-public tree-sitter-lua
+  (tree-sitter-grammar
+   "lua" "Lua"
+   "1hhffz45ilni4g9idsbpp1aymdjkjnyd4491xbdmr47bglf9cgc1"
+   "0.0.14"
+   #:repository-url "https://github.com/MunifTanjim/tree-sitter-lua"))
+
 (define-public tree-sitter-ocaml
   (tree-sitter-grammar
    "ocaml" "OCaml (.ml and .mli)"
-- 
2.39.1

[bug#49946] [PATCH v8 2/7] gnu: Add tree-sitter-rust@0.20.2.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-rust-0.20.2): New variable.
---
 gnu/packages/tree-sitter.scm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 395ae10d88..b202ebf4d3 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -419,6 +419,13 @@ (define-public tree-sitter-rust
    "149jhy01mqvavwa8jlxb8bnn7sxpfq2x1w35si6zn60b7kqjlx8f"
    "0.20.3"))
 
+;; This version is needed by emacs-tree-sitter-langs.
+(define-public tree-sitter-rust-0.20.2
+  (tree-sitter-grammar
+   "rust" "Rust"
+   "03sd5hv9hc6ww1d53f4w5lbx3bbnkbm4hd7k38n2dmn9ldnmdf39"
+   "0.20.2"))
+
 (define-public tree-sitter-clojure
   (tree-sitter-grammar
    "clojure" "Clojure"
-- 
2.39.1

[bug#49946] [PATCH v8 3/7] gnu: Add rust-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 37 ++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index b202ebf4d3..7256ad02e4 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -30,6 +30,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
+  #:use-module (guix download)
   #:use-module (guix gexp)
   #:use-module (guix git-download)
   #:use-module (guix packages)
@@ -188,6 +189,42 @@ (define-public tree-sitter-cli
 This package includes the @command{tree-sitter} command-line tool.")
     (license license:expat)))
 
+(define-public rust-tree-sitter
+  (package
+    (name "rust-tree-sitter")
+    (version "0.20.7")
+    (source (origin
+              (method url-fetch)
+              (uri (crate-uri "tree-sitter" version))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "183niy9malzr4dm81swcgl05xkwqamim6ck0gw5xb6k78nprz6jl"))
+              (modules '((guix build utils)))
+              (snippet
+               #~(begin
+                   ;; Remove the runtime library code and dynamically link to
+                   ;; it instead.
+                   (delete-file-recursively "src")
+                   (delete-file "binding_rust/build.rs")
+                   (with-output-to-file "binding_rust/build.rs"
+                     (lambda _
+                       (format #t "fn main() {~@
+                              println!(\"cargo:rustc-link-lib=tree-sitter\");~@
+                              }~%")))))))
+    (build-system cargo-build-system)
+    (inputs (list tree-sitter))
+    (arguments
+     `(#:cargo-inputs
+       (("rust-cc" ,rust-cc-1)
+        ("rust-lazy-static" ,rust-lazy-static-1)
+        ("rust-regex" ,rust-regex-1))))
+    (home-page "https://tree-sitter.github.io/tree-sitter/")
+    (synopsis "Rust bindings to the Tree-sitter parsing library")
+    (description "This package provides Rust bindings to the Tree-sitter
+parsing library.")
+    (license license:expat)))
+
 (define (tree-sitter-delete-generated-files grammar-directories)
   #~(begin
       (use-modules (guix build utils))
-- 
2.39.1

[bug#49946] [PATCH v8 4/7] gnu: Add rust-tree-sitter-for-emacs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (rust-tree-sitter-for-emacs): New variable.
* gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
---
 gnu/local.mk                                  |  1 +
 .../rust-tree-sitter-text-provider-fix.patch  | 98 +++++++++++++++++++
 gnu/packages/tree-sitter.scm                  | 14 +++
 3 files changed, 113 insertions(+)
 create mode 100644 gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 5fcbdd4586..1dd98592b5 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1831,6 +1831,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/rust-nettle-disable-vendor.patch		 \
   %D%/packages/patches/rust-nettle-sys-disable-vendor.patch	 \
   %D%/packages/patches/rust-openssl-sys-no-vendor.patch	\
+  %D%/packages/patches/rust-tree-sitter-text-provider-fix.patch	\
   %D%/packages/patches/rust-wl-clipboard-rs-newer-wl.patch      \
   %D%/packages/patches/rw-igraph-0.10.patch			\
   %D%/packages/patches/sbc-fix-build-non-x86.patch		\
diff --git a/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
new file mode 100644
index 0000000000..215e7fc18b
--- /dev/null
+++ b/gnu/packages/patches/rust-tree-sitter-text-provider-fix.patch
@@ -0,0 +1,98 @@
+From e3576b180488b8231e1fc0ca130748577579d129 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Tu=E1=BA=A5n-Anh=20Nguy=E1=BB=85n?= <ubolonton@gmail.com>
+Date: Sun, 25 Jul 2021 13:11:52 +0700
+Subject: [PATCH] Allow TextProvider's iterators to generate owned text
+
+---
+ binding_rust/lib.rs | 33 ++++++++++++++++++---------------
+ 1 file changed, 18 insertions(+), 15 deletions(-)
+
+diff --git a/lib/binding_rust/lib.rs b/lib/binding_rust/lib.rs
+index e88a411c..cf214d92 100644
+--- a/binding_rust/lib.rs
++++ b/binding_rust/lib.rs
+@@ -5,6 +5,7 @@ mod util;
+ use std::os::unix::io::AsRawFd;
+ 
+ use std::{
++    borrow::Cow,
+     char, error,
+     ffi::CStr,
+     fmt, hash, iter,
+@@ -183,7 +184,8 @@ pub struct QueryCaptures<'a, 'tree: 'a, T: TextProvider<'a>> {
+ }
+ 
+ pub trait TextProvider<'a> {
+-    type I: Iterator<Item = &'a [u8]> + 'a;
++    type I: Iterator<Item = Cow<'a, [u8]>>;
++
+     fn text(&mut self, node: Node) -> Self::I;
+ }
+ 
+@@ -1840,19 +1842,19 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+         buffer2: &mut Vec<u8>,
+         text_provider: &mut impl TextProvider<'a>,
+     ) -> bool {
+-        fn get_text<'a, 'b: 'a, I: Iterator<Item = &'b [u8]>>(
++        fn get_text<'a, 'b: 'a, I: Iterator<Item = Cow<'b, [u8]>>>(
+             buffer: &'a mut Vec<u8>,
+             mut chunks: I,
+-        ) -> &'a [u8] {
+-            let first_chunk = chunks.next().unwrap_or(&[]);
++        ) -> Cow<'a, [u8]> {
++            let first_chunk = chunks.next().unwrap_or(Cow::Owned(vec![0u8; 0]));
+             if let Some(next_chunk) = chunks.next() {
+                 buffer.clear();
+-                buffer.extend_from_slice(first_chunk);
+-                buffer.extend_from_slice(next_chunk);
++                buffer.extend_from_slice(&first_chunk);
++                buffer.extend_from_slice(&next_chunk);
+                 for chunk in chunks {
+-                    buffer.extend_from_slice(chunk);
++                    buffer.extend_from_slice(&chunk);
+                 }
+-                buffer.as_slice()
++                Cow::Borrowed(buffer.as_slice())
+             } else {
+                 first_chunk
+             }
+@@ -1888,7 +1890,7 @@ impl<'a, 'tree> QueryMatch<'a, 'tree> {
+                     match node {
+                         Some(node) => {
+                             let text = get_text(buffer1, text_provider.text(node));
+-                            r.is_match(text) == *is_positive
++                            r.is_match(&text) == *is_positive
+                         }
+                         None => true,
+                     }
+@@ -2002,23 +2004,24 @@ impl<'cursor, 'tree> fmt::Debug for QueryMatch<'cursor, 'tree> {
+     }
+ }
+ 
+-impl<'a, F, I> TextProvider<'a> for F
++impl<'a, F, I, T> TextProvider<'a> for F
+ where
+     F: FnMut(Node) -> I,
+-    I: Iterator<Item = &'a [u8]> + 'a,
++    T: Into<Cow<'a, [u8]>>,
++    I: Iterator<Item = T>,
+ {
+-    type I = I;
++    type I = iter::Map<I, fn(T) -> Cow<'a, [u8]>>;
+ 
+     fn text(&mut self, node: Node) -> Self::I {
+-        (self)(node)
++        (self)(node).map(T::into)
+     }
+ }
+ 
+ impl<'a> TextProvider<'a> for &'a [u8] {
+-    type I = iter::Once<&'a [u8]>;
++    type I = iter::Once<Cow<'a, [u8]>>;
+ 
+     fn text(&mut self, node: Node) -> Self::I {
+-        iter::once(&self[node.byte_range()])
++        iter::once(Cow::Borrowed(&self[node.byte_range()]))
+     }
+ }
+ 
diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 7256ad02e4..bb9972cb00 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -22,6 +22,7 @@
 
 (define-module (gnu packages tree-sitter)
   #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (gnu packages)
   #:use-module (gnu packages crates-graphics)
   #:use-module (gnu packages crates-io)
   #:use-module (gnu packages graphviz)
@@ -225,6 +226,19 @@ (define-public rust-tree-sitter
 parsing library.")
     (license license:expat)))
 
+;; We need to apply a patch in order to compile the rust bindings against the
+;; emacs tree-sitter module.
+;; See https://github.com/tree-sitter/tree-sitter/pull/1294
+(define-public rust-tree-sitter-for-emacs
+  (package (inherit rust-tree-sitter)
+    (source (origin
+              (inherit (package-source rust-tree-sitter))
+              (patches (search-patches
+                        "rust-tree-sitter-text-provider-fix.patch"))))
+    ;; Do not show this package in the UI as it's only meant to be used for
+    ;; emacs's tree-sitter module.
+    (properties '((hidden? . #t)))))
+
 (define (tree-sitter-delete-generated-files grammar-directories)
   #~(begin
       (use-modules (guix build utils))
-- 
2.39.1

[bug#49946] [PATCH v8 5/7] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
(emacs-tree-sitter-core): New variable.
---
 gnu/packages/tree-sitter.scm | 85 ++++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index bb9972cb00..12f2d108d9 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -29,6 +29,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (gnu packages icu4c)
   #:use-module (gnu packages node)
   #:use-module (guix build-system cargo)
+  #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
   #:use-module (guix download)
@@ -538,3 +539,87 @@ (define-public tree-sitter-racket
      (git-version "0.1.0" revision commit)
      #:repository-url "https://github.com/6cdh/tree-sitter-racket"
      #:commit commit)))
+
+;; Local package definition solely for building the native emacs module
+;; written in Rust.
+(define tree-sitter-emacs-module
+  (package
+    (name "tree-sitter-emacs-module")
+    (version "0.18.0")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1sdvz827v436qijs6xafakkfw2d16bvp8frymd818rppjc7a9dif"))))
+    (build-system cargo-build-system)
+    (inputs
+     (list tree-sitter))
+    (arguments
+     (list
+      #:cargo-inputs
+      `(("rust-anyhow" ,rust-anyhow-1)
+        ("rust-emacs" ,rust-emacs-0.18)
+        ("rust-libloading" ,rust-libloading-0.7)
+        ("rust-once-cell" ,rust-once-cell-1)
+        ("rust-tree-sitter" ,rust-tree-sitter-for-emacs))
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'chdir
+            (lambda _ (chdir "core")))
+          (add-after 'chdir 'delete-cargo.lock
+            (lambda _ (delete-file "Cargo.lock")))
+          (add-after 'delete-cargo.lock 'do-not-fetch-from-github
+            (lambda _
+              (substitute* "Cargo.toml"
+                (("\\[patch.*") "")
+                (("git = .*") ""))))
+          (replace 'install
+            (lambda _
+              (let ((lib (string-append #$output "/lib")))
+                (mkdir-p lib)
+                (copy-file "target/release/libtsc_dyn.so"
+                           (string-append lib "/tsc-dyn.so"))))))))
+    (home-page #f)
+    (synopsis #f)
+    (description #f)
+    (license license:expat)))
+
+(define-public emacs-tree-sitter-core
+  (package
+    (name "emacs-tree-sitter-core")
+    (version (package-version tree-sitter-emacs-module))
+    (source (package-source tree-sitter-emacs-module))
+    (build-system emacs-build-system)
+    (native-inputs
+     (list tree-sitter-emacs-module))
+    (arguments
+     (list
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'chdir
+            (lambda _ (chdir "core")))
+          (add-before 'install 'find-tsc-dyn
+            (lambda* (#:key inputs #:allow-other-keys)
+              (let ((module (search-input-file inputs "/lib/tsc-dyn.so")))
+                ;; Install the tsc-dyn module in site-lisp and the current
+                ;; directory for test to pass.
+                (install-file module (elpa-directory #$output))
+                (copy-file module "tsc-dyn.so")
+                ;; We replace the tsc-dyn-get.el file with an empty stub to
+                ;; prevent the code from downloading the module.
+                (call-with-output-file "tsc-dyn-get.el"
+                  (lambda (port)
+                    (for-each
+                     (lambda (sexp) (write sexp port))
+                     '((defun tsc-dyn-get-ensure (requested)
+                         nil)
+                       (provide 'tsc-dyn-get)))))))))))
+    (home-page "https://emacs-tree-sitter.github.io")
+    (synopsis "Tree-sitter bindings for Emacs Lisp, core library")
+    (description "This package provides core APIs of the Emacs binding for
+Tree-sitter, an incremental parsing system.")
+    (license license:expat)))
-- 
2.39.1

[bug#49946] [PATCH v8 6/7] gnu: Add emacs-tree-sitter.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter): New variable.
---
 gnu/packages/tree-sitter.scm | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index 12f2d108d9..f8be967e21 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -623,3 +623,31 @@ (define-public emacs-tree-sitter-core
     (description "This package provides core APIs of the Emacs binding for
 Tree-sitter, an incremental parsing system.")
     (license license:expat)))
+
+(define-public emacs-tree-sitter
+  (package
+    (name "emacs-tree-sitter")
+    (version (package-version emacs-tree-sitter-core))
+    (source (package-source emacs-tree-sitter-core))
+    (build-system emacs-build-system)
+    (propagated-inputs
+     (list emacs-tree-sitter-core))
+    (arguments
+     (list
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'chdir
+            (lambda _ (chdir "lisp"))))))
+    (home-page "https://emacs-tree-sitter.github.io")
+    (synopsis "Tree-sitter bindings for Emacs Lisp")
+    (description "This package provides Emacs bindings for Tree-sitter, an
+incremental parsing library.  It aims to be the foundation for a new breed of
+Emacs packages that understand code structurally.  For example:
+
+@enumerate
+@item Faster, fine-grained code highlighting.
+@item More flexible code folding.
+@item Structural editing (like Paredit, or even better) for non-Lisp code.
+@item More informative indexing for imenu.
+@end enumerate")
+    (license license:expat)))
-- 
2.39.1

[bug#49946] [PATCH v8 7/7] gnu: Add emacs-tree-sitter-langs.

[Pierre Langlois]

* gnu/packages/tree-sitter.scm (emacs-tree-sitter-langs-grammar-bundle): New
variable.
(emacs-tree-sitter-langs): New variable.
---
 gnu/packages/tree-sitter.scm | 200 +++++++++++++++++++++++++++++++++++
 1 file changed, 200 insertions(+)

diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
index f8be967e21..2a05179bb4 100644
--- a/gnu/packages/tree-sitter.scm
+++ b/gnu/packages/tree-sitter.scm
@@ -32,6 +32,7 @@ (define-module (gnu packages tree-sitter)
   #:use-module (guix build-system emacs)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system tree-sitter)
+  #:use-module (guix build-system trivial)
   #:use-module (guix download)
   #:use-module (guix gexp)
   #:use-module (guix git-download)
@@ -651,3 +652,202 @@ (define-public emacs-tree-sitter
 @item More informative indexing for imenu.
 @end enumerate")
     (license license:expat)))
+
+;; tree-sitter-langs expects all supported grammars to be bundled in a single
+;; directory. Doing this statically also allows us to check that grammars are
+;; compatible.
+(define emacs-tree-sitter-langs-grammar-bundle
+  (package
+    (name "emacs-tree-sitter-langs-grammar-bundle")
+    (source #f)
+    (version (package-version tree-sitter))
+    (build-system trivial-build-system)
+    (inputs
+     ;; When adding support for new languages, also add their names into the
+     ;; list in the package emacs-tree-sitter-langs package description.
+     ;; FIXME: Support for some languages is still left to package.
+     (list tree-sitter-bash
+           tree-sitter-c
+           tree-sitter-c-sharp
+           tree-sitter-cpp
+           tree-sitter-clojure
+           tree-sitter-css
+           tree-sitter-elixir
+           tree-sitter-elm
+           tree-sitter-go
+           tree-sitter-haskell
+           tree-sitter-html
+           tree-sitter-java
+           tree-sitter-javascript
+           tree-sitter-json
+           tree-sitter-julia
+           tree-sitter-lua
+           tree-sitter-markdown-gfm
+           tree-sitter-ocaml
+           tree-sitter-php
+           tree-sitter-python
+           tree-sitter-r
+           tree-sitter-ruby
+           tree-sitter-rust-0.20.2
+           tree-sitter-typescript))
+    (arguments
+     (list
+      #:builder
+      (with-imported-modules '((guix build union)
+                               (guix build utils))
+        #~(begin
+            (use-modules (ice-9 match)
+                         (srfi srfi-1)
+                         (guix build union)
+                         (guix build utils))
+            (mkdir-p #$output)
+            (for-each
+             (lambda (lib)
+               ;; tree-sitter-langs expects grammars to be as "<lang>.so"
+               ;; rather than "libtree-sitter-<lang>.so".
+               (symlink lib
+                        (string-append
+                         #$output
+                         "/"
+                         (substring (basename lib)
+                                    (string-length "libtree-sitter-")))))
+             (append-map (match-lambda
+                           ((name directory)
+                            (find-files directory "libtree-sitter-.*\\.so$")))
+                         '#$(package-inputs this-package)))))))
+    (synopsis #f)
+    (description #f)
+    (home-page #f)
+    (license #f)))
+
+(define-public emacs-tree-sitter-langs
+  (package
+    (name "emacs-tree-sitter-langs")
+    (version "0.12.16")
+    (source (origin
+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/emacs-tree-sitter/tree-sitter-langs")
+                    (commit version)))
+              (file-name (git-file-name name version))
+              (sha256
+               (base32
+                "1i8lhv404ijj52jjxrmljnrhby9jlrdgpymn2jfs4kml0x7qhsna"))))
+    (build-system emacs-build-system)
+    (inputs
+     (list emacs-tree-sitter-langs-grammar-bundle))
+    (propagated-inputs
+     (list emacs-tree-sitter))
+    (arguments
+     (list
+      #:tests? #t
+      #:test-command ''("script/test")
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'disable-downloader
+            (lambda _
+              (call-with-output-file "tree-sitter-langs-build.el"
+                (lambda (port)
+                  (let ((on-load-message
+                         (string-append
+                          "tree-sitter-langs: Grammar bundle already installed "
+                          "via Guix.  Installing external grammars via this "
+                          "function isn't supported, if a language you need is "
+                          "missing please report a bug at bug-guix@gnu.org.")))
+                    (format
+                     port
+                     ";;;###autoload
+                      (defun tree-sitter-langs-install-grammars
+                             (&optional skip-if-installed version os
+                                        keep-bundle)
+                        (interactive)
+                        (message \"~a\"))
+                      (defconst tree-sitter-langs--queries-dir
+                        (file-name-as-directory
+                          (concat (file-name-directory
+                                   (locate-library \"tree-sitter-langs.el\"))
+                                  \"queries\")))
+                      (defun tree-sitter-langs--bin-dir () \"~a\")
+                      (provide 'tree-sitter-langs-build)"
+                     on-load-message
+                     #$emacs-tree-sitter-langs-grammar-bundle))))))
+          (add-after 'unpack 'remove-cask
+            (lambda _
+              (substitute* "script/test"
+                (("cask") ""))))
+          (add-before 'check 'patch-tree-sitter-require-test
+            (lambda _
+              (use-modules (ice-9 regex))
+              ;; This test needs a git repositories with submodules for
+              ;; each languages in order to map all repositories.  We patch
+              ;; the mapping function with one that invokes the tests for
+              ;; each packaged language.
+              (let ((supported-languages
+                     (map (lambda (lib)
+                            (match:substring
+                             (string-match "(.*)\\.so$" (basename lib))
+                             1))
+                          (find-files "bin" "\\.so$"))))
+                (substitute* "tree-sitter-langs-tests.el"
+                  (("tree-sitter-langs--map-repos")
+                   (call-with-output-string
+                     (lambda (port)
+                       (write `(lambda (fn)
+                                 (dolist (lang ',supported-languages)
+                                         (funcall fn lang)))
+                              port))))))))
+          ;; Tests for queries will fail given those languages are not
+          ;; packaged yet.
+          (add-before 'check 'remove-unused-highlight-queries
+            (lambda _
+              (delete-file-recursively "queries/hcl")
+              (delete-file-recursively "queries/pgn")
+              (delete-file-recursively "queries/yaml")))
+          (add-after 'install 'install-queries
+            (lambda _
+              (copy-recursively
+               "queries"
+               (string-append (elpa-directory #$output) "/queries")))))))
+    (home-page "https://emacs-tree-sitter.github.io/languages/")
+    (synopsis "Language support bundle for Tree-sitter")
+    (description "This package is a convenient language bundle for
+Tree-sitter.  For each supported language, this package provides:
+
+@enumerate
+@item Pre-compiled grammar binaries.
+@item An optional highlights.scm file that provides highlighting patterns.
+This is mainly intended for major modes that are not aware of tree-sitter.
+@item Optional query patterns for other minor modes that provide high-level
+functionalities on top of tree-sitter, such as code folding, evil text
+objects, ...etc.
+@end enumerate
+
+The following languages are currently supported in Guix:
+
+@itemize
+@item Bash
+@item C
+@item C#
+@item C++
+@item Clojure
+@item CSS
+@item Elixir
+@item Elm
+@item Go
+@item Haskell
+@item HTML
+@item Java
+@item JavaScript
+@item JSON
+@item Julia
+@item Lua
+@item Markdown
+@item Ocaml
+@item PHP
+@item Python
+@item R
+@item Ruby
+@item Rust
+@item Typescript
+@end itemize")
+    (license license:expat)))
-- 
2.39.1

[bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 10679 bytes --]

Hi,

Andrew Tropin <andrew@trop.in> writes:

> [[PGP Signed Part:Undecided]]
> On 2023-02-12 12:07, Pierre Langlois wrote:
>
>> Hi,
>>
>> Andrew Tropin <andrew@trop.in> writes:
>>
>>> [[PGP Signed Part:Undecided]]
>>> On 2023-02-10 15:48, Pierre Langlois wrote:
>>>
>>>> Hi Andrew, thanks for pushing this along! It's great to see things
>>>> getting merged.
>>>>
>>>> Andrew Tropin <andrew@trop.in> writes:
>>>>
>>>>> [[PGP Signed Part:Undecided]]
>>>>> On 2023-02-09 18:04, Andrew Tropin wrote:
>>>>>
>>>>>> On 2023-02-09 13:39, zimoun wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin <andrew@trop.in> wrote:
>>>>>>>
>>>>>>>> I applied tree-sitter and tree-sitter-cli patches,
>>>>>>>
>>>>>>> Just to be sure to understand, you have only applied 02/32 and 05/32,
>>>>>>> right?
>>>>>>>
>>>>>>>
>>>>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7.
>>>>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com
>>>>>>>
>>>>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli.
>>>>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com
>>>>>>>
>>>>>>> Leaving out all the others, right?
>>>>>>
>>>>>> Merged first 5 patches from 01 to 05, also added one more commit, which
>>>>>> addresses some things from reviews and one commit, which adds html
>>>>>> grammar.
>>>>>>
>>>>>> The html grammar is added for the testing purposes.  It relies on
>>>>>> generated parser.c and scanner.c and we will need to repackage it using
>>>>>> grammar.js instead.  I'm not sure if a separate build system is needed
>>>>>> for this, I guess we can just rewrite tree-sitter-grammar function,
>>>>>> which generates packages as in example with tree-sitter-grammar-html:
>>>>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158
>>>>>>
>>>>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to
>>>>>> read it carefully, evaluate and either introduce new build system or
>>>>>> just move all needed parts to tree-sitter-grammar function.  WDYT?
>>>>>> After we done with it we can package all other grammars.
>>>>>
>>>>> Ok, I realized that the proper build process for tree-sitter grammars is
>>>>> a little harder than I expected, tree-sitter-build system make sense.  I
>>>>> reviewed it, made a small change:
>>>>
>>>> Ah great, I was going to comment to try and push for us to keep the
>>>> build system. I originally went with a template package and inheritance,
>>>> but Maxime suggested moving to a build-system which ended up making the
>>>> package definitions a *lot* nicer IMO (see previous discussion here
>>>> https://issues.guix.gnu.org/49946#144). It also allows us to deal with
>>>> grammars that depend on each other more nicely I think.
>>>>
>>>>>
>>>>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system)
>>>>>  ;; Commentary:
>>>>>  ;;
>>>>>  ;; Build procedures for tree-sitter grammar packages.  This is the
>>>>> -;; builder-side code, which builds on top fo the node build-system.
>>>>> +;; builder-side code, which builds on top of the node build-system.
>>>>>  ;;
>>>>>  ;; Tree-sitter grammars are written in JavaScript and compiled to a native
>>>>>  ;; shared object.  The `tree-sitter generate' command invokes `node' in order
>>>>> @@ -114,7 +114,7 @@ (define (compile-language dir)
>>>>>                     "-fno-exceptions"
>>>>>                     "-O2"
>>>>>                     "-g"
>>>>> -                   "-o" ,(string-append lib "/" lang ".so")
>>>>> +                   "-o" ,(string-append lib "/libtree-sitter-" lang ".so")
>>>>>                     ;; An additional `scanner.{c,cc}' file is sometimes
>>>>>                     ;; provided.
>>>>>                     ,@(cond
>>>>>
>>>>>
>>>>> rewrote html grammar to use this build system and made it work with
>>>>> built-in treesit package.  Also, tried examples of c and cpp grammars
>>>>> from patches in this thread.
>>>>>
>>>>> If you ok with it, I'll push the build system to master and update the
>>>>> html grammar accordingly.
>>>>>
>>>>> The final result will look like this:
>>>>>
>>>>> (define tree-sitter-delete-generated-files
>>>>>   #~(begin
>>>>>       (delete-file "binding.gyp")
>>>>>       (delete-file-recursively "bindings")
>>>>>       (delete-file "src/grammar.json")
>>>>>       (delete-file "src/node-types.json")
>>>>>       (delete-file "src/parser.c")
>>>>>       (delete-file-recursively "src/tree_sitter")))
>>>>>
>>>>> (define* (tree-sitter-grammar
>>>>>           language language-for-synopsis version commit hash
>>>>>           #:key
>>>>>           (repository-url
>>>>>            (format #f "https://github.com/tree-sitter/tree-sitter-~a" language))
>>>>>           (inputs '()))
>>>>>   (let ((synopsis (string-append language-for-synopsis
>>>>>                                  " grammar for tree-sitter"))
>>>>>         (name (string-append "tree-sitter-grammar-" language)))
>>>>>     (package
>>>>>       (name name)
>>>>>       (version version)
>>>>>       (home-page repository-url)
>>>>>       (source (origin
>>>>>                 (method git-fetch)
>>>>>                 (uri (git-reference
>>>>>                       (url repository-url)
>>>>>                       (commit commit)))
>>>>>                 (file-name (git-file-name name version))
>>>>>                 (sha256 (base32 hash))
>>>>>                 (modules '((guix build utils)))
>>>>>                 (snippet tree-sitter-delete-generated-files)))
>>>>>       (build-system tree-sitter-build-system)
>>>>>       (inputs inputs)
>>>>>       (synopsis synopsis)
>>>>>       (description (string-append synopsis "."))
>>>>>       (license license:expat))))
>>>>>
>>>>> (define-public tree-sitter-grammar-html
>>>>>   (tree-sitter-grammar
>>>>>    "html" "HTML"
>>>>>    "0.19.0" "v0.19.0"
>>>>>    "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
>>>>>
>>>>> After that we can bring the rest of the grammars.
>>>>
>>>> I would suggest to rmeove the `tree-sitter-grammar' function, and keep
>>>> grammars as "regular" package records, even though it's a little bit
>>>> more verbose:
>>>>
>>>> --8<---------------cut here---------------start------------->8---
>>>> (define-public tree-sitter-html
>>>>   (package
>>>>     (name "tree-sitter-html")
>>>
>>> It seems tree-sitter-html mimics upstream package name and probably make
>>> more sense than tree-sitter-grammar-html used by me.
>>
>> Yeah, at some point I think I had named the packages with "grammar" as
>> well, but thought it was a bit of a mouthful.  I'm also thinking one day
>> we may build language bindings as part of the build system (Rust and
>> NodeJS I think ATM), so those packages could do more than ship the
>> grammar in the future (although we don't know if we'll ever really need
>> that).
>>
>>>
>>>>     (version "0.19.0")
>>>>     (source (origin
>>>>               (method git-fetch)
>>>>               (uri (git-reference
>>>>                     (url "https://github.com/tree-sitter/tree-sitter-html")
>>>>                     (commit (string-append "v" version))))
>>>>               (file-name (git-file-name name version))
>>>>               (sha256
>>>>                (base32
>>>>                 "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy"))
>>>>               (modules '((guix build utils)))
>>>>               (snippet tree-sitter-delete-generated-files)))
>>>>     (build-system tree-sitter-build-system)
>>>>     (home-page "https://github.com/tree-sitter/tree-sitter-html")
>>>>     (synopsis "Tree-sitter HTML grammar")
>>>>     (description
>>>>      "This package provides a HTML grammar for the Tree-sitter library.")
>>>>     (license license:expat)))
>>>> --8<---------------cut here---------------end--------------->8---
>>>>
>>>> This way, they look like any other package in Guix, which makes it
>>>> easier for us to apply automatic changes in the future if needed (for
>>>> example like how the input format could be automically updated for all
>>>> "simple" package definitions, but had to be manual whenever custom code
>>>> refactoring was done). Does this make sense?
>>>
>>> Make sense, but on the other hand we already have hunspell, aspell
>>> dictionaries and probably a few more others, which are very similiar in
>>> spirit and we already have to keep in mind their existence on such
>>> automatic code updates.
>>>
>>> It looks that the packages differ only in url for the source code, lang
>>> name and sometimes in inputs.  Having template package function can make
>>> management of shared parts more centralized, reduce possibility of
>>> copy-paste mistakes, when the description wasn't updated and so on and
>>> can reduce the amount of the code overall (which also reduces the change
>>> of introducing an error).
>>>
>>> I don't have a strong opinion on this topic, but leaning towards the
>>> template function slightly more, however I'm completely ok with the
>>> standalone package definitions as well.  WDYT?
>>
>> I can think of both cost/benefits to the template so I don't have a
>> strong opinion either :-).
>>
>> I do like the template to make sure people don't forget to delete
>> generated files, that's quite important as it seems upstream packages
>> often check-in the generated C code.  Although, we could probably assert
>> that with in the build-system phase?  I'll think about that.
>>
>> On the other hand, I wonder how the template works for packages that
>> provide multiple grammars (see ocaml and typescript for example).  I
>> guess we could use the template for trivial packages, and standalone
>> definitions for more complex ones?  In general, if we keep the template
>> interface really simple, then I'm happy with it.
>
> Hi Pierre!
>
> I spend two days trying grammars with and without helper function and
> found hepler quite helpful to reduce boilerplate and errors from
> copypaste, so I went the way with helper.  The logic inside is quite
> trivial, the only downside I found so far is that in cases when
> repository url constructed automatically I can't easily open the repo
> url in the browser.
>
> I packaged all the grammars from this thread and a few more on top of
> it.  Updated them to usually latest versions, added some comments, when
> needed.
>
> If I forgot to reply on something or you have any comments/ideas, let me
> know! :)
>
> Kudos to Pierre and everyone, who helped with all the tree-sitter stuff.

Thank you for landing all this work!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

[bug#49946] [PATCH v8 5/7] gnu: Add emacs-tree-sitter-core.

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 4734 bytes --]


Pierre Langlois <pierre.langlois@gmx.com> writes:

> * gnu/packages/tree-sitter.scm (tree-sitter-emacs-module): New local variable.
> (emacs-tree-sitter-core): New variable.
> ---
>  gnu/packages/tree-sitter.scm | 85 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 85 insertions(+)
>
> diff --git a/gnu/packages/tree-sitter.scm b/gnu/packages/tree-sitter.scm
> index bb9972cb00..12f2d108d9 100644
> --- a/gnu/packages/tree-sitter.scm
> +++ b/gnu/packages/tree-sitter.scm
> @@ -29,6 +29,7 @@ (define-module (gnu packages tree-sitter)
>    #:use-module (gnu packages icu4c)
>    #:use-module (gnu packages node)
>    #:use-module (guix build-system cargo)
> +  #:use-module (guix build-system emacs)
>    #:use-module (guix build-system gnu)
>    #:use-module (guix build-system tree-sitter)
>    #:use-module (guix download)
> @@ -538,3 +539,87 @@ (define-public tree-sitter-racket
>       (git-version "0.1.0" revision commit)
>       #:repository-url "https://github.com/6cdh/tree-sitter-racket"
>       #:commit commit)))
> +
> +;; Local package definition solely for building the native emacs module
> +;; written in Rust.
> +(define tree-sitter-emacs-module
> +  (package
> +    (name "tree-sitter-emacs-module")
> +    (version "0.18.0")
> +    (source (origin
> +              (method git-fetch)
> +              (uri (git-reference
> +                    (url "https://github.com/emacs-tree-sitter/elisp-tree-sitter")
> +                    (commit version)))
> +              (file-name (git-file-name name version))
> +              (sha256
> +               (base32
> +                "1sdvz827v436qijs6xafakkfw2d16bvp8frymd818rppjc7a9dif"))))
> +    (build-system cargo-build-system)
> +    (inputs
> +     (list tree-sitter))
> +    (arguments
> +     (list
> +      #:cargo-inputs
> +      `(("rust-anyhow" ,rust-anyhow-1)
> +        ("rust-emacs" ,rust-emacs-0.18)
> +        ("rust-libloading" ,rust-libloading-0.7)
> +        ("rust-once-cell" ,rust-once-cell-1)
> +        ("rust-tree-sitter" ,rust-tree-sitter-for-emacs))
> +      #:phases
> +      #~(modify-phases %standard-phases
> +          (add-after 'unpack 'chdir
> +            (lambda _ (chdir "core")))
> +          (add-after 'chdir 'delete-cargo.lock
> +            (lambda _ (delete-file "Cargo.lock")))
> +          (add-after 'delete-cargo.lock 'do-not-fetch-from-github
> +            (lambda _
> +              (substitute* "Cargo.toml"
> +                (("\\[patch.*") "")
> +                (("git = .*") ""))))
> +          (replace 'install
> +            (lambda _
> +              (let ((lib (string-append #$output "/lib")))
> +                (mkdir-p lib)
> +                (copy-file "target/release/libtsc_dyn.so"
> +                           (string-append lib "/tsc-dyn.so"))))))))
> +    (home-page #f)
> +    (synopsis #f)
> +    (description #f)
> +    (license license:expat)))
> +
> +(define-public emacs-tree-sitter-core
> +  (package
> +    (name "emacs-tree-sitter-core")
> +    (version (package-version tree-sitter-emacs-module))
> +    (source (package-source tree-sitter-emacs-module))
> +    (build-system emacs-build-system)
> +    (native-inputs
> +     (list tree-sitter-emacs-module))

Wait, I think this should have been a regular input, not a native-input,
otherwise it'll break if we ever add cross-compilation support.

> +    (arguments
> +     (list
> +      #:phases
> +      #~(modify-phases %standard-phases
> +          (add-after 'unpack 'chdir
> +            (lambda _ (chdir "core")))
> +          (add-before 'install 'find-tsc-dyn
> +            (lambda* (#:key inputs #:allow-other-keys)
> +              (let ((module (search-input-file inputs "/lib/tsc-dyn.so")))
> +                ;; Install the tsc-dyn module in site-lisp and the current
> +                ;; directory for test to pass.
> +                (install-file module (elpa-directory #$output))
> +                (copy-file module "tsc-dyn.so")
> +                ;; We replace the tsc-dyn-get.el file with an empty stub to
> +                ;; prevent the code from downloading the module.
> +                (call-with-output-file "tsc-dyn-get.el"
> +                  (lambda (port)
> +                    (for-each
> +                     (lambda (sexp) (write sexp port))
> +                     '((defun tsc-dyn-get-ensure (requested)
> +                         nil)
> +                       (provide 'tsc-dyn-get)))))))))))
> +    (home-page "https://emacs-tree-sitter.github.io")
> +    (synopsis "Tree-sitter bindings for Emacs Lisp, core library")
> +    (description "This package provides core APIs of the Emacs binding for
> +Tree-sitter, an incremental parsing system.")
> +    (license license:expat)))


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]