From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 2KofAx5q5mOydAAAbAwnHQ (envelope-from ) for ; Fri, 10 Feb 2023 17:00:30 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id SEnLAh5q5mMJKgAAauVa8A (envelope-from ) for ; Fri, 10 Feb 2023 17:00:30 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 958062CB6B for ; Fri, 10 Feb 2023 17:00:29 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pQVoi-0005I3-Ax; Fri, 10 Feb 2023 11:00:04 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pQVog-0005Ho-Cv for guix-patches@gnu.org; Fri, 10 Feb 2023 11:00:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pQVog-00013r-2Y for guix-patches@gnu.org; Fri, 10 Feb 2023 11:00:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pQVof-0008OT-Qh for guix-patches@gnu.org; Fri, 10 Feb 2023 11:00:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module. Resent-From: Pierre Langlois Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 10 Feb 2023 16:00:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49946 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Andrew Tropin Cc: "\(" , Pierre Langlois , 49946@debbugs.gnu.org, Luis Henrique Gomes Higino , zimoun Received: via spool by 49946-submit@debbugs.gnu.org id=B49946.167604477732206 (code B ref 49946); Fri, 10 Feb 2023 16:00:01 +0000 Received: (at 49946) by debbugs.gnu.org; 10 Feb 2023 15:59:37 +0000 Received: from localhost ([127.0.0.1]:37928 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pQVoG-0008NO-Sk for submit@debbugs.gnu.org; Fri, 10 Feb 2023 10:59:37 -0500 Received: from mout.gmx.net ([212.227.17.22]:39505) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pQVoE-0008N9-Cx for 49946@debbugs.gnu.org; Fri, 10 Feb 2023 10:59:35 -0500 Received: from labiere ([82.69.64.142]) by mail.gmx.net (mrgmx104 [212.227.17.174]) with ESMTPSA (Nemesis) id 1MRCKC-1pCAJO1sEW-00N9WA; Fri, 10 Feb 2023 16:59:26 +0100 References: <87mtfi63ut.fsf@gmx.com> <20221125012142.22579-1-pierre.langlois@gmx.com> <20221125012142.22579-2-pierre.langlois@gmx.com> <87bkovcp1d.fsf@gmx.com> <87h6vvgnd6.fsf@trop.in> <86pmaj3td2.fsf@gmail.com> <87cz6jgcku.fsf@trop.in> <87a61lr8cp.fsf@trop.in> User-agent: mu4e 1.8.13; emacs 28.2 From: Pierre Langlois Date: Fri, 10 Feb 2023 15:48:46 +0000 In-reply-to: <87a61lr8cp.fsf@trop.in> Message-ID: <87o7q1qzol.fsf@gmx.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Provags-ID: V03:K1:RDLMLiR7kEGD7IWso76iJf4ZM2MWc4d+d1x8uznjbdW5Bi/edUz HiMyhAvgjRw3kKq4P89m9olDpjEGY1UtpzO7qb3+vartjihDiUd+mtOQgQTlyjF6Jy13qyF yhpEUlPGpHQin67y9emKHc3BgRBXZ18z6dY7OPMChsTHQLkG5QLrKAayRu+v0ZUdfRZwfAW dgyD3rYewA689ELY3XQ8w== UI-OutboundReport: notjunk:1;M01:P0:X4J5wsg8c5Y=;r6yXaw0z3lVcMBcj8F6IEjfbI3C QNQpwwp4JqjN6/mjY7o/d4F5RpEzAycxwRXTEyMf4G5F8LONmBR0YGufY3hVcGK+8rwuIawws kK7DrQK05576XHNgQA5ErZzGpmC+/wM5wMd2aROuZWL/APJnOdFPlpC6jQuT4AZwh9QO5mVCE kfQHTDvTPFTNhyWOHobATvWOryY6uEKZLVQIodJsYoDiUS+Yk1f2Zk24ty7TTLrEf+U2CdgpS x4HJ1Nhx9j5/K822iwKAzTZcJGFXVN/dAUAtv/fBPKynweWcoQrLUjHOefV5wHGYWc8fgZSib vpPW2F8jiObnzOp97hcf85jPqnd8f7HOIUQbhKtt8vHOmOwQAfnGI1osXJBZRNrS3N9dbtd7A Qc9DilEefM/RrMYnZ+z4lTpQ/QQUwJA+LBlNhSUx3dnky1h8JZ7vANfAayu7aq5or0X61sEnc uOz1yRkuG10ObVoGxbPp0Gvh4mtH8WGHx0U8CUbtgKOdltRq1JxFswC9u4tlnYd51dFmtiTif VZzC5x03h++0xqI3/sDNydpPTV1+a8CrOQ/aaK60orxbWYODV8zjo8N7F9Iud9RP4WPSkYy0z 2/kmJ+VrAjFBwQlT9iKy2kyY3RwZqDlOJjgqrcMCQtx27zYqUwaEt29K26YMZzkAr/OkXkmaS HEPOdDdvabmWTcd+2aRhL6Jrtm/hTwPPWjrb2gFUVAqUKXt0ioqFeZCO7rqkENwMx31SwHGg+ JZ4HxMaOoSRPA7LSLudi0Tuv2skQRa49ksm/gsy429OWQHn/PZ3qn0a+B6NxySxyApMhfsUQJ tUrtdzWRa7jUG8iHkDkN1Tbjniifr6LBc7cU5DefxC3+/8eIEkaM4C6bfBpqeUW+FajLnXsUD zHC5uLDRiDcy9xb9EqN3yWdd/pRSnYB5AAPUFMcqLJQsW+qDvo1o1FuL3J+pL73ezO9atDhhN 9e/BjgktJ5BX+fmjLhtj4dDEZcc= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1676044829; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=vs6254DzEjz+4AFp9nJURPEbJ/xgpzO8Z1ADp17+3rU=; b=q9VYJ0WmZjtkAiGZVSeuc6TpgI8fq6AeTOf1u6GnAYRwSpgnDBIM+cNB4DsgX9rE7xc4lP lIqYxwVO5vn9DtkcbyDfitNc7GdQoT85RseC862u8195QRL+bmh2PWCLpwAnjSw8nRCCks 9ftorL0efjAm+/6p/wFJWdhGkSmz55I2zhX47D4OtkPQBmhYsGrntdOC+2RaNhF461qaLQ alun9/Koa+Q/VDmX5JfkuvBMHD34ODV9UyenGbUKKJGhi1jLJFRYHV2eJE4knCUoNiZDuH TL6RThhSWEby+iXhaUhIwLRfq4h6Cr3Som6vCNIdZcxKPkbj7ieZhvC3P0EmFg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmx.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1676044829; a=rsa-sha256; cv=none; b=tP4LJdvbFw1dtqAygS6zy3NVcN4vxdaVeVW3rv17a2waVLpMDtnSpD/zW2xyTKPH0CFL9j 8T/aJ/1Oc4c4mZJ27P/28V4t7CTiYhG0UUeU/8rBFyGkeBlVYVIYXaRRlD/7nH6a2JV4Jc dpHJvRtPIDm1Rqa73O1JZnGXvcrTqPVSsNB6BzVNi+EE+9FBcZJ5vNJMUlheptuXa2uWj6 9q2Kvz0RbW2YdH3hLYLmh4lh6BWs8TXO2ZO4NkeASYOwsWXg/DYV8Jd2wYGhGssDqqXNWV eflvVch3OI6uFKKZ6QBygxvWQCZoWhy8C4tvslHPLXGgysD0Ao50V+RyntjImg== Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmx.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 1.19 X-Spam-Score: 1.19 X-Migadu-Queue-Id: 958062CB6B X-Migadu-Scanner: scn1.migadu.com X-TUID: rSXYy+OvmUNl --=-=-= Content-Type: text/plain Hi Andrew, thanks for pushing this along! It's great to see things getting merged. Andrew Tropin writes: > [[PGP Signed Part:Undecided]] > On 2023-02-09 18:04, Andrew Tropin wrote: > >> On 2023-02-09 13:39, zimoun wrote: >> >>> Hi, >>> >>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin wrote: >>> >>>> I applied tree-sitter and tree-sitter-cli patches, >>> >>> Just to be sure to understand, you have only applied 02/32 and 05/32, >>> right? >>> >>> >>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7. >>> id:20221125012142.22579-3-pierre.langlois@gmx.com >>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com >>> >>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli. >>> id:20221125012142.22579-6-pierre.langlois@gmx.com >>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com >>> >>> Leaving out all the others, right? >> >> Merged first 5 patches from 01 to 05, also added one more commit, which >> addresses some things from reviews and one commit, which adds html >> grammar. >> >> The html grammar is added for the testing purposes. It relies on >> generated parser.c and scanner.c and we will need to repackage it using >> grammar.js instead. I'm not sure if a separate build system is needed >> for this, I guess we can just rewrite tree-sitter-grammar function, >> which generates packages as in example with tree-sitter-grammar-html: >> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158 >> >> Anyway, I only skimmed tree-sitter-build-system source code, and plan to >> read it carefully, evaluate and either introduce new build system or >> just move all needed parts to tree-sitter-grammar function. WDYT? >> After we done with it we can package all other grammars. > > Ok, I realized that the proper build process for tree-sitter grammars is > a little harder than I expected, tree-sitter-build system make sense. I > reviewed it, made a small change: Ah great, I was going to comment to try and push for us to keep the build system. I originally went with a template package and inheritance, but Maxime suggested moving to a build-system which ended up making the package definitions a *lot* nicer IMO (see previous discussion here https://issues.guix.gnu.org/49946#144). It also allows us to deal with grammars that depend on each other more nicely I think. > > @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system) > ;; Commentary: > ;; > ;; Build procedures for tree-sitter grammar packages. This is the > -;; builder-side code, which builds on top fo the node build-system. > +;; builder-side code, which builds on top of the node build-system. > ;; > ;; Tree-sitter grammars are written in JavaScript and compiled to a native > ;; shared object. The `tree-sitter generate' command invokes `node' in order > @@ -114,7 +114,7 @@ (define (compile-language dir) > "-fno-exceptions" > "-O2" > "-g" > - "-o" ,(string-append lib "/" lang ".so") > + "-o" ,(string-append lib "/libtree-sitter-" lang ".so") > ;; An additional `scanner.{c,cc}' file is sometimes > ;; provided. > ,@(cond > > > rewrote html grammar to use this build system and made it work with > built-in treesit package. Also, tried examples of c and cpp grammars > from patches in this thread. > > If you ok with it, I'll push the build system to master and update the > html grammar accordingly. > > The final result will look like this: > > (define tree-sitter-delete-generated-files > #~(begin > (delete-file "binding.gyp") > (delete-file-recursively "bindings") > (delete-file "src/grammar.json") > (delete-file "src/node-types.json") > (delete-file "src/parser.c") > (delete-file-recursively "src/tree_sitter"))) > > (define* (tree-sitter-grammar > language language-for-synopsis version commit hash > #:key > (repository-url > (format #f "https://github.com/tree-sitter/tree-sitter-~a" language)) > (inputs '())) > (let ((synopsis (string-append language-for-synopsis > " grammar for tree-sitter")) > (name (string-append "tree-sitter-grammar-" language))) > (package > (name name) > (version version) > (home-page repository-url) > (source (origin > (method git-fetch) > (uri (git-reference > (url repository-url) > (commit commit))) > (file-name (git-file-name name version)) > (sha256 (base32 hash)) > (modules '((guix build utils))) > (snippet tree-sitter-delete-generated-files))) > (build-system tree-sitter-build-system) > (inputs inputs) > (synopsis synopsis) > (description (string-append synopsis ".")) > (license license:expat)))) > > (define-public tree-sitter-grammar-html > (tree-sitter-grammar > "html" "HTML" > "0.19.0" "v0.19.0" > "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy")) > > After that we can bring the rest of the grammars. I would suggest to rmeove the `tree-sitter-grammar' function, and keep grammars as "regular" package records, even though it's a little bit more verbose: --8<---------------cut here---------------start------------->8--- (define-public tree-sitter-html (package (name "tree-sitter-html") (version "0.19.0") (source (origin (method git-fetch) (uri (git-reference (url "https://github.com/tree-sitter/tree-sitter-html") (commit (string-append "v" version)))) (file-name (git-file-name name version)) (sha256 (base32 "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy")) (modules '((guix build utils))) (snippet tree-sitter-delete-generated-files))) (build-system tree-sitter-build-system) (home-page "https://github.com/tree-sitter/tree-sitter-html") (synopsis "Tree-sitter HTML grammar") (description "This package provides a HTML grammar for the Tree-sitter library.") (license license:expat))) --8<---------------cut here---------------end--------------->8--- This way, they look like any other package in Guix, which makes it easier for us to apply automatic changes in the future if needed (for example like how the input format could be automically updated for all "simple" package definitions, but had to be manual whenever custom code refactoring was done). Does this make sense? > >> >> According to emacs integration: I already have a working prototype on >> my local setup, which uses builtin treesit package, it's similiar to >> what Luis mentioned with search-paths. We just need to figure out a >> few minor details and I'll add it. > > I've sent a patch for emacs+tree-sitter integration in a different reply > in this thread. > > Thank you very much for all the work, it's really valuable! :-) Thanks, Pierre --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQFMBAEBCgA2FiEEctU9gYy29KFyWDdMqPyeRH9PfVQFAmPmadoYHHBpZXJyZS5s YW5nbG9pc0BnbXguY29tAAoJEKj8nkR/T31UcrMIAIoyUPydzsRC71FQXoR04D20 mLjtBKgsGhP+GNkZZFIyJgr8GmxdUE2fAW/CCegUgigVpfOLCdBkMphp8v1W9ij7 l8PQktmn+vPuRnu6gbitrvfUlz0Mfa2E3t0fO3xdEV8TUpHbJLomShL7cRjU1IIy H1gaUrNDv8F5zEY1D5lEmMWzek+za8BewO9u7EPUR8AMv7i1fnyvWFIsqNJt3Mwl Scm4drDAA2cEbergnh8tBrLO3k87W6DoHV2ullhzCKCWWoHNEgff+sAoRmGu59B7 P942Jg2kekzKy+Hc8rl+c/2yuv/lEo6VpF6AKXWJBNqlLFkM+TKrys8jvnR0YMc= =5EOX -----END PGP SIGNATURE----- --=-=-=--