From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 0MqmE7ba6GOpUAEAbAwnHQ (envelope-from ) for ; Sun, 12 Feb 2023 13:25:26 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id IODJErba6GNBagEAG6o9tA (envelope-from ) for ; Sun, 12 Feb 2023 13:25:26 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id EE66D79A7 for ; Sun, 12 Feb 2023 13:25:25 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pRBPn-0005Nf-5E; Sun, 12 Feb 2023 07:25:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pRBPl-0005NG-0j for guix-patches@gnu.org; Sun, 12 Feb 2023 07:25:05 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pRBPi-0006m0-DQ for guix-patches@gnu.org; Sun, 12 Feb 2023 07:25:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pRBPi-0008Vq-7H for guix-patches@gnu.org; Sun, 12 Feb 2023 07:25:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module. Resent-From: Pierre Langlois Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 12 Feb 2023 12:25:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49946 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Andrew Tropin Cc: "\(" , Pierre Langlois , 49946@debbugs.gnu.org, Luis Henrique Gomes Higino , zimoun Received: via spool by 49946-submit@debbugs.gnu.org id=B49946.167620467832682 (code B ref 49946); Sun, 12 Feb 2023 12:25:02 +0000 Received: (at 49946) by debbugs.gnu.org; 12 Feb 2023 12:24:38 +0000 Received: from localhost ([127.0.0.1]:44275 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pRBPJ-0008V4-Me for submit@debbugs.gnu.org; Sun, 12 Feb 2023 07:24:38 -0500 Received: from mout.gmx.net ([212.227.17.20]:37235) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pRBPH-0008Uq-4a for 49946@debbugs.gnu.org; Sun, 12 Feb 2023 07:24:36 -0500 Received: from labiere ([82.69.64.142]) by mail.gmx.net (mrgmx105 [212.227.17.174]) with ESMTPSA (Nemesis) id 1MZTqg-1p6mh50yEY-00Wa1l; Sun, 12 Feb 2023 13:24:25 +0100 References: <87mtfi63ut.fsf@gmx.com> <20221125012142.22579-1-pierre.langlois@gmx.com> <20221125012142.22579-2-pierre.langlois@gmx.com> <87bkovcp1d.fsf@gmx.com> <87h6vvgnd6.fsf@trop.in> <86pmaj3td2.fsf@gmail.com> <87cz6jgcku.fsf@trop.in> <87a61lr8cp.fsf@trop.in> <87o7q1qzol.fsf@gmx.com> <87mt5jfmvl.fsf@trop.in> User-agent: mu4e 1.8.13; emacs 28.2 From: Pierre Langlois Date: Sun, 12 Feb 2023 12:07:27 +0000 In-reply-to: <87mt5jfmvl.fsf@trop.in> Message-ID: <871qmvm5qh.fsf@gmx.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Provags-ID: V03:K1:wcl8aDuvBMut0XAlLHTfdy/hrogDkPVe89XyPNmY5v1IGiNZYSV yCf/lTRkvC9thsR/Q/05+DvVGCvyVX9NH+X+AlGf/ied3U+J+dqTA4n4i7EsM3b0MjizFiA KPu7S5BbKeeYM4vt/BdpKwLsOtOdg1XouD8yGm+/zmXrTrM3MPubqyqYMAoy/5oOJURjlz3 hXLGMnSXXjxGyzQO9oPNQ== UI-OutboundReport: notjunk:1;M01:P0:4zGSbKBpz88=;eWenBOq0qVt8fc8/iTqzqwyeHso WKAizmu5OrzY2cNfFpf5JuhEKWL7nLKPLC6o58s+3E6uMAJi1tRGIjZtpvuKz4crxRAn8nqpd 1hGUAvxBOQymvfHU8v0uOf93GsYMl0ImEnzGw0Ud8AccVxNM1q6Ng2EUgh4knXyYbFKSQGjtA SdvSDNur3rLNJrAdDqk2TDoyODDBrHm/XURrT1AkGN3yNQ4N7305deyDJF5iqG5IREwW225Qe 44ALi9YUr3XYlmXMi1Gv/P+KYROVUimR2jtcr/ejSE5oocQLn5B0qr36YiO5uboySpV6zP3d1 Zws/Zf4h8wl+8fY8+k+/jepDX6dMOrB6336t+Lgl6OhA/PkGsm6+ZNfclODz7zpRm/qKtmQaY FP8MvGMeqI3mkIDtQexJF8aXQ3CAhImwHqLfJ1xR6DbZS8x1HEEsNuD25pxrPIWnuq+xNfEZk Z4C9QH94u1g+7BWt886+vOH1c4uZghVMGbIDfS2VwQA7kqP+Spng1uo/tOva9PwNPQMt7H6e6 sdQLqnTM12Zeh1/4S3HNmPUzKoRugW0eiMN1ETDlw6oIPawAg8iyrbcs1BM+oXfUpS/UWbic/ 1OtOFy3Nu6vanxWHUHBJUk7HMuo6bPQFn92Dy4lokhSVj5ryxGi6t/n0XwcQp8nVg41qPZPGD vWYrFsBUx9JHiJp685tLMjbGOFZ2xpeMIQ5JBBoumYOihg0W4RxYEYEbvPbzmIPxws502RTLl qLZSMAPJkzPz2JPkpodN7LYFJ+qZDE5XfcGdEoyHCXqIjmrkvMv4xQx2DdjkHf/z/OwgnCS53 1dKaRL231rE0yQlRZ7jIGiiQ/nKp2qOSi3U4MJShYJ9UupCnMS7MAz99IZTubhSEy1LUYAjly naBcM7S4Xmappia1fSIpDdkt71GYMJAidAEVbyPFraz2n4mKLHl1V8cojFdmkPd599BEY5AKj 8qmnSUsMe6b7JSYNAaTqcEiIUc4= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmx.com (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1676204726; a=rsa-sha256; cv=none; b=WW2kqOoIY7nHqTl3ScFQxSY2VkYSw7kudZLsrFJafTAPti6JZQXtnXeRlykFUmTYs0t5Aa eIod0FfT+hG0Nzo5W8kWOD0hga+YvBqQzuXiDtseiY1rGZ2UuZr0AROaDi0ND0BZMRlSq1 tYOff8UCMKvXC2LUyw1gCMoVp6JjBf6Dl0IwGfkQRWW+ZvztMw+fq7ybg/GdXrHY8tiySk PggV2Xh6zfeVWS47vB14E4S2v7166KCAZ2bSiCKdwbv2mSgTd/ItrFRZuJ0/r+NJ9DcEvy naLytsdpIzuNGKS8gCbfyU97D6PCxGwAIvCXtPYzn39busm3+5ZK7/mNZ5oezw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1676204726; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=fGckC2Vo7DdThnjJrl2MGsknoklHeau267zcbhpf/0c=; b=HsfncNMCo+sLLpZWnpWclSIevxtPmKDORiDAecw8dXFFHE1MbSJ6CGHAbKSKTb29CwaRfq Ocz/wLFj8DLwqBD9ywpzLoWpkTBv16GFK7+PWrsAIrZ8yVFkjc0vt2Ev3gp8/Z+1jOKiwu e9eSNjRmxzuQ6MQxdxT2Dak5aBw5pmgkKZkW78icMEySHFiJNW7SntkPqaL/kLk6uLRB9o ZaSXP7ZLDtWcAPA2WbKljeN5GGYp00dZzcgZsWv4625ixt38sBDNOIKy0PaVBqVo3duyDa gWxSOoLmGTengHxGVdS1H9PhgMucciJgmv7diINYY/0jru5k9XjkAJUyozOeSg== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -3.72 X-Spam-Score: -3.72 X-Migadu-Queue-Id: EE66D79A7 Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmx.com (policy=none) X-TUID: ay/LNEZgflVz --=-=-= Content-Type: text/plain Hi, Andrew Tropin writes: > [[PGP Signed Part:Undecided]] > On 2023-02-10 15:48, Pierre Langlois wrote: > >> Hi Andrew, thanks for pushing this along! It's great to see things >> getting merged. >> >> Andrew Tropin writes: >> >>> [[PGP Signed Part:Undecided]] >>> On 2023-02-09 18:04, Andrew Tropin wrote: >>> >>>> On 2023-02-09 13:39, zimoun wrote: >>>> >>>>> Hi, >>>>> >>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin wrote: >>>>> >>>>>> I applied tree-sitter and tree-sitter-cli patches, >>>>> >>>>> Just to be sure to understand, you have only applied 02/32 and 05/32, >>>>> right? >>>>> >>>>> >>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7. >>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com >>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com >>>>> >>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli. >>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com >>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com >>>>> >>>>> Leaving out all the others, right? >>>> >>>> Merged first 5 patches from 01 to 05, also added one more commit, which >>>> addresses some things from reviews and one commit, which adds html >>>> grammar. >>>> >>>> The html grammar is added for the testing purposes. It relies on >>>> generated parser.c and scanner.c and we will need to repackage it using >>>> grammar.js instead. I'm not sure if a separate build system is needed >>>> for this, I guess we can just rewrite tree-sitter-grammar function, >>>> which generates packages as in example with tree-sitter-grammar-html: >>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158 >>>> >>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to >>>> read it carefully, evaluate and either introduce new build system or >>>> just move all needed parts to tree-sitter-grammar function. WDYT? >>>> After we done with it we can package all other grammars. >>> >>> Ok, I realized that the proper build process for tree-sitter grammars is >>> a little harder than I expected, tree-sitter-build system make sense. I >>> reviewed it, made a small change: >> >> Ah great, I was going to comment to try and push for us to keep the >> build system. I originally went with a template package and inheritance, >> but Maxime suggested moving to a build-system which ended up making the >> package definitions a *lot* nicer IMO (see previous discussion here >> https://issues.guix.gnu.org/49946#144). It also allows us to deal with >> grammars that depend on each other more nicely I think. >> >>> >>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system) >>> ;; Commentary: >>> ;; >>> ;; Build procedures for tree-sitter grammar packages. This is the >>> -;; builder-side code, which builds on top fo the node build-system. >>> +;; builder-side code, which builds on top of the node build-system. >>> ;; >>> ;; Tree-sitter grammars are written in JavaScript and compiled to a native >>> ;; shared object. The `tree-sitter generate' command invokes `node' in order >>> @@ -114,7 +114,7 @@ (define (compile-language dir) >>> "-fno-exceptions" >>> "-O2" >>> "-g" >>> - "-o" ,(string-append lib "/" lang ".so") >>> + "-o" ,(string-append lib "/libtree-sitter-" lang ".so") >>> ;; An additional `scanner.{c,cc}' file is sometimes >>> ;; provided. >>> ,@(cond >>> >>> >>> rewrote html grammar to use this build system and made it work with >>> built-in treesit package. Also, tried examples of c and cpp grammars >>> from patches in this thread. >>> >>> If you ok with it, I'll push the build system to master and update the >>> html grammar accordingly. >>> >>> The final result will look like this: >>> >>> (define tree-sitter-delete-generated-files >>> #~(begin >>> (delete-file "binding.gyp") >>> (delete-file-recursively "bindings") >>> (delete-file "src/grammar.json") >>> (delete-file "src/node-types.json") >>> (delete-file "src/parser.c") >>> (delete-file-recursively "src/tree_sitter"))) >>> >>> (define* (tree-sitter-grammar >>> language language-for-synopsis version commit hash >>> #:key >>> (repository-url >>> (format #f "https://github.com/tree-sitter/tree-sitter-~a" language)) >>> (inputs '())) >>> (let ((synopsis (string-append language-for-synopsis >>> " grammar for tree-sitter")) >>> (name (string-append "tree-sitter-grammar-" language))) >>> (package >>> (name name) >>> (version version) >>> (home-page repository-url) >>> (source (origin >>> (method git-fetch) >>> (uri (git-reference >>> (url repository-url) >>> (commit commit))) >>> (file-name (git-file-name name version)) >>> (sha256 (base32 hash)) >>> (modules '((guix build utils))) >>> (snippet tree-sitter-delete-generated-files))) >>> (build-system tree-sitter-build-system) >>> (inputs inputs) >>> (synopsis synopsis) >>> (description (string-append synopsis ".")) >>> (license license:expat)))) >>> >>> (define-public tree-sitter-grammar-html >>> (tree-sitter-grammar >>> "html" "HTML" >>> "0.19.0" "v0.19.0" >>> "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy")) >>> >>> After that we can bring the rest of the grammars. >> >> I would suggest to rmeove the `tree-sitter-grammar' function, and keep >> grammars as "regular" package records, even though it's a little bit >> more verbose: >> >> --8<---------------cut here---------------start------------->8--- >> (define-public tree-sitter-html >> (package >> (name "tree-sitter-html") > > It seems tree-sitter-html mimics upstream package name and probably make > more sense than tree-sitter-grammar-html used by me. Yeah, at some point I think I had named the packages with "grammar" as well, but thought it was a bit of a mouthful. I'm also thinking one day we may build language bindings as part of the build system (Rust and NodeJS I think ATM), so those packages could do more than ship the grammar in the future (although we don't know if we'll ever really need that). > >> (version "0.19.0") >> (source (origin >> (method git-fetch) >> (uri (git-reference >> (url "https://github.com/tree-sitter/tree-sitter-html") >> (commit (string-append "v" version)))) >> (file-name (git-file-name name version)) >> (sha256 >> (base32 >> "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy")) >> (modules '((guix build utils))) >> (snippet tree-sitter-delete-generated-files))) >> (build-system tree-sitter-build-system) >> (home-page "https://github.com/tree-sitter/tree-sitter-html") >> (synopsis "Tree-sitter HTML grammar") >> (description >> "This package provides a HTML grammar for the Tree-sitter library.") >> (license license:expat))) >> --8<---------------cut here---------------end--------------->8--- >> >> This way, they look like any other package in Guix, which makes it >> easier for us to apply automatic changes in the future if needed (for >> example like how the input format could be automically updated for all >> "simple" package definitions, but had to be manual whenever custom code >> refactoring was done). Does this make sense? > > Make sense, but on the other hand we already have hunspell, aspell > dictionaries and probably a few more others, which are very similiar in > spirit and we already have to keep in mind their existence on such > automatic code updates. > > It looks that the packages differ only in url for the source code, lang > name and sometimes in inputs. Having template package function can make > management of shared parts more centralized, reduce possibility of > copy-paste mistakes, when the description wasn't updated and so on and > can reduce the amount of the code overall (which also reduces the change > of introducing an error). > > I don't have a strong opinion on this topic, but leaning towards the > template function slightly more, however I'm completely ok with the > standalone package definitions as well. WDYT? I can think of both cost/benefits to the template so I don't have a strong opinion either :-). I do like the template to make sure people don't forget to delete generated files, that's quite important as it seems upstream packages often check-in the generated C code. Although, we could probably assert that with in the build-system phase? I'll think about that. On the other hand, I wonder how the template works for packages that provide multiple grammars (see ocaml and typescript for example). I guess we could use the template for trivial packages, and standalone definitions for more complex ones? In general, if we keep the template interface really simple, then I'm happy with it. Thanks, Pierre --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQFMBAEBCgA2FiEEctU9gYy29KFyWDdMqPyeRH9PfVQFAmPo2nYYHHBpZXJyZS5s YW5nbG9pc0BnbXguY29tAAoJEKj8nkR/T31UEu4H/28hYTTd7UXk3XP09t/L9LAK J6ZU8tN0dghON14h5vy/KhMGXhbBDpqHLTPmMPcyjRiQbkcGhh7bPaUDtoQ1P916 Ytmfgc9vpqjznFL2YmqcUzAW6pklnhvTJt8wz3RVEwg/7DO7piMSkW9zW7eM7S43 +EhhIHfC+My3+hQKaKjTxIzCo+J51cFFYALnrY+eaoxOOb3/zg6RFdEVjj/fxOYc tX3T1MTeAOtgQIShKjQj6TF9qUGLvT5B2t6WNDH9OE+zsXCLtsMbXnfCsWOjvwUl ARug7+XwQKVVMjlCPbqVkGRQoUwRqwIZBljWCeDyJzrF17Vp6st2UHRpA2kLN38= =4Eku -----END PGP SIGNATURE----- --=-=-=--