From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id oHr3Aax172O62AAAbAwnHQ (envelope-from ) for ; Fri, 17 Feb 2023 13:40:12 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id sMQGAqx172MjfQAA9RJhRA (envelope-from ) for ; Fri, 17 Feb 2023 13:40:12 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9A3E71068C for ; Fri, 17 Feb 2023 13:40:11 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pT020-0005MH-Dm; Fri, 17 Feb 2023 07:40:04 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pT01y-0005Lq-D3 for guix-patches@gnu.org; Fri, 17 Feb 2023 07:40:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pT01y-0001Km-2a for guix-patches@gnu.org; Fri, 17 Feb 2023 07:40:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pT01x-0001Lo-Sx for guix-patches@gnu.org; Fri, 17 Feb 2023 07:40:01 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module. Resent-From: Pierre Langlois Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 17 Feb 2023 12:40:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49946 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Andrew Tropin Cc: "\(" , Pierre Langlois , 49946@debbugs.gnu.org, Luis Henrique Gomes Higino , zimoun Received: via spool by 49946-submit@debbugs.gnu.org id=B49946.16766375565130 (code B ref 49946); Fri, 17 Feb 2023 12:40:01 +0000 Received: (at 49946) by debbugs.gnu.org; 17 Feb 2023 12:39:16 +0000 Received: from localhost ([127.0.0.1]:38725 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pT01D-0001Kf-8Y for submit@debbugs.gnu.org; Fri, 17 Feb 2023 07:39:15 -0500 Received: from mout.gmx.net ([212.227.17.22]:54441) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pT01B-0001KR-21 for 49946@debbugs.gnu.org; Fri, 17 Feb 2023 07:39:13 -0500 Received: from labiere ([82.69.64.142]) by mail.gmx.net (mrgmx104 [212.227.17.174]) with ESMTPSA (Nemesis) id 1MfpSb-1ovr3g2zmL-00gJuk; Fri, 17 Feb 2023 13:39:05 +0100 References: <87mtfi63ut.fsf@gmx.com> <20221125012142.22579-1-pierre.langlois@gmx.com> <20221125012142.22579-2-pierre.langlois@gmx.com> <87bkovcp1d.fsf@gmx.com> <87h6vvgnd6.fsf@trop.in> <86pmaj3td2.fsf@gmail.com> <87cz6jgcku.fsf@trop.in> <87a61lr8cp.fsf@trop.in> <87o7q1qzol.fsf@gmx.com> <87mt5jfmvl.fsf@trop.in> <871qmvm5qh.fsf@gmx.com> <87zg9g753h.fsf@trop.in> User-agent: mu4e 1.8.13; emacs 28.2 From: Pierre Langlois Date: Fri, 17 Feb 2023 12:38:05 +0000 In-reply-to: <87zg9g753h.fsf@trop.in> Message-ID: <873574v53t.fsf@gmx.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Provags-ID: V03:K1:/4nl6r7J0Xpr5Mj6WWMB3avK2pT8wRoQWwAiAfRe7YXCWW98V00 fv7xpWIzuiBbvi7ataOx+KtrBHO5arOuRWHkMt3k6Ip8E6LYsUWUnuJEUkNYEiFkaXydUdE HbXqFfGy7JUrts67lS+jMxYiywgSQrH99cI7YvadCq0t1fxNgYLwS1/vDRdTibduDvxU8UB iR1u3E8bjmrMM1iVaVpwg== UI-OutboundReport: notjunk:1;M01:P0:BI1HgXf+Ioc=;niAmgAqmPuZEnbU3LOTJw/JGJSq +I+XeT+q6Gvf/4+5a6M18UGZpIYCvIfcSUE7d77/syxlxHVsXbxV03e10LocvcbBhB2eVi/+P nNcbQSKe+G4sBcbVU8q20C8IZ+mplLiZr0vKEWfstOy2Rl+gN8V1IPWrf2P2qecZh3F39uh7D tqjuVgUlk2IhHetxUZz1o1MJzgFihF6EkQMUGNoq+BIUVH3LQce06YcX1SpAib8d8xskQQMDA q7+OdS9/1nKswspa/Xh5vDRLjS4QuZFpPGTHZ/CQsF76GfmXPJ+Y8AiBhs+sbRAEzyvVUpjkI fxdmgPknhGFJqsu/T4hnxkZmzX58ZHaiDTQX8ij8fdzDFiP5Zp0s2Lgx0exVrDTs1485CHYm0 zl0UfFSq3yEi4eRZJXss1dsIq4+DUzMMpIE8DL6iQiIzS/86mYJHMCNobpgS3TC1lrwls7Ggq ohQj272BxX9wuGK2jRr+5PmWnVchveTD6GfVMOQ113HfnVkYu99r45jPzmc/sl9itDl5dZF6n 7n6ErMoTIufleEJu5mydTuYMutdDCUWdptjF7gjI7aSwoQ2h3qbyjinUgvX9B9f8Sm/Y0Sx/P W6rsg6SvZBW562VOIDwTkh6q8UiBmbNKlW0fwi4yqzhJCd50f75Hn2PbEeMfAIzJYK8BfJwNk FsU91sRO0/axcT7O0T2OUkDQDM4jyNktMnFsuSmPzjLPvOwa0K+dwKP6XE2ucquNb+/Tdlmhe gIraYkN0mLJ8LRDyL+yhOjWpijnQG0VJUEr83Ap58lFoKyPIvzmhCY6ipxvzqcxF3pGsG+36t yRSA99EGnG5y9JytL3kCD4wpCHYrX8MOpreL9rBMgBPfyuToylsEI2bQrEOPHhMHPzbEuuYXp H+GpfuJY/eTOFFYW21OtQP9d+G6wl6oIUiU51VPZoHefzfy7b97WYA2IKH6Lfi9FE1Eq9CxJZ K20q6nz8mweegNoh115cAZO3cYs= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1676637611; a=rsa-sha256; cv=none; b=RGzpks9B96dtHGGkakvqMASAmM9pwylFhFdZatDsA+JXbAvTdG3r7DzmljNmN5atrc3js4 Wy3vf2W+wEAf7qFOZDGoMHFvK4BuwrT152o4ApU1zixrLBkfETgydoQppgDGU1UjnWaEdP ui2Xv4sPY8d0Y4IlVb4IFsWZ5neACDTcjb33PqiY7OFeKECU5Bz4gG9MbipctKBQSUDFO2 mb1xK+gZa8Opwki4wSxgkYKIAhPh7Sz7xWOY11Fd9rrFbzp6FtBGBHQ8EtTVMkb9Ul3p0D pByV9wwNqP46Uhg6f4m80OuG4622jE+fpJO/+Qw2kuW/CyUOYwXtqwlbo81o5g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmx.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1676637611; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=LZWz65xDcGlTco1qQrCeBGaC2bfTmL8pXWAtW1Zno5Q=; b=ko5hWXvui0Vgboc+rd2lVDkNdq5nnb0U4WYub5ZjuIB+HT6Kb4UxfrzOCkeABrJz76iypE eCJ337wt49qKf/wErBuQ1EuSatKwmPkAxDO77SD6Y1SgYpWUUS7Yx4Vf3DzrlbaE41a0dG 5fnDCdhPyWpAokSlbQ1uA3T2YTFn9ps28p8dKwyK9hPeMh+gGQtXXS5JVCfg2+C6GTU5He L1q52obi5DwpWg8VNu3RCvozIQvSq3vPyCMtxNm5w/slM07+DpRGHufpRIXfVAiZMB9xgQ s1VJap1cT+DuLWQFbS4KpNbKVKQwyu8GfJhNMuNHw0Bq2sQjEf/pTaqh96ILog== X-Migadu-Spam-Score: -3.73 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmx.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 9A3E71068C X-Migadu-Scanner: scn1.migadu.com X-Spam-Score: -3.73 X-TUID: OPx1x2zDMKXW --=-=-= Content-Type: text/plain Hi, Andrew Tropin writes: > [[PGP Signed Part:Undecided]] > On 2023-02-12 12:07, Pierre Langlois wrote: > >> Hi, >> >> Andrew Tropin writes: >> >>> [[PGP Signed Part:Undecided]] >>> On 2023-02-10 15:48, Pierre Langlois wrote: >>> >>>> Hi Andrew, thanks for pushing this along! It's great to see things >>>> getting merged. >>>> >>>> Andrew Tropin writes: >>>> >>>>> [[PGP Signed Part:Undecided]] >>>>> On 2023-02-09 18:04, Andrew Tropin wrote: >>>>> >>>>>> On 2023-02-09 13:39, zimoun wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin wrote: >>>>>>> >>>>>>>> I applied tree-sitter and tree-sitter-cli patches, >>>>>>> >>>>>>> Just to be sure to understand, you have only applied 02/32 and 05/32, >>>>>>> right? >>>>>>> >>>>>>> >>>>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7. >>>>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com >>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langlois@gmx.com >>>>>>> >>>>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli. >>>>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com >>>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langlois@gmx.com >>>>>>> >>>>>>> Leaving out all the others, right? >>>>>> >>>>>> Merged first 5 patches from 01 to 05, also added one more commit, which >>>>>> addresses some things from reviews and one commit, which adds html >>>>>> grammar. >>>>>> >>>>>> The html grammar is added for the testing purposes. It relies on >>>>>> generated parser.c and scanner.c and we will need to repackage it using >>>>>> grammar.js instead. I'm not sure if a separate build system is needed >>>>>> for this, I guess we can just rewrite tree-sitter-grammar function, >>>>>> which generates packages as in example with tree-sitter-grammar-html: >>>>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sitter.scm?h=53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158 >>>>>> >>>>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan to >>>>>> read it carefully, evaluate and either introduce new build system or >>>>>> just move all needed parts to tree-sitter-grammar function. WDYT? >>>>>> After we done with it we can package all other grammars. >>>>> >>>>> Ok, I realized that the proper build process for tree-sitter grammars is >>>>> a little harder than I expected, tree-sitter-build system make sense. I >>>>> reviewed it, made a small change: >>>> >>>> Ah great, I was going to comment to try and push for us to keep the >>>> build system. I originally went with a template package and inheritance, >>>> but Maxime suggested moving to a build-system which ended up making the >>>> package definitions a *lot* nicer IMO (see previous discussion here >>>> https://issues.guix.gnu.org/49946#144). It also allows us to deal with >>>> grammars that depend on each other more nicely I think. >>>> >>>>> >>>>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system) >>>>> ;; Commentary: >>>>> ;; >>>>> ;; Build procedures for tree-sitter grammar packages. This is the >>>>> -;; builder-side code, which builds on top fo the node build-system. >>>>> +;; builder-side code, which builds on top of the node build-system. >>>>> ;; >>>>> ;; Tree-sitter grammars are written in JavaScript and compiled to a native >>>>> ;; shared object. The `tree-sitter generate' command invokes `node' in order >>>>> @@ -114,7 +114,7 @@ (define (compile-language dir) >>>>> "-fno-exceptions" >>>>> "-O2" >>>>> "-g" >>>>> - "-o" ,(string-append lib "/" lang ".so") >>>>> + "-o" ,(string-append lib "/libtree-sitter-" lang ".so") >>>>> ;; An additional `scanner.{c,cc}' file is sometimes >>>>> ;; provided. >>>>> ,@(cond >>>>> >>>>> >>>>> rewrote html grammar to use this build system and made it work with >>>>> built-in treesit package. Also, tried examples of c and cpp grammars >>>>> from patches in this thread. >>>>> >>>>> If you ok with it, I'll push the build system to master and update the >>>>> html grammar accordingly. >>>>> >>>>> The final result will look like this: >>>>> >>>>> (define tree-sitter-delete-generated-files >>>>> #~(begin >>>>> (delete-file "binding.gyp") >>>>> (delete-file-recursively "bindings") >>>>> (delete-file "src/grammar.json") >>>>> (delete-file "src/node-types.json") >>>>> (delete-file "src/parser.c") >>>>> (delete-file-recursively "src/tree_sitter"))) >>>>> >>>>> (define* (tree-sitter-grammar >>>>> language language-for-synopsis version commit hash >>>>> #:key >>>>> (repository-url >>>>> (format #f "https://github.com/tree-sitter/tree-sitter-~a" language)) >>>>> (inputs '())) >>>>> (let ((synopsis (string-append language-for-synopsis >>>>> " grammar for tree-sitter")) >>>>> (name (string-append "tree-sitter-grammar-" language))) >>>>> (package >>>>> (name name) >>>>> (version version) >>>>> (home-page repository-url) >>>>> (source (origin >>>>> (method git-fetch) >>>>> (uri (git-reference >>>>> (url repository-url) >>>>> (commit commit))) >>>>> (file-name (git-file-name name version)) >>>>> (sha256 (base32 hash)) >>>>> (modules '((guix build utils))) >>>>> (snippet tree-sitter-delete-generated-files))) >>>>> (build-system tree-sitter-build-system) >>>>> (inputs inputs) >>>>> (synopsis synopsis) >>>>> (description (string-append synopsis ".")) >>>>> (license license:expat)))) >>>>> >>>>> (define-public tree-sitter-grammar-html >>>>> (tree-sitter-grammar >>>>> "html" "HTML" >>>>> "0.19.0" "v0.19.0" >>>>> "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy")) >>>>> >>>>> After that we can bring the rest of the grammars. >>>> >>>> I would suggest to rmeove the `tree-sitter-grammar' function, and keep >>>> grammars as "regular" package records, even though it's a little bit >>>> more verbose: >>>> >>>> --8<---------------cut here---------------start------------->8--- >>>> (define-public tree-sitter-html >>>> (package >>>> (name "tree-sitter-html") >>> >>> It seems tree-sitter-html mimics upstream package name and probably make >>> more sense than tree-sitter-grammar-html used by me. >> >> Yeah, at some point I think I had named the packages with "grammar" as >> well, but thought it was a bit of a mouthful. I'm also thinking one day >> we may build language bindings as part of the build system (Rust and >> NodeJS I think ATM), so those packages could do more than ship the >> grammar in the future (although we don't know if we'll ever really need >> that). >> >>> >>>> (version "0.19.0") >>>> (source (origin >>>> (method git-fetch) >>>> (uri (git-reference >>>> (url "https://github.com/tree-sitter/tree-sitter-html") >>>> (commit (string-append "v" version)))) >>>> (file-name (git-file-name name version)) >>>> (sha256 >>>> (base32 >>>> "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy")) >>>> (modules '((guix build utils))) >>>> (snippet tree-sitter-delete-generated-files))) >>>> (build-system tree-sitter-build-system) >>>> (home-page "https://github.com/tree-sitter/tree-sitter-html") >>>> (synopsis "Tree-sitter HTML grammar") >>>> (description >>>> "This package provides a HTML grammar for the Tree-sitter library.") >>>> (license license:expat))) >>>> --8<---------------cut here---------------end--------------->8--- >>>> >>>> This way, they look like any other package in Guix, which makes it >>>> easier for us to apply automatic changes in the future if needed (for >>>> example like how the input format could be automically updated for all >>>> "simple" package definitions, but had to be manual whenever custom code >>>> refactoring was done). Does this make sense? >>> >>> Make sense, but on the other hand we already have hunspell, aspell >>> dictionaries and probably a few more others, which are very similiar in >>> spirit and we already have to keep in mind their existence on such >>> automatic code updates. >>> >>> It looks that the packages differ only in url for the source code, lang >>> name and sometimes in inputs. Having template package function can make >>> management of shared parts more centralized, reduce possibility of >>> copy-paste mistakes, when the description wasn't updated and so on and >>> can reduce the amount of the code overall (which also reduces the change >>> of introducing an error). >>> >>> I don't have a strong opinion on this topic, but leaning towards the >>> template function slightly more, however I'm completely ok with the >>> standalone package definitions as well. WDYT? >> >> I can think of both cost/benefits to the template so I don't have a >> strong opinion either :-). >> >> I do like the template to make sure people don't forget to delete >> generated files, that's quite important as it seems upstream packages >> often check-in the generated C code. Although, we could probably assert >> that with in the build-system phase? I'll think about that. >> >> On the other hand, I wonder how the template works for packages that >> provide multiple grammars (see ocaml and typescript for example). I >> guess we could use the template for trivial packages, and standalone >> definitions for more complex ones? In general, if we keep the template >> interface really simple, then I'm happy with it. > > Hi Pierre! > > I spend two days trying grammars with and without helper function and > found hepler quite helpful to reduce boilerplate and errors from > copypaste, so I went the way with helper. The logic inside is quite > trivial, the only downside I found so far is that in cases when > repository url constructed automatically I can't easily open the repo > url in the browser. > > I packaged all the grammars from this thread and a few more on top of > it. Updated them to usually latest versions, added some comments, when > needed. > > If I forgot to reply on something or you have any comments/ideas, let me > know! :) > > Kudos to Pierre and everyone, who helped with all the tree-sitter stuff. Thank you for landing all this work! --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQFMBAEBCgA2FiEEctU9gYy29KFyWDdMqPyeRH9PfVQFAmPvdWYYHHBpZXJyZS5s YW5nbG9pc0BnbXguY29tAAoJEKj8nkR/T31Uyb0IALOneatOvzeS+nrtS0ryGEga hwPLzlJH4ZJfPd9zMHz0p1z0H191C155dP1ejGYnTK0QJzepqjVypzvbUl8MKVkt p4aCm3Aid9yYW2QJ9pfmE0mc4PNH0je82Fps9w8QwYZwT2lc7rRDer1UAzM22LBD GLTiLeTLWbAlA8/klpaOURkPs7E57BrHLbO554B0Vf5OHC+zp9SD7WdohTfgUC3O gjXGCDgTrF+n4S+YR40fZRX+AEClfAYL6GGXgn6OelmX6Z1F2qOWTTcmCbgG6em3 ge/EbPSqJfvjQySLUXYUx8wEwAdvs/n8XI6znnuSH14IEVM+H6ce3gXPv3PcZNs= =2XjL -----END PGP SIGNATURE----- --=-=-=--