From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id iPLvDdKL62MQYAAAbAwnHQ (envelope-from ) for ; Tue, 14 Feb 2023 14:25:38 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id +J7sDdKL62MxugAAauVa8A (envelope-from ) for ; Tue, 14 Feb 2023 14:25:38 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 95AE3267 for ; Tue, 14 Feb 2023 14:25:37 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=trop.in header.s=gm1 header.b="lqGyw/aX"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1676381138; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=zUptAXUHeMhAiDhC8bHsygrr7KJUYzsQ/UIw6ba3m3Y=; b=rkMTcLmgRRyTVzJntgolwDsQj8Cnc1TU2Np70k3eKPpfu3FxmBHd1a1n5LLSlIEdL31hQu ZWrbK7s8UkiX1jdFETO9ELd7C27la76es1ta9VwHsR6XQ25WZsriCBRcz0zfKQnalNc/46 S+l2lOu1bZ9Be2bRI0epBYVVtqBMaSDJntlQDqs6stwVszQA1R2IC1Cna9G6htAuUvaZ70 ghEY/qZFkt+CnHXdzdhZQrDrhDqzt8475YlOriDjMz6/6t3tTLGlm+f5WobBaNp8pcRjf/ tAykHTLKk4n82XoxXuwILRouuDxB8vt+8EzSd2hy5SdicunLJLH1v0KQJdsHGQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=trop.in header.s=gm1 header.b="lqGyw/aX"; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Seal: i=1; s=key1; d=yhetil.org; t=1676381138; a=rsa-sha256; cv=none; b=lSCILFVck3ZQksCZRa6cd3bPUBQrQaNmEyzKKqfT0mOOZFYqjSnPc1SMJOJ4PiVYZ53PP1 Kaq7BNApvKUbRc7TXgugc5wS30j8s3IBB4KrCipbw2tM7HKTQoCnS2+l2QWrgn3uAzHrbC L0qPLMOOsR0lJXWtZCGrbpyXt0ZlH3OJKdKkjtDi9nfFi+x2I3BVH8rOPTkBChuDrTw1Ay iZYOfT5PGAF+T7EhJlG8c0zB1JsLsGWB5KoUTAJW4ATl8W1ZLlFMKp1OcUQ1A/QaL9wR14 1weWQU0INEMlPUSk5SJI8nLUUoVJHm6M4clmWJyvBN5J3uBe5HLNxLHWKNxJEw== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pRvJH-0002WE-CY; Tue, 14 Feb 2023 08:25:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pRvIt-00020Y-Mn for guix-patches@gnu.org; Tue, 14 Feb 2023 08:25:07 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pRvIs-0006Gm-Nz for guix-patches@gnu.org; Tue, 14 Feb 2023 08:25:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pRvIs-0000bX-B0 for guix-patches@gnu.org; Tue, 14 Feb 2023 08:25:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#49946] [PATCH v7 01/32] gnu: tree-sitter: Move to its own module. Resent-From: Andrew Tropin Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 14 Feb 2023 13:25:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49946 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Pierre Langlois Cc: "\(" , Pierre Langlois , 49946@debbugs.gnu.org, Luis Henrique Gomes Higino , zimoun Received: via spool by 49946-submit@debbugs.gnu.org id=B49946.16763810602251 (code B ref 49946); Tue, 14 Feb 2023 13:25:02 +0000 Received: (at 49946) by debbugs.gnu.org; 14 Feb 2023 13:24:20 +0000 Received: from localhost ([127.0.0.1]:52879 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pRvIB-0000aE-Tf for submit@debbugs.gnu.org; Tue, 14 Feb 2023 08:24:20 -0500 Received: from relay3-d.mail.gandi.net ([217.70.183.195]:38229) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pRvI8-0000Zw-KK for 49946@debbugs.gnu.org; Tue, 14 Feb 2023 08:24:18 -0500 Received: (Authenticated sender: andrew@trop.in) by mail.gandi.net (Postfix) with ESMTPSA id 3D27560018; Tue, 14 Feb 2023 13:24:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=trop.in; s=gm1; t=1676381050; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=zUptAXUHeMhAiDhC8bHsygrr7KJUYzsQ/UIw6ba3m3Y=; b=lqGyw/aXS6nCcGajwxGLrjbXnVBUzX+ZK+5GXYpHSRQ9pDNbptG2WzYDmxCSH54H5Z+nPQ DOdwzEvOAo9kPPOekbScG6xTE3xHVdRp6hTkcLjkHdYsEO4sPK5v+CB08FV+3kPwk/e0tT HyVjrpKD/avKSXk7tbmYCc3lGcVVf+03O+0ZYm2czY66IEZppP61t2JhRGGDte3w5yBr4P ooavEfPrHrvyw9rmkqf96ZZR7kSu9Sq6QdqSJKnm2bpqxWEXmMsJ1zWcrtwfiV7KHxCA0p 4qsJrbu7DQbJBDWNAiCU6pMORno+IpyGYfF+gY6W6glU2eTr1nH8VNcp1y2tQg== From: Andrew Tropin In-Reply-To: <871qmvm5qh.fsf@gmx.com> References: <87mtfi63ut.fsf@gmx.com> <20221125012142.22579-1-pierre.langlois@gmx.com> <20221125012142.22579-2-pierre.langlois@gmx.com> <87bkovcp1d.fsf@gmx.com> <87h6vvgnd6.fsf@trop.in> <86pmaj3td2.fsf@gmail.com> <87cz6jgcku.fsf@trop.in> <87a61lr8cp.fsf@trop.in> <87o7q1qzol.fsf@gmx.com> <87mt5jfmvl.fsf@trop.in> <871qmvm5qh.fsf@gmx.com> Date: Tue, 14 Feb 2023 17:24:02 +0400 Message-ID: <87zg9g753h.fsf@trop.in> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: X-Migadu-Queue-Id: 95AE3267 X-Spam-Score: 1.18 X-Migadu-Spam-Score: 1.18 X-Migadu-Scanner: scn0.migadu.com List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-TUID: YC84/Kxhd5JX --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On 2023-02-12 12:07, Pierre Langlois wrote: > Hi, > > Andrew Tropin writes: > >> [[PGP Signed Part:Undecided]] >> On 2023-02-10 15:48, Pierre Langlois wrote: >> >>> Hi Andrew, thanks for pushing this along! It's great to see things >>> getting merged. >>> >>> Andrew Tropin writes: >>> >>>> [[PGP Signed Part:Undecided]] >>>> On 2023-02-09 18:04, Andrew Tropin wrote: >>>> >>>>> On 2023-02-09 13:39, zimoun wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> On Thu, 09 Feb 2023 at 14:11, Andrew Tropin wrote: >>>>>> >>>>>>> I applied tree-sitter and tree-sitter-cli patches, >>>>>> >>>>>> Just to be sure to understand, you have only applied 02/32 and 05/32, >>>>>> right? >>>>>> >>>>>> >>>>>> [bug#49946] [PATCH v7 02/32] gnu: tree-sitter: Update to 0.20.7. >>>>>> id:20221125012142.22579-3-pierre.langlois@gmx.com >>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-3-pierre.langl= ois@gmx.com >>>>>> >>>>>> [bug#49946] [PATCH v7 05/32] gnu: Add tree-sitter-cli. >>>>>> id:20221125012142.22579-6-pierre.langlois@gmx.com >>>>>> http://issues.guix.gnu.org/msgid/20221125012142.22579-6-pierre.langl= ois@gmx.com >>>>>> >>>>>> Leaving out all the others, right? >>>>> >>>>> Merged first 5 patches from 01 to 05, also added one more commit, whi= ch >>>>> addresses some things from reviews and one commit, which adds html >>>>> grammar. >>>>> >>>>> The html grammar is added for the testing purposes. It relies on >>>>> generated parser.c and scanner.c and we will need to repackage it usi= ng >>>>> grammar.js instead. I'm not sure if a separate build system is needed >>>>> for this, I guess we can just rewrite tree-sitter-grammar function, >>>>> which generates packages as in example with tree-sitter-grammar-html: >>>>> https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tree-sit= ter.scm?h=3D53b00b91b73bd60412d5bd057e22e6d63194a7f7#n158 >>>>> >>>>> Anyway, I only skimmed tree-sitter-build-system source code, and plan= to >>>>> read it carefully, evaluate and either introduce new build system or >>>>> just move all needed parts to tree-sitter-grammar function. WDYT? >>>>> After we done with it we can package all other grammars. >>>> >>>> Ok, I realized that the proper build process for tree-sitter grammars = is >>>> a little harder than I expected, tree-sitter-build system make sense. = I >>>> reviewed it, made a small change: >>> >>> Ah great, I was going to comment to try and push for us to keep the >>> build system. I originally went with a template package and inheritance, >>> but Maxime suggested moving to a build-system which ended up making the >>> package definitions a *lot* nicer IMO (see previous discussion here >>> https://issues.guix.gnu.org/49946#144). It also allows us to deal with >>> grammars that depend on each other more nicely I think. >>> >>>> >>>> @@ -29,7 +29,7 @@ (define-module (guix build tree-sitter-build-system) >>>> ;; Commentary: >>>> ;; >>>> ;; Build procedures for tree-sitter grammar packages. This is the >>>> -;; builder-side code, which builds on top fo the node build-system. >>>> +;; builder-side code, which builds on top of the node build-system. >>>> ;; >>>> ;; Tree-sitter grammars are written in JavaScript and compiled to a n= ative >>>> ;; shared object. The `tree-sitter generate' command invokes `node' = in order >>>> @@ -114,7 +114,7 @@ (define (compile-language dir) >>>> "-fno-exceptions" >>>> "-O2" >>>> "-g" >>>> - "-o" ,(string-append lib "/" lang ".so") >>>> + "-o" ,(string-append lib "/libtree-sitter-" lang "= .so") >>>> ;; An additional `scanner.{c,cc}' file is sometimes >>>> ;; provided. >>>> ,@(cond >>>> >>>> >>>> rewrote html grammar to use this build system and made it work with >>>> built-in treesit package. Also, tried examples of c and cpp grammars >>>> from patches in this thread. >>>> >>>> If you ok with it, I'll push the build system to master and update the >>>> html grammar accordingly. >>>> >>>> The final result will look like this: >>>> >>>> (define tree-sitter-delete-generated-files >>>> #~(begin >>>> (delete-file "binding.gyp") >>>> (delete-file-recursively "bindings") >>>> (delete-file "src/grammar.json") >>>> (delete-file "src/node-types.json") >>>> (delete-file "src/parser.c") >>>> (delete-file-recursively "src/tree_sitter"))) >>>> >>>> (define* (tree-sitter-grammar >>>> language language-for-synopsis version commit hash >>>> #:key >>>> (repository-url >>>> (format #f "https://github.com/tree-sitter/tree-sitter-~a" = language)) >>>> (inputs '())) >>>> (let ((synopsis (string-append language-for-synopsis >>>> " grammar for tree-sitter")) >>>> (name (string-append "tree-sitter-grammar-" language))) >>>> (package >>>> (name name) >>>> (version version) >>>> (home-page repository-url) >>>> (source (origin >>>> (method git-fetch) >>>> (uri (git-reference >>>> (url repository-url) >>>> (commit commit))) >>>> (file-name (git-file-name name version)) >>>> (sha256 (base32 hash)) >>>> (modules '((guix build utils))) >>>> (snippet tree-sitter-delete-generated-files))) >>>> (build-system tree-sitter-build-system) >>>> (inputs inputs) >>>> (synopsis synopsis) >>>> (description (string-append synopsis ".")) >>>> (license license:expat)))) >>>> >>>> (define-public tree-sitter-grammar-html >>>> (tree-sitter-grammar >>>> "html" "HTML" >>>> "0.19.0" "v0.19.0" >>>> "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy")) >>>> >>>> After that we can bring the rest of the grammars. >>> >>> I would suggest to rmeove the `tree-sitter-grammar' function, and keep >>> grammars as "regular" package records, even though it's a little bit >>> more verbose: >>> >>> --8<---------------cut here---------------start------------->8--- >>> (define-public tree-sitter-html >>> (package >>> (name "tree-sitter-html") >> >> It seems tree-sitter-html mimics upstream package name and probably make >> more sense than tree-sitter-grammar-html used by me. > > Yeah, at some point I think I had named the packages with "grammar" as > well, but thought it was a bit of a mouthful. I'm also thinking one day > we may build language bindings as part of the build system (Rust and > NodeJS I think ATM), so those packages could do more than ship the > grammar in the future (although we don't know if we'll ever really need > that). > >> >>> (version "0.19.0") >>> (source (origin >>> (method git-fetch) >>> (uri (git-reference >>> (url "https://github.com/tree-sitter/tree-sitter-ht= ml") >>> (commit (string-append "v" version)))) >>> (file-name (git-file-name name version)) >>> (sha256 >>> (base32 >>> "1hg7vbcy7bir6b8x11v0a4x0glvqnsqc3i2ixiarbxmycbgl3axy")) >>> (modules '((guix build utils))) >>> (snippet tree-sitter-delete-generated-files))) >>> (build-system tree-sitter-build-system) >>> (home-page "https://github.com/tree-sitter/tree-sitter-html") >>> (synopsis "Tree-sitter HTML grammar") >>> (description >>> "This package provides a HTML grammar for the Tree-sitter library.= ") >>> (license license:expat))) >>> --8<---------------cut here---------------end--------------->8--- >>> >>> This way, they look like any other package in Guix, which makes it >>> easier for us to apply automatic changes in the future if needed (for >>> example like how the input format could be automically updated for all >>> "simple" package definitions, but had to be manual whenever custom code >>> refactoring was done). Does this make sense? >> >> Make sense, but on the other hand we already have hunspell, aspell >> dictionaries and probably a few more others, which are very similiar in >> spirit and we already have to keep in mind their existence on such >> automatic code updates. >> >> It looks that the packages differ only in url for the source code, lang >> name and sometimes in inputs. Having template package function can make >> management of shared parts more centralized, reduce possibility of >> copy-paste mistakes, when the description wasn't updated and so on and >> can reduce the amount of the code overall (which also reduces the change >> of introducing an error). >> >> I don't have a strong opinion on this topic, but leaning towards the >> template function slightly more, however I'm completely ok with the >> standalone package definitions as well. WDYT? > > I can think of both cost/benefits to the template so I don't have a > strong opinion either :-). > > I do like the template to make sure people don't forget to delete > generated files, that's quite important as it seems upstream packages > often check-in the generated C code. Although, we could probably assert > that with in the build-system phase? I'll think about that. > > On the other hand, I wonder how the template works for packages that > provide multiple grammars (see ocaml and typescript for example). I > guess we could use the template for trivial packages, and standalone > definitions for more complex ones? In general, if we keep the template > interface really simple, then I'm happy with it. Hi Pierre! I spend two days trying grammars with and without helper function and found hepler quite helpful to reduce boilerplate and errors from copypaste, so I went the way with helper. The logic inside is quite trivial, the only downside I found so far is that in cases when repository url constructed automatically I can't easily open the repo url in the browser. I packaged all the grammars from this thread and a few more on top of it. Updated them to usually latest versions, added some comments, when needed. If I forgot to reply on something or you have any comments/ideas, let me know! :) Kudos to Pierre and everyone, who helped with all the tree-sitter stuff. =2D-=20 Best regards, Andrew Tropin --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEKEGaxlA4dEDH6S/6IgjSCVjB3rAFAmPri3IACgkQIgjSCVjB 3rC06g//XvLbq36ylpj8Um6MOevrYtlA45HNQGZwSQx1SYQmEcl5x+toCRlobD5Q KcQWhFHXr7Fz29fdhC4lFbI3mJNe0+RIwkRiG0rpA0X7Z90rbSfvzziSygg8voGu 4/k+vShxTJbbDcdA6WuscaidGW3ZLi2AOM/zD1EwPhgf4T6x0Heh4C5cVxsaunLU ydH36yz9/RXQHQR/yHfN74lMt43/+RRtHFgoz74ULreCOgYRIdbPF6Yr5UCbeYeN qbhsRoFZLm/RsTxkWisiw7+pjm1oLnfJc33gOd+F/n/LmBW4Lnup9VXJo/jiCgZR +50ZvMsDgJh+aYpng5kLtJHYJLtHTWNyukS0Xdx6gcZ6UP3GNJI2JN96dXRfjPQe DX0rgWRfWIJDcBtpsUq1oIVZDzjaNXo4TpMed2XVLNE+H4deF2UgzR9IwGnaWPbc Sw7cdcugMy9CxBYanz79EUtrzt5J20f4xE6uGYkXrSCwkdpgMscx3wO1i03QOCcA AxvsALkayQrRSAR7qTuZBTPF57pjQedZgP1aKft6BkHN8PPcz7DKAerzcfDdxdTt 6iV932tmR4r2S7JeU9fF8q/7R7vtPLES4lxZivPnmrzN1d6tAvmTDhFQVEqXGTbS xsjeSAxuzQvcOWioRDVxlmcZtfZBr2kZDBv+wOL5+yVWUZ13SQY= =LF4M -----END PGP SIGNATURE----- --=-=-=--