From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.devel Subject: Re: Tree-sitter integration in python.el Date: Tue, 27 Sep 2022 15:16:14 -0700 Message-ID: <20432C20-8FB9-44D9-A91C-ACEEAE1CAD09@gmail.com> References: <3A68545C-2E40-4BB4-8563-8041A5452634@gmail.com> <761f064e-3cbf-5ca7-2edb-4de6670345bf@secure.kjonigsen.net> Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="15271"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel To: jostein@kjonigsen.net Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Wed Sep 28 00:18:50 2022 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1odIuf-0003m0-Rh for ged-emacs-devel@m.gmane-mx.org; Wed, 28 Sep 2022 00:18:49 +0200 Original-Received: from localhost ([::1]:32956 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1odIud-0000QF-P3 for ged-emacs-devel@m.gmane-mx.org; Tue, 27 Sep 2022 18:18:47 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:51860) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1odIsH-0007Jp-4s for emacs-devel@gnu.org; Tue, 27 Sep 2022 18:16:21 -0400 Original-Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]:45924) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1odIsF-0004es-8E for emacs-devel@gnu.org; Tue, 27 Sep 2022 18:16:20 -0400 Original-Received: by mail-pl1-x62e.google.com with SMTP id w20so10265387ply.12 for ; Tue, 27 Sep 2022 15:16:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date; bh=9jLlSgH3WfPvWmeBQzovgZyIPtxTfojSV4uMS0WDNfU=; b=hhRhUBCOsWgrm1oCvDuS9PD0dhk19LYxYAoa6xqOJpF6JH6lDV1tfcGZ5h9R6338Kh DG/3rqC7PEIa9W31LTHyggiZqXiXBNbOWhh0QS8U6eVQJRAXrJ89OPzFj67sCoWR9Xby ZJQunvcYrte9nP2w7GPJI5BOn4mFpT/CxNq23CKgbgqGENeqkYAZa4jghoSYkDbLuVyN dJVunw2u52xS/XY6OgXEB+MTIJdl3jAq/VYW4+1t1+Zi1MgXT2gK3/ZcFFWDQCeNid7u wLrmxNKkrQd7v9DpoSSgnwVEoYZvXuhDU/0N3yankQ/B/R2/PjvX4fvA7yEMefQwP4ov NI5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date; bh=9jLlSgH3WfPvWmeBQzovgZyIPtxTfojSV4uMS0WDNfU=; b=E3gLK2WBqjhv9YSdppIx5w4H3UeqO8DTkB2BxbCP1OXXYJZ2E1/bStTx2hS87eV+Rv KAg7VsQWCl7NkuykIBKFhql95L/6AIK8obYLTxE4DjniWGIfnD9uepatYpi1FZdXuwP3 mOeus6z4pT88s/YtsS0KOzmKca5H82J21tlxBrynvuEYy8Fh7Y9+7eyHzAJu6ZxI2Xm/ NEcPOMYnXPngL9VdfmaBRFFBBIDGylGlSm4a9LAVCzr9vFxw4xHENLFiIe0P0fVDoTRx udBUZQOYwfbEWK8mDlpI59Ri0Uuv/YVS4NDlnD5lznIHJrbss9my7WpFmdT7YdoU0Inh FmlA== X-Gm-Message-State: ACrzQf0e4zoe5cH06ZeLxgUemf6vqQm5fRjW7dLk3NLDAA/5zCcUtsPL TxM7WR3m5b5g73XRfdX/WgY= X-Google-Smtp-Source: AMsMyM6oYFTusOM2q+g3iccuOBeDmSztSAnVBqfUfouaMlUCsdj0ZxHYlUwqoizqhE0BkmWb7x90nA== X-Received: by 2002:a17:902:ecc5:b0:178:3217:e464 with SMTP id a5-20020a170902ecc500b001783217e464mr29256180plh.75.1664316977131; Tue, 27 Sep 2022 15:16:17 -0700 (PDT) Original-Received: from smtpclient.apple (cpe-172-117-161-177.socal.res.rr.com. [172.117.161.177]) by smtp.gmail.com with ESMTPSA id y26-20020aa79e1a000000b00537daf64e8esm2296616pfq.188.2022.09.27.15.16.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Sep 2022 15:16:16 -0700 (PDT) In-Reply-To: <761f064e-3cbf-5ca7-2edb-4de6670345bf@secure.kjonigsen.net> X-Mailer: Apple Mail (2.3696.120.41.1.1) Received-SPF: pass client-ip=2607:f8b0:4864:20::62e; envelope-from=casouri@gmail.com; helo=mail-pl1-x62e.google.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, FREEMAIL_REPLY=1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:296373 Archived-At: > On Sep 26, 2022, at 12:10 PM, Jostein Kj=C3=B8nigsen = wrote: >=20 > On 22.09.2022 20:42, Yuan Fu wrote: >> Hi,=20 >>=20 >> I=E2=80=99ve added tree-sitter version for font-lock and which-func = in python.el. And I=E2=80=99d love to hear some feedback from python.el = maintainers. Specifically, does it look right and which other part of = python.el could actually benefit from a parse tree? I wrote a = tree-sitter imenu indexer for python and it performed worse than the = current one, presumably because it traverses the whole parse tree = whereas the current one only scans the buffer once or so and do some = regex matching. >>=20 >> Here is the commit:=20 >> = https://git.savannah.gnu.org/cgit/emacs.git/commit/?h=3Dfeature/tree-sitte= r&id=3D1cdb24fe35a9ff2e4f92c5acc93a5a5b0e70d93f >>=20 >>=20 >> Yuan >>=20 >>=20 >>=20 > Hey Yuan. >=20 > Thanks for putting in all this work into tree-sitter in Emacs! >=20 > Trying the latest Emacs-version in feature/tree-sitter I got an error = I tend to get "a lot" with tree-sitter based modes, which I hoped = bundling things with Emacs would solve, namely obtaining the original = shared-object containing the compiled grammer. >=20 > Like for your python-mode, I get this: >=20 > File mode specification error: (treesit-load-language-error python = (/home/jostein/.emacs.d/tree-sitter/libtree-sitter-python: cannot open = shared object file: No such file or directory = /home/jostein/.emacs.d/tree-sitter/libtree-sitter-python.so: cannot open = shared object file: No such file or directory libtree-sitter-python: = cannot open shared object file: No such file or directory = libtree-sitter-python.so: cannot open shared object file: No such file = or directory)) >=20 > I realize third-party modes are on their own, but when tree-sitter is = compiled with Emacs, I would at least expect the Emacs-build to also = produce these .so-files. >=20 > What are your thoughts on how we can best, across the Emacs-verse, = provide these libraries? Or at least for the modes which are bundled = with Emacs itself? >=20 > Being a tree-sitter based-developer myself, I know where I can go to = get this compiled to make the mode runnable, but surely that's not how = we can deploy this en-masse. Most people will be stuck at this point, = and we will need to come up with a better answer. >=20 > --=20 > Kind regards > Jostein Kj=C3=B8nigsen >=20 > jostein@kjonigsen.net =F0=9F=8D=B5 jostein@gmail.com > https://jostein.kj=C3=B8nigsen.no This has been discussed before and our conclusion is to treat language = definitions like other dynamic libraries, expecting package manager to = install them. There are a couple of reasons. Tree-sitter library and = language definitions must be on the same =E2=80=9Cprotocol=E2=80=9D = version to work. I expect this version to change slowly, but in = principle we don=E2=80=99t want to bundle a language definition that = could conflict with the tree-sitter library provided by the system. = Also, as a dynamic object file, it is machine-dependent. I don=E2=80=99t = know if we bundle anything machine-dependent in Emacs distribution, but = at any rate it is unusual.=20 I do have some ideas to make it easier for users, like hosting them on = ELPA or NONGNU ELPA, and user can install them by package-install. Eg, a = package tree-sitter-installer-linux, and M-x tree-sitter-installer-linux = RET c RET will install C language definition for you. Stefan, WDYT? Yuan=