From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.ciao.gmane.io!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: Re: Using incremental parsing in Emacs (via: emacs rendering comparisson between emacs23 and emacs26.3) Date: Sun, 29 Mar 2020 14:46:01 -0400 Message-ID: Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="ciao.gmane.io:159.69.161.202"; logging-data="36694"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Mar 29 20:47:01 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jIcxZ-0009Sa-Cd for ged-emacs-devel@m.gmane-mx.org; Sun, 29 Mar 2020 20:47:01 +0200 Original-Received: from localhost ([::1]:40392 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jIcxY-0007Ee-F8 for ged-emacs-devel@m.gmane-mx.org; Sun, 29 Mar 2020 14:47:00 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:59242) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jIcwm-0006ZE-5e for emacs-devel@gnu.org; Sun, 29 Mar 2020 14:46:13 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jIcwk-0007Wc-2o for emacs-devel@gnu.org; Sun, 29 Mar 2020 14:46:11 -0400 Original-Received: from mailscanner.iro.umontreal.ca ([132.204.25.50]:21574) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jIcwi-0007Qs-4c; Sun, 29 Mar 2020 14:46:08 -0400 Original-Received: from pmg3.iro.umontreal.ca (localhost [127.0.0.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id 229E144F9F6; Sun, 29 Mar 2020 14:46:05 -0400 (EDT) Original-Received: from mail01.iro.umontreal.ca (unknown [172.31.2.1]) by pmg3.iro.umontreal.ca (Proxmox) with ESMTP id 85F3344F800; Sun, 29 Mar 2020 14:46:03 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iro.umontreal.ca; s=mail; t=1585507563; bh=zdh14mcyJ4um9wQgYEvgknZFp5K6avUTLK2O6Zq2jdQ=; h=From:To:Cc:Subject:Date:From; b=Wxbxcjas5uZvTSnPGanL98nLRWB/mxlWa+13k5Az0RYFr+rD3S50Qe9y+wlESbJVW Nxer5k4XIdDE3O8pyKFhG+fP2rq6N7A31BL/tz0ytBeRx8+TjFaGtxpTUW+33R3yd/ V7MmfPRmQIpxvWTsKXXoaG2O6Ec0nxX8T5Vk1DW1IzAR0Co5nHinN4YnoWoxTMr7Yf TmEf0sYVyhznDiOkkYvrGjvBs4+LGcpQvf0MIKjcl5NriGlrlsaAwLhkQ9WpcSMN2K XDPOtUpkOed4fTrVBnZQmP98aIQy3o5omDPGLznSXikVV5YucW1r89wyEkaerrCJjS gZleJFiTto+KA== Original-Received: from alfajor (unknown [104.247.241.114]) by mail01.iro.umontreal.ca (Postfix) with ESMTPSA id 432F31208C1; Sun, 29 Mar 2020 14:46:03 -0400 (EDT) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 132.204.25.50 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:245959 Archived-At: >> tree-sitter, like LSP, is something Emacs should embrace. > https://lists.gnu.org/archive/html/emacs-devel/2020-01/msg00059.html Ah, thanks Eli: I guess I skipped over that while catching up. > Would someone like to try to figure out how we could use the > incremental parsing technology in Emacs for making our > programming-language support more accurate and efficient? One package > that implements this technology is tree-sitter: > > https://tree-sitter.github.io/tree-sitter/ Yes, adding support for this would be great. > AFAIU, these capabilities could be used as an alternative to > regexp- and syntax-pps-based font-lock, better code folding, > completion, refactoring, and other similar features; in general, any > feature which would benefit from having a parse tree for the source > code in a buffer. Some of those features could be provided by LSP as well, but IIUC the way LSP is designed and usually used makes it somewhat inadequate for synchronous use, when you want an immediate answer. tree-sitter is designed exactly for that: it can parse "immediately", in the same sense as `syntax-ppss`, so LSP seems inapplicable (in the near future at least) for things like font-lock and navigation, and indentation, whereas tree-sitter should work great for that. [ W.r.t disucssions around LSP's use of JSON: AFAICT, parsing and emitting json can be done as efficiently as any other format, AFAICT, so I don't see the use of JSON as a problem in the protocol. ] > To be able to use such libraries, we need to figure out how to > integrate them into the core, what kind of interfaces would be needed > for that, and what kind of infrastructure we would need for basing > Lisp features on those libraries. The existing third party packages should be good starting points to come up with a design. But I think an important issue is to figure out how to make tree-sitter usable for the end users: AFAICT the main issue being how to let end users download and install new grammars. IIUC grammars are written in Javascript (or some subset thereof?) and then somehow compiled to C code. Having them as C code implies either the end-user need to have a C compiler or distributing pre-compiled binaries with all the trouble this entails (with all the variations of OSes, and architectures, and ABIs, ..., plus issues related to licensing, security, ...). Maybe those grammars could be compiled to some other representation (I don't know if it is made mostly of data-tables or actual code or what)? Stefan