From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.ciao.gmane.io!not-for-mail From: Stephen Leake Newsgroups: gmane.emacs.devel Subject: Re: Reliable after-change-functions (via: Using incremental parsing in Emacs) Date: Thu, 02 Apr 2020 18:27:59 -0800 Message-ID: <86tv21fgls.fsf@stephe-leake.org> References: <83369o1khx.fsf@gnu.org> <83imijz68s.fsf@gnu.org> <831rp7ypam.fsf@gnu.org> <86wo6yhj4d.fsf@stephe-leake.org> <83r1x6x8df.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="ciao.gmane.io:159.69.161.202"; logging-data="115557"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (windows-nt) To: emacs-devel Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Apr 03 04:28:49 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jKC4f-000Tyj-LK for ged-emacs-devel@m.gmane-mx.org; Fri, 03 Apr 2020 04:28:49 +0200 Original-Received: from localhost ([::1]:49436 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jKC4e-0006tC-Kl for ged-emacs-devel@m.gmane-mx.org; Thu, 02 Apr 2020 22:28:48 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:34251) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jKC3z-0006Q9-S0 for emacs-devel@gnu.org; Thu, 02 Apr 2020 22:28:08 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jKC3y-0003up-FY for emacs-devel@gnu.org; Thu, 02 Apr 2020 22:28:07 -0400 Original-Received: from gateway21.websitewelcome.com ([192.185.45.140]:28990) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jKC3y-0003uB-7E for emacs-devel@gnu.org; Thu, 02 Apr 2020 22:28:06 -0400 Original-Received: from cm16.websitewelcome.com (cm16.websitewelcome.com [100.42.49.19]) by gateway21.websitewelcome.com (Postfix) with ESMTP id 08C89400CAC9C for ; Thu, 2 Apr 2020 21:28:05 -0500 (CDT) Original-Received: from host2007.hostmonster.com ([67.20.76.71]) by cmsmtp with SMTP id KC3wjkEXc8vkBKC3wjYjl3; Thu, 02 Apr 2020 21:28:05 -0500 X-Authority-Reason: nr=8 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=stephe-leake.org; s=default; h=Content-Type:MIME-Version:Message-ID: In-Reply-To:Date:References:Subject:To:From:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=Of71ReB9iubmbVCaZVo4HBoVNvdkU0/RMNaI08yAr40=; b=nYFalASZIM/gU2g9r9oSS5J0E a5BtHuOU2yc8tYr8OwF59dtVRMh5CuGlBPmQMKh0M2QHd+TTrZU1AuKF65/OnwKF/Zvr5VLMSZ8oI st3ANN0JP6JwE0xFy6iRe0RgKIOlO1hfbjBR3fxH1vdMvfq/ryaCR0FZ6ozr3syNaToAcQcktqkEm MZSvDEHloaBbzUFkAfTQc1WSKf/MB4CNsfOXVOLlwsx1U/XMN4NZBZgcreLbFrgSbcHak1FF3Ougu YR9dEG7VUTXaOqPofRIno3Zz8TjSKsNS7/TsPTw1856T5kOBtWtonwBcItO+9T3DO45RuD3CzK9eq Vd6J6qDxw==; Original-Received: from [76.77.182.20] (port=63754 helo=Takver4) by host2007.hostmonster.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92) (envelope-from ) id 1jKC3w-0032lW-BR for emacs-devel@gnu.org; Thu, 02 Apr 2020 20:28:04 -0600 In-Reply-To: <83r1x6x8df.fsf@gnu.org> (Eli Zaretskii's message of "Thu, 02 Apr 2020 17:36:12 +0300") X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - host2007.hostmonster.com X-AntiAbuse: Original Domain - gnu.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - stephe-leake.org X-BWhitelist: no X-Source-IP: 76.77.182.20 X-Source-L: No X-Exim-ID: 1jKC3w-0032lW-BR X-Source-Sender: (Takver4) [76.77.182.20]:63754 X-Source-Auth: stephen_leake@stephe-leake.org X-Email-Count: 3 X-Source-Cap: c3RlcGhlbGU7c3RlcGhlbGU7aG9zdDIwMDcuaG9zdG1vbnN0ZXIuY29t X-Local-Domain: yes X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 192.185.45.140 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:246297 Archived-At: Eli Zaretskii writes: >> From: Stephen Leake >> Date: Wed, 01 Apr 2020 15:38:26 -0800 >> >> Eli Zaretskii writes: >> >> > Also, direct access to buffer text generally means we must make sure >> > GC never runs as long as pointers to buffer text are lying around. >> > Can any Lisp run between calls to the reader function that the >> > tree-sitter parser calls to access the buffer text? >> >> If the parser copies the text into an internal buffer, that reader >> function should only be called once per call to the parser. > > Such copying is not really scalable, and IMO should be avoided. > During active editing, redisplay runs very frequently, and having to > copy portions of the buffer, let alone all of it, each time, which > necessarily requires memory allocation, consing of Lisp objects, etc., > will produce significant memory pressure, expensive heap > allocations/deallocations, and a lot of GC. Recall that on many > modern platforms Emacs doesn't really return memory to the system, > which means we risk increasing the memory footprint, and create > system-wide memory pressure. It isn't a catastrophe, but we should > try to avoid it if possible. Ok. I know very little about the internal storage of text in Emacs. There is at least two strings with a gap at the current edit point; if we pass a simple pointer to tree-sitter, it will have to handle the gap. You mention "consing of Lisp objects" above, which says to me that the text is stored in a more complex structure. How can we provide direct access of that to tree-sitter? Avoid _all_ copying is impossible; the parser must store the contents of each token in some way. Typically that is done by storing pointers/indices into the text buffer that contains the entire text. >> In sum, the short answer is "yes, you must parse the whole file, unless >> your language is particularly simple". > > Funny, my conclusion from reading your detailed description was > entirely different. I need more than that to respond in a helpful way. >> In general, each parser library, and even each grammar author, will have >> different representations for the syntax tree. >> >> So if we want to support different parsers, I think it is best to define >> the Emacs "parser API" as "give text to parser; accept text properties >> from parser". > > Yes, something like that. It's probably enough to accept a list of > regions with syntactic attributes. Ok, good. -- -- Stephe