From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.ciao.gmane.io!not-for-mail From: Stephen Leake Newsgroups: gmane.emacs.devel Subject: Re: Reliable after-change-functions (via: Using incremental parsing in Emacs) Date: Fri, 03 Apr 2020 09:45:44 -0800 Message-ID: <868sjcfoon.fsf@stephe-leake.org> References: <83369o1khx.fsf@gnu.org> <83imijz68s.fsf@gnu.org> <831rp7ypam.fsf@gnu.org> <86wo6yhj4d.fsf@stephe-leake.org> <83r1x6x8df.fsf@gnu.org> <86tv21fgls.fsf@stephe-leake.org> <83zhbtvwsm.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="ciao.gmane.io:159.69.161.202"; logging-data="39069"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (windows-nt) To: emacs-devel Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Apr 03 19:46:59 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jKQPC-000A2E-Rm for ged-emacs-devel@m.gmane-mx.org; Fri, 03 Apr 2020 19:46:58 +0200 Original-Received: from localhost ([::1]:58986 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jKQPB-0000iP-S5 for ged-emacs-devel@m.gmane-mx.org; Fri, 03 Apr 2020 13:46:57 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:44019) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jKQO9-0008Vz-HJ for emacs-devel@gnu.org; Fri, 03 Apr 2020 13:45:54 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jKQO7-0002Qa-BW for emacs-devel@gnu.org; Fri, 03 Apr 2020 13:45:52 -0400 Original-Received: from gateway24.websitewelcome.com ([192.185.50.93]:13222) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jKQO7-0002P3-2J for emacs-devel@gnu.org; Fri, 03 Apr 2020 13:45:51 -0400 Original-Received: from cm11.websitewelcome.com (cm11.websitewelcome.com [100.42.49.5]) by gateway24.websitewelcome.com (Postfix) with ESMTP id 507E241A67 for ; Fri, 3 Apr 2020 12:45:50 -0500 (CDT) Original-Received: from host2007.hostmonster.com ([67.20.76.71]) by cmsmtp with SMTP id KQO5jfghdSl8qKQO6jAzZT; Fri, 03 Apr 2020 12:45:50 -0500 X-Authority-Reason: nr=8 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=stephe-leake.org; s=default; h=Content-Type:MIME-Version:Message-ID: In-Reply-To:Date:References:Subject:To:From:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=PAjXYfhdOv+vXxtox9B9k550YUtb7asb/ofn/uZdBfI=; b=NG2fA5A9KvRcbKZbYchOUX+lo Sh7twiDJrdABa1kUiIQC7WDTKbuHYcKB2KZTO/AiZFMdC4kx4qbBXHaL97hCHfrJ35pYR3MmFv+D/ cI9zRfzNpiF2sNEu6h3rSkW5aEQL1E6GN7ju+mtK6uyJhLW8/CSTVlzIL8EgIUBh/SDCbvTsOhaPb 3sdYAfkb5hwYjrZFkdLdZYywh16eSNpxrArQHv5qJZc2Cnvz05HzkgyIrZqnxJ2D39nNkALPfjm/5 PdCpylGgDpbZEVg4xLc6l0/+tgqhXchaWBoaznnYSijFqRkmMx0vV3FT4IcrsOys3XIdgOQCktilB PXIHFwdYw==; Original-Received: from [76.77.182.20] (port=64216 helo=Takver4) by host2007.hostmonster.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92) (envelope-from ) id 1jKQO5-000JMy-Os for emacs-devel@gnu.org; Fri, 03 Apr 2020 11:45:49 -0600 In-Reply-To: <83zhbtvwsm.fsf@gnu.org> (Eli Zaretskii's message of "Fri, 03 Apr 2020 10:43:53 +0300") X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - host2007.hostmonster.com X-AntiAbuse: Original Domain - gnu.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - stephe-leake.org X-BWhitelist: no X-Source-IP: 76.77.182.20 X-Source-L: No X-Exim-ID: 1jKQO5-000JMy-Os X-Source-Sender: (Takver4) [76.77.182.20]:64216 X-Source-Auth: stephen_leake@stephe-leake.org X-Email-Count: 4 X-Source-Cap: c3RlcGhlbGU7c3RlcGhlbGU7aG9zdDIwMDcuaG9zdG1vbnN0ZXIuY29t X-Local-Domain: yes X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 192.185.50.93 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:246343 Archived-At: Eli Zaretskii writes: >> From: Stephen Leake >> Date: Thu, 02 Apr 2020 18:27:59 -0800 >> >> > Such copying is not really scalable, and IMO should be avoided. >> > During active editing, redisplay runs very frequently, and having to >> > copy portions of the buffer, let alone all of it, each time, which >> > necessarily requires memory allocation, consing of Lisp objects, etc., >> > will produce significant memory pressure, expensive heap >> > allocations/deallocations, and a lot of GC. Recall that on many >> > modern platforms Emacs doesn't really return memory to the system, >> > which means we risk increasing the memory footprint, and create >> > system-wide memory pressure. It isn't a catastrophe, but we should >> > try to avoid it if possible. >> >> Ok. I know very little about the internal storage of text in Emacs. >> There is at least two strings with a gap at the current edit point; if >> we pass a simple pointer to tree-sitter, it will have to handle the gap. > > Tree-sitter allows the application to define a "reader" function that > it will then call to access buffer text. That function should cope > with the gap. and also with the encoding, which you did not address. I don't see how that is different from the C level buffer-substring. Certainly there should be a module function buffer-substring that is as efficient as possible. >> You mention "consing of Lisp objects" above, which says to me that the >> text is stored in a more complex structure. > > I meant the consing that is necessary to make a buffer-substring that > will be passed to the parser. Since are are calling the parser from C (if it is linked into Emacs, or in a module), I still don't understand. Does C code have to cons to create a string? It will have to allocate if the requested range is not contiguous in the buffer. >> Avoid _all_ copying is impossible; the parser must store the contents of >> each token in some way. Typically that is done by storing >> pointers/indices into the text buffer that contains the entire text. > > I don't think tree-sitter does that, because the text it gets is > ephemeral. If we pass it a buffer-substring, it's a temporary string > which will be GCed after it's used; if we pass it pointers to buffer > text, those pointers can be invalid after GC, because GC can relocate > buffer text to a different memory region. Hmm. https://tree-sitter.github.io/tree-sitter/using-parsers#providing-the-code says: Syntax nodes store their position in the source code both in terms of raw bytes and row/column coordinates In the case of passing a pointer to a string (or buffer, etc), those positions are relative to that original buffer. So the Emacs buffer is serving as the parse buffer. Ok, that avoids any copying. If we pass a buffer-substring to the parser, we are then responsible for mapping positions relative to the substring into positions relative to the full buffer. wisi delegates that to the parser; it can pass start-char-pos and start-byte-pos to the parser along with a string. >> >> In sum, the short answer is "yes, you must parse the whole file, unless >> >> your language is particularly simple". >> > >> > Funny, my conclusion from reading your detailed description was >> > entirely different. >> >> I need more than that to respond in a helpful way. > > Well, you said: > >> To some extent, that depends on the language. > > and then went on to describing how each language might _not_ need a > full parse in many cases. Thus the conclusion sounded a bit radical > to me. Ok, we are putting different spins on what "particularly simple" means. A more neutral phrasing would be: Some languages require parsing the whole file, some do not. -- -- Stephe