From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.ciao.gmane.io!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: Reliable after-change-functions (via: Using incremental parsing in Emacs) Date: Fri, 03 Apr 2020 10:43:53 +0300 Message-ID: <83zhbtvwsm.fsf@gnu.org> References: <83369o1khx.fsf@gnu.org> <83imijz68s.fsf@gnu.org> <831rp7ypam.fsf@gnu.org> <86wo6yhj4d.fsf@stephe-leake.org> <83r1x6x8df.fsf@gnu.org> <86tv21fgls.fsf@stephe-leake.org> Injection-Info: ciao.gmane.io; posting-host="ciao.gmane.io:159.69.161.202"; logging-data="18228"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel@gnu.org To: Stephen Leake Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Apr 03 09:45:17 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jKH0r-0004ar-Pg for ged-emacs-devel@m.gmane-mx.org; Fri, 03 Apr 2020 09:45:13 +0200 Original-Received: from localhost ([::1]:51476 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jKH0q-0008KS-Q1 for ged-emacs-devel@m.gmane-mx.org; Fri, 03 Apr 2020 03:45:12 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:40122) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jKGzp-0007qL-Lu for emacs-devel@gnu.org; Fri, 03 Apr 2020 03:44:10 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]:47752) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1jKGzo-0000vU-Ui; Fri, 03 Apr 2020 03:44:08 -0400 Original-Received: from [176.228.60.248] (port=1687 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jKGzo-0003My-D2; Fri, 03 Apr 2020 03:44:08 -0400 In-Reply-To: <86tv21fgls.fsf@stephe-leake.org> (message from Stephen Leake on Thu, 02 Apr 2020 18:27:59 -0800) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:246308 Archived-At: > From: Stephen Leake > Date: Thu, 02 Apr 2020 18:27:59 -0800 > > > Such copying is not really scalable, and IMO should be avoided. > > During active editing, redisplay runs very frequently, and having to > > copy portions of the buffer, let alone all of it, each time, which > > necessarily requires memory allocation, consing of Lisp objects, etc., > > will produce significant memory pressure, expensive heap > > allocations/deallocations, and a lot of GC. Recall that on many > > modern platforms Emacs doesn't really return memory to the system, > > which means we risk increasing the memory footprint, and create > > system-wide memory pressure. It isn't a catastrophe, but we should > > try to avoid it if possible. > > Ok. I know very little about the internal storage of text in Emacs. > There is at least two strings with a gap at the current edit point; if > we pass a simple pointer to tree-sitter, it will have to handle the gap. Tree-sitter allows the application to define a "reader" function that it will then call to access buffer text. That function should cope with the gap. > You mention "consing of Lisp objects" above, which says to me that the > text is stored in a more complex structure. I meant the consing that is necessary to make a buffer-substring that will be passed to the parser. > How can we provide direct access of that to tree-sitter? See above: by writing our function to access buffer text. > Avoid _all_ copying is impossible; the parser must store the contents of > each token in some way. Typically that is done by storing > pointers/indices into the text buffer that contains the entire text. I don't think tree-sitter does that, because the text it gets is ephemeral. If we pass it a buffer-substring, it's a temporary string which will be GCed after it's used; if we pass it pointers to buffer text, those pointers can be invalid after GC, because GC can relocate buffer text to a different memory region. They definitely do copy portions of the text they get for internal processing purposes, but I doubt that they duplicate all of it, because that would not be scalable to huge buffers. And in any case, any copying we do would be _in_addition_ to what tree-sitter does internally. > >> In sum, the short answer is "yes, you must parse the whole file, unless > >> your language is particularly simple". > > > > Funny, my conclusion from reading your detailed description was > > entirely different. > > I need more than that to respond in a helpful way. Well, you said: > To some extent, that depends on the language. and then went on to describing how each language might _not_ need a full parse in many cases. Thus the conclusion sounded a bit radical to me.