From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.devel Subject: Re: Tree-sitter api Date: Sat, 18 Dec 2021 23:52:35 -0800 Message-ID: <9C5A86D6-0E7D-4DDF-B211-278EF9AC7E01@gmail.com> References: <83r1f7hydn.fsf@gnu.org> <83lf3zdh4z.fsf@gnu.org> <8965C4A0-79D3-4D77-A6BA-D07A6C93F7FE@gmail.com> <83ilz3cs4k.fsf@gnu.org> <04D19C1A-CD64-4156-B932-1C9FEEE4EC7B@gmail.com> <83zgsebc0r.fsf@gnu.org> <1F752923-F357-4A18-B6E2-0120F1B9BD37@gmail.com> <83fsu5bzem.fsf@gnu.org> <83zgsdad5j.fsf@gnu.org> <83sfy391ni.fsf@gnu.org> <03386E3C-A975-4ECD-BF89-6AC62F751725@gmail.com> <83ilyz8xdl.fsf@gnu.org> <86czp6ukpj.fsf@stephe-leake.org> <1AEED8A5-BF5E-45E5-AE22-9B405A710F9B@gmail.com> <83czp313tf.fsf@gnu.org> <7447C582-70CB-4599-BC76-AD5687D64250@gmail.com> <784947A6-6164-4158-993B-01E881690ADF@gmail.com> <83zgrxji7q.fsf@gnu.org> <8E8D28F9-258E-490B-BF48-7EF8D6CA0817@gmail.com> <838rzgk6ll.fsf@gnu.org> <83k0g8mznc.fsf@gnu.org> <83tuf69d06.fsf@gnu.org> <83czlt9ii5.fsf@gnu.org> Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.40.0.1.81\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33043"; mail-complaints-to="usenet@ciao.gmane.io" Cc: =?utf-8?B?VHXhuqVuLUFuaCBOZ3V54buFbg==?= , Theodor Thornhill , =?utf-8?Q?Cl=C3=A9ment_Pit-Claudel?= , Emacs developers , Philipp , Stefan Monnier , Stephen Leake , john@yates-sheets.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Dec 19 08:53:45 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1myr0r-0008QX-06 for ged-emacs-devel@m.gmane-mx.org; Sun, 19 Dec 2021 08:53:45 +0100 Original-Received: from localhost ([::1]:48130 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1myr0p-00072D-Q3 for ged-emacs-devel@m.gmane-mx.org; Sun, 19 Dec 2021 02:53:43 -0500 Original-Received: from eggs.gnu.org ([209.51.188.92]:47446) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1myqzo-0006LK-VT for emacs-devel@gnu.org; Sun, 19 Dec 2021 02:52:41 -0500 Original-Received: from [2607:f8b0:4864:20::f31] (port=42587 helo=mail-qv1-xf31.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1myqzn-0007S1-Fs; Sun, 19 Dec 2021 02:52:40 -0500 Original-Received: by mail-qv1-xf31.google.com with SMTP id p3so6544944qvj.9; Sat, 18 Dec 2021 23:52:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=xvx6zfwbkp51PfOvvArrOxebkeBuKa1J5KPypHwzIVU=; b=IR/Tl49wpUptwidoBoFAOd5utMGQwt1IwMh8C01SWE0poc9yV8EOdHHWs1a7D/i6QR /KJD3AHbQ8b8CKBHBP9R63Qk28lMxH3qL4PLQjROfjMy1cXIFlZ5ahdnFLC4U+mn345V MBe3ddHTVMlQtPw1nbbJM0QlDdQFaYrDgNhmo8uofHpFYDERDNrjoGgyzv+4YtpyMS0N TNWBaz/VCya3kv2YNUbMPMNLmZH3y1bckfNddlWjfSZLCVikxxiooiIQT3ZMzloUX6kY ZSIUSFXI7DWQgc0oh9vymh3bvNQXPOysYn7JAJ3oLOYjHy2yoj6XoBgEtzbUOBK6IihO 3HVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=xvx6zfwbkp51PfOvvArrOxebkeBuKa1J5KPypHwzIVU=; b=t3KbsIjtITW5BJs/coknuBi+/K4QPTfmtSlKGfSKv4Pu/MDctvmkFjMziUPzhqAdP1 WWDXytnOpHATDqScfKKOijPAeWw5MzixZQQaJv+1SKqA/JrGKaQWLyinojB5XgY2T86v e3Q05qsDLjE9/GgiLJoMYsTXcgiwYX8IpudHwFFvV9Y90W2VGdHSDZDTZD0R7QWYGIjD zgrtLh8hZsluQxTfF+KgxkemKTFifKIrVOIObiRm4/p7BcCifsJATBV4EAm+guRkq8u5 jxrXYVSzIrZnpQehCGeOU8pBWqJ44cThb7po8vN03TDDxsT8XDGAr0G+4Uaib8ONYvEA WtaA== X-Gm-Message-State: AOAM5304KaiFJfmX8tdfrOg5RQfaVChHJUjoLhdswUd8XhyGNgYZmzEm LC5qPPzq8vuRjcXIWtqmMlKWy+yYjWXCRA== X-Google-Smtp-Source: ABdhPJwYhmBxxRBjIAryjjncHn2OvTHucSjzKgENyx/ba+jpEMuncJTDT5J5+33JnrZIEbzzCHmE6w== X-Received: by 2002:ad4:5ba4:: with SMTP id 4mr1156110qvq.100.1639900357726; Sat, 18 Dec 2021 23:52:37 -0800 (PST) Original-Received: from smtpclient.apple ([2600:1700:2ec7:8c90:5407:cb3:bd28:ba6a]) by smtp.gmail.com with ESMTPSA id v21sm11514212qta.0.2021.12.18.23.52.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 18 Dec 2021 23:52:37 -0800 (PST) In-Reply-To: <83czlt9ii5.fsf@gnu.org> X-Mailer: Apple Mail (2.3693.40.0.1.81) X-Host-Lookup-Failed: Reverse DNS lookup failed for 2607:f8b0:4864:20::f31 (failed) Received-SPF: pass client-ip=2607:f8b0:4864:20::f31; envelope-from=casouri@gmail.com; helo=mail-qv1-xf31.google.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:282374 Archived-At: > On Dec 18, 2021, at 11:11 PM, Eli Zaretskii wrote: >=20 >> From: Yuan Fu >> Date: Sat, 18 Dec 2021 18:51:25 -0800 >> Cc: Philipp , >> ubolonton@gmail.com, >> theo@thornhill.no, >> cpitclaudel@gmail.com, >> emacs-devel@gnu.org, >> monnier@iro.umontreal.ca, >> stephen_leake@stephe-leake.org, >> john@yates-sheets.org >>=20 >>>> That feature in itself won't be enough. Even with it, TreeSitter = will have the same problem as GMP: allocation isn't allowed to fail, and = longjmp'ing out of it isn't allowed and generally causes undefined = behavior. >>>=20 >>> It may not be enough to satisfy purists, but it's enough to allow = the >>> user to save the session and shut down Emacs in an orderly fashion, >>> instead of abruptly exiting and losing all the edits. >>=20 >> Uses can set tree-sitter-maximum-size to limit memory usage of = tree-sitter. Buffers with size larger than that cannot enable = tree-sitter. That doesn=E2=80=99t solve the problem directly but should = let users avoid allocation failing most of the time. >=20 > Btw, we should have a good idea how frequent this out-of-memory > problem could be with tree-sitter. Did someone try to scroll through > all of xdisp.c, using tree-sitter for C Mode fontifications, and > measured the memory footprint that produces? If not, I think it would > be a good idea to try. >=20 > If the OOM problem happens frequently with large source files, it may > indeed be the case that we will need to disable tree-sitter up front > based on some size criteria. =46rom the author=E2=80=99s quote and my experiments, tree-sitter uses = about 10=E2=80=9320x memory of the buffer size. So xdisp.c is fine. Also = you don=E2=80=99t need to scroll through the buffer, tree-sitter parses = the whole buffer up-front. Yuan=