From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#60953: The :match predicate with large regexp in tree-sitter font-lock seems inefficient Date: Thu, 26 Jan 2023 20:07:30 +0200 Message-ID: <31559c1f-1a12-691d-3d03-f566019a0aab@yandex.ru> References: <7624dddc-4600-9a03-ac8b-d3c9e0ab618c@yandex.ru> <04729838-b7d4-8a08-2b71-12536a28aebb@yandex.ru> <83wn5ag4nc.fsf@gnu.org> <01b5d074-fb12-6b1f-cbfb-5e759833b854@yandex.ru> <838rhpg57n.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="18267"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Cc: casouri@gmail.com, 60953@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Jan 26 19:08:31 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pL6fl-0004an-Or for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 26 Jan 2023 19:08:29 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL6fN-0000rG-3q; Thu, 26 Jan 2023 13:08:05 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL6fK-0000p3-KU for bug-gnu-emacs@gnu.org; Thu, 26 Jan 2023 13:08:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pL6fK-0005Lk-CK for bug-gnu-emacs@gnu.org; Thu, 26 Jan 2023 13:08:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pL6fJ-0007Bb-U2 for bug-gnu-emacs@gnu.org; Thu, 26 Jan 2023 13:08:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 26 Jan 2023 18:08:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 60953 X-GNU-PR-Package: emacs Original-Received: via spool by 60953-submit@debbugs.gnu.org id=B60953.167475646027587 (code B ref 60953); Thu, 26 Jan 2023 18:08:01 +0000 Original-Received: (at 60953) by debbugs.gnu.org; 26 Jan 2023 18:07:40 +0000 Original-Received: from localhost ([127.0.0.1]:36142 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pL6ex-0007As-P0 for submit@debbugs.gnu.org; Thu, 26 Jan 2023 13:07:40 -0500 Original-Received: from mail-wm1-f52.google.com ([209.85.128.52]:51815) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pL6ew-0007Ac-7U for 60953@debbugs.gnu.org; Thu, 26 Jan 2023 13:07:38 -0500 Original-Received: by mail-wm1-f52.google.com with SMTP id fl24so1715489wmb.1 for <60953@debbugs.gnu.org>; Thu, 26 Jan 2023 10:07:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=M2pH88xOb9V43g6yrp2i59Thp/jKnq2pxz+qExBtkQQ=; b=OQABbPBv4fDrd+EREjWAvCYCcN/vfadA/FzVWW2j/orpJS93Lk3smf/13imzGake6Z KsNA33TghsspuQHZtYtb92QKLdIoP3lwgSe4a84DALTOBSOQFkjw7dXaDhB1QSD6456K 4+aDI9z8s1T+GeI2OJEXyzyIPWSPY4k8aO7/DMlIuAnpozvZoTmm482e8dgUjW48V2bI sZhpWYLYcBrrdrl+TCIHjGuzn7Jab6dcc7AmAIrnM9MKEgthW0AJgn0c06Eh/B70YpDF elLuIpUevCXWgLUY28jGrwbQ7xDxNPyFvV9wHHfuxmtoCgLbmU483f/iKboRtMQaC6Ay /BHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=M2pH88xOb9V43g6yrp2i59Thp/jKnq2pxz+qExBtkQQ=; b=vaFL+ZPRP6N6vjFCAaTuDhtqiy5gfidbQS8cpHNHYgJ1oHbUmLepBcD2tEqLhjuSfu 5Ienq9qQPpZ8aAUN9Q8cnad9H1qzQRk5MPSsnCgxopVx3ZVkBk3dLOeQBfGLOkO+a9O5 QPzrlxpBlSS/bGAy6YdvIUeWjYtfxZtLau+zO1AnPkoGIyb+FyAJ3M7h3/OjpETNhX7i +MSTh16uGfukf9/pVauoqhG8vhC6pJgHsCngx/7xIaZPN1meA5ZnfhlQL1+mGwsB4amD F+tagnJ+xZ67FWIh1gl1nT8e2V6HiEgSQLYlPCK/uOZpzrbm8ejANYBJJb20HOWt3dN5 apbw== X-Gm-Message-State: AFqh2koBdT+Imi6TPq6GAepXwIWbQ3UzmNa+qLKub3Ex5+TOnkszXeph M39daiSz8/s8mDwI9QMZWy8= X-Google-Smtp-Source: AMrXdXs9Lb31+THfr5sq3oJDwuXys4R0YMMcmnlct1OMEr9FDUJ3V9KMhZko+xPA0nJQ+7cI/tRj7Q== X-Received: by 2002:a05:600c:1e1d:b0:3cf:674a:aefe with SMTP id ay29-20020a05600c1e1d00b003cf674aaefemr35976420wmb.22.1674756452289; Thu, 26 Jan 2023 10:07:32 -0800 (PST) Original-Received: from [192.168.0.2] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id 18-20020a05600c26d200b003da28dfdedcsm2473698wmv.5.2023.01.26.10.07.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 26 Jan 2023 10:07:31 -0800 (PST) Content-Language: en-US In-Reply-To: <838rhpg57n.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:254223 Archived-At: On 26/01/2023 08:50, Eli Zaretskii wrote: >> Date: Thu, 26 Jan 2023 01:21:08 +0200 >> Cc: casouri@gmail.com, 60953@debbugs.gnu.org >> From: Dmitry Gutov >> >> Thank you. Unfortunately, the performance improvement from this patch is >> still fairly negligible. > > This is quite strange, since all of the approaches basically use the > same primitives under the hood. Perhaps the reason for the slowness > is that the code which computes the text span of a node is slow? That code seems to be the same between the two options: treesit_predicate_capture_name_to_text basically does the same as treesit-node-text (except in C) after iterating through a Lisp list to find the node. ruby-ts--builtin-method-p calls treesit-node-text. And treesit_predicate_pred does the same iteration, so the :pred option should just be slower, due to Lisp-related overhead. funcalls and stuff. > Otherwise, I must be missing something here, since the rest of the > code on the C level is basically the same, give or take some wrappers > that should not change the overall picture. The query object is smaller, though. That's basically my only remaining hypothesis. >> Switching to using :pred with function (like I did in commit >> d94dc606a0934) which still uses buffer-substring inside is significantly >> faster. > > If the performance issue is fixed, then the only aspect that we should > perhaps try to improve is consing. I wouldn't say it's "fixed", just improved. And :match really should be able to be made faster than :pred, since it'll probably be used for similar cases (where a lot/most of nodes match). There seems to be a fair amount of consing going on inside treesit-query-capture already: we wrap every TS node in our objects, we turn the captured nodes into a Lisp alist, and we turn the predicates into a list, turning the strings into "our" strings. The 'make_string' function creates a new copy in the memory, right? One could hope to avoid recreating the list of predicates on every match, but that seems to be a limitation of the TS API: ts_query_predicates_for_pattern requires a second argument, match.pattern_index. Maybe we could memoize that, though? In any case, that seems to explain why adding or avoiding one buffer-substring call per match isn't moving the needle very much. > Consing a string each time you > need to fontify increases the GC pressure, so if there's a good way of > avoiding that without performance degradation, we should take it. Is > it possible to use your :pred technique in a way that doesn't need to > produce strings from buffer text? The only version I managed to get some (very minor) performance improvement is this: (defun ruby-ts--builtin-method-p (node) (goto-char (treesit-node-start node)) (let ((inhibit-changing-match-data t)) (re-search-forward ruby-ts--builtin-methods (treesit-node-end node) t))) The improvement is like 200-300ms, whereas the difference between :match and :pred in this benchmark is several seconds. And if I try to bring it back to 100% correctness, to ensure that the whole of node text is matched, I have to use narrowing (and string-start and string-end anchors in regexp): (defvar ruby-ts--builtin-methods (format "\\`%s\\'" (regexp-opt (append ruby-builtin-methods-no-reqs ruby-builtin-methods-with-reqs))) "Ruby built-in methods.") (defun ruby-ts--builtin-method-p (node) (save-restriction (goto-char (treesit-node-start node)) (narrow-to-region (point) (treesit-node-end node)) (let ((inhibit-changing-match-data t)) (re-search-forward ruby-ts--builtin-methods nil t)))) And with that, the performance is again no better than the current version. If I also add save-excursion, it's worse.