From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Yuan Fu Newsgroups: gmane.emacs.devel Subject: Re: How to add pseudo vector types Date: Thu, 15 Jul 2021 14:23:02 -0400 Message-ID: <1AAB1BCC-362B-4249-B785-4E0530E15C60@gmail.com> References: <83h7gw6pyj.fsf@gnu.org> <45EBF16A-C953-42C7-97D1-3A2BFEF7DD01@gmail.com> <83y2a764oy.fsf@gnu.org> <83v95b60fn.fsf@gnu.org> <00DD5BFE-D14E-449A-9319-E7B725DEBFB3@gmail.com> <83r1fz5xr9.fsf@gnu.org> Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="18724"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Stefan Monnier , emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Thu Jul 15 20:24:08 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1m461n-0004bW-Ky for ged-emacs-devel@m.gmane-mx.org; Thu, 15 Jul 2021 20:24:07 +0200 Original-Received: from localhost ([::1]:39088 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m461m-0001VP-Gz for ged-emacs-devel@m.gmane-mx.org; Thu, 15 Jul 2021 14:24:06 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:43632) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m460r-0008Uc-HT for emacs-devel@gnu.org; Thu, 15 Jul 2021 14:23:09 -0400 Original-Received: from mail-qv1-xf2b.google.com ([2607:f8b0:4864:20::f2b]:40726) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m460o-0004hI-Im; Thu, 15 Jul 2021 14:23:09 -0400 Original-Received: by mail-qv1-xf2b.google.com with SMTP id p10so3318251qvk.7; Thu, 15 Jul 2021 11:23:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=/QMFsAJT9X34BTLQxBb51kqH/Q/lj48FMPG3y8lMCAU=; b=sY/VHHtuZ6/uLkPojUGZKuDuxxYoC2Lwc5iz6eKYqhJQxtpFok4z7ZjEQULmerzzuu 3JY7MpjlDSBPpoG+vsC8eC+Ymbn7LKbJnVI67oeasAXqRMZY8qqBILdVQiCBGNGkKIJS yv1E4dmZv+uVYDh7Tg0V+fxZ0hi8J7Yiy+hbuEOcKUno0XBUfG0ETU0eFbK/RLmlyjT2 m14TR7Hi11I/SIEjcGiFyH99ddvQbL7NmlA2QVWxbRfftq6I/QdqVz6uxXowIrQEQBOb Ad6KOhRgb32QYysZ3Vt38pDSOfHyuEbySrbnCezaqQS7OweUolZj4hlyMdmCfcqONZwd viaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=/QMFsAJT9X34BTLQxBb51kqH/Q/lj48FMPG3y8lMCAU=; b=bGwSxb6injn5b8DyXe25Wky5QRsTGrUyTEaS2W5BoXrijPUc1aD8RbFvde+xC8aUl9 lmSBm/cMlNi+RAWDUq2MMG/zdQ8YWLnTSmkjx6BNOsYRMIZJit1+BCIfO3K0l3/Be1Zv glzyRM0qZRkkEymwdh76gx5TdhLCS8POYUZhttOzEUBAdCABHp0qDFht3I5gtXytsioT CmMCO3qPrP8nwr5VNPMKRshxeiQRgf3kMcJh5LvxitSxFyIT+e1KqMTpywRboVnbmE9B Et2HXE4ZKUbK8gy3Mi4Qy9YAX61jNUWHJ6IXZ2MI+L2kphFqfDDpIE4+7Bvv8rVZk6Um sb/g== X-Gm-Message-State: AOAM532SnstqHDJi2+9LdHEwJb9u0+ttm6oFRdzE6klZ1pNQxdK+Y6lc AZKRxS29n2XZW1N/ILNn0bKkL9rYybYHSw== X-Google-Smtp-Source: ABdhPJzoPs80ligtdW/QTF4YLxTIopZDl18t4iU3t26YJtUvC9biJfRxyH9Ya7VKYNrY2/HUZdQ2/g== X-Received: by 2002:ad4:5fca:: with SMTP id jq10mr5884407qvb.36.1626373384279; Thu, 15 Jul 2021 11:23:04 -0700 (PDT) Original-Received: from ?IPv6:2601:98a:4200:9210:b02d:e0d8:dab6:21b? ([2601:98a:4200:9210:b02d:e0d8:dab6:21b]) by smtp.gmail.com with ESMTPSA id v16sm2248203qta.47.2021.07.15.11.23.03 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Jul 2021 11:23:03 -0700 (PDT) In-Reply-To: <83r1fz5xr9.fsf@gnu.org> X-Mailer: Apple Mail (2.3654.60.0.2.21) Received-SPF: pass client-ip=2607:f8b0:4864:20::f2b; envelope-from=casouri@gmail.com; helo=mail-qv1-xf2b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:271272 Archived-At: > On Jul 15, 2021, at 12:48 PM, Eli Zaretskii wrote: >=20 >> From: Yuan Fu >> Date: Thu, 15 Jul 2021 12:19:31 -0400 >> Cc: monnier@iro.umontreal.ca, >> emacs-devel@gnu.org >>=20 >>> Why do you need to do this when a buffer is updated? why not use >>> display as the trigger? Large portions of a buffer will never be >>> displayed, and some buffers will not be displayed at all. Why waste >>> cycles on them? Redisplay is perfectly equipped to tell you when = some >>> chunk of buffer text is going to be redrawn, and it already knows to >>> do nothing if the buffer haven't changed. >>=20 >> Tree-sitter expects you to tell it every single change to the parsed = text. >=20 > That cannot be true, because the parsed text could be in a state where > parsing it will fail. When you are in the middle of writing the code, > this is what will happen many times, even if you pass the whole buffer > to the parser. And since tree-sitter _must_ be able to deal with this > problem, it also must be able to receive incomplete parts of the > buffer text, and do the best it can with it. >=20 >> Say you have a buffer with some content and scrolled through it, so = tree-sitter has parsed the whole buffer. Then some elisp edited some = text outside the visible portion. Redisplay doesn=E2=80=99t happen, we = don=E2=80=99t tell this edit to tree-sitter. Then I scroll to the place = that has been edited. What now? >=20 > Now you call tree-sitter passing it the part of the buffer that needs > to be parsed (e.g., the chunk that is about to be displayed). If > tree-sitter needs to look back, it will. >=20 >> I=E2=80=99ve lost the change information, and tree-sitter=E2=80=99s = tree is out-dated. >=20 > No information is lost because the updated buffer text is available. >=20 >> We can fontify on-demand, but we can=E2=80=99t parse on-demand. >=20 > Sorry, I don't believe this is true. tree-sitter _must_ be able to > deal with these situations, because it must be able to deal with > incomplete text that cannot be parsed without parse errors. >=20 I think my assertion was too strong. By =E2=80=9Ccan=E2=80=99t parse = on-demand=E2=80=9D I mean we can=E2=80=99t easily pass tree-sitter a = random chunk of text and not letting it to parse from BOB.=20 > In addition, Emacs records (for redisplay purposes) two places in each > buffer related to changes: the minimum buffer position before which no > changes were done since last redisplay, and the maximum buffer > position beyond which there were no changes. This can also be used to > pass only a small part of the buffer to the parser, because the rest > didn't change. >=20 >> What we can do is to only parse the portion from BOB to the visible = portion. So we won=E2=80=99t parse the whole buffer unless you scroll to = the bottom. >=20 > My primary worry is the fact that you want to use buffer-change hooks > (and will soon enough want to use post-command-hook as well). They > slow down editing, sometimes tremendously, so I'd very much prefer not > to use those hooks for fontification/parsing. The original font-lock > mechanism in Emacs 19 used these hooks; we switched to jit-lock and > its redisplay-triggered fontifications because the original design had > problems which couldn't be solved reliably and with reasonable > performance. I hope we will not make the mistake of going back to > that sub-optimal design. I understand. I want to point out that parsing is separated from = fontification, and syntax-pass flushes its cache in before-change-hook. = I was hoping to use the parse tree for more than fontification, e.g., = motion commands like sexp-forward/backward or structural editing = commands like expand-region. Another scenario: some elisp edited some = text before the visible portion, the tree is not updated, now I want to = select the node at point (like expand-region), I look for the leave node = that contains the byte position of point. However, because the tree is = out-dated, the byte position of point will not correspond to the node I = want. We can still fontify with jit-lock, it=E2=80=99s just parsing cannot = easily work like fontification, I expect tree-sitter to work similarly = to syntax-pass rather than jit-lock. >=20 >>>> And, for tree-sitter to take the buffer=E2=80=99s content directly, = we need to tell it to skip the gap. >>>=20 >>> AFAIR, tree-sitter allows the calling package to provide a function = to >>> access the text, isn't that so? If so, you could write a function >>> that accesses buffer text via BYTE_POS_ADDR etc., and that knows how >>> to skip the gap already. >>=20 >> Yes, that function returns a char*. But what if the gap is in the = middle of the portion that tree-sitter wants to read? >=20 > If you provide the function that returns text one character at a time, > as AFAIR tree-sitter allows, you will be able to skip the gap > automagically by using BYTE_POS_ADDR. If that's not possible for some > reason, or not performant enough, we could ask tree-sitter developers > to add an API that access buffer text in two chunks, in which case it > will be called first with text before the gap, and then with text > after the gap. Like we do when we call regex search functions. Yes, I make a mistake reading the api. Indeed we can read one character = at a time, and gap is not an issue anymore. Yuan=