From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Re: tree-sitter: conceptional problem solvable at Emacs' level? Date: Sat, 11 Feb 2023 08:58:52 -0500 Message-ID: References: <87zg9n45ig.fsf@yahoo.com> <0DDF6978-D75A-4137-9D93-6200908675B6@gmail.com> <837cwplxni.fsf@gnu.org> <87bkm0czpp.fsf@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="16391"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Eli Zaretskii , Yuan Fu , luangruo@yahoo.com, holgerschurig@gmail.com, Emacs-devel@gnu.org To: Ihor Radchenko Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Feb 11 15:00:04 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pQqQ7-00042T-Pi for ged-emacs-devel@m.gmane-mx.org; Sat, 11 Feb 2023 15:00:03 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pQqPH-0003nf-4S; Sat, 11 Feb 2023 08:59:11 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pQqPF-0003nX-JW for Emacs-devel@gnu.org; Sat, 11 Feb 2023 08:59:09 -0500 Original-Received: from mail-pj1-x1031.google.com ([2607:f8b0:4864:20::1031]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pQqPD-0004mY-Eq; Sat, 11 Feb 2023 08:59:09 -0500 Original-Received: by mail-pj1-x1031.google.com with SMTP id nh19-20020a17090b365300b00233ceae8407so798806pjb.3; Sat, 11 Feb 2023 05:59:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=CWedZ4RsfdEgYd7HDxsyvXLfi3UE5mPHtPMQn1pRunI=; b=YjfgLUGKC9PqE+xrEb/dRqJUW3erUlSm5XCDY96N+SCSPzrVE9mo0G5Wrc3HkzY9Ns 8FSrHQ+FQUWOqHyRTmNp9+NKIHGnKhyw6xW/0fSM7y1X99MD8ZgF+1bB+Ws0RZyv1uFt gYUpO5exTTIjt/g3d3oxuagjoSLdkeSjnO26/toZV3eXF9HcYS2CwzsbbicjY0LCPzPE T5/y68z1FUPpHOtxwouR4x6sWMXUrlK/MlQp4rTxZ6J2Ua4lqH/MDQ0txoU1UBzthqnD pVT90I7GZbUbOd4/kKOuSy8DJfflm7k4OMESnKsuMpLJtuMcf2+BTQq2Zszj0u+7dsdl MWWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CWedZ4RsfdEgYd7HDxsyvXLfi3UE5mPHtPMQn1pRunI=; b=7NVGzLbAH9FBZzzvlHpSmji4r/q9iS5DCeHokPZOwgReFIbM7sCXqfNVdm10SuohtW NfJfawg4EcvIvo1swaqSX+B269PMCbf24SfkAb3JAHm67Xv9CweH4VatILkwt+btQZ1+ /eBlSRawBgM7y6vZcJnyhKyck57ozfeE760BBN7yJteTReQBo/gfb1goGoLtGtjrm+70 f+XaOGRX0SXKa8ik3s5j4QqlK7YQVunylv6Yj0uHcwLUInKsZAnQU64Hul49gnU32llW n9u7PnpIalWWBln8MEGTnpNAf0X9MyEgTp+3AeGs7i+xxQ+4Zhcl69x01ydppU1FCdEH GxYA== X-Gm-Message-State: AO0yUKUPBZShkHcdskiTUCSYiVqyDGkBYFlxuJ0z0HRwPqWWbRq0uyC1 qJBomFlY9K8kWXvYZ49/1mOpR21hQ8JGg+512EY= X-Google-Smtp-Source: AK7set9WvYYJhrfPSeSSsCmhIXq5ONbO4/F4WnqaXCCPAUcQTliXUT6Jdbe2YGcSgcW9SQvRAd+CZpS1dBY2XsNPOL8= X-Received: by 2002:a17:90a:1912:b0:230:7ffb:cead with SMTP id 18-20020a17090a191200b002307ffbceadmr3472476pjg.14.1676123945074; Sat, 11 Feb 2023 05:59:05 -0800 (PST) In-Reply-To: <87bkm0czpp.fsf@localhost> Received-SPF: pass client-ip=2607:f8b0:4864:20::1031; envelope-from=owinebar@gmail.com; helo=mail-pj1-x1031.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:303139 Archived-At: On Sat, Feb 11, 2023 at 4:34 AM Ihor Radchenko wrote: > > Eli Zaretskii writes: > > > Another possibility is to complicate the function we pass to > > tree-sitter with which to read buffer text, in a way that replaces the > > text of a macro with something else (in the simplest case, just space > > characters), so as to avoid errors in the parser, and again analyze > > the macros in our own code. > > Another idea is delegating parts of buffer to Elisp/alternative parser. > > Tree sitter provides support to documents written using a mixture of > grammars: https://tree-sitter.github.io/tree-sitter/using-parsers#multi-language-documents > Macros can be considered such a "mixed" grammar with macros being a > grammar of their own. > > AFAIU, tree sitter allows excluding certain file ranges from parsing > and instead parse the excluded ranges using alternative grammar. If > Elisp can somehow tell tree-sitter backend not skip parsing > macro-looking lines, it should solve the problem at least partially. > What's needed is (a) a generic "macro keyword" terminal in tree-sitter grammar, recognized by the lexer because they can appear anywhere, and (b) a parser for macro definitions. Then (b) maintains the set of macros that the lexer uses to recognize instances of (a). For extra credit, the macros could be hypothetically expanded, the results parsed, and the annotations generated on instances of the macro arguments mapped back to their occurrence as arguments. Maybe some kind of "unfold" notation could be used to see the results of the expansion and the resulting annotations in context. Lynn