From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Danny Freeman via "Bug reports for GNU Emacs, the Swiss army knife of text editors" Newsgroups: gmane.emacs.bugs Subject: bug#66004: [PATCH] Offset ranges before applying embeded treesit parsers Date: Fri, 15 Sep 2023 11:45:00 -0400 Message-ID: <87o7i32zxz.fsf@dfreeman.email> Reply-To: Danny Freeman Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="35336"; mail-complaints-to="usenet@ciao.gmane.io" To: 66004@debbugs.gnu.org, casouri@gmail.com Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Fri Sep 15 17:57:09 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qhBBs-0008z3-10 for geb-bug-gnu-emacs@m.gmane-mx.org; Fri, 15 Sep 2023 17:57:08 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qhBBh-00037t-Hb; Fri, 15 Sep 2023 11:56:57 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qhBBf-00037I-Ls for bug-gnu-emacs@gnu.org; Fri, 15 Sep 2023 11:56:55 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qhBBf-0003h8-Dp for bug-gnu-emacs@gnu.org; Fri, 15 Sep 2023 11:56:55 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qhBBl-00075c-Rg for bug-gnu-emacs@gnu.org; Fri, 15 Sep 2023 11:57:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Danny Freeman Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 15 Sep 2023 15:57:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 66004 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch X-Debbugs-Original-To: bug-gnu-emacs@gnu.org, Yuan Fu Original-Received: via spool by submit@debbugs.gnu.org id=B.169479339027208 (code B ref -1); Fri, 15 Sep 2023 15:57:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 15 Sep 2023 15:56:30 +0000 Original-Received: from localhost ([127.0.0.1]:44525 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qhBBF-00074m-WE for submit@debbugs.gnu.org; Fri, 15 Sep 2023 11:56:30 -0400 Original-Received: from lists.gnu.org ([2001:470:142::17]:49812) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qhBBD-00074V-58 for submit@debbugs.gnu.org; Fri, 15 Sep 2023 11:56:29 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qhBB0-0002ur-Sd for bug-gnu-emacs@gnu.org; Fri, 15 Sep 2023 11:56:14 -0400 Original-Received: from out-217.mta1.migadu.com ([2001:41d0:203:375::d9]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qhBAx-0003cg-Je for bug-gnu-emacs@gnu.org; Fri, 15 Sep 2023 11:56:14 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dfreeman.email; s=key1; t=1694793367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=ta4ocfaATcWMkwyozPEoXJNFwfWx61zjDBfdEMyjy4M=; b=Du96DDFh6nrbaAO6etGklLatbV6l/vx1jGZB12SNkB5EkoNL1/XBkZ1uhXF+AP77xcy1k4 9nnWunWi+ehwLgVgaJFxxDAUOpKy/56/OcYqeG5QG12IXSxU0cMYlqDxE7koHCrykuzDmP 0MCfBquL4AzgaH6M6A7Nj+Ep2WFFF5U= X-Migadu-Flow: FLOW_OUT Received-SPF: pass client-ip=2001:41d0:203:375::d9; envelope-from=danny@dfreeman.email; helo=out-217.mta1.migadu.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:270544 Archived-At: --=-=-= Content-Type: text/plain Background: In clojure-ts-mode I've been capturing docstrings and applying some limited syntax highlighting using an embedded markdown parser. I'm only able to capture the full string, "quotes included". I would like to be able to easily adjust the ranges captured to only include the contents of the string, delimiters excluded. I have a similar desire to capture the contents of a regular expression literal and apply a nested regex grammar. I've seen an offset mechanism used by the neovim tree-sitter integration for similar purposes. I believe the javascript/typescript modes could take advantage of this with template strings. I've included a small test in the patch that demonstrates this. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Offset-ranges-before-applying-embedded-tree-sitter-p.patch >From 61b89cf08ff8eb6e984d862519b9a0750f7f0cd0 Mon Sep 17 00:00:00 2001 From: Danny Freeman Date: Fri, 15 Sep 2023 11:29:05 -0400 Subject: [PATCH] Offset ranges before applying embedded tree-sitter parser * lisp/treesit.el (treesit-query-range): Accept an optional offest arg, apply the offset to all returned ranges (treesit-range-rules): Accept an optional :offset keyword arg to adjust ranges an embded parser is applied to (treesit-update-ranges): Forward optional :offset setting from `treesit-range-rules' to `treesit-query-rang' * test/lisp/treesit-tests.el (treesit-range-offset): Tests the new offset functioanlity This is feature would allow treesitter major modes to easily specify offsets when using embeded parsers. A potential use case for this is javascript template strings, when we want to apply a different parser to the string's contents, but do not want to include the template string's delmiters. --- lisp/treesit.el | 49 +++++++++++++++++++++++++++------------ test/src/treesit-tests.el | 14 +++++++++++ 2 files changed, 48 insertions(+), 15 deletions(-) diff --git a/lisp/treesit.el b/lisp/treesit.el index 78bd149b7e2..0b257c93c44 100644 --- a/lisp/treesit.el +++ b/lisp/treesit.el @@ -449,21 +449,25 @@ treesit-query-string (treesit-parser-root-node parser) query)))) -(defun treesit-query-range (node query &optional beg end) +(defun treesit-query-range (node query &optional beg end offset) "Query the current buffer and return ranges of captured nodes. QUERY, NODE, BEG, END are the same as in `treesit-query-capture'. This function returns a list of (START . END), where START and -END specifics the range of each captured node. Capture names -generally don't matter, but names that starts with an underscore -are ignored." - (cl-loop for capture - in (treesit-query-capture node query beg end) - for name = (car capture) - for node = (cdr capture) - if (not (string-prefix-p "_" (symbol-name name))) - collect (cons (treesit-node-start node) - (treesit-node-end node)))) +END specifics the range of each captured node. OFFSET is an +optional pair of numbers (START-OFFSET . END-OFFSET). The +respective offset values are added to each (START . END) range +being returned. Capture names generally don't matter, but names +that starts with an underscore are ignored." + (let ((offset-left (or (car offset) 0)) + (offset-right (or (cdr offset) 0))) + (cl-loop for capture + in (treesit-query-capture node query beg end) + for name = (car capture) + for node = (cdr capture) + if (not (string-prefix-p "_" (symbol-name name))) + collect (cons (+ (treesit-node-start node) offset-left) + (+ (treesit-node-end node) offset-right))))) ;;; Range API supplement @@ -509,6 +513,7 @@ treesit-range-rules (treesit-range-rules :embed \\='javascript :host \\='html + :offset \\='(1 . -1) \\='((script_element (raw_text) @cap))) The `:embed' keyword specifies the embedded language, and the @@ -521,13 +526,20 @@ treesit-range-rules this QUERY is given a dedicated local parser. Otherwise, the range shares the same parser with other ranges. +If there's an `:offset' keyword with a pair of numbers, each +captured range is offset by those numbers. For example, an +offset of (1 . -1) will update a captured range of (2 . 8) to +be (3 . 7). This can be used to exclude things like surrounding +delimiters from being included in the range covered by an +embedded parser. + QUERY can also be a function that takes two arguments, START and END. If QUERY is a function, it doesn't need the :KEYWORD VALUE pair preceding it. This function should set the ranges for parsers in the current buffer in the region between START and END. It is OK for this function to set ranges in a larger region that encompasses the region between START and END." - (let (host embed result local) + (let (host embed offset result local) (while query-specs (pcase (pop query-specs) (:local (when (eq t (pop query-specs)) @@ -540,6 +552,12 @@ treesit-range-rules (unless (symbolp embed-lang) (signal 'treesit-error (list "Value of :embed option should be a symbol" embed-lang))) (setq embed embed-lang))) + (:offset (let ((range-offset (pop query-specs))) + (unless (and (consp range-offset) + (numberp (car range-offset)) + (numberp (cdr range-offset))) + (signal 'treesit-error (list "Value of :offset option should be a pair of numbers" range-offset))) + (setq offset range-offset))) (query (if (functionp query) (push (list query nil nil) result) (when (null embed) @@ -547,9 +565,9 @@ treesit-range-rules (when (null host) (signal 'treesit-error (list "Value of :host option cannot be omitted"))) (push (list (treesit-query-compile host query) - embed local) + embed local offset) result)) - (setq host nil embed nil)))) + (setq host nil embed nil offset nil)))) (nreverse result))) (defun treesit--merge-ranges (old-ranges new-ranges start end) @@ -676,6 +694,7 @@ treesit-update-ranges (let ((query (nth 0 setting)) (language (nth 1 setting)) (local (nth 2 setting)) + (offset (nth 3 setting)) (beg (or beg (point-min))) (end (or end (point-max)))) (cond @@ -687,7 +706,7 @@ treesit-update-ranges (parser (treesit-parser-create language)) (old-ranges (treesit-parser-included-ranges parser)) (new-ranges (treesit-query-range - host-lang query beg end)) + host-lang query beg end offset)) (set-ranges (treesit--clip-ranges (treesit--merge-ranges old-ranges new-ranges beg end) diff --git a/test/src/treesit-tests.el b/test/src/treesit-tests.el index 65994ce608f..4308e4048f6 100644 --- a/test/src/treesit-tests.el +++ b/test/src/treesit-tests.el @@ -662,6 +662,20 @@ treesit-range ;; TODO: More tests. ))) +(ert-deftest treesit-range-offset () + "Tests if range offsets work." + (skip-unless (treesit-language-available-p 'javascript)) + (with-temp-buffer + (let ((query '(((call_expression (identifier) @_html_template_fn + (template_string) @capture) + (:equal "html" @_html_template_fn))))) + (progn + (insert "const x = html`

Hello

`;") + (treesit-parser-create 'javascript)) + (should (equal '((15 . 29)) (treesit-query-range 'javascript query))) + (should (equal '((16 . 28)) (treesit-query-range + 'javascript query nil nil '(1 . -1))))))) + ;;; Multiple language (ert-deftest treesit-multi-lang () -- 2.40.1 --=-=-= Content-Type: text/plain Let me know what you think. Thank you, -- Danny Freeman --=-=-=--