Re: [PATCH] feat: add markdown-ts-mode

[Rahul Martim Juliato <rahuljuliato@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 13187 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> Rahul Martim Juliato <rahuljuliato@gmail.com> writes:
>
>> Thanks Eli and Jostein!
>>
>> No problem criticising :)
>>
>>
>> Please find a new patch attached. Things I've made:
>>
>> - changed the file to the textmodes folder
>>
>> - made it inherit from `text-mode'
>>
>> - changed the news with +++
>>
>> - changed `emacs.texi' (is it here?) to add Markdown
>>
>> - scratched my head until 2:30AM thinking why it would not work for
>>   Jostein test.
>>
>>
>> Well Jostein, I've been played by the tree-sitter github repository
>> maintainer, lol.
>>
>> First, a little note. There's actually no major changes to the version
>> on github. I only changed documenting strings and made some small name
>> changes to more "alike" already existing modes.
>>
>> It happens that on tree-sitter repository, unlike any other of their
>> repositories (that I've seen), "main" branch is NOT the current branch,
>> but the "we moved from here bye" one.
>>
>> The current default branch is "split_parser" where
>> tree-sitter-markdown/src/ resides, hence the confusion.
>>
>> So, I wrote the mode with the "main" one in mind.
>>
>> I already checked the "new" (split_parser) one, and it seems it is
>> possible to convert the current work to use this one instead. I'll try
>> to do this in the middle of the week or sooner :)
>>
>> In the mean time, I realized more people would like to test this patch
>> and tree-sitter setup is really trick, so I made this little "guide" on
>> how to apply it until the end result (this might work now, pointing to
>> the "main" branch with the current patch).
>>
>> If you could please try the path (B) on the guide and tell me if this
>> works for you, It would be nice, Jostein.
>>
>>
>> Thanks!
>>
>> Rahul
>>
>>
>>
>> --- beggining of guide on how to apply this patch and test it
>>
>> Git apply this patch.
>>
>> Run the autogen script:
>>
>> ./autogen.sh
>>
>> Make sure configure uses tree-sitter:
>>
>> ./configure --with-tree-sitter
>>
>> Compile:
>>
>> make bootstrap
>>
>> Check the build:
>>
>> ./src/emacs -Q --version
>>
>> Open emacs
>>
>> ./src/emacs -Q --init-dir=~/tmp_emacs_dir/
>>
>>
>> A) The Hard Way
>>
>> Visit a .md file
>>
>> C-x f TEST.md
>>
>> Note the mode will not automatically load.
>>
>> This behavior is the same as typescript-ts-mode or other
>> treesitter modes I've been using.
>>
>> M-x markdown-ts-mode
>>
>> It will fail since you have no tree sitter grammar installed.
>> and will suggest installing it with `treesit-install-language-grammar'.
>>
>> This behaviour is also standard for the tree-sitter modes I
>> currently use (typescript, tsx, rust).
>>
>> Issuing "M-x treesit-install-language-grammar RET".
>>
>> It asks for language, complete with "markdown RET".
>>
>> It says there's no recipie for it, if you want to build it
>> interactivelly. Anwser "yes".
>>
>> It asks for the URL where the grammar is hosted, enter:
>> "https://github.com/tree-sitter-grammars/tree-sitter-markdown RET"
>>
>> It asks for the tag or branch, setting the default, enter "main RET".
>>
>> It asks for the subfolder, leave the default (src), enter "RET".
>>
>> It asks for the C compiler to use (default: auto-detect), just "RET".
>>
>> It asks for the C++ compiler to use (default: auto-detect), just "RET".
>>
>> Install to (default "~/tmp_emacs_dir/treesitter"), just "RET".
>>
>> It will clone the repository, compile the library and tell in the
>> minibuffer the library is installed to your folder.
>>
>>
>> B) The Easier Way.
>>
>> Copy-paste and eval this use-package definition.
>>
>> (use-package markdown-ts-mode
>>   :mode ("\\.md\\'" . markdown-ts-mode)
>>   :defer 't
>>   :config
>>   (add-to-list 'treesit-language-source-alist '(markdown "https://github.com/tree-sitter-grammars/tree-sitter-markdown" "main" "src")))
>>
>>
>> Visit your markdown test file.
>>
>> It will probably fail due to the missing grammar.
>>
>> Issue "M-x treesit-install-language-grammar RET".
>>
>> Now with TAB it should complete "markdown", if not, type it.
>>
>> As we already have the source now defined, just hit "RET" to install
>> to the default treemacs folder.
>>
>> It will clone the repository and say it installed on your folder.
>>
>> Just reload the mode with "M-x markdown-ts-mode".
>>
>> From now on (with the use-package definition on `init.el') you should
>> just open .md and have the highlight and stuff.
>>
>> --- end of the guide
>>
>> From 45796df36129ec77c532b3fa21cdd0b8033c9777 Mon Sep 17 00:00:00 2001
>> From: Rahul Martim Juliato <rahul.juliato@gmail.com>
>> Date: Fri, 19 Apr 2024 23:21:20 -0300
>> Subject: [PATCH] feat: add markdown-ts-mode
>>
>>  * lisp/textmodes/markdown-ts-mode.el: New file.
>>  * doc/emacs/emacs.texi: Add Markdown to the manual
>>  * etc/NEWS: Announce markdown-ts-mode
>> ---
>>  doc/emacs/emacs.texi               |   1 +
>>  etc/NEWS                           |   5 ++
>>  lisp/textmodes/markdown-ts-mode.el | 106 +++++++++++++++++++++++++++++
>>  3 files changed, 112 insertions(+)
>>  create mode 100644 lisp/textmodes/markdown-ts-mode.el
>>
>> diff --git a/doc/emacs/emacs.texi b/doc/emacs/emacs.texi
>> index 7d77f13ab21..244c822ee04 100644
>> --- a/doc/emacs/emacs.texi
>> +++ b/doc/emacs/emacs.texi
>> @@ -618,6 +618,7 @@ Top
>>  * Enriched Text::       Editing text enriched with fonts, colors, etc.
>>  * Text Based Tables::   Commands for editing text-based tables.
>>  * Two-Column::          Splitting text columns into separate windows.
>> +* Markdown::            Major mode for editing Markdown files.
>>  
>>  Filling Text
>>  
>> diff --git a/etc/NEWS b/etc/NEWS
>> index 8ad1e78ca60..06fbaa03b55 100644
>> --- a/etc/NEWS
>> +++ b/etc/NEWS
>> @@ -1688,6 +1688,11 @@ A major mode based on the tree-sitter library for editing Elixir files.
>>  *** New major mode 'lua-ts-mode'.
>>  A major mode based on the tree-sitter library for editing Lua files.
>>  
>> ++++
>> +*** New major mode 'markdown-ts-mode'.
>> +A major mode based on the tree-sitter library for editing Markdown files.
>> +
>> +
>>  ** Minibuffer and Completions
>>  
>>  +++
>> diff --git a/lisp/textmodes/markdown-ts-mode.el b/lisp/textmodes/markdown-ts-mode.el
>> new file mode 100644
>> index 00000000000..063780a772f
>> --- /dev/null
>> +++ b/lisp/textmodes/markdown-ts-mode.el
>> @@ -0,0 +1,106 @@
>> +;;; markdown-ts-mode.el --- tree sitter support for Markdown  -*- lexical-binding: t; -*-
>> +
>> +;; Copyright (C) 2024 Free Software Foundation, Inc.
>> +
>> +;; Author     : Rahul Martim Juliato <rahul.juliato@gmail.com>
>> +;; Maintainer : Rahul Martim Juliato <rahul.juliato@gmail.com>
>> +;; Created    : April 2024
>> +;; Keywords   : markdown md languages tree-sitter
>> +
>> +;; This file is part of GNU Emacs.
>> +
>> +;; GNU Emacs is free software: you can redistribute it and/or modify
>> +;; it under the terms of the GNU General Public License as published by
>> +;; the Free Software Foundation, either version 3 of the License, or
>> +;; (at your option) any later version.
>> +
>> +;; GNU Emacs is distributed in the hope that it will be useful,
>> +;; but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +;; GNU General Public License for more details.
>> +
>> +;; You should have received a copy of the GNU General Public License
>> +;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
>> +
>> +;;; Commentary:
>> +;;
>
> You should probably briefly explain the package here.
>
>> +;;; Code:
>> +
>> +(require 'treesit)
>> +(require 'subr-x)
>> +
>> +(declare-function treesit-node-parent "treesit.c")
>> +(declare-function treesit-node-type "treesit.c")
>> +(declare-function treesit-parser-create "treesit.c")
>> +
>> +(defvar markdown-ts--treesit-settings
>> +  (treesit-font-lock-rules
>> +   :language 'markdown
>> +   :override t
>> +   :feature 'delimiter
>> +   '([ "[" "]" "(" ")" ] @shadow)
>> +
>> +   :language 'markdown
>> +   :feature 'paragraph
>> +   '([((atx_heading) @font-lock-keyword-face)
>> +      ((block_quote_marker) @font-lock-string-face)
>> +      ((code_span) @font-lock-string-face)
>> +      ((emphasis) @underline)
>> +      ((image_description) @link)
>> +      ((indented_code_block) @font-lock-string-face)
>> +      ((link_destination) @font-lock-string-face)
>> +      ((setext_heading) @font-lock-keyword-face)
>> +      ((strong_emphasis) @bold)
>> +      ((thematic_break) @shadow)
>> +      (block_quote (block_quote_marker) @font-lock-string-face)
>> +      (block_quote (paragraph) @font-lock-string-face)
>> +      (fenced_code_block (code_fence_content) @font-lock-string-face)
>> +      (fenced_code_block (fenced_code_block_delimiter) @font-lock-doc-face)
>> +      (inline_link (link_destination) @font-lock-string-face)
>> +      (inline_link (link_text) @link)
>> +      (list_item (list_marker_dot) @font-lock-keyword-face)
>> +      (list_item (list_marker_minus) @font-lock-keyword-face)
>> +      (list_item (list_marker_plus) @font-lock-keyword-face)
>> +      (list_item (list_marker_star) @font-lock-keyword-face)
>> +      (shortcut_link (link_text) @link)
>> +      ])))
>> +
>> +(defun markdown-ts-imenu-node-p (node)
>> +  "Check if NODE is a valid entry to imenu."
>> +  (equal (treesit-node-type (treesit-node-parent node))
>> +         "atx_heading"))
>> +
>> +(defun markdown-ts-imenu-name-function (node)
>> +  "Return an imenu entry if NODE is a valid header."
>> +  (let ((name (treesit-node-text node)))
>> +    (if (markdown-ts-imenu-node-p node)
>> +	(thread-first (treesit-node-parent node)(treesit-node-text))
>> +      name)))
>> +
>> +(defun markdown-ts-setup ()
>> +  "Setup treesit for `markdown-ts-mode'."
>> +  (setq-local treesit-font-lock-settings markdown-ts--treesit-settings)
>> +  (treesit-major-mode-setup))
>> +
>> +;;;###autoload
>> +(define-derived-mode markdown-ts-mode text-mode "Markdown"
>> +  "Major mode for editing Markdown using tree-sitter grammar."
>> +  (setq-local font-lock-defaults nil
>> +	      treesit-font-lock-feature-list '((delimiter)
>> +					       (paragraph)))
>> +
>> +  (setq-local treesit-simple-imenu-settings
>> +              `(("Headings" markdown-ts-imenu-node-p nil markdown-ts-imenu-name-function)))
>> +
>> +  (when (treesit-ready-p 'markdown)
>> +    (treesit-parser-create 'markdown)
>> +    (markdown-ts-setup)))
>> +
>> +(derived-mode-add-parents 'markdown-ts-mode '(markdown-mode))
>> +
>> +(if (treesit-ready-p 'markdown)
>> +    (add-to-list 'auto-mode-alist '("\\.md\\'" . markdown-ts-mode)))
>> +
>> +(provide 'markdown-ts-mode)
>> +;;; markdown-ts-mode.el ends here
>
> So if I understand correctly, there is no keymap or any convenience
> features that markdown-mode provides, because this is just the
> beginning, right?  A large part of major modes for me is not just the
> syntax highlighting and integration into Emacs systems, but also
> bindings that in the case of Markdown would insert links or add/remove
> emphasis.  I think it would be nice, if we could add these features in
> the future, and re-use bindings from a package like AucTeX to build on
> existing intuition (org-mode would be an alternative, but I am not a fan
> or their choice of bindings).  Some commands to "compile" and preview a
> document would also be nice.


Hello there again!

Please find attached the current version of this patch.

It now works with the "official" tree-sitter grammar for markdown:
https://github.com/tree-sitter-grammars/tree-sitter-markdown

Please note (as explained above) the `main branch' is not the default
anymore, but `split_parser' is.

They splitted the parser into two, one for the body of the document and
another to inline.

So, in order to test this patch, configure it with:

(use-package markdown-ts-mode
    :ensure nil
    :defer t  
    :mode ("\\.md\\'" . markdown-ts-mode)
    :config
    (add-to-list 'treesit-language-source-alist '(markdown "https://github.com/tree-sitter-grammars/tree-sitter-markdown" "split_parser" "tree-sitter-markdown/src"))
    (add-to-list 'treesit-language-source-alist '(markdown-inline "https://github.com/tree-sitter-grammars/tree-sitter-markdown" "split_parser" "tree-sitter-markdown-inline/src")))

If you visit a markdown file treesit issues an error, as always, asking
for the grammars.

Then, install BOTH grammars:

M-x treesit-install-language-grammar RET markdown RET
M-x treesit-install-language-grammar RET markdown-inline RET

Reload the `markdown-ts-mode'.

And everything should be working fine :)


@Philip, thanks for bringing that up! I completely agree with you;
aiming for a fully-featured mode is our long-term objective.

The idea here is to establish a foundation for supporting markdown files
within Emacs, without relying on external packages.

From there, we can delve into discussions about editing and exporting
features. For instance, we could consider mimicking bindings and prompts
from org-mode, as well as exploring integrations if users opt to also
utilize third-party markdown-mode.

However, simply having the mode built-in reduces friction significantly
for new ideas and collaborations :)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-feat-add-markdown-ts-mode.patch --]
[-- Type: text/x-diff, Size: 5872 bytes --]

From a1d071285b3ea989f8722b0b2915cf881b59b84c Mon Sep 17 00:00:00 2001
From: Rahul Martim Juliato <rahul.juliato@gmail.com>
Date: Fri, 19 Apr 2024 23:21:20 -0300
Subject: [PATCH] feat: add markdown-ts-mode

 * lisp/textmodes/markdown-ts-mode.el: New file
 * doc/emacs/emacs.texi: Add Markdown to the manual
 * etc/NEWS: Announce markdown-ts-mode
---
 doc/emacs/emacs.texi               |   1 +
 etc/NEWS                           |   5 ++
 lisp/textmodes/markdown-ts-mode.el | 113 +++++++++++++++++++++++++++++
 3 files changed, 119 insertions(+)
 create mode 100644 lisp/textmodes/markdown-ts-mode.el

diff --git a/doc/emacs/emacs.texi b/doc/emacs/emacs.texi
index 7d77f13ab21..244c822ee04 100644
--- a/doc/emacs/emacs.texi
+++ b/doc/emacs/emacs.texi
@@ -618,6 +618,7 @@ Top
 * Enriched Text::       Editing text enriched with fonts, colors, etc.
 * Text Based Tables::   Commands for editing text-based tables.
 * Two-Column::          Splitting text columns into separate windows.
+* Markdown::            Major mode for editing Markdown files.
 
 Filling Text
 
diff --git a/etc/NEWS b/etc/NEWS
index 8ad1e78ca60..06fbaa03b55 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1688,6 +1688,11 @@ A major mode based on the tree-sitter library for editing Elixir files.
 *** New major mode 'lua-ts-mode'.
 A major mode based on the tree-sitter library for editing Lua files.
 
++++
+*** New major mode 'markdown-ts-mode'.
+A major mode based on the tree-sitter library for editing Markdown files.
+
+
 ** Minibuffer and Completions
 
 +++
diff --git a/lisp/textmodes/markdown-ts-mode.el b/lisp/textmodes/markdown-ts-mode.el
new file mode 100644
index 00000000000..49fe1baf011
--- /dev/null
+++ b/lisp/textmodes/markdown-ts-mode.el
@@ -0,0 +1,113 @@
+;;; markdown-ts-mode.el --- tree sitter support for Markdown  -*- lexical-binding: t; -*-
+
+;; Copyright (C) 2024 Free Software Foundation, Inc.
+
+;; Author     : Rahul Martim Juliato <rahul.juliato@gmail.com>
+;; Maintainer : Rahul Martim Juliato <rahul.juliato@gmail.com>
+;; Created    : April 2024
+;; Keywords   : markdown md languages tree-sitter
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;
+
+;;; Code:
+
+(require 'treesit)
+(require 'subr-x)
+
+(declare-function treesit-node-parent "treesit.c")
+(declare-function treesit-node-type "treesit.c")
+(declare-function treesit-parser-create "treesit.c")
+
+(defvar markdown-ts--treesit-settings
+  (treesit-font-lock-rules
+   :language 'markdown
+   :override t
+   :feature 'delimiter
+   '([ "[" "]" "(" ")" ] @shadow)
+
+
+   :language 'markdown
+   :feature 'paragraph
+   '([((setext_heading) @font-lock-keyword-face)
+      ((atx_heading) @font-lock-keyword-face)
+      ((thematic_break) @shadow)
+      ((indented_code_block) @font-lock-string-face)
+      (list_item (list_marker_star) @font-lock-keyword-face)
+      (list_item (list_marker_plus) @font-lock-keyword-face)
+      (list_item (list_marker_minus) @font-lock-keyword-face)
+      (list_item (list_marker_dot) @font-lock-keyword-face)
+      (fenced_code_block (fenced_code_block_delimiter) @font-lock-doc-face)
+      (fenced_code_block (code_fence_content) @font-lock-string-face)
+      ((block_quote_marker) @font-lock-string-face)
+      (block_quote (paragraph) @font-lock-string-face)
+      (block_quote (block_quote_marker) @font-lock-string-face)
+      ])
+
+   :language 'markdown-inline
+   :feature 'paragraph-inline
+   '([
+      ((image_description) @link)
+      ((link_destination) @font-lock-string-face)
+      ((code_span) @font-lock-string-face)
+      ((emphasis) @underline)
+      ((strong_emphasis) @bold)
+      (inline_link (link_text) @link)
+      (inline_link (link_destination) @font-lock-string-face)
+      (shortcut_link (link_text) @link)])))
+
+(defun markdown-ts-imenu-node-p (node)
+  "Check if NODE is a valid entry to imenu."
+  (equal (treesit-node-type (treesit-node-parent node))
+         "atx_heading"))
+
+(defun markdown-ts-imenu-name-function (node)
+  "Return an imenu entry if NODE is a valid header."
+  (let ((name (treesit-node-text node)))
+    (if (markdown-ts-imenu-node-p node)
+	(thread-first (treesit-node-parent node)(treesit-node-text))
+      name)))
+
+(defun markdown-ts-setup ()
+  "Setup treesit for `markdown-ts-mode'."
+  (setq-local treesit-font-lock-settings markdown-ts--treesit-settings)
+  (treesit-major-mode-setup))
+
+;;;###autoload
+(define-derived-mode markdown-ts-mode text-mode "Markdown"
+  "Major mode for editing Markdown using tree-sitter grammar."
+  (setq-local font-lock-defaults nil
+	          treesit-font-lock-feature-list '((delimiter)
+					           (paragraph)
+					           (paragraph-inline)))
+
+  (setq-local treesit-simple-imenu-settings
+              `(("Headings" markdown-ts-imenu-node-p nil markdown-ts-imenu-name-function)))
+
+  (when (treesit-ready-p 'markdown)
+    (treesit-parser-create 'markdown-inline)
+    (treesit-parser-create 'markdown)
+    (markdown-ts-setup)))
+
+(derived-mode-add-parents 'markdown-ts-mode '(markdown-mode))
+
+(if (treesit-ready-p 'markdown)
+    (add-to-list 'auto-mode-alist '("\\.md\\'" . markdown-ts-mode)))
+
+(provide 'markdown-ts-mode)
+;;; markdown-ts-mode.el ends here
-- 
2.39.2