From: "Clément Pit--Claudel" <clement.pit@gmail.com>
To: emacs-devel@gnu.org
Subject: Re: Feature request/RFC: proper highlighting of code embedded in comments
Date: Sun, 16 Oct 2016 17:10:07 -0400 [thread overview]
Message-ID: <df97a14e-d63b-db5f-12ea-e6ee5b536196@gmail.com> (raw)
In-Reply-To: <930446db-6dad-72a0-d0cd-a0710e5c6ec3@mit.edu>
[-- Attachment #1.1.1: Type: text/plain, Size: 5499 bytes --]
After writing my original email I thought about something a bit different, and I managed (with suggestions and help from Anders Lindgren) to write a convincing (to me :) proof of concept. The idea is to use a separate buffer to do the fontification. I've attached the code; after loading it, it's enough to run
(font-lock-add-keywords nil '(("^ *>>> \\(.*\\)" (0 (indirect-font-lock-highlighter 1 'python-mode)))))
Stefan (and emacs-devel!), do you think I should add this to ELPA? Are there downsides I should be aware of?
Cheers,
Clément.
On 2016-10-15 11:19, Clément Pit--Claudel wrote:
> Hi emacs-devel,
>
> Some languages have a way to quote code in comments. Some examples:
>
> * Python
>
> def example(foo, *bars):
> """Foo some bars"""
>
> >>> example(1,
> ... 2,
> ... 3)
> 3
>
> >>> example(4, 8)
> 67
> """
>
> * Coq
>
> Definition example foo bars :=
> (* [example foo bars] uses [foo] to foo some [bars]. For example:
> <<
> Compute (example 1 [2, 3]).
> (* 3 *)
> >> *)
>
> In Python, ‘>>>’ indicates a doctest (a small bit of example code). In Coq, ‘[…]’ and ‘<<…>>’ serve as markers (inside of comments) of single-line (resp multi-line) code snippets. At the moment, Emacs doesn't highlight these snippets. I originally asked about this in http://emacs.stackexchange.com/questions/19998/code-blocks-in-font-lock-comments , but received no answers.
>
> There are multiple currently-available workarounds, but none of them that I know of are satisfactory:
>
> * Duplicate all font-lock rules, creating anchored matchers that recognize code in comments. The duplication is very unpleasant, and it will require adding ‘prepend’ to a bunch of font-lock rules, which will break some of them.
>
> * Use a custom syntax-propertize-function to recognize these code snippets and escape out of strings. This has some potential, but it confuses existing tools. For example, in Python, one can do the following; it works fine for ‘>>>’ in comments, but in strings it seems to break eldoc, among others:
>
> syntax-ppss()
> python-util-forward-comment(1)
> python-nav-end-of-defun()
> python-info-current-defun()
> (let ((current-defun (python-info-current-defun))) (if current-defun (progn (format "In: %s()" current-defun))))
>
> (defconst litpy--doctest-re
> "^#*\\s-*\\(>>>\\|\\.\\.\\.\\)\\s-*\\(.+\\)$"
> "Regexp matching doctests.")
>
> (defun litpy--syntax-propertize-function (start end)
> "Mark doctests in START..END."
> (goto-char start)
> (while (re-search-forward litpy--doctest-re end t)
> (let* ((old-syntax (save-excursion (syntax-ppss (match-beginning 1))))
> (in-docstring-p (eq (nth 3 old-syntax) t))
> (in-comment-p (eq (nth 4 old-syntax) t))
> (closing-syntax (cond (in-docstring-p "|") (in-comment-p ">")))
> (reopening-syntax (cond (in-docstring-p "|") (in-comment-p "<")))
> (reopening-char (char-after (match-end 2)))
> (no-reopen (eq (and reopening-char (char-syntax reopening-char))
> (cond (in-comment-p ?>)))))
> (when closing-syntax
> (put-text-property (1- (match-end 1)) (match-end 1)
> 'syntax-table (string-to-syntax closing-syntax))
> (when (and reopening-char (not no-reopen))
> (put-text-property (match-end 2) (1+ (match-end 2))
> 'syntax-table (string-to-syntax reopening-syntax)))))))
>
>
> Maybe the second approach can be made to more-or-less work for Python, despite the issue above — I'm not entirely sure. The idea there is to detect chunks of code, and mark their starting and ending characters in a way that escapes from the surrounding comment or string.
>
> But this doesn't solve the problem for Coq, for example, because it confuses comment-forward and the like. Some coq tools depend on Emacs to identify comments and skip over them when running a file (code is sent bit by bit, so if ‘(* foo [some code here] bar *)’ is annotated with syntax properties to make Emacs think that it should be understood as ‘(* foo *) some code here (* bar *)’, then Proof General (a Coq IDE based on Emacs) won't realize that “some code here” is part of a comment, and things will break.
>
> I'm not sure what the right approach is. I guess there are two approaches:
>
> * Mark embedded code in comments as actual code using syntax-propertize-function, and add a way for tools to detect this "code but not really code" situation. Pros: things like company, eldoc, prettify-symbols-mode, etc. will work in embedded code comments without having to opt them in. Cons: some things will break, and will need to be fixed (comment-forward, Proof General, Elpy, indentation functions…).
>
> * Add new "code block starter"/"code-block-ender" syntax classes? Then font-lock would know that it has to highlight these. Pros: few things would break. Cons: Tools would have to be opted-in (company-mode, eldoc, prettify-symbols-mode, …).
>
> Am I missing another obvious solution? Has this topic been discussed before?
>
> Cheers,
> Clément.
>
>
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.2: indirect-font-lock.el --]
[-- Type: text/x-emacs-lisp; name="indirect-font-lock.el", Size: 3550 bytes --]
;;; indirect-font-lock.el --- Highlight parts of comments and strings as code -*- lexical-binding: t; -*-
;; Copyright (C) 2016 Clément Pit-Claudel
;; Author: Clément Pit-Claudel <clement.pitclaudel@live.com>
;; Keywords: faces
;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;; GNU General Public License for more details.
;; You should have received a copy of the GNU General Public License
;; along with this program. If not, see <http://www.gnu.org/licenses/>.
;;; Commentary:
;;
;;; Code:
(defvar-local indirect-font-lock--temp-buffers nil
"Alist of (MODE-FN . BUFFER).
These are temporary buffers, used for highlighting.")
(defun indirect-font-lock--kill-temp-buffers ()
"Kill buffers in `indirect-font-lock--temp-buffers'."
(mapc #'kill-buffer (mapcar #'cdr indirect-font-lock--temp-buffers))
(setq indirect-font-lock--temp-buffers nil))
(defun indirect-font-lock--make-buffer-for-mode (mode-fn)
"Create a temporary buffer for MODE-FN.
The buffer is created and initialized with MODE-FN only once;
further calls with the same MODE-FN reuse the same buffer."
(let ((buffer (cdr (assoc mode-fn indirect-font-lock--temp-buffers))))
(unless buffer
(setq buffer (generate-new-buffer (format " *%S-highlight*" mode-fn)))
(push (cons mode-fn buffer) indirect-font-lock--temp-buffers)
(with-current-buffer buffer
(funcall mode-fn)
(setq-local kill-buffer-query-functions nil)))
(with-current-buffer buffer
(setq buffer-read-only nil)
(erase-buffer))
buffer))
(defun indirect-font-lock--copy-faces-to (buffer offset)
"Copy faces from current buffer to BUFFER, starting at OFFSET."
(let ((start (point-min))
(making-progress t)
(offset (- offset (point-min))))
(while making-progress
(let ((end (next-single-property-change start 'face nil (point-max))))
(if (< start end)
(font-lock-prepend-text-property (+ start offset) (+ end offset)
'face (get-text-property start 'face)
buffer)
(setq making-progress nil))
(setq start end)))))
(defun indirect-font-lock--fontify-as (mode-fn from to)
"Use buffer in MODE-FN to fontify FROM..TO.
In other word, fontify FROM..TO would as if it had been alone in its own
buffer, in major mode MODE-FN."
(let ((str (buffer-substring-no-properties from to))
(original-buffer (current-buffer)))
(with-current-buffer (indirect-font-lock--make-buffer-for-mode mode-fn)
(insert str)
(font-lock-fontify-region (point-min) (point-max))
(indirect-font-lock--copy-faces-to original-buffer from))))
(defun indirect-font-lock-highlighter (group mode-fn)
"Font-lock highlighter using an indirect buffer.
Fontify GROUP as if it had been alone in its own buffer, in major
mode MODE-FN."
(save-match-data
(indirect-font-lock--fontify-as mode-fn (match-beginning group) (match-end group)))
'(face nil))
(provide 'indirect-font-lock)
;;; indirect-font-lock.el ends here
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
next prev parent reply other threads:[~2016-10-16 21:10 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-15 15:19 Feature request/RFC: proper highlighting of code embedded in comments Clément Pit--Claudel
2016-10-15 20:22 ` Dmitry Gutov
2016-10-15 21:21 ` Clément Pit--Claudel
2016-10-16 17:42 ` Stefan Monnier
2016-10-16 21:05 ` Clément Pit--Claudel
2016-10-17 13:02 ` Stefan Monnier
2016-10-17 14:19 ` Clément Pit--Claudel
2016-10-16 21:10 ` Clément Pit--Claudel [this message]
2016-10-17 13:12 ` Stefan Monnier
2016-10-17 14:25 ` Clément Pit--Claudel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=df97a14e-d63b-db5f-12ea-e6ee5b536196@gmail.com \
--to=clement.pit@gmail.com \
--cc=emacs-devel@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this external index
https://git.savannah.gnu.org/cgit/emacs.git
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.