Feature request/RFC: proper highlighting of code embedded in comments

Feature request/RFC: proper highlighting of code embedded in comments

[Clément Pit--Claudel]

Hi emacs-devel,

Some languages have a way to quote code in comments.  Some examples:

* Python

    def example(foo, *bars):
        """Foo some bars"""

        >>> example(1,
        ...         2,
        ...         3)
        3

        >>> example(4, 8)
        67
        """

* Coq

    Definition example foo bars :=
        (* [example foo bars] uses [foo] to foo some [bars].  For example:
           <<
             Compute (example 1 [2, 3]).
             (* 3 *)
           >> *)

In Python, ‘>>>’ indicates a doctest (a small bit of example code).  In Coq, ‘[…]’ and ‘<<…>>’ serve as markers (inside of comments) of single-line (resp multi-line) code snippets.  At the moment, Emacs doesn't highlight these snippets.  I originally asked about this in http://emacs.stackexchange.com/questions/19998/code-blocks-in-font-lock-comments , but received no answers.

There are multiple currently-available workarounds, but none of them that I know of are satisfactory:

* Duplicate all font-lock rules, creating anchored matchers that recognize code in comments.  The duplication is very unpleasant, and it will require adding ‘prepend’ to a bunch of font-lock rules, which will break some of them.

* Use a custom syntax-propertize-function to recognize these code snippets and escape out of strings.  This has some potential, but it confuses existing tools.  For example, in Python, one can do the following; it works fine for ‘>>>’ in comments, but in strings it seems to break eldoc, among others:

    syntax-ppss()
    python-util-forward-comment(1)
    python-nav-end-of-defun()
    python-info-current-defun()
    (let ((current-defun (python-info-current-defun))) (if current-defun (progn (format "In: %s()" current-defun))))

    (defconst litpy--doctest-re
      "^#*\\s-*\\(>>>\\|\\.\\.\\.\\)\\s-*\\(.+\\)$"
      "Regexp matching doctests.")

    (defun litpy--syntax-propertize-function (start end)
      "Mark doctests in START..END."
      (goto-char start)
      (while (re-search-forward litpy--doctest-re end t)
        (let* ((old-syntax (save-excursion (syntax-ppss (match-beginning 1))))
               (in-docstring-p (eq (nth 3 old-syntax) t))
               (in-comment-p (eq (nth 4 old-syntax) t))
               (closing-syntax (cond (in-docstring-p "|") (in-comment-p ">")))
               (reopening-syntax (cond (in-docstring-p "|") (in-comment-p "<")))
               (reopening-char (char-after (match-end 2)))
               (no-reopen (eq (and reopening-char (char-syntax reopening-char))
                              (cond (in-comment-p ?>)))))
          (when closing-syntax
            (put-text-property (1- (match-end 1)) (match-end 1)
                               'syntax-table (string-to-syntax closing-syntax))
            (when (and reopening-char (not no-reopen))
              (put-text-property (match-end 2) (1+ (match-end 2))
                                 'syntax-table (string-to-syntax reopening-syntax)))))))


Maybe the second approach can be made to more-or-less work for Python, despite the issue above — I'm not entirely sure.  The idea there is to detect chunks of code, and mark their starting and ending characters in a way that escapes from the surrounding comment or string.

But this doesn't solve the problem for Coq, for example, because it confuses comment-forward and the like.  Some coq tools depend on Emacs to identify comments and skip over them when running a file (code is sent bit by bit, so if ‘(* foo [some code here] bar *)’ is annotated with syntax properties to make Emacs think that it should be understood as ‘(* foo *) some code here (* bar *)’, then Proof General (a Coq IDE based on Emacs) won't realize that “some code here” is part of a comment, and things will break.

I'm not sure what the right approach is.  I guess there are two approaches:

* Mark embedded code in comments as actual code using syntax-propertize-function, and add a way for tools to detect this "code but not really code" situation.  Pros: things like company, eldoc, prettify-symbols-mode, etc. will work in embedded code comments without having to opt them in.  Cons: some things will break, and will need to be fixed (comment-forward, Proof General, Elpy, indentation functions…).

* Add new "code block starter"/"code-block-ender" syntax classes?  Then font-lock would know that it has to highlight these.  Pros: few things would break.  Cons: Tools would have to be opted-in (company-mode, eldoc, prettify-symbols-mode, …).

Am I missing another obvious solution?  Has this topic been discussed before?

Cheers,
Clément.

Re: Feature request/RFC: proper highlighting of code embedded in comments

[Dmitry Gutov]

On 15.10.2016 18:19, Clément Pit--Claudel wrote:

> Am I missing another obvious solution?  Has this topic been discussed before?

mmm-mode?

And other tools and discussions about mixed-mode solutions.

Re: Feature request/RFC: proper highlighting of code embedded in comments

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 480 bytes --]

On 2016-10-15 16:22, Dmitry Gutov wrote:
> On 15.10.2016 18:19, Clément Pit--Claudel wrote:
> 
>> Am I missing another obvious solution?  Has this topic been discussed before?
> 
> mmm-mode?
> 
> And other tools and discussions about mixed-mode solutions.

Thanks :) I was hoping that this kind of complexity wouldn't be needed, since this is all a single mode.  But maybe that perceived complexity just reflects my lack of experience with mmm.

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Feature request/RFC: proper highlighting of code embedded in comments

[Stefan Monnier]

FWIW, in sm-c-mode.el (in elpa.git), CPP directives are treated as
comments, and since they do contain code, I have to solve the same kind
of problem.

I (ab)use for that purpose a syntactic face function.  The starting
point is:

  (setq-local font-lock-syntactic-face-function #'sm-c-syntactic-face-function)

Take a look at sm-c-syntactic-face-function and especially
sm-c--cpp-fontify-syntactically to see how I try to re-use the existing
font-lock functionality.

It's a bit gross, tho.


        Stefan

Re: Feature request/RFC: proper highlighting of code embedded in comments

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1547 bytes --]

Thanks!  How is the package called? I don't see it list-packages :/

This sounds pretty similar to the solution I outlined before, though.  The problem that I ran into with python is that I also need to "reopen" the quotes or comments.  For example. a bit of code might be in the middle of a docstring, like this:

    def example():
        """Blah

        >>> blah(xyz)
        bluh!

        What a great example!
        """

The issue here is that “What a great example” is a string.  I tried using a syntactic face function to mark the last ‘>’ as a strong closer and the newline as a string opener, but that confused the existing function, which expects the docstring starter to be ‘"""’, not ‘\n’.  Even after fixing this, python-mode was unusable: it inflooped when trying to find a whole defun, because the nav-end-of-defun function isn't ready to accept ‘\n’ as a string starter.

Cheers,
Clément.

On 2016-10-16 13:42, Stefan Monnier wrote:
> FWIW, in sm-c-mode.el (in elpa.git), CPP directives are treated as
> comments, and since they do contain code, I have to solve the same kind
> of problem.
> 
> I (ab)use for that purpose a syntactic face function.  The starting
> point is:
> 
>   (setq-local font-lock-syntactic-face-function #'sm-c-syntactic-face-function)
> 
> Take a look at sm-c-syntactic-face-function and especially
> sm-c--cpp-fontify-syntactically to see how I try to re-use the existing
> font-lock functionality.
> 
> It's a bit gross, tho.
> 
> 
>         Stefan
> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Feature request/RFC: proper highlighting of code embedded in comments

[Stefan Monnier]

> Thanks!  How is the package called? I don't see it list-packages :/

It's called sm-c-mode.  And no, it's not in GNU ELPA, it's in elpa.git.

> The issue here is that “What a great example” is a string.  I tried
> using a syntactic face function to mark the last ‘>’ as a string
> closer and the newline as a string opener,

Never set `syntax-table` text properties from
font-lock-syntactic-face-function (this will bring nothing but
problems that are difficult to track down).  Been there, done that.
Do it from syntax-propertize-function.

> but that confused the existing function, which expects the docstring
> starter to be ‘"""’, not ‘\n’.  Even after fixing this, python-mode
> was unusable: it inflooped when trying to find a whole defun, because
> the nav-end-of-defun function isn't ready to accept ‘\n’ as
> a string starter.

Sounds like you need to adjust other parts of the code, yes.

Alternatively, don't use `syntax-table` text properties at all, and
highlight the nested strings "by hand" (with regexps and/or manual
parsing).


        Stefan

Re: Feature request/RFC: proper highlighting of code embedded in comments

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1360 bytes --]

On 2016-10-17 09:02, Stefan Monnier wrote:
>> Thanks!  How is the package called? I don't see it list-packages :/
> 
> It's called sm-c-mode.  And no, it's not in GNU ELPA, it's in elpa.git.

Thanks! I'll look.

>> The issue here is that “What a great example” is a string.  I tried
>> using a syntactic face function to mark the last ‘>’ as a string
>> closer and the newline as a string opener,
> 
> Never set `syntax-table` text properties from
> font-lock-syntactic-face-function (this will bring nothing but
> problems that are difficult to track down).  Been there, done that.
> Do it from syntax-propertize-function.

Woops, this was a typo.  I'm using a syntax-propertize-function.

>> but that confused the existing function, which expects the docstring
>> starter to be ‘"""’, not ‘\n’.  Even after fixing this, python-mode
>> was unusable: it inflooped when trying to find a whole defun, because
>> the nav-end-of-defun function isn't ready to accept ‘\n’ as
>> a string starter.
> 
> Sounds like you need to adjust other parts of the code, yes.
> 
> Alternatively, don't use `syntax-table` text properties at all, and
> highlight the nested strings "by hand" (with regexps and/or manual
> parsing).

OK; I think I'll go with the solution I posted in the other subthread :)

Thanks!
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Feature request/RFC: proper highlighting of code embedded in comments

[Clément Pit--Claudel]

[-- Attachment #1.1.1: Type: text/plain, Size: 5499 bytes --]

After writing my original email I thought about something a bit different, and I managed (with suggestions and help from Anders Lindgren) to write a convincing (to me :) proof of concept.  The idea is to use a separate buffer to do the fontification.  I've attached the code; after loading it, it's enough to run

    (font-lock-add-keywords nil '(("^ *>>> \\(.*\\)" (0 (indirect-font-lock-highlighter 1 'python-mode)))))

Stefan (and emacs-devel!), do you think I should add this to ELPA?  Are there downsides I should be aware of?

Cheers,
Clément.

On 2016-10-15 11:19, Clément Pit--Claudel wrote:
> Hi emacs-devel,
> 
> Some languages have a way to quote code in comments.  Some examples:
> 
> * Python
> 
>     def example(foo, *bars):
>         """Foo some bars"""
> 
>         >>> example(1,
>         ...         2,
>         ...         3)
>         3
> 
>         >>> example(4, 8)
>         67
>         """
> 
> * Coq
> 
>     Definition example foo bars :=
>         (* [example foo bars] uses [foo] to foo some [bars].  For example:
>            <<
>              Compute (example 1 [2, 3]).
>              (* 3 *)
>            >> *)
> 
> In Python, ‘>>>’ indicates a doctest (a small bit of example code).  In Coq, ‘[…]’ and ‘<<…>>’ serve as markers (inside of comments) of single-line (resp multi-line) code snippets.  At the moment, Emacs doesn't highlight these snippets.  I originally asked about this in http://emacs.stackexchange.com/questions/19998/code-blocks-in-font-lock-comments , but received no answers.
> 
> There are multiple currently-available workarounds, but none of them that I know of are satisfactory:
> 
> * Duplicate all font-lock rules, creating anchored matchers that recognize code in comments.  The duplication is very unpleasant, and it will require adding ‘prepend’ to a bunch of font-lock rules, which will break some of them.
> 
> * Use a custom syntax-propertize-function to recognize these code snippets and escape out of strings.  This has some potential, but it confuses existing tools.  For example, in Python, one can do the following; it works fine for ‘>>>’ in comments, but in strings it seems to break eldoc, among others:
> 
>     syntax-ppss()
>     python-util-forward-comment(1)
>     python-nav-end-of-defun()
>     python-info-current-defun()
>     (let ((current-defun (python-info-current-defun))) (if current-defun (progn (format "In: %s()" current-defun))))
> 
>     (defconst litpy--doctest-re
>       "^#*\\s-*\\(>>>\\|\\.\\.\\.\\)\\s-*\\(.+\\)$"
>       "Regexp matching doctests.")
> 
>     (defun litpy--syntax-propertize-function (start end)
>       "Mark doctests in START..END."
>       (goto-char start)
>       (while (re-search-forward litpy--doctest-re end t)
>         (let* ((old-syntax (save-excursion (syntax-ppss (match-beginning 1))))
>                (in-docstring-p (eq (nth 3 old-syntax) t))
>                (in-comment-p (eq (nth 4 old-syntax) t))
>                (closing-syntax (cond (in-docstring-p "|") (in-comment-p ">")))
>                (reopening-syntax (cond (in-docstring-p "|") (in-comment-p "<")))
>                (reopening-char (char-after (match-end 2)))
>                (no-reopen (eq (and reopening-char (char-syntax reopening-char))
>                               (cond (in-comment-p ?>)))))
>           (when closing-syntax
>             (put-text-property (1- (match-end 1)) (match-end 1)
>                                'syntax-table (string-to-syntax closing-syntax))
>             (when (and reopening-char (not no-reopen))
>               (put-text-property (match-end 2) (1+ (match-end 2))
>                                  'syntax-table (string-to-syntax reopening-syntax)))))))
> 
> 
> Maybe the second approach can be made to more-or-less work for Python, despite the issue above — I'm not entirely sure.  The idea there is to detect chunks of code, and mark their starting and ending characters in a way that escapes from the surrounding comment or string.
> 
> But this doesn't solve the problem for Coq, for example, because it confuses comment-forward and the like.  Some coq tools depend on Emacs to identify comments and skip over them when running a file (code is sent bit by bit, so if ‘(* foo [some code here] bar *)’ is annotated with syntax properties to make Emacs think that it should be understood as ‘(* foo *) some code here (* bar *)’, then Proof General (a Coq IDE based on Emacs) won't realize that “some code here” is part of a comment, and things will break.
> 
> I'm not sure what the right approach is.  I guess there are two approaches:
> 
> * Mark embedded code in comments as actual code using syntax-propertize-function, and add a way for tools to detect this "code but not really code" situation.  Pros: things like company, eldoc, prettify-symbols-mode, etc. will work in embedded code comments without having to opt them in.  Cons: some things will break, and will need to be fixed (comment-forward, Proof General, Elpy, indentation functions…).
> 
> * Add new "code block starter"/"code-block-ender" syntax classes?  Then font-lock would know that it has to highlight these.  Pros: few things would break.  Cons: Tools would have to be opted-in (company-mode, eldoc, prettify-symbols-mode, …).
> 
> Am I missing another obvious solution?  Has this topic been discussed before?
> 
> Cheers,
> Clément.
> 
> 

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.2: indirect-font-lock.el --]
[-- Type: text/x-emacs-lisp; name="indirect-font-lock.el", Size: 3550 bytes --]

;;; indirect-font-lock.el --- Highlight parts of comments and strings as code  -*- lexical-binding: t; -*-

;; Copyright (C) 2016  Clément Pit-Claudel

;; Author: Clément Pit-Claudel <clement.pitclaudel@live.com>
;; Keywords: faces

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:

;;

;;; Code:

(defvar-local indirect-font-lock--temp-buffers nil
  "Alist of (MODE-FN . BUFFER).
These are temporary buffers, used for highlighting.")

(defun indirect-font-lock--kill-temp-buffers ()
  "Kill buffers in `indirect-font-lock--temp-buffers'."
  (mapc #'kill-buffer (mapcar #'cdr indirect-font-lock--temp-buffers))
  (setq indirect-font-lock--temp-buffers nil))

(defun indirect-font-lock--make-buffer-for-mode (mode-fn)
  "Create a temporary buffer for MODE-FN.
The buffer is created and initialized with MODE-FN only once;
further calls with the same MODE-FN reuse the same buffer."
  (let ((buffer (cdr (assoc mode-fn indirect-font-lock--temp-buffers))))
    (unless buffer
      (setq buffer (generate-new-buffer (format " *%S-highlight*" mode-fn)))
      (push (cons mode-fn buffer) indirect-font-lock--temp-buffers)
      (with-current-buffer buffer
        (funcall mode-fn)
        (setq-local kill-buffer-query-functions nil)))
    (with-current-buffer buffer
      (setq buffer-read-only nil)
      (erase-buffer))
    buffer))

(defun indirect-font-lock--copy-faces-to (buffer offset)
  "Copy faces from current buffer to BUFFER, starting at OFFSET."
  (let ((start (point-min))
        (making-progress t)
        (offset (- offset (point-min))))
    (while making-progress
      (let ((end (next-single-property-change start 'face nil (point-max))))
        (if (< start end)
            (font-lock-prepend-text-property (+ start offset) (+ end offset)
                                     'face (get-text-property start 'face)
                                     buffer)
          (setq making-progress nil))
        (setq start end)))))

(defun indirect-font-lock--fontify-as (mode-fn from to)
  "Use buffer in MODE-FN to fontify FROM..TO.

In other word, fontify FROM..TO would as if it had been alone in its own
buffer, in major mode MODE-FN."
  (let ((str (buffer-substring-no-properties from to))
        (original-buffer (current-buffer)))
    (with-current-buffer (indirect-font-lock--make-buffer-for-mode mode-fn)
      (insert str)
      (font-lock-fontify-region (point-min) (point-max))
      (indirect-font-lock--copy-faces-to original-buffer from))))

(defun indirect-font-lock-highlighter (group mode-fn)
  "Font-lock highlighter using an indirect buffer.
Fontify GROUP as if it had been alone in its own buffer, in major
mode MODE-FN."
  (save-match-data
    (indirect-font-lock--fontify-as mode-fn (match-beginning group) (match-end group)))
  '(face nil))

(provide 'indirect-font-lock)
;;; indirect-font-lock.el ends here

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Feature request/RFC: proper highlighting of code embedded in comments

[Stefan Monnier]

>     (font-lock-add-keywords nil '(("^ *>>> \\(.*\\)" (0
> (indirect-font-lock-highlighter 1 'python-mode)))))

IIUC you additionally want to handle the "... " lines.

> Stefan (and emacs-devel!), do you think I should add this to ELPA?

Feel free.

> Are there downsides I should be aware of?

The usual:
- running the major mode might run "side-effecting code" from the user's
  hook.  I consider it a bug in the user's setup, but it does happen.
- it fails to take into account local modifications of the font-lock rules.
- ...

Nothing too terrible, I think.


        Stefan

Re: Feature request/RFC: proper highlighting of code embedded in comments

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 2240 bytes --]

On 2016-10-17 09:12, Stefan Monnier wrote:
>>     (font-lock-add-keywords nil '(("^ *>>> \\(.*\\)" (0
>> (indirect-font-lock-highlighter 1 'python-mode)))))
> 
> IIUC you additionally want to handle the "... " lines.

Yup; this was just a POC :)

>> Stefan (and emacs-devel!), do you think I should add this to ELPA?
> 
> Feel free.

Will do it soon!

>> Are there downsides I should be aware of?
> 
> The usual:
> - running the major mode might run "side-effecting code" from the user's
>   hook.  I consider it a bug in the user's setup, but it does happen.
> - it fails to take into account local modifications of the font-lock rules.

This second point is interesting.  I ran into this issue with company-coq.  The problem was that entering the coq major mode took a long time (maybe .2 seconds?), so fontifying small things on the fly was slow.  I ended up copying font-lock related variables.  Something like

    (defconst company-coq--font-lock-vars '(font-lock-keywords
                                 font-lock-keywords-only
                                 font-lock-keywords-case-fold-search
                                 font-lock-syntax-table
                                 font-lock-syntactic-face-function)
      "Font-lock variables that influence fontification.")

    (defun company-coq--fontify-buffer-with (&optional ref-buffer)
      "Fontify current buffer according to settings in REF-BUFFER."
      (cl-loop for var in company-coq--font-lock-vars
               do (set (make-local-variable var)
                       (buffer-local-value var ref-buffer)))
      (ignore-errors
        ;; Some modes, such as whitespace-mode, rely on buffer-local state to do
        ;; their fontification.  Thus copying only font-lock variables is not
        ;; enough; one would need to copy these modes private variables as well.
        ;; See GH-124.
        (font-lock-default-fontify-region (point-min) (point-max) nil)))

But this is a bit broken, because fontification rules can run essentially anything.  It would be nice to have a feature like "fontify as if this code was at position 0 in this buffer".  I don't know whether it would make sense.

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]