From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: =?UTF-8?Q?Cl=c3=a9ment_Pit--Claudel?= Newsgroups: gmane.emacs.devel Subject: Feature request/RFC: proper highlighting of code embedded in comments Date: Sat, 15 Oct 2016 11:19:24 -0400 Message-ID: <930446db-6dad-72a0-d0cd-a0710e5c6ec3@mit.edu> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Trace: blaine.gmane.org 1476545051 18731 195.159.176.226 (15 Oct 2016 15:24:11 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sat, 15 Oct 2016 15:24:11 +0000 (UTC) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 To: Emacs developers Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sat Oct 15 17:24:06 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bvQoO-0001tF-J0 for ged-emacs-devel@m.gmane.org; Sat, 15 Oct 2016 17:23:48 +0200 Original-Received: from localhost ([::1]:52324 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bvQoN-0000cg-9n for ged-emacs-devel@m.gmane.org; Sat, 15 Oct 2016 11:23:47 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:60102) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bvQkG-0008EO-Hy for emacs-devel@gnu.org; Sat, 15 Oct 2016 11:19:33 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bvQkC-0007Cl-9i for emacs-devel@gnu.org; Sat, 15 Oct 2016 11:19:31 -0400 Original-Received: from mout.kundenserver.de ([212.227.17.13]:60102) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1bvQkB-0007Bo-Tu for emacs-devel@gnu.org; Sat, 15 Oct 2016 11:19:28 -0400 Original-Received: from [192.168.1.109] ([67.186.135.89]) by mrelayeu.kundenserver.de (mreue104) with ESMTPSA (Nemesis) id 0LgGqC-1cj7UH34Ng-00nlWj for ; Sat, 15 Oct 2016 17:19:26 +0200 X-Provags-ID: V03:K0:Eu/p4iYv4SW9ZpfHVCb9TrgzcksLOD1zBbnHEj08pl7oOhO126g k3ka5VlZzpYHp4igcF2XoMX67Bz+ZHvzx+AqtL4p1fr1wfO1msoihBMI/AUOWSmVyWqWfu6 t7FY3mrkkAAiL6h1JvSTtxtOvxl5uA4zDQy2BkKavVZF+WTXdcFRBS8rBfGjfPRawoC856h 7EzdpmApYqd4s/0xISpOg== X-UI-Out-Filterresults: notjunk:1;V01:K0:a7Q2US77hn8=:vyw8W9hHRnuaSm7IzXV8wB S/d7fosHmCedsQt6mPpkA1ww85wp6PyUka9H/A9eGvvQtsauH3JZduSS5ae8tA2tvWHU3j3bH wmlFI4bYaRDw09Y4l2K3gBlF/mtJH5J+dIo0RJ9Bpj5ekpqRDanQ/OoBD3k907eSJ2U38SVcR t1xXX33jYwy/dYWtGT4BAn5VA8v8b3vJI0pAonGD6T7U15FOz1g7Qv2x9jyDPpCxKNatHaVkB LTB1FybDeGyulsiATO6JPumlWS8v0/Mq+qqbOczC86s43MQprtjAkDhkYHm0wYNYkhKbIusKD ULbzABVJywKOcKIZC8b5eFSgPqoic7z+FQkO9PvBWzgTydnSIklPYfhvue4w7ipTwcJL4NjbW IwoUTa0jco5HBlE+VjCafIhzKln6a41d3a78B44AsXwu1DqXfTq3Z2vYnhRO8k0J8O6Eo3mQd e4bbzaTJYu4iXDAdGuZur+hOj7Z4kn1g07bbmZ+vKMBoTELGKX1NjlNsG9ETR0RO2cvMi9DmB /yAWNBj61mUqQ52rZtS7l2qmJElzOXyV3H0qN1j72d2RfqsOTRb1ZRxej5Ect+9JlhGMKddc7 lgvA4VONc6hSbR5OG7LCje6Qmm+baMRwNFoy/pLnLaZ0usAyXi/XnyFf3sNGm3hd7pk5Vdbw4 BDxRygBAAyk9tbSIBaT7XtXFjXnIiF6V+FhdVymX293TAWwcODylptoz67/CYY7mkLHBYO2jq 2T768Dsd/Ck8Vaaw X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 212.227.17.13 X-Mailman-Approved-At: Sat, 15 Oct 2016 11:23:42 -0400 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:208296 Archived-At: Hi emacs-devel, Some languages have a way to quote code in comments. Some examples: * Python def example(foo, *bars): """Foo some bars""" >>> example(1, ... 2, ... 3) 3 >>> example(4, 8) 67 """ * Coq Definition example foo bars := (* [example foo bars] uses [foo] to foo some [bars]. For example: << Compute (example 1 [2, 3]). (* 3 *) >> *) In Python, ‘>>>’ indicates a doctest (a small bit of example code). In Coq, ‘[…]’ and ‘<<…>>’ serve as markers (inside of comments) of single-line (resp multi-line) code snippets. At the moment, Emacs doesn't highlight these snippets. I originally asked about this in http://emacs.stackexchange.com/questions/19998/code-blocks-in-font-lock-comments , but received no answers. There are multiple currently-available workarounds, but none of them that I know of are satisfactory: * Duplicate all font-lock rules, creating anchored matchers that recognize code in comments. The duplication is very unpleasant, and it will require adding ‘prepend’ to a bunch of font-lock rules, which will break some of them. * Use a custom syntax-propertize-function to recognize these code snippets and escape out of strings. This has some potential, but it confuses existing tools. For example, in Python, one can do the following; it works fine for ‘>>>’ in comments, but in strings it seems to break eldoc, among others: syntax-ppss() python-util-forward-comment(1) python-nav-end-of-defun() python-info-current-defun() (let ((current-defun (python-info-current-defun))) (if current-defun (progn (format "In: %s()" current-defun)))) (defconst litpy--doctest-re "^#*\\s-*\\(>>>\\|\\.\\.\\.\\)\\s-*\\(.+\\)$" "Regexp matching doctests.") (defun litpy--syntax-propertize-function (start end) "Mark doctests in START..END." (goto-char start) (while (re-search-forward litpy--doctest-re end t) (let* ((old-syntax (save-excursion (syntax-ppss (match-beginning 1)))) (in-docstring-p (eq (nth 3 old-syntax) t)) (in-comment-p (eq (nth 4 old-syntax) t)) (closing-syntax (cond (in-docstring-p "|") (in-comment-p ">"))) (reopening-syntax (cond (in-docstring-p "|") (in-comment-p "<"))) (reopening-char (char-after (match-end 2))) (no-reopen (eq (and reopening-char (char-syntax reopening-char)) (cond (in-comment-p ?>))))) (when closing-syntax (put-text-property (1- (match-end 1)) (match-end 1) 'syntax-table (string-to-syntax closing-syntax)) (when (and reopening-char (not no-reopen)) (put-text-property (match-end 2) (1+ (match-end 2)) 'syntax-table (string-to-syntax reopening-syntax))))))) Maybe the second approach can be made to more-or-less work for Python, despite the issue above — I'm not entirely sure. The idea there is to detect chunks of code, and mark their starting and ending characters in a way that escapes from the surrounding comment or string. But this doesn't solve the problem for Coq, for example, because it confuses comment-forward and the like. Some coq tools depend on Emacs to identify comments and skip over them when running a file (code is sent bit by bit, so if ‘(* foo [some code here] bar *)’ is annotated with syntax properties to make Emacs think that it should be understood as ‘(* foo *) some code here (* bar *)’, then Proof General (a Coq IDE based on Emacs) won't realize that “some code here” is part of a comment, and things will break. I'm not sure what the right approach is. I guess there are two approaches: * Mark embedded code in comments as actual code using syntax-propertize-function, and add a way for tools to detect this "code but not really code" situation. Pros: things like company, eldoc, prettify-symbols-mode, etc. will work in embedded code comments without having to opt them in. Cons: some things will break, and will need to be fixed (comment-forward, Proof General, Elpy, indentation functions…). * Add new "code block starter"/"code-block-ender" syntax classes? Then font-lock would know that it has to highlight these. Pros: few things would break. Cons: Tools would have to be opted-in (company-mode, eldoc, prettify-symbols-mode, …). Am I missing another obvious solution? Has this topic been discussed before? Cheers, Clément.