From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: =?UTF-8?Q?Cl=c3=a9ment_Pit--Claudel?= Newsgroups: gmane.emacs.devel Subject: Re: Feature request/RFC: proper highlighting of code embedded in comments Date: Sun, 16 Oct 2016 17:10:07 -0400 Message-ID: References: <930446db-6dad-72a0-d0cd-a0710e5c6ec3@mit.edu> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Wj5L5kCdDk6EwVUP1R5Qb8rPhXCoC7GoR" X-Trace: blaine.gmane.org 1476652239 4140 195.159.176.226 (16 Oct 2016 21:10:39 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sun, 16 Oct 2016 21:10:39 +0000 (UTC) User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Oct 16 23:10:35 2016 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bvshM-0007fU-9S for ged-emacs-devel@m.gmane.org; Sun, 16 Oct 2016 23:10:24 +0200 Original-Received: from localhost ([::1]:57800 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bvshO-0000kq-Dx for ged-emacs-devel@m.gmane.org; Sun, 16 Oct 2016 17:10:26 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:60589) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bvshG-0000kj-U3 for emacs-devel@gnu.org; Sun, 16 Oct 2016 17:10:20 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bvshD-00024G-Iy for emacs-devel@gnu.org; Sun, 16 Oct 2016 17:10:18 -0400 Original-Received: from mout.kundenserver.de ([212.227.126.187]:62059) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1bvshD-000244-7R for emacs-devel@gnu.org; Sun, 16 Oct 2016 17:10:15 -0400 Original-Received: from [18.189.6.80] ([18.189.6.80]) by mrelayeu.kundenserver.de (mreue002) with ESMTPSA (Nemesis) id 0MeXc7-1cKIGY3nFH-00QAAD for ; Sun, 16 Oct 2016 23:10:14 +0200 In-Reply-To: <930446db-6dad-72a0-d0cd-a0710e5c6ec3@mit.edu> X-Provags-ID: V03:K0:Z1FLQSLMD49PgIl5nmp1w79Q/GOth/ShJhXnxKNfYiWMLN4UFFi KhLsJIyXoDdqSl0MCm/Mrhhs7Cm9FNVvGguN2jU/DfzyQhPgzIpVTZDJ0sDDbkViAIkOwIC sgi5N1q1gsYMwqm5pjcwSL2E14v7PB+1+KWzuR/L7X4U88HQZxpvHLV/OElby5gktQFhInJ FSkFIncmGeAftiRQmpKqA== X-UI-Out-Filterresults: notjunk:1;V01:K0:q88UJdHZgWk=:CDKHXSVej+f217qjR7f4dn nz9TAEeHH0GuKFuppkFNVBnMW7V1pe2JYOS7geEs2DZBx389SuKP7bZEwxWL9eVT+16RgflQ9 vyVKW5VonG4sg57EYqtf2xilz7mPi2hW963o3mO9/FadUGxYpBSmGGmgonlQMMUDMEsdvhxTn SZc85A2VU9N4Fo0KRXBfVpHCge5Rd2MO7hMIQYIm6cwGuKVAH91K87KCQaUEFjBT8m3o4QDc6 IpXRO5RiP5oOALVVThv90Vt7edWhCHO+jMraP284n4k/aVnF34DGd3BzOO647SGHbMV3hsIT0 a2lr+S5GWTcsWTXx523FLI0Yjv7hkZ/SrBQVwvPLzfSHva2wbMoeDKu6oL8D761IUgF4/3ErP ee2TCMh1zU/nZP94Kz33OdzDaed6FZWq7YgT7Ro4Fa/YulJ7woHDBapfWFXoK71+NfwS+eIbN NIyYCZDV8Ly8uNQqjRu2qKXEIQkO8K0Z6eywxZof+Ep5dOdhz9K5QBnuSEt3v4UahyhpbS/Xa p05ZvGAE1w1wcFgvJbt/z7snlYlklNHRdjaDLZobwjpdDiAzizlHOpD0PgJ3M14Gm7xrYkhmU kwHRSPnmt6RIttgCG95DCCCCHFtvqWX9E0Y20YDT0HoNfWFvdPG74aTnuccgJdn97j+d9t7Up 7ZH5ntZLb6A8b7lws/634d3jgvhcMLkbxOkslv5TSwgEuIbCLxYUKG+aV7yWx+AHg76I= X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 212.227.126.187 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:208342 Archived-At: This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Wj5L5kCdDk6EwVUP1R5Qb8rPhXCoC7GoR Content-Type: multipart/mixed; boundary="ptXJbQFweomJ7aI3h6Cc26rVKD1H7CLkh"; protected-headers="v1" From: =?UTF-8?Q?Cl=c3=a9ment_Pit--Claudel?= To: emacs-devel@gnu.org Message-ID: Subject: Re: Feature request/RFC: proper highlighting of code embedded in comments References: <930446db-6dad-72a0-d0cd-a0710e5c6ec3@mit.edu> In-Reply-To: <930446db-6dad-72a0-d0cd-a0710e5c6ec3@mit.edu> --ptXJbQFweomJ7aI3h6Cc26rVKD1H7CLkh Content-Type: multipart/mixed; boundary="------------6E8D3102BFEEBF74A3754C69" This is a multi-part message in MIME format. --------------6E8D3102BFEEBF74A3754C69 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable After writing my original email I thought about something a bit different= , and I managed (with suggestions and help from Anders Lindgren) to write= a convincing (to me :) proof of concept. The idea is to use a separate = buffer to do the fontification. I've attached the code; after loading it= , it's enough to run (font-lock-add-keywords nil '(("^ *>>> \\(.*\\)" (0 (indirect-font-lo= ck-highlighter 1 'python-mode))))) Stefan (and emacs-devel!), do you think I should add this to ELPA? Are t= here downsides I should be aware of? Cheers, Cl=C3=A9ment. On 2016-10-15 11:19, Cl=C3=A9ment Pit--Claudel wrote: > Hi emacs-devel, >=20 > Some languages have a way to quote code in comments. Some examples: >=20 > * Python >=20 > def example(foo, *bars): > """Foo some bars""" >=20 > >>> example(1, > ... 2, > ... 3) > 3 >=20 > >>> example(4, 8) > 67 > """ >=20 > * Coq >=20 > Definition example foo bars :=3D > (* [example foo bars] uses [foo] to foo some [bars]. For examp= le: > << > Compute (example 1 [2, 3]). > (* 3 *) > >> *) >=20 > In Python, =E2=80=98>>>=E2=80=99 indicates a doctest (a small bit of ex= ample code). In Coq, =E2=80=98[=E2=80=A6]=E2=80=99 and =E2=80=98<<=E2=80= =A6>>=E2=80=99 serve as markers (inside of comments) of single-line (resp= multi-line) code snippets. At the moment, Emacs doesn't highlight these= snippets. I originally asked about this in http://emacs.stackexchange.c= om/questions/19998/code-blocks-in-font-lock-comments , but received no an= swers. >=20 > There are multiple currently-available workarounds, but none of them th= at I know of are satisfactory: >=20 > * Duplicate all font-lock rules, creating anchored matchers that recogn= ize code in comments. The duplication is very unpleasant, and it will re= quire adding =E2=80=98prepend=E2=80=99 to a bunch of font-lock rules, whi= ch will break some of them. >=20 > * Use a custom syntax-propertize-function to recognize these code snipp= ets and escape out of strings. This has some potential, but it confuses = existing tools. For example, in Python, one can do the following; it wor= ks fine for =E2=80=98>>>=E2=80=99 in comments, but in strings it seems to= break eldoc, among others: >=20 > syntax-ppss() > python-util-forward-comment(1) > python-nav-end-of-defun() > python-info-current-defun() > (let ((current-defun (python-info-current-defun))) (if current-defu= n (progn (format "In: %s()" current-defun)))) >=20 > (defconst litpy--doctest-re > "^#*\\s-*\\(>>>\\|\\.\\.\\.\\)\\s-*\\(.+\\)$" > "Regexp matching doctests.") >=20 > (defun litpy--syntax-propertize-function (start end) > "Mark doctests in START..END." > (goto-char start) > (while (re-search-forward litpy--doctest-re end t) > (let* ((old-syntax (save-excursion (syntax-ppss (match-beginnin= g 1)))) > (in-docstring-p (eq (nth 3 old-syntax) t)) > (in-comment-p (eq (nth 4 old-syntax) t)) > (closing-syntax (cond (in-docstring-p "|") (in-comment-p= ">"))) > (reopening-syntax (cond (in-docstring-p "|") (in-comment= -p "<"))) > (reopening-char (char-after (match-end 2))) > (no-reopen (eq (and reopening-char (char-syntax reopenin= g-char)) > (cond (in-comment-p ?>))))) > (when closing-syntax > (put-text-property (1- (match-end 1)) (match-end 1) > 'syntax-table (string-to-syntax closing-= syntax)) > (when (and reopening-char (not no-reopen)) > (put-text-property (match-end 2) (1+ (match-end 2)) > 'syntax-table (string-to-syntax reopen= ing-syntax))))))) >=20 >=20 > Maybe the second approach can be made to more-or-less work for Python, = despite the issue above =E2=80=94 I'm not entirely sure. The idea there = is to detect chunks of code, and mark their starting and ending character= s in a way that escapes from the surrounding comment or string. >=20 > But this doesn't solve the problem for Coq, for example, because it con= fuses comment-forward and the like. Some coq tools depend on Emacs to id= entify comments and skip over them when running a file (code is sent bit = by bit, so if =E2=80=98(* foo [some code here] bar *)=E2=80=99 is annotat= ed with syntax properties to make Emacs think that it should be understoo= d as =E2=80=98(* foo *) some code here (* bar *)=E2=80=99, then Proof Gen= eral (a Coq IDE based on Emacs) won't realize that =E2=80=9Csome code her= e=E2=80=9D is part of a comment, and things will break. >=20 > I'm not sure what the right approach is. I guess there are two approac= hes: >=20 > * Mark embedded code in comments as actual code using syntax-propertize= -function, and add a way for tools to detect this "code but not really co= de" situation. Pros: things like company, eldoc, prettify-symbols-mode, = etc. will work in embedded code comments without having to opt them in. = Cons: some things will break, and will need to be fixed (comment-forward,= Proof General, Elpy, indentation functions=E2=80=A6). >=20 > * Add new "code block starter"/"code-block-ender" syntax classes? Then= font-lock would know that it has to highlight these. Pros: few things w= ould break. Cons: Tools would have to be opted-in (company-mode, eldoc, = prettify-symbols-mode, =E2=80=A6). >=20 > Am I missing another obvious solution? Has this topic been discussed b= efore? >=20 > Cheers, > Cl=C3=A9ment. >=20 >=20 --------------6E8D3102BFEEBF74A3754C69 Content-Type: text/x-emacs-lisp; name="indirect-font-lock.el" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="indirect-font-lock.el" ;;; indirect-font-lock.el --- Highlight parts of comments and strings as = code -*- lexical-binding: t; -*- ;; Copyright (C) 2016 Cl=C3=A9ment Pit-Claudel ;; Author: Cl=C3=A9ment Pit-Claudel ;; Keywords: faces ;; This program is free software; you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation, either version 3 of the License, or ;; (at your option) any later version. ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; You should have received a copy of the GNU General Public License ;; along with this program. If not, see . ;;; Commentary: ;; ;;; Code: (defvar-local indirect-font-lock--temp-buffers nil "Alist of (MODE-FN . BUFFER). These are temporary buffers, used for highlighting.") (defun indirect-font-lock--kill-temp-buffers () "Kill buffers in `indirect-font-lock--temp-buffers'." (mapc #'kill-buffer (mapcar #'cdr indirect-font-lock--temp-buffers)) (setq indirect-font-lock--temp-buffers nil)) (defun indirect-font-lock--make-buffer-for-mode (mode-fn) "Create a temporary buffer for MODE-FN. The buffer is created and initialized with MODE-FN only once; further calls with the same MODE-FN reuse the same buffer." (let ((buffer (cdr (assoc mode-fn indirect-font-lock--temp-buffers)))) (unless buffer (setq buffer (generate-new-buffer (format " *%S-highlight*" mode-fn= ))) (push (cons mode-fn buffer) indirect-font-lock--temp-buffers) (with-current-buffer buffer (funcall mode-fn) (setq-local kill-buffer-query-functions nil))) (with-current-buffer buffer (setq buffer-read-only nil) (erase-buffer)) buffer)) (defun indirect-font-lock--copy-faces-to (buffer offset) "Copy faces from current buffer to BUFFER, starting at OFFSET." (let ((start (point-min)) (making-progress t) (offset (- offset (point-min)))) (while making-progress (let ((end (next-single-property-change start 'face nil (point-max)= ))) (if (< start end) (font-lock-prepend-text-property (+ start offset) (+ end offs= et) 'face (get-text-property start 'face= ) buffer) (setq making-progress nil)) (setq start end))))) (defun indirect-font-lock--fontify-as (mode-fn from to) "Use buffer in MODE-FN to fontify FROM..TO. In other word, fontify FROM..TO would as if it had been alone in its own buffer, in major mode MODE-FN." (let ((str (buffer-substring-no-properties from to)) (original-buffer (current-buffer))) (with-current-buffer (indirect-font-lock--make-buffer-for-mode mode-f= n) (insert str) (font-lock-fontify-region (point-min) (point-max)) (indirect-font-lock--copy-faces-to original-buffer from)))) (defun indirect-font-lock-highlighter (group mode-fn) "Font-lock highlighter using an indirect buffer. Fontify GROUP as if it had been alone in its own buffer, in major mode MODE-FN." (save-match-data (indirect-font-lock--fontify-as mode-fn (match-beginning group) (matc= h-end group))) '(face nil)) (provide 'indirect-font-lock) ;;; indirect-font-lock.el ends here --------------6E8D3102BFEEBF74A3754C69-- --ptXJbQFweomJ7aI3h6Cc26rVKD1H7CLkh-- --Wj5L5kCdDk6EwVUP1R5Qb8rPhXCoC7GoR Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIbBAEBCAAGBQJYA+yvAAoJEPqg+cTm90wjVUcP+L9vsIE+55lyF6xG/fjlpIll GFQ1T5xpIhkzZgUDTZyG+TDYCYKqSdUOg0WjmYM3uVb9LneI+RJMSaHkdmFu3KIk fsQasvZm3L4mCbhFE0SJsLvnZwO0yg4QI7sHNH7wAsdgVC4ZiW26d/+IOPx3c9DG dG43RLEFTSMZ85VcB16uazkpf8V3lIX4fa/rprMyfEy0XjyJEKc/qvt5SDYmSA+p Wlonht/M351TGYrLujsiR8bhNK/7iCulMB1exIMIzR8et4Og84GA2YpWG5hAraeG ejTWDogFyZQJK/CP8VK2T6M4ef+od8BKRzgjk6n0Bb/SJfL2ohQ2zgO4UlrDdEd6 sUyj3J/Ia7LtyoSXofK4Fmpg7FiWRFjeXYpuSq6182312KurSgKzR8iChYJTBZYF +Jv0T80glnf0Y3vCABeRMJwiW9LHYtyVBPT3x7YltIB98shuefbf7EHc220+H7AZ hYy8426Ps19luqbcFybSoor+IubQmi/lapf8Mzajuk79BGRbV/pwHaV5HrwWS3ip shkKS5ssU+PS1miGPrOTNt1uRikrMoeCRm5vzA1ZKFFALX26hbbHtACX0oBWK77n 2EsOM4jYc5DbOiZFihMYMsuOtUl7USkV6XZX3rPVo7fV3O8P2lIXSZn9TpMZpa3+ ieyAafi02SVOwxmtOJ4= =hGmo -----END PGP SIGNATURE----- --Wj5L5kCdDk6EwVUP1R5Qb8rPhXCoC7GoR--