From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id 2G9xMjcap2WkOgEAqHPOHw:P1 (envelope-from ) for ; Wed, 17 Jan 2024 01:07:19 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id 2G9xMjcap2WkOgEAqHPOHw (envelope-from ) for ; Wed, 17 Jan 2024 01:07:19 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20230601 header.b=RVgDv7vF; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1705450039; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=5IJDygsctSGGjtObp39W4HefMzRvb/m06GGbsWtNFFo=; b=N1vBU9sTr+ljjHMGHKTzWK/g6JeQltn2Gl5s6CWbj6HcADmLiLfs7jQrhHgYYXqJa5mb8k EhiHki1Ad2dcD046PyV6GGJEFoVpKpSnIBf7cGyWIq2ngBu3JDFvEn8QrY5Y35fkRogIGs H0128fAHdUvWWpL90c5MhO5CeI7xAIPc/CrVVGE5Gxl45UUdMRWTPJ0EG/vH59TQRM2Z72 FbWwNqZ8Z81CsVLz9/dyOV2Npo/NOFiGQo2aRmVj7EvHoO2v2eJ2HpPNQfC+RZQ0WyAInE hziG6Oy2IFeKZ5p1ezG69nidiGGasqmWwj5UhFoiWi9Lw3a777MH+B4MsB3CuQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20230601 header.b=RVgDv7vF; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1705450039; a=rsa-sha256; cv=none; b=gSioygCToLZo5hjixDUkDirMOVW1uUNO1OZCMLTyCXIxnPi/C6F+oYux8lRz26UC6DKunQ sFuYJCWgLO8be+UgRIUt2gMvygLdmyq/DKr8V/MlZO822InhjktXjchbkWPxhzIuTam9KO RBmCnK55+mhrElCo3TizQkx9aIpzPf1wDXhwYKcHnxlqfc2kME/NnuF9DNsFzUbfGBow8T PVjn4atvHaR25Ta3RQIbqjdwiQ5UbFAljn32H3V87P3KkzOVW24EJXd1/xk3mpBllXgeMj ZYMEfJ70xV4NG+BXIG/jUf9UoyLvxUPqBhMsULbnnvyGu06R0VIyzUtZcNLiMg== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 421F126627 for ; Wed, 17 Jan 2024 01:07:19 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rPtRu-00071H-Dd; Tue, 16 Jan 2024 19:06:30 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rPtRq-00070u-MP for emacs-orgmode@gnu.org; Tue, 16 Jan 2024 19:06:28 -0500 Received: from mail-il1-x132.google.com ([2607:f8b0:4864:20::132]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rPtRn-0004vu-7S for emacs-orgmode@gnu.org; Tue, 16 Jan 2024 19:06:26 -0500 Received: by mail-il1-x132.google.com with SMTP id e9e14a558f8ab-35d374bebe3so7833585ab.1 for ; Tue, 16 Jan 2024 16:06:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705449979; x=1706054779; darn=gnu.org; h=mime-version:message-id:in-reply-to:date:subject:cc:to:from :user-agent:references:from:to:cc:subject:date:message-id:reply-to; bh=flQyig9AOPFe3n1q3lVmc3uwr+GwHUrvpLQBwxkBOMs=; b=RVgDv7vF7RFXb9H4Sop/QvpGzA3KHqV5J8mByAtg+NMf+z7aOjBiE2TDESYu9DBiol qCfpgPCMASZlghBDnorH6oa6cYP4drrThqzLeDI4zvsdhfevu20rzLTZEB4pt+kcHJic cuzamm5I1aeaUEpxcEnDJKeoGDfdfSymTJNlYrIkbPVtwLvfl6l//Em/Q9cIGJNrFFoi 4/skdr3YZDOtqtpoKs6EryPPgMK7BCiKMDNOlF5xx2zW7McapCtxkptpmMvneureGgaV k+VYIAsJuf8BtXsjgNAE2zDrN9H0F3mc7xH7M++lbYG+zvVNaj1EpQKgjZlQt2M20PM1 SL7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705449979; x=1706054779; h=mime-version:message-id:in-reply-to:date:subject:cc:to:from :user-agent:references:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=flQyig9AOPFe3n1q3lVmc3uwr+GwHUrvpLQBwxkBOMs=; b=nnoEaQST4UcqvH9lIXYX8QzlAw71dUA0C7NUAtP+f9X6nNI05hChrHMzqi4gHQr1c3 2qmy3K2yRhmZfOxU8XJQsvShx+Aiidx0ikJgNKbnRcjzXkmw+8l6FM8ngpXy1Uf8pq8O OgQvssAFIu608LSY0cyS+0gtNiEI9LyWgYZoAERBjnhogKIYYcu+LH6n2zTnSNElLNuI +fouh3hYpA/A9wVBt5hT5NeI92Hd2XNasixKHeJ8InMhXu+oUhuUmhCGx847D0HNuTW3 OUsdSRm7X9STHRe/qDJ76pAQo2/llBxytIuHjz35J0Zt+3oDgOA0at0yu7Tql0RLVvKb Przg== X-Gm-Message-State: AOJu0YwkFAwz0/qOkrzITfDHukIQwwUxhiAoeBP5gJ/XeDbhPYkL/TxJ wwq2/41pT4mhn4LfwlA3yENKJfqQ4o9hXw== X-Google-Smtp-Source: AGHT+IEvTY7NDVCNXT+5uAcLbRlIef76VL4Po8OpmKzk0zYG+I7LlkzrIP2uo8rKwxBAAkBr8u9kQw== X-Received: by 2002:a92:c561:0:b0:35f:fa79:644 with SMTP id b1-20020a92c561000000b0035ffa790644mr14900282ilj.3.1705449979210; Tue, 16 Jan 2024 16:06:19 -0800 (PST) Received: from entropy (c-71-239-116-158.hsd1.il.comcast.net. [71.239.116.158]) by smtp.gmail.com with ESMTPSA id bp5-20020a056e02348500b00360882d77adsm3893375ilb.44.2024.01.16.16.06.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jan 2024 16:06:18 -0800 (PST) References: <874jpuijpc.fsf@gmail.com> <87y1n6igvo.fsf@localhost> <878rev1q0k.fsf@gmail.com> <877cueonkj.fsf@localhost> <87zg6dez93.fsf@gmail.com> <871qjobhwa.fsf@localhost> <877ct5fzt6.fsf@gmail.com> <87a5y1mnj0.fsf@localhost> <87msvcgjgv.fsf@gmail.com> <87le9wq2dg.fsf@localhost> User-agent: mu4e 1.8.13; emacs 30.0.50 From: Nathaniel Nicandro To: Ihor Radchenko Cc: Nathaniel Nicandro , emacs-orgmode Subject: Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements) Date: Tue, 16 Jan 2024 18:02:14 -0600 In-reply-to: <87le9wq2dg.fsf@localhost> Message-ID: <8734uwhlhj.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2607:f8b0:4864:20::132; envelope-from=nathanielnicandro@gmail.com; helo=mail-il1-x132.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: 2.41 X-Spam-Score: 2.41 X-Migadu-Queue-Id: 421F126627 X-Migadu-Scanner: mx12.migadu.com X-TUID: 0HlLPGVCaV4L --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ihor Radchenko writes: Hello, attached is another updated patch with the following changes: - Made it possible to add headlines or inline tasks to `org-ansi-highlightable-elements', these are added by default now. - To tackle the issue discussed previously about highlights spanning multiple lines (or elements) being removed when a line is modified I went ahead and used the font-lock-multiline property (see font-lock-extend-region-multiline and font-lock-extend-region-functions) across those regions so that on any edit of one of the lines, the region including all of the ANSI sequences that affect that line will be re-fontified. This was the easier solution, but the downside is that it can cause large regions to be re-fontified when really all we want to do is apply the highlighting face to a small line change, for example. An alternative solution would, when no ANSI sequences are being edited in the region being fontified and assuming a previous fontification cycle has applied highlights due to ANSI sequences already, only apply the highlighting face to the edited region instead of expanding the region before fontification. The expansion unnecessarily wastes the fontification cycle on a region larger than what it needs to be since the information needed for highlighting the region according to ANSI sequences has already been computed on a previous fontification cycle. In practice I don't think this inefficiency will matter much since I would assume most of these ANSI sequences will be inserted due to the results of code block execution or inserted by users who want to highlight small regions of the document so I would consider this problem solved by using font-lock-multiline for the time being. WDYT? - To tackle the issue of editing around the invisible ANSI sequences I left it up to the font-lock process to catch the invisible edits. Whenever an edit deletes a character of the sequence that renders the sequence invalid, the font-lock process will reveal the partial sequence. But I had to limit what was considered a valid ANSI sequence to get it working in a somewhat acceptable way. The problem that I found was that if the buffer contains something like =20=20 ^[[43mfoo =20=20 (where ^[ is the ESC character and can be inserted with "C-q ESC" and the whole sequence ^[[43m is the ANSI sequence) what was happening was that deleting into the hidden sequence would leave the region in the state =20=20 ^[[43foo =20=20 and because the end byte of the ANSI sequence can be any character in the ASCII range [@A-Z[\]^_`a=E2=80=93z{|}~], ^[[43f would still be a valid ANSI sequence and would be hidden during the fontification process after the edit. Since `ansi-color-apply-on-region' only really handles the sequences that end in an m byte, just rendering all other ones invisible, I limited the ANSI sequences handled by this patch to be only those sequences that end in m. This way, after deleting into the sequence like in the above example the fontification process would not recognize the region as containing any sequence. The downside to this solution is that sequences that end in any other end byte won't get conveniently hidden and the problem still persists if you have text that starts with an m and you delete into a hidden sequence. =20=20 An alternative solution that doesn't constrain the end byte could be to add in some extra invisible character like a zero width space and then use something like the `modification-hooks' text property on the character to signify that a deletion at the boundary between the sequence and the text should really delete part of the sequence instead of the zero width space. I haven't really worked out the details of this, for example how would it be detected which direction a deletion is coming from, the front or behind, but I'm throwing it out there to see if there are any other solutions other people might be aware of for a similar problem. =20=20 - Finally, code has been added to delete the overlays on the hidden sequences in `org-unfontify-region' so that multiple overlays are not created on re-fontifying regions containing those sequences. Other than that, the code is the same as the last patch. > P.S. I am not yet commenting on the details in the code. Please let me know what you think of this patch and where I should be focusing my efforts moving forward to get this submitted to Org. One thing I would like to start doing is writing some tests for this feature. It would be great if someone could point me to some tests that I can peruse so that I can get an idea of how I can go about writing some of my own. Also, are there any procedures or things I should be aware of when trying to write my own tests? --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Highlight-ANSI-escape-sequences.patch Content-Description: Patch >From 506e8c1e5a177b797a541b1541ea98c95668d5e1 Mon Sep 17 00:00:00 2001 From: Nathaniel Nicandro Date: Tue, 9 May 2023 19:58:11 -0500 Subject: [PATCH] Highlight ANSI escape sequences * etc/ORG-NEWS: Describe the new feature. * org.el (org-fontify-ansi-sequences): New customization variable and function which does the work of fontifying the sequences. (org-ansi-highlightable-elements) (org-ansi-highlightable-objects): New customization variables. (org-ansi-new-context, org-ansi-process-region) (org-ansi-process-block, org-ansi-process-paragraph) (org-ansi-process-fixed-width) (org-fontify-ansi-sequences-1): New functions. (org-ansi--control-seq-regexp): New variable. (org-set-font-lock-defaults): Add the `org-fontify-ansi-sequences` function to the font-lock keywords. (org-unfontify-region): Delete ANSI specific overlays. (org-ansi-mode): New minor mode to enable/disable highlighting of the sequences. Enabled in Org buffers by default. --- etc/ORG-NEWS | 12 +++ lisp/org.el | 269 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 281 insertions(+) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 1207d6f..76a81e3 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -492,6 +492,18 @@ Currently implemented options are: iCalendar programs support this usage. ** New features +*** ANSI escape sequences are now highlighted in the whole buffer + +A new customization ~org-fontify-ansi-sequences~ is available which +tells Org to highlight all ANSI sequences in the buffer if non-nil and +the new minor mode ~org-ansi-mode~ is enabled. + +To disable highlighting of the sequences you can either +disable ~org-ansi-mode~ or set ~org-fontify-ansi-sequences~ to ~nil~ +and =M-x org-mode-restart RET=. Doing the latter will disable +highlighting of sequences in all newly opened Org buffers whereas +doing the former disables highlighting locally to the current buffer. + *** =ob-plantuml.el=: Support tikz file format output =ob-plantuml.el= now output =tikz= :file format via diff --git a/lisp/org.el b/lisp/org.el index d2cd0b9..6e4744e 100644 --- a/lisp/org.el +++ b/lisp/org.el @@ -81,6 +81,7 @@ (eval-when-compile (require 'gnus-sum)) (require 'calendar) (require 'find-func) (require 'format-spec) +(require 'ansi-color) (condition-case nil (load (concat (file-name-directory load-file-name) @@ -3608,6 +3609,12 @@ (defcustom org-fontify-whole-block-delimiter-line t :group 'org-appearance :type 'boolean) +(defcustom org-fontify-ansi-sequences t + "Non-nil means to highlight ANSI escape sequences." + :group 'org-appearance + :type 'boolean + :package-version '(Org . "9.7")) + (defcustom org-highlight-latex-and-related nil "Non-nil means highlight LaTeX related syntax in the buffer. When non-nil, the value should be a list containing any of the @@ -5598,6 +5605,243 @@ (defun org-fontify-extend-region (beg end _old-len) (cons beg (or (funcall extend "end" "]" 1) end))) (t (cons beg end)))))) +(defcustom org-ansi-highlightable-elements + '(plain-list drawer headline inlinetask + example-block export-block fixed-width paragraph) + "A list of element types that will have ANSI sequences processed." + :type '(list (symbol :tag "Element Type")) + :version "9.7" + :group 'org-appearance) + +(defcustom org-ansi-highlightable-objects + '(bold code export-snippet italic macro + strike-through table-cell underline verbatim) + "A list of object types that will have ANSI sequences processed." + :type '(list (symbol :tag "Object Type")) + :version "9.7" + :group 'org-appearance) + +(defun org-ansi-new-context (pos) + (list (list (make-bool-vector 8 nil) + nil nil) + (copy-marker pos))) + +;; Only match color sequences (escape codes ending with an m). +;; +;; This effectively means that other control sequences won't get +;; conveniently hidden. +(defvar org-ansi--control-seq-regexp "\e\\[[\x30-\x3F]*[\x20-\x2F]*m") + +(defun org-ansi-process-region (beg end &optional context) + (or context (setq context (org-ansi-new-context beg))) + (move-marker (cadr context) beg) + (let ((ansi-color-context-region context) + (ansi-color-control-seq-regexp org-ansi--control-seq-regexp) + (ansi-color-apply-face-function + (lambda (beg end face) + (when face + (font-lock-prepend-text-property beg end 'face face) + (add-text-properties beg end '(font-lock-multiline t)))))) + (ansi-color-apply-on-region beg end t)) + (save-excursion + (goto-char beg) + (while (re-search-forward org-ansi--control-seq-regexp end t) + (let ((beg (match-beginning 0)) + (end (point))) + (dolist (ov (overlays-at beg)) + (when (and (= beg (overlay-start ov)) + (= end (overlay-end ov)) + (overlay-get ov 'invisible)) + ;; Assume this is the overlay added by + ;; `ansi-color-apply-on-region' + (overlay-put ov 'org-ansi t))))))) + +(defun org-ansi-process-block (el &optional context) + (let ((beg (org-element-property :begin el)) + (end (org-element-property :end el))) + (save-excursion + (goto-char beg) + (while (org-at-keyword-p) + (forward-line)) + (setq beg (line-beginning-position 2))) + (save-excursion + (goto-char end) + (skip-chars-backward " \t\n") + (setq end (line-beginning-position))) + (org-ansi-process-region beg end context))) + +(defun org-ansi-process-paragraph (el &optional context) + ;; Compute the regions of the paragraph excluding inline + ;; source blocks. + (let ((pend (org-element-property :contents-end el)) beg end) + (push (point) beg) + (while (re-search-forward + "\\