Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)

[Nathaniel Nicandro <nathanielnicandro@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6999 bytes --]


Ihor Radchenko <yantar92@posteo.net> writes:

> Nathaniel Nicandro <nathanielnicandro@gmail.com> writes:
>
>>> Ideally, fontifying ANSI sequences should be fully controlled by users:
>>> 1. We may not want to touch src blocks by default, when
>>>    `org-src-fontify-natively' is set to t. Only, maybe, provide an
>>>    option. Or you may better publish a minor mode that does this for
>>>    shell scripts.
>>> 2. We may allow all the ANSI sequences to be fontified in the whole
>>>    buffer.
>>
>> I've updated my patch to be a combination of (1) and (2), see the
>> attached patch.  Essentially every sequence is fontified except those in
>> source blocks and a minor mode has been created to allow users to
>> disable or enable fontification whenever they want.
>>
>> I've also attached an example Org file with some ANSI sequences in it
>> for testing purposes that you can try out.
>
> Thanks!
>
>> One issue that remains is how to handle sequences within inline source
>> blocks.  Those don't have a src-block property so any sequences within
>> an inline source block are currently fontified.
>
> You should not use 'src-block property at all. There are scenarios when
> jit-lock defers source block fontification (in particular, when source
> block spans beyond the screen) and 'src-block property is not yet
> applied.
>
> Instead, you should query `org-element-at-point' or
> `org-element-context'.

The attached patch now uses `org-element-at-point' and
`org-element-context' to query for the bounds of elements.

Note, I've also attached an updated example file which shows that the
escape sequences in inline source blocks are now handled similarly to
regular source blocks, i.e. they are not fontified.

>
>> +*** ANSI escape sequences are now highlighted in the whole buffer
>> +
>> +A new customization ~org-fontify-ansi-sequences~ is available which
>> +tells Org to highlight all ANSI sequences in the buffer if non-nil and
>> +the new minor mode ~org-ansi-mode~ is enabled.
>> +
>> +To disable highlighting of the sequences you can either
>> +disable ~org-ansi-mode~ or set ~org-fontify-ansi-sequences~ to ~nil~
>> +and =M-x revert-buffer RET=.  Doing the latter will disable
>> +highlighting of sequences in all newly opened Org buffers whereas
>> +doing the former disables highlighting locally to the current buffer.
>
> Rather than asking to use revert-buffer, we usually suggest M-x
> org-mode-restart.

Done.

>
>> +(defun org-fontify-ansi-sequences (limit)
>> +  "Fontify ANSI sequences."
>> +  (when (and org-fontify-ansi-sequences org-ansi-mode)
>> +    (let (end)
>> +      (while (/= (point) limit)
>
> Instead of this strict condition and later juggle with
> `narrow-to-region', just use the usual (while (< (point) limit) ...).
>

Done.

>> +        (cond
>> +         ((get-text-property (point) 'src-block)
>
> As I mentioned above, please use `org-element-at-point'. This function
> will also give you information about the block boundaries.
>
>> +            (ansi-color-apply-on-region (point) end t)
>
> We should probably limit ANSI colour pairs to a single Org element. It
> does not make much sense to have text in-between the quotes below
> coloured:
>
> #+begin_quote
> ... <opening ANSI def> ...
> #+end_quote
>
>
> ....
>
> #+begin_quote
> ... <closing ANSI def> ...
> #+end_quote
>

Makes sense. Done.

>> +            ;; Reset the context before every fontification cycle.  This
>> +            ;; avoids issues where `ansi-color-apply-on-region' attempts to
>> +            ;; use an old starting point that may be from a different part
>> +            ;; of the buffer, leading to "wrong side of point" errors.
>> +            (setq ansi-color-context-region nil)
>
> This looks fragile. AFAIU, `ansi-color-context-region' is used to track
> currently active ANSI colour settings. Since your fontification function
> may be called with various LIMITs, depending on what is displayed on the
> user screen, the fontification results might be unpredictable for ANSI
> defs spanning across multiple screens.
>

It seems to be safe to reset `ansi-color-context-region' now given that
org-element is used to find the bounds of the element at
`point'. Although the fontification limits are dependent on screen size,
the org-element functions are not and so the bounds used when applying
the fontification for the ANSI sequences won't depend on screen size
either.

Also, re-setting `ansi-color-context-region' has the effect of not
propagating previously applied color settings to other Org elements.

>> +(defvar org-ansi-colors
>> +  '(black red green yellow blue purple cyan white))
>> +
>> +(defun org-ansi-highlight (beg end seq)
>> +  (save-excursion
>> +    (goto-char end)
>> +    (insert "\e")
>> +    (insert "[0m")
>> +    (goto-char beg)
>> +    (insert "\e")
>> +    (insert (format "[%sm" seq))))
>> +
>> +(defun org-ansi-highlight-foreground (beg end color)
>> +  "Highlight the foreground between BEG and END with COLOR."
>> +  (interactive
>> +   (let ((bounds (car (region-bounds))))
>> +     (list (car bounds) (cdr bounds) 
>> +           (completing-read "Color: " org-ansi-colors nil t))))
>> +  (let ((n (- (length org-ansi-colors)
>> +              (length (memq color org-ansi-colors)))))
>> +    (org-ansi-highlight beg end (+ 30 n))))
>> +
>> +(defun org-ansi-highlight-background (beg end color)
>> +  "Highlight the background between BEG and END with COLOR."
>> +  (interactive
>> +   (let ((bounds (car (region-bounds))))
>> +     (list (car bounds) (cdr bounds) 
>> +           (completing-read "Color: " org-ansi-colors nil t))))
>> +  (let ((n (- (length org-ansi-colors)
>> +              (length (memq color org-ansi-colors)))))
>> +    (org-ansi-highlight beg end (+ 40 n))))
>
> The above has no relation to fontification and does not belong to Org in
> general. Org syntax has no notion of ANSI escapes. We may support them
> as a useful feature, but no more. Editing ANSI escapes would make more
> sense in shell-script-mode or similar.

Removed.

>
>> +  :lighter " OANSI"
>> +  (cond
>> +   ((and org-fontify-ansi-sequences org-ansi-mode)
>> +    (remove-text-properties (point-min) (point-max) '(fontified t))
>> +    (font-lock-ensure))
>
> Just use `org-restart-font-lock'.
>

Thanks. Done.

>> +   (t
>> +    (dolist (ov (overlays-in (point-min) (point-max)))
>> +      ;; Attempt to find ANSI specific overlays.  See
>> +      ;; `ansi-color-make-extent'.
>> +      (when (eq (car-safe (overlay-get ov 'insert-behind-hooks))
>> +                'ansi-color-freeze-overlay)
>
> This is extremely awkward and relies on internal implementation details
> of ansi-color. Moreover, we must avoid overlays, if possible - they do
> not scale well. I recommend re-defining `ansi-color-apply-face-function'
> to something that uses text properties. Using text properties will also
> make restarting font-lock sufficient to clear the fontification.

I've re-defined `ansi-color-apply-face-function' as you've
said. 


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: test-ansi.org --]
[-- Type: text/x-org, Size: 879 bytes --]

* This is a ^[[42mtest^[[0m
:PROPERTIES:
:CUSTOM_ID: 123
:END:

Of ^[[31mANSI^[[0m ^[[33mcolor^[[0m sequences

#+begin_src python
for x in y:
    print(x + "this is a ^[[43mtest^[[0m")
#+end_src

: this ^[[42mis a^[[0m td

=testing=

In paragraph a ~color ^[[44msequ^[[0mence~ is ^[[41mhere^[[0m.

^[[43mThis is a sequence that covers a block

#+begin_example
shouldn't be colored
#+end_example

there should be an end here^[[0m there is the end.

begin  ^[[43m
sequence
without end
#+begin_src python
1 + 1
#+end_src

#+begin_quote
open ^[[43m
#+end_quote

should not be highlighted

#+begin_quote
close ^[[0m
#+end_quote

This is a paragraph src_python{return "t^[[43mest^[[0ming"} {{{results(=t^[[43mest^[[0ming=)}}} with
multiple inline src_python{return 5*4} {{{results(=20=)}}} source blocks.

An inline source block src_python{return 1+ 1 without an
end.  src_python{return "t^[[43mest^[[0ming"}.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: Patch --]
[-- Type: text/x-patch, Size: 6527 bytes --]

From c59d39d76266670200f9cfe70a1e1c2dad04c8bc Mon Sep 17 00:00:00 2001
From: Nathaniel Nicandro <nathanielnicandro@gmail.com>
Date: Tue, 9 May 2023 19:58:11 -0500
Subject: [PATCH] Highlight ANSI escape sequences

* etc/ORG-NEWS: Describe the new feature.
* org.el (org-fontify-ansi-sequences): New customization variable and
function which does the work of fontifying the sequences.
(org-set-font-lock-defaults): Add the `org-fontify-ansi-sequences`
function to the font-lock keywords.
(org-ansi-mode): New minor mode to enable/disable highlighting of the
sequences.  Enabled in Org buffers by default.
---
 etc/ORG-NEWS | 12 ++++++++
 lisp/org.el  | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 94 insertions(+)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index b7c88fd..2c28785 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -169,6 +169,18 @@ official [[https://clojure.org/guides/deps_and_cli][Clojure CLI tools]].
 The command can be customized with ~ob-clojure-cli-command~.
 
 ** New features
+*** ANSI escape sequences are now highlighted in the whole buffer
+
+A new customization ~org-fontify-ansi-sequences~ is available which
+tells Org to highlight all ANSI sequences in the buffer if non-nil and
+the new minor mode ~org-ansi-mode~ is enabled.
+
+To disable highlighting of the sequences you can either
+disable ~org-ansi-mode~ or set ~org-fontify-ansi-sequences~ to ~nil~
+and =M-x org-mode-restart RET=.  Doing the latter will disable
+highlighting of sequences in all newly opened Org buffers whereas
+doing the former disables highlighting locally to the current buffer.
+
 *** Add support for ~logind~ idle time in ~org-user-idle-seconds~
 
 When Emacs is built with =dbus= support and
diff --git a/lisp/org.el b/lisp/org.el
index 26d2a86..6742449 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -81,6 +81,7 @@ (eval-when-compile (require 'gnus-sum))
 (require 'calendar)
 (require 'find-func)
 (require 'format-spec)
+(require 'ansi-color)
 
 (condition-case nil
     (load (concat (file-name-directory load-file-name)
@@ -3582,6 +3583,12 @@ (defcustom org-fontify-whole-block-delimiter-line t
   :group 'org-appearance
   :type 'boolean)
 
+(defcustom org-fontify-ansi-sequences t
+  "Non-nil means to highlight ANSI escape sequences."
+  :group 'org-appearance
+  :type 'boolean
+  :package-version '(Org . "9.7"))
+
 (defcustom org-highlight-latex-and-related nil
   "Non-nil means highlight LaTeX related syntax in the buffer.
 When non-nil, the value should be a list containing any of the
@@ -5543,6 +5550,66 @@ (defun org-fontify-extend-region (beg end _old-len)
 	     (cons beg (or (funcall extend "end" "]" 1) end)))
 	    (t (cons beg end))))))
 
+(defvar org-ansi-mode)
+
+(defun org-fontify-ansi-sequences (limit)
+  "Fontify ANSI sequences."
+  (when (and org-fontify-ansi-sequences org-ansi-mode)
+    (while (< (point) limit)
+      (let ((el (org-element-at-point)) beg end next)
+        (pcase (org-element-type el)
+          (`src-block
+           (setq beg (org-element-property :end el)
+                 end beg
+                 next end))
+          (`headline
+           (setq beg (org-element-property :begin el)
+                 end (org-element-property :contents-begin el)
+                 next end))
+          (`paragraph
+           ;; Compute the regions of the paragraph excluding inline
+           ;; source blocks.
+           (setq beg nil end nil)
+           (let ((pbeg (org-element-property :begin el))
+                 (pend (org-element-property :end el)))
+             (goto-char pbeg)
+             (push pbeg beg)
+             (while (re-search-forward
+                     "\\<src_\\([^ \t\n[{]+\\)[{[]" pend t)
+               (let ((el (org-element-context)))
+                 (when (eq (org-element-type el) 'inline-src-block)
+                   (push (org-element-property :begin el) end)
+                   (goto-char (org-element-property :end el))
+                   (push (point) beg))))
+             (push pend end)
+             (setq beg (nreverse beg)
+                   end (nreverse end)
+                   next pend)))
+          (_
+           (setq beg (or (org-element-property :contents-begin el)
+                         (org-element-property :begin el))
+                 end (or (org-element-property :contents-end el)
+                         (org-element-property :end el))
+                 next (org-element-property :end el))))
+        (cl-letf (((symbol-function #'delete-region)
+                   (lambda (beg end)
+                     (add-text-properties beg end '(invisible t))))
+                  (ansi-color-apply-face-function
+                   (lambda (beg end face)
+                     (font-lock-prepend-text-property beg end 'face face))))
+          (if (consp beg)
+              (while (consp beg)
+                (ansi-color-apply-on-region (pop beg) (pop end)))
+            (ansi-color-apply-on-region beg end)))
+        ;; Reset the context after applying the color to prevent color
+        ;; settings from propagating to other elements.  This also
+        ;; avoids issues where `ansi-color-apply-on-region' attempts
+        ;; to use an old starting point that may be from a different
+        ;; part of the buffer, leading to "wrong side of point"
+        ;; errors.
+        (setq ansi-color-context-region nil)
+        (goto-char next)))))
+
 (defun org-activate-footnote-links (limit)
   "Add text properties for footnotes."
   (let ((fn (org-footnote-next-reference-or-definition limit)))
@@ -5861,6 +5928,7 @@ (defun org-set-font-lock-defaults ()
 	  ;; Blocks and meta lines
 	  '(org-fontify-meta-lines-and-blocks)
           '(org-fontify-inline-src-blocks)
+          '(org-fontify-ansi-sequences)
           ;; Citations.  When an activate processor is specified, if
           ;; specified, try loading it beforehand.
           (progn
@@ -15455,6 +15523,20 @@ (defun org-agenda-prepare-buffers (files)
     (when org-agenda-file-menu-enabled
       (org-install-agenda-files-menu))))
 
+\f
+;;;; ANSI minor mode
+
+(define-minor-mode org-ansi-mode
+  "Toggle the minor `org-ansi-mode'.
+This mode adds support to highlight ANSI sequences in Org mode.
+The sequences are highlighted only if the customization
+`org-fontify-ansi-sequences' is non-nil when the mode is enabled.
+\\{org-ansi-mode-map}"
+  :lighter " OANSI"
+  (org-restart-font-lock))
+
+(add-hook 'org-mode-hook #'org-ansi-mode)
+
 \f
 ;;;; CDLaTeX minor mode
 
-- 
2.39.1


[-- Attachment #4: Type: text/plain, Size: 15 bytes --]


-- 
Nathaniel