Re: [PATCH] Highlight ANSI sequences in the whole buffer (was [PATCH] ANSI color on example blocks and fixed width elements)

[Ihor Radchenko <yantar92@posteo.net>]

Nathaniel Nicandro <nathanielnicandro@gmail.com> writes:

> Hello, attached is another updated patch with the following changes:

Thanks!

> - To tackle the issue discussed previously about highlights spanning
>   multiple lines (or elements) being removed when a line is modified I
>   went ahead and used the font-lock-multiline property (see
>   font-lock-extend-region-multiline and
>   font-lock-extend-region-functions) across those regions so that on
>   any edit of one of the lines, the region including all of the ANSI
>   sequences that affect that line will be re-fontified.  This was the
>   easier solution, but the downside is that it can cause large regions
>   to be re-fontified when really all we want to do is apply the
>   highlighting face to a small line change, for example.

This is fine.

> - To tackle the issue of editing around the invisible ANSI sequences I
>   left it up to the font-lock process to catch the invisible edits.
>   Whenever an edit deletes a character of the sequence that renders
>   the sequence invalid, the font-lock process will reveal the partial
>   sequence.  But I had to limit what was considered a valid ANSI
>   sequence to get it working in a somewhat acceptable way.
>
>   The problem that I found was that if the buffer contains something
>   like
>   
>   ^[[43mfoo
>   
>   (where ^[ is the ESC character and can be inserted with "C-q ESC" and
>   the whole sequence ^[[43m is the ANSI sequence) what was happening was
>   that deleting into the hidden sequence would leave the region in the
>   state
>   
>   ^[[43foo
>   
>   and because the end byte of the ANSI sequence can be any character
>   in the ASCII range [@A-Z[\]^_`a–z{|}~], ^[[43f would still be a
>   valid ANSI sequence and would be hidden during the fontification
>   process after the edit.  Since `ansi-color-apply-on-region' only
>   really handles the sequences that end in an m byte, just rendering
>   all other ones invisible, I limited the ANSI sequences handled by
>   this patch to be only those sequences that end in m.  This way,
>   after deleting into the sequence like in the above example the
>   fontification process would not recognize the region as containing
>   any sequence.  The downside to this solution is that sequences that
>   end in any other end byte won't get conveniently hidden and the
>   problem still persists if you have text that starts with an m and
>   you delete into a hidden sequence.

Makes sense. We may also make hiding ^[[43foo as customization disabled
by default.
   
>   An alternative solution that doesn't constrain the end byte could be
>   to add in some extra invisible character like a zero width space and
>   then use something like the `modification-hooks' text property on
>   the character to signify that a deletion at the boundary between the
>   sequence and the text should really delete part of the sequence
>   instead of the zero width space.  I haven't really worked out the
>   details of this, for example how would it be detected which
>   direction a deletion is coming from, the front or behind, but I'm
>   throwing it out there to see if there are any other solutions other
>   people might be aware of for a similar problem.

If you want to go in this direction, check out
`org-fold-check-before-invisible-edit'. We can unfontify the escape
sequence from there and font-lock will re-apply only during the next
editing cycle, making the sequence visible temporarily.
Not mandatory though.

>> P.S. I am not yet commenting on the details in the code.
>
> Please let me know what you think of this patch and where I should be
> focusing my efforts moving forward to get this submitted to Org.

I tried to test your newest patch with the example file you provided and
I notice two things that would be nice:

1. It is a bit confusing to understand why one or other text is colored
   without seeing the escape characters. Some customization like
   `org-link-descriptive' and a command like `org-toggle-link-display'
   would be nice. I can see some users prefer seeing the escape codes.

2. Using overlays for fontification is problematic. In your example
   file, table alignment becomes broken when escape sequences are hidden
   inside overlays:

   | [31mcell 1 | cell 2 |
   | cell 3       | cell 4 |

   looks like

   |       cell 1 | cell 2 |
   | cell 3       | cell 4 |

   Using text properties would make table alignment work without
   adjustments in the org-table.el code.

> One thing I would like to start doing is writing some tests for this
> feature.  It would be great if someone could point me to some tests
> that I can peruse so that I can get an idea of how I can go about
> writing some of my own.  Also, are there any procedures or things I
> should be aware of when trying to write my own tests?

Check out testing/README file in the Org repository.

Unfortunately, we do not yet have any existing tests for font-locking in
Org tests. You may still refer to the files in testing/lisp/ to see some
example tests.

Also, Emacs has built-in library to help writing font-lock tests -
faceup.el. You may consider using it. Its top comment also contains a
number of references to various tools that could be useful to diagnose
font-locking code.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>