bug#73332: 31.0.50; Whitespace-mode breaking font-locking

[Eli Zaretskii <eliz@gnu.org>]

> Date: Wed, 18 Sep 2024 21:44:53 +0900
> From:  Jordan Ellis Coppard via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Starting from `emacs -Q`:
> 
> With whitespace-mode enabled (no other configuration at all) in an Elisp
> buffer (or any other I've tested such as Org) the last character at the
> end of the buffer has all fontification removed (confirmed with 
> describe-char) while typing and said character stays un-fontified until 
> the buffer is re-formatted (say by killing it and re-opening, or by 
> having whitespace cleanup occur).
> 
> Whitespace-mode should not be stripping all fontification of the last
> character in a buffer while typing.
> 
> Open Emacs, create an Elisp buffer, enable whitespace-mode, and type a 
> comment (easiest to visualise): `;; Foo bar baz`. Notice as you type 
> this comment the last character has no fontification until a subsequent 
> one is typed or the aforementioned buffer re-formatting occurs.

Thank you for your report.

Richard, could you please look into this?

There are at least two problems here, AFAICT.  First, whitespace-mode
does not refontify the character at EOB when there's no newline at EOB
and you move point to or from EOB.  This can be fixed by the patch
below; any comments to it?

The other problem is with fontification of the last character in the
buffer when point is at EOB: the foreground color is removed from the
last character, but the font-lock face for it is not restored.  If I
go to that character and invoke describe-text-properties, I see that
it has the 'face' property of nil, and I think that's the reason.  The
nil value comes from this part of whitespace-font-lock-keywords:

       ,@(when (memq 'missing-newline-at-eof whitespace-active-style)
           ;; Show missing newline.
           `(("[^\n]\\'" 0
              ;; Don't mark the end of the buffer is point is there --
              ;; it probably means that the user is typing something
              ;; at the end of the buffer.
              (and (/= whitespace-point (point-max))
                   'whitespace-missing-newline-at-eof)
              t)))

That 'and' expression yields nil when point is at EOB.

Stefan, is there a way of avoiding this effect, perhaps by using a
function in whitespace-font-lock-keywords?

This missing-newline-at-eof feature is new in Emacs 30, so I'd like to
fix it on the emacs-30 release branch.

Here's the patch that fixes the refontification of the last character
when there's no newline at EOB:

diff --git a/lisp/whitespace.el b/lisp/whitespace.el
index 28d131b..d1b1f84 100644
--- a/lisp/whitespace.el
+++ b/lisp/whitespace.el
@@ -2333,10 +2333,22 @@ whitespace-post-command-hook
         (font-lock-flush whitespace-eob-marker (1+ (buffer-size)))))
     (setq-local whitespace-buffer-changed nil)
     (setq whitespace-point (point))	; current point position
-    (let ((refontify (and (eolp) ; It is at end of line ...
-                          ;; ... with trailing SPACE or TAB
-                          (or (memq (preceding-char) '(?\s ?\t)))
-                          (line-beginning-position)))
+    (let ((refontify (or (and (eolp) ; It is at end of line ...
+                              ;; ... with trailing SPACE or TAB
+                              (or (memq (preceding-char) '(?\s ?\t)))
+                              (line-beginning-position))
+                         (and (memq 'missing-newline-at-eof
+                                    ;; If user requested to highlight
+                                    ;; EOB without a newline...
+                                    whitespace-active-style)
+                              ;; ...and the buffer is not empty...
+                              (not (= (point-min) (point-max)))
+                              (= (point-max) (without-restriction (point-max)))
+                              ;; ...and no newline at EOB...
+                              (not (eq (char-before (point-max)) ?\n))
+                              ;; ...then refontify the last character in
+                              ;; the buffer
+                              (max (1- (point-max)) (point-min)))))
           (ostart (overlay-start whitespace-point--used)))
       (cond
        ((not refontify)