bug#73332: 31.0.50; Whitespace-mode breaking font-locking

bug#73332: 31.0.50; Whitespace-mode breaking font-locking

[Jordan Ellis Coppard via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hello,


Starting from `emacs -Q`:

With whitespace-mode enabled (no other configuration at all) in an Elisp
buffer (or any other I've tested such as Org) the last character at the
end of the buffer has all fontification removed (confirmed with 
describe-char) while typing and said character stays un-fontified until 
the buffer is re-formatted (say by killing it and re-opening, or by 
having whitespace cleanup occur).

Whitespace-mode should not be stripping all fontification of the last
character in a buffer while typing.

Open Emacs, create an Elisp buffer, enable whitespace-mode, and type a 
comment (easiest to visualise): `;; Foo bar baz`. Notice as you type 
this comment the last character has no fontification until a subsequent 
one is typed or the aforementioned buffer re-formatting occurs.

In GNU Emacs 31.0.50 (build 1, aarch64-apple-darwin23.5.0, NS
appkit-2487.60 Version 14.5 (Build 23F79)) of 2024-09-16 built on
yote.local
Windowing system distributor 'Apple', version 10.3.2487
System Description: macOS 14.5

Configured using:
'configure --prefix=/opt/local --disable-silent-rules --without-dbus
--without-gconf --without-libotf --without-m17n-flt --with-libgmp
--with-gnutls --with-xml2 --with-modules --with-sqlite3 --with-webp
--infodir /opt/local/share/info/emacs --with-native-compilation=no
--with-ns --with-lcms2 --without-harfbuzz --without-imagemagick
--without-xaw3d --with-rsvg --with-xwidgets
--with-native-compilation=aot --with-tree-sitter 'CFLAGS=-pipe -Os
-Wno-attributes
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -arch
arm64' 'CPPFLAGS=-I/opt/local/include
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk'
'LDFLAGS=-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-no_pie
-Wl,-rpath /opt/local/lib/gcc14 -Wl,-rpath /opt/local/lib
-Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk
-arch arm64''

Configured features:
ACL GIF GLIB GMP GNUTLS JPEG LCMS2 LIBXML2 MODULES NATIVE_COMP NOTIFY
KQUEUE NS PDUMPER PNG RSVG SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS
TREE_SITTER WEBP XIM XWIDGETS ZLIB

Important settings:
value of $LC_ALL: en_US.UTF-8
value of $LC_CTYPE: en_US.UTF-8
value of $LANG: en_AU.UTF-8
locale-coding-system: utf-8-unix

Major mode: Help

Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
isearch-fold-quotes-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
minibuffer-regexp-mode: t
buffer-read-only: t
line-number-mode: t
indent-tabs-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr cl-extra pp wid-edit descr-text help-mode
emacsbug message mailcap yank-media puny dired dired-loaddefs rfc822 mml
mml-sec password-cache epa derived epg rfc6068 epg-config gnus-util
text-property-search mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils time-date subr-x cl-loaddefs cl-lib
disp-table whitespace rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/ns-win ns-win ucs-normalize mule-util term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads xwidget-internal kqueue cocoa
ns lcms2 multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 63491 38364) (symbols 48 22693 0) (strings 32 84115 2344)
(string-bytes 1 1921637) (vectors 16 10958)
(vector-slots 8 222175 45778) (floats 8 139 3) (intervals 56 774 32)
(buffers 992 14))

bug#73332: 31.0.50; Whitespace-mode breaking font-locking

[Eli Zaretskii]

> Date: Wed, 18 Sep 2024 21:44:53 +0900
> From:  Jordan Ellis Coppard via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Starting from `emacs -Q`:
> 
> With whitespace-mode enabled (no other configuration at all) in an Elisp
> buffer (or any other I've tested such as Org) the last character at the
> end of the buffer has all fontification removed (confirmed with 
> describe-char) while typing and said character stays un-fontified until 
> the buffer is re-formatted (say by killing it and re-opening, or by 
> having whitespace cleanup occur).
> 
> Whitespace-mode should not be stripping all fontification of the last
> character in a buffer while typing.
> 
> Open Emacs, create an Elisp buffer, enable whitespace-mode, and type a 
> comment (easiest to visualise): `;; Foo bar baz`. Notice as you type 
> this comment the last character has no fontification until a subsequent 
> one is typed or the aforementioned buffer re-formatting occurs.

Thank you for your report.

Richard, could you please look into this?

There are at least two problems here, AFAICT.  First, whitespace-mode
does not refontify the character at EOB when there's no newline at EOB
and you move point to or from EOB.  This can be fixed by the patch
below; any comments to it?

The other problem is with fontification of the last character in the
buffer when point is at EOB: the foreground color is removed from the
last character, but the font-lock face for it is not restored.  If I
go to that character and invoke describe-text-properties, I see that
it has the 'face' property of nil, and I think that's the reason.  The
nil value comes from this part of whitespace-font-lock-keywords:

       ,@(when (memq 'missing-newline-at-eof whitespace-active-style)
           ;; Show missing newline.
           `(("[^\n]\\'" 0
              ;; Don't mark the end of the buffer is point is there --
              ;; it probably means that the user is typing something
              ;; at the end of the buffer.
              (and (/= whitespace-point (point-max))
                   'whitespace-missing-newline-at-eof)
              t)))

That 'and' expression yields nil when point is at EOB.

Stefan, is there a way of avoiding this effect, perhaps by using a
function in whitespace-font-lock-keywords?

This missing-newline-at-eof feature is new in Emacs 30, so I'd like to
fix it on the emacs-30 release branch.

Here's the patch that fixes the refontification of the last character
when there's no newline at EOB:

diff --git a/lisp/whitespace.el b/lisp/whitespace.el
index 28d131b..d1b1f84 100644
--- a/lisp/whitespace.el
+++ b/lisp/whitespace.el
@@ -2333,10 +2333,22 @@ whitespace-post-command-hook
         (font-lock-flush whitespace-eob-marker (1+ (buffer-size)))))
     (setq-local whitespace-buffer-changed nil)
     (setq whitespace-point (point))	; current point position
-    (let ((refontify (and (eolp) ; It is at end of line ...
-                          ;; ... with trailing SPACE or TAB
-                          (or (memq (preceding-char) '(?\s ?\t)))
-                          (line-beginning-position)))
+    (let ((refontify (or (and (eolp) ; It is at end of line ...
+                              ;; ... with trailing SPACE or TAB
+                              (or (memq (preceding-char) '(?\s ?\t)))
+                              (line-beginning-position))
+                         (and (memq 'missing-newline-at-eof
+                                    ;; If user requested to highlight
+                                    ;; EOB without a newline...
+                                    whitespace-active-style)
+                              ;; ...and the buffer is not empty...
+                              (not (= (point-min) (point-max)))
+                              (= (point-max) (without-restriction (point-max)))
+                              ;; ...and no newline at EOB...
+                              (not (eq (char-before (point-max)) ?\n))
+                              ;; ...then refontify the last character in
+                              ;; the buffer
+                              (max (1- (point-max)) (point-min)))))
           (ostart (overlay-start whitespace-point--used)))
       (cond
        ((not refontify)

bug#73332: 31.0.50; Whitespace-mode breaking font-locking

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

> Stefan, is there a way of avoiding this effect, perhaps by using a
> function in whitespace-font-lock-keywords?

We could of course replace the regexp with a function and thus do
"whatever we want", as in:

       ,@(when (memq 'missing-newline-at-eof whitespace-active-style)
           ;; Show missing newline.
           `(,(lambda (limit)
                (when (and (re-search-forward ".\\'" nil t)
                           ;; Don't mark the end of the buffer if point is
                           ;; there -- it probably means that the user is
                           ;; typing something at the end of the buffer.
                           (/= whitespace-point (point-max)))
                  (put-text-property (match-beginning 0)
                                     (match-end 0)
                                     'face 'whitespace-missing-newline-at-eof)
                  nil))))

But personally I think the root cause is the use of OVERRIDE and I'd
replace it with `prepend` as in the patch below, but whitespace uses
OVERRIDE "everywhere", whereas my argument would suggest it should use
`prepend` at all those places, which is likely beyond the scope of `emacs-30`.


        Stefan

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: whitespace.patch --]
[-- Type: text/x-diff, Size: 1759 bytes --]

diff --git a/lisp/whitespace.el b/lisp/whitespace.el
index 28d131b054c..3693125ec7b 100644
--- a/lisp/whitespace.el
+++ b/lisp/whitespace.el
@@ -739,7 +739,7 @@ whitespace-space-before-tab-regexp
 
 (defcustom whitespace-indentation-regexp
   '("^\t*\\(\\( \\{%d\\}\\)+\\)[^\n\t]"
-    . "^ *\\(\t+\\)[^\n]")
+    . "^ *\\(\t+\\).")
   "Regexps to match indentation whitespace that should be visualized.
 
 The value should be a cons whose car specifies the regexp to match
@@ -1752,7 +1752,7 @@ whitespace-report-region
                        ((eq (car option) 'space-after-tab::space)
                         (whitespace-space-after-tab-regexp 'space))
                        ((eq (car option) 'missing-newline-at-eof)
-                        "[^\n]\\'")
+                        ".\\'")
                        (t
                         (cdr option)))))
                  (when (re-search-forward regexp rend t)
@@ -2188,14 +2188,14 @@ whitespace-color-on
               1 whitespace-space-after-tab t)))
        ,@(when (memq 'missing-newline-at-eof whitespace-active-style)
            ;; Show missing newline.
-           `(("[^\n]\\'" 0
-              ;; Don't mark the end of the buffer is point is there --
+           `((".\\'" 0
+              ;; Don't mark the end of the buffer if point is there --
               ;; it probably means that the user is typing something
               ;; at the end of the buffer.
               (and (/= whitespace-point (point-max))
                    'whitespace-missing-newline-at-eof)
-              t)))))
-    (font-lock-add-keywords nil whitespace-font-lock-keywords t)
+              prepend)))))
+    (font-lock-add-keywords nil whitespace-font-lock-keywords 'append)
     (font-lock-flush)))
 
 

bug#73332: 31.0.50; Whitespace-mode breaking font-locking

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Jordan Ellis Coppard <jc+o.emacs@wz.ht>,  Richard Hansen
>  <rhansen@rhansen.org>,  73332@debbugs.gnu.org
> Date: Thu, 19 Sep 2024 11:50:26 -0400
> 
> But personally I think the root cause is the use of OVERRIDE and I'd
> replace it with `prepend` as in the patch below, but whitespace uses
> OVERRIDE "everywhere", whereas my argument would suggest it should use
> `prepend` at all those places, which is likely beyond the scope of `emacs-30`.

Thanks.  I installed on the the change in your name (in addition to my
part,macs-30 branch which fixes refontification of the last character
at EOB when point moves).  I hope I didn't misunderstand you, and
removing OVERRIDE in that one place together with using 'prepend' will
not cause any unintended consequences.

bug#73332: 31.0.50; Whitespace-mode breaking font-locking

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Thanks.  I installed on the the change in your name (in addition to my
> part,macs-30 branch which fixes refontification of the last character
> at EOB when point moves).  I hope I didn't misunderstand you, and
> removing OVERRIDE in that one place together with using 'prepend' will
> not cause any unintended consequences.

LGTM, thank you,


        Stefan