bug#56818: 28.1; c-mode font-lock issues in Emacs 28

bug#56818: 28.1; c-mode font-lock issues in Emacs 28

[Bill Sacks]

[-- Attachment #1.1: Type: text/plain, Size: 5621 bytes --]

Starting with Emacs 28, I have been seeing font-lock issues when editing 
C and C++ code. The situation where I see this the most (though I'm not 
sure if it's the only situation) is when I am writing a comment and 
currently have a space at the end of the comment line: in this 
situation, the fontification of a variable name or function name on the 
next line becomes broken until I type a non-space character to end the 
current line.

The attached screen shots illustrate the problem: nospaces.png shows the 
correct fontification; space_before_var.png and 
space_before_function.png show that variable and function names lose 
their fontification when there is a space at the end of the previous 
comment line. Running M-x font-lock-fontify-buffer temporarily fixes the 
issue.

The problem occurs even when using emacs -Q. I have tried the latest 
emacs28 pretest and the latest nightly build available from 
emacsformacosx (though with my customizations – NOT with emacs -Q) and 
those also exhibit the problem. The latest emacs27 from emacsformacosx 
does NOT have this issue.

(I have also been seeing slightly similar font-lock issues when editing 
python code with Emacs 28. There, I get inconsistent fontification of 
variables: variables sometimes are not fontified with 
font-lock-variable-name-face when I initially load a file, but then if I 
edit a line – e.g., by adding a space at the end of the line – then the 
variable defined on that line will become properly fontified. I realize 
that this may be a separate issue that may warrant its own bug report, 
but I thought I'd mention it in case it's related.)

Thank you,
Bill


In GNU Emacs 28.1 (build 2, aarch64-apple-darwin21.5.0, NS 
appkit-2113.50 Version 12.4 (Build 21F79))
of 2022-07-19 built on cgdm-green
Windowing system distributor 'Apple', version 10.3.2113
System Description: macOS 12.4

Configured using:
'configure --disable-dependency-tracking --disable-silent-rules
--enable-locallisppath=/opt/homebrew/share/emacs/site-lisp
--infodir=/opt/homebrew/Cellar/emacs-plus@28/28.1/share/info/emacs
--prefix=/opt/homebrew/Cellar/emacs-plus@28/28.1 --with-xml2
--with-gnutls --with-native-compilation --without-dbus
--without-imagemagick --with-modules --with-rsvg --with-ns
--disable-ns-self-contained 'CFLAGS=-Os -w -pipe
-mmacosx-version-min=12
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk'
'CPPFLAGS=-I/opt/homebrew/opt/zlib/include
-I/opt/homebrew/opt/icu4c/include -I/opt/homebrew/opt/sqlite/include
-I/opt/homebrew/opt/openssl@1.1/include
-I/opt/homebrew/opt/readline/include -I/opt/homebrew/opt/libffi/include
-isystem/opt/homebrew/include -F/opt/homebrew/Frameworks
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk'
'LDFLAGS=-L/opt/homebrew/opt/zlib/lib -L/opt/homebrew/opt/icu4c/lib
-L/opt/homebrew/opt/sqlite/lib -L/opt/homebrew/opt/openssl@1.1/lib
-L/opt/homebrew/opt/readline/lib -L/opt/homebrew/opt/libffi/lib
-L/opt/homebrew/lib -F/opt/homebrew/Frameworks
-Wl,-headerpad_max_install_names
-isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk''

Configured features:
ACL GLIB GMP GNUTLS JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP NOTIFY
KQUEUE NS PDUMPER PNG RSVG THREADS TIFF TOOLKIT_SCROLL_BARS XIM ZLIB

Important settings:
value of $LANG: en_US.UTF-8
locale-coding-system: utf-8-unix

Major mode: C/*l

Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
line-number-mode: t
indent-tabs-mode: t
transient-mark-mode: t
abbrev-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search seq byte-opt
gv bytecomp byte-compile cconv mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils time-date subr-x jka-compr info
vc-dispatcher vc-svn cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs cl-loaddefs cl-lib
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/ns-win ns-win ucs-normalize
mule-util term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads kqueue cocoa ns lcms2
multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 107172 9071)
(symbols 48 8698 0)
(strings 32 28953 1830)
(string-bytes 1 1020321)
(vectors 16 16903)
(vector-slots 8 352351 17895)
(floats 8 29 38)
(intervals 56 546 0)
(buffers 992 13))


[-- Attachment #1.2: Type: text/html, Size: 6330 bytes --]

[-- Attachment #2: space_before_var.png --]
[-- Type: image/png, Size: 211283 bytes --]

[-- Attachment #3: space_before_function.png --]
[-- Type: image/png, Size: 211237 bytes --]

[-- Attachment #4: nospaces.png --]
[-- Type: image/png, Size: 216824 bytes --]

bug#56818: 28.1; c-mode font-lock issues in Emacs 28

[Eli Zaretskii]

> From: Bill Sacks <sacks@ucar.edu>
> Date: Thu, 28 Jul 2022 14:32:09 -0600
> 
> Starting with Emacs 28, I have been seeing font-lock issues when editing C and C++ code. The situation
> where I see this the most (though I'm not sure if it's the only situation) is when I am writing a comment and
> currently have a space at the end of the comment line: in this situation, the fontification of a variable name or
> function name on the next line becomes broken until I type a non-space character to end the current line.
> 
> The attached screen shots illustrate the problem: nospaces.png shows the correct fontification;
> space_before_var.png and space_before_function.png show that variable and function names lose their
> fontification when there is a space at the end of the previous comment line. Running M-x
> font-lock-fontify-buffer temporarily fixes the issue.
> 
> The problem occurs even when using emacs -Q. I have tried the latest emacs28 pretest and the latest
> nightly build available from emacsformacosx (though with my customizations – NOT with emacs -Q) and
> those also exhibit the problem. The latest emacs27 from emacsformacosx does NOT have this issue.

Alan, this seems to be a regression in Emacs 28, so could you please
look into it?

bug#56818: 28.1; c-mode font-lock issues in Emacs 28

[Alan Mackenzie]

Hello, Bill and Eli.

On Fri, Jul 29, 2022 at 08:56:21 +0300, Eli Zaretskii wrote:
> > From: Bill Sacks <sacks@ucar.edu>
> > Date: Thu, 28 Jul 2022 14:32:09 -0600

First of all (Bill), thanks for taking the trouble to report this bug,
and thanks even more for cutting the test case down to the short fragment
in your screenshots.

> > Starting with Emacs 28, I have been seeing font-lock issues when
> > editing C and C++ code. The situation where I see this the most
> > (though I'm not sure if it's the only situation) is when I am writing
> > a comment and currently have a space at the end of the comment line:
> > in this situation, the fontification of a variable name or function
> > name on the next line becomes broken until I type a non-space
> > character to end the current line.

> > The attached screen shots illustrate the problem: nospaces.png shows
> > the correct fontification; space_before_var.png and
> > space_before_function.png show that variable and function names lose
> > their fontification when there is a space at the end of the previous
> > comment line. Running M-x font-lock-fontify-buffer temporarily fixes
> > the issue.

> > The problem occurs even when using emacs -Q. I have tried the latest
> > emacs28 pretest and the latest nightly build available from
> > emacsformacosx (though with my customizations – NOT with emacs -Q)
> > and those also exhibit the problem. The latest emacs27 from
> > emacsformacosx does NOT have this issue.

This is a coding bug in an optimisation from March 2020, where the
complaint was that scrolling over a 2,000 line macro was slow.  The fix
neglected the possibility of spaces at the end of comment lines.

Could you please apply the following patch in your Emacs-28.1, byte
compile the file ..../lisp/progmodes/cc-engine.el, then try out the
result on your real code.  (If you want any help with the patching or
byte compiling, feel free to send me private mail.)  Then please confirm
that the bug is fixed, or tell us how it's not fixed.  Thanks!



diff -r 9c649274b259 cc-engine.el
--- a/cc-engine.el	Tue Jul 26 20:08:39 2022 +0000
+++ b/cc-engine.el	Fri Jul 29 17:25:16 2022 +0000
@@ -1679,9 +1679,13 @@
 Return the result of `forward-comment' if it gets called, nil otherwise."
   `(if (not comment-end-can-be-escaped)
        (forward-comment -1)
-     (when (and (< (skip-syntax-backward " >") 0)
-		(eq (char-after) ?\n))
-       (forward-char))
+     (let ((dist (skip-syntax-backward " >")))
+       (when (and
+	      (< dist 0)
+	      (progn
+		(skip-syntax-forward " " (- (point) dist 1))
+		(eq (char-after) ?\n)))
+	 (forward-char)))
      (cond
       ((and (eq (char-before) ?\n)
 	    (eq (char-before (1- (point))) ?\\))




> Alan, this seems to be a regression in Emacs 28, so could you please
> look into it?

Eli, Do I understand you want the fix in the release branch?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#56818: 28.1; c-mode font-lock issues in Emacs 28

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 17:44:58 +0000
> Cc: 56818@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> diff -r 9c649274b259 cc-engine.el
> --- a/cc-engine.el	Tue Jul 26 20:08:39 2022 +0000
> +++ b/cc-engine.el	Fri Jul 29 17:25:16 2022 +0000
> @@ -1679,9 +1679,13 @@
>  Return the result of `forward-comment' if it gets called, nil otherwise."
>    `(if (not comment-end-can-be-escaped)
>         (forward-comment -1)
> -     (when (and (< (skip-syntax-backward " >") 0)
> -		(eq (char-after) ?\n))
> -       (forward-char))
> +     (let ((dist (skip-syntax-backward " >")))
> +       (when (and
> +	      (< dist 0)
> +	      (progn
> +		(skip-syntax-forward " " (- (point) dist 1))
> +		(eq (char-after) ?\n)))
> +	 (forward-char)))
>       (cond
>        ((and (eq (char-before) ?\n)
>  	    (eq (char-before (1- (point))) ?\\))
> 
> 
> 
> 
> > Alan, this seems to be a regression in Emacs 28, so could you please
> > look into it?
> 
> Eli, Do I understand you want the fix in the release branch?

Yes, please.

bug#56818: 28.1; c-mode font-lock issues in Emacs 28

[Bill Sacks]

[-- Attachment #1: Type: text/plain, Size: 3873 bytes --]

Thank you very much for this quick reply and fix, Alan! Yes, from a 
couple of tests, this fixes the issue I was having (and stops me from 
getting distracted by constantly changing fontification while writing a 
comment – thank you!). I'll let you know if I notice any remaining issues.

This does not fix the issue I'm seeing with fontification of variable 
names in python-mode (which also appears to be an Emacs 28 regression), 
but given that this fix is specific to cc-engine, the python-mode issue 
is apparently different. That one seems a little harder to reproduce, so 
it may take me a while to develop a simple reproducer for it, especially 
since I haven't been doing much python programming recently. However, 
please let me know if you would like me to prioritize opening an issue 
for that one: I can do so sooner if it would be helpful.

Thanks again,
Bill

Alan Mackenzie wrote on 7/29/22 11:44 AM:
> Hello, Bill and Eli.
>
> On Fri, Jul 29, 2022 at 08:56:21 +0300, Eli Zaretskii wrote:
>>> From: Bill Sacks <sacks@ucar.edu>
>>> Date: Thu, 28 Jul 2022 14:32:09 -0600
> First of all (Bill), thanks for taking the trouble to report this bug,
> and thanks even more for cutting the test case down to the short fragment
> in your screenshots.
>
>>> Starting with Emacs 28, I have been seeing font-lock issues when
>>> editing C and C++ code. The situation where I see this the most
>>> (though I'm not sure if it's the only situation) is when I am writing
>>> a comment and currently have a space at the end of the comment line:
>>> in this situation, the fontification of a variable name or function
>>> name on the next line becomes broken until I type a non-space
>>> character to end the current line.
>>> The attached screen shots illustrate the problem: nospaces.png shows
>>> the correct fontification; space_before_var.png and
>>> space_before_function.png show that variable and function names lose
>>> their fontification when there is a space at the end of the previous
>>> comment line. Running M-x font-lock-fontify-buffer temporarily fixes
>>> the issue.
>>> The problem occurs even when using emacs -Q. I have tried the latest
>>> emacs28 pretest and the latest nightly build available from
>>> emacsformacosx (though with my customizations – NOT with emacs -Q)
>>> and those also exhibit the problem. The latest emacs27 from
>>> emacsformacosx does NOT have this issue.
> This is a coding bug in an optimisation from March 2020, where the
> complaint was that scrolling over a 2,000 line macro was slow.  The fix
> neglected the possibility of spaces at the end of comment lines.
>
> Could you please apply the following patch in your Emacs-28.1, byte
> compile the file ..../lisp/progmodes/cc-engine.el, then try out the
> result on your real code.  (If you want any help with the patching or
> byte compiling, feel free to send me private mail.)  Then please confirm
> that the bug is fixed, or tell us how it's not fixed.  Thanks!
>
>
>
> diff -r 9c649274b259 cc-engine.el
> --- a/cc-engine.el	Tue Jul 26 20:08:39 2022 +0000
> +++ b/cc-engine.el	Fri Jul 29 17:25:16 2022 +0000
> @@ -1679,9 +1679,13 @@
>   Return the result of `forward-comment' if it gets called, nil otherwise."
>     `(if (not comment-end-can-be-escaped)
>          (forward-comment -1)
> -     (when (and (< (skip-syntax-backward " >") 0)
> -		(eq (char-after) ?\n))
> -       (forward-char))
> +     (let ((dist (skip-syntax-backward " >")))
> +       (when (and
> +	      (< dist 0)
> +	      (progn
> +		(skip-syntax-forward " " (- (point) dist 1))
> +		(eq (char-after) ?\n)))
> +	 (forward-char)))
>        (cond
>         ((and (eq (char-before) ?\n)
>   	    (eq (char-before (1- (point))) ?\\))
>
>
>
>
>> Alan, this seems to be a regression in Emacs 28, so could you please
>> look into it?
> Eli, Do I understand you want the fix in the release branch?
>


[-- Attachment #2: Type: text/html, Size: 4660 bytes --]

bug#56818: 28.1; c-mode font-lock issues in Emacs 28

[Bill Sacks]

[-- Attachment #1.1: Type: text/plain, Size: 5652 bytes --]

Unfortunately, I still see an issue in a slightly different context. The 
issue appears when function arguments appear on a line following the 
function name. This one is harder to provide screen shots for, because 
it is a bit more dynamic, but I'll try to walk through how to reproduce 
it, referencing two attachments: The starting point is testing_start.c 
and the end point is testing.c. If you start with testing_start.c and 
then take some steps to get to testing.c, you will notice a few issues 
with fontification of "somevar":

(1) Position the cursor at the start of line 2 and hit Tab to put the 
cursor on line 2, column 12. Type "int somevar" and notice that 
"somevar" remains in the default face rather than being given 
font-lock-variable-name-face.

(2) Run M-x font-lock-fontify-buffer to make "somevar" correctly take on 
font-lock-variable-name-face. (Interestingly, it seems fixed as soon as 
I type "M-x" and get into the minibuffer.)

(3) Delete then retype the "r" in "somevar", noticing that it again 
reverts to default face

(4) Repeat step (2) to temporarily fix the fontification

(5) Type " // comment", noticing that "somevar" again loses its 
fontification

(6) Repeat step (2) to temporarily fix the fontification

(7) Delete and retype the "t" in "comment", noticing that "somevar" 
again loses its fontification.

If my instructions are unclear or you cannot reproduce this, I can 
provide a screencast where I illustrate this behavior.

I tested briefly with Emacs 27, and it appears that the problems in (1) 
and (3) are new issues in Emacs 28, but (5) and (7) appear in Emacs 27 
as well.

Bill

Bill Sacks wrote on 7/29/22 2:23 PM:
> Thank you very much for this quick reply and fix, Alan! Yes, from a 
> couple of tests, this fixes the issue I was having (and stops me from 
> getting distracted by constantly changing fontification while writing 
> a comment – thank you!). I'll let you know if I notice any remaining 
> issues.
>
> This does not fix the issue I'm seeing with fontification of variable 
> names in python-mode (which also appears to be an Emacs 28 
> regression), but given that this fix is specific to cc-engine, the 
> python-mode issue is apparently different. That one seems a little 
> harder to reproduce, so it may take me a while to develop a simple 
> reproducer for it, especially since I haven't been doing much python 
> programming recently. However, please let me know if you would like me 
> to prioritize opening an issue for that one: I can do so sooner if it 
> would be helpful.
>
> Thanks again,
> Bill
>
> Alan Mackenzie wrote on 7/29/22 11:44 AM:
>> Hello, Bill and Eli.
>>
>> On Fri, Jul 29, 2022 at 08:56:21 +0300, Eli Zaretskii wrote:
>>>> From: Bill Sacks<sacks@ucar.edu>
>>>> Date: Thu, 28 Jul 2022 14:32:09 -0600
>> First of all (Bill), thanks for taking the trouble to report this bug,
>> and thanks even more for cutting the test case down to the short fragment
>> in your screenshots.
>>
>>>> Starting with Emacs 28, I have been seeing font-lock issues when
>>>> editing C and C++ code. The situation where I see this the most
>>>> (though I'm not sure if it's the only situation) is when I am writing
>>>> a comment and currently have a space at the end of the comment line:
>>>> in this situation, the fontification of a variable name or function
>>>> name on the next line becomes broken until I type a non-space
>>>> character to end the current line.
>>>> The attached screen shots illustrate the problem: nospaces.png shows
>>>> the correct fontification; space_before_var.png and
>>>> space_before_function.png show that variable and function names lose
>>>> their fontification when there is a space at the end of the previous
>>>> comment line. Running M-x font-lock-fontify-buffer temporarily fixes
>>>> the issue.
>>>> The problem occurs even when using emacs -Q. I have tried the latest
>>>> emacs28 pretest and the latest nightly build available from
>>>> emacsformacosx (though with my customizations – NOT with emacs -Q)
>>>> and those also exhibit the problem. The latest emacs27 from
>>>> emacsformacosx does NOT have this issue.
>> This is a coding bug in an optimisation from March 2020, where the
>> complaint was that scrolling over a 2,000 line macro was slow.  The fix
>> neglected the possibility of spaces at the end of comment lines.
>>
>> Could you please apply the following patch in your Emacs-28.1, byte
>> compile the file ..../lisp/progmodes/cc-engine.el, then try out the
>> result on your real code.  (If you want any help with the patching or
>> byte compiling, feel free to send me private mail.)  Then please confirm
>> that the bug is fixed, or tell us how it's not fixed.  Thanks!
>>
>>
>>
>> diff -r 9c649274b259 cc-engine.el
>> --- a/cc-engine.el	Tue Jul 26 20:08:39 2022 +0000
>> +++ b/cc-engine.el	Fri Jul 29 17:25:16 2022 +0000
>> @@ -1679,9 +1679,13 @@
>>   Return the result of `forward-comment' if it gets called, nil otherwise."
>>     `(if (not comment-end-can-be-escaped)
>>          (forward-comment -1)
>> -     (when (and (< (skip-syntax-backward " >") 0)
>> -		(eq (char-after) ?\n))
>> -       (forward-char))
>> +     (let ((dist (skip-syntax-backward " >")))
>> +       (when (and
>> +	      (< dist 0)
>> +	      (progn
>> +		(skip-syntax-forward " " (- (point) dist 1))
>> +		(eq (char-after) ?\n)))
>> +	 (forward-char)))
>>        (cond
>>         ((and (eq (char-before) ?\n)
>>   	    (eq (char-before (1- (point))) ?\\))
>>
>>
>>
>>
>>> Alan, this seems to be a regression in Emacs 28, so could you please
>>> look into it?
>> Eli, Do I understand you want the fix in the release branch?
>>
>


[-- Attachment #1.2: Type: text/html, Size: 6614 bytes --]

[-- Attachment #2: testing.c --]
[-- Type: text/plain, Size: 52 bytes --]

void myfunc(
	    int somevar // comment
  ) {
  
}

[-- Attachment #3: testing_start.c --]
[-- Type: text/plain, Size: 25 bytes --]

void myfunc(

  ) {
  
}

bug#56818: 28.1; c-mode font-lock issues in Emacs 28

[Alan Mackenzie]

Hello, Bill.

On Fri, Jul 29, 2022 at 14:38:53 -0600, Bill Sacks wrote:
> Unfortunately, I still see an issue in a slightly different context. The 
> issue appears when function arguments appear on a line following the 
> function name. This one is harder to provide screen shots for, because 
> it is a bit more dynamic, but I'll try to walk through how to reproduce 
> it, referencing two attachments: The starting point is testing_start.c 
> and the end point is testing.c. If you start with testing_start.c and 
> then take some steps to get to testing.c, you will notice a few issues 
> with fontification of "somevar":

> (1) Position the cursor at the start of line 2 and hit Tab to put the 
> cursor on line 2, column 12. Type "int somevar" and notice that 
> "somevar" remains in the default face rather than being given 
> font-lock-variable-name-face.

> (2) Run M-x font-lock-fontify-buffer to make "somevar" correctly take on 
> font-lock-variable-name-face. (Interestingly, it seems fixed as soon as 
> I type "M-x" and get into the minibuffer.)

> (3) Delete then retype the "r" in "somevar", noticing that it again 
> reverts to default face

> (4) Repeat step (2) to temporarily fix the fontification

> (5) Type " // comment", noticing that "somevar" again loses its 
> fontification

> (6) Repeat step (2) to temporarily fix the fontification

> (7) Delete and retype the "t" in "comment", noticing that "somevar" 
> again loses its fontification.

> If my instructions are unclear or you cannot reproduce this, I can 
> provide a screencast where I illustrate this behavior.

Those directions are admirably clear, thanks!

> I tested briefly with Emacs 27, and it appears that the problems in (1) 
> and (3) are new issues in Emacs 28, but (5) and (7) appear in Emacs 27 
> as well.

What is happening here is a different bug, distinct from bug #56818,
with different causes.  I've opened a fresh bug for this, bug #56841,
putting you on the Cc:.

So I'm closing bug #56818 with this post, as it appears to be fixed.

[ .... ]

> Bill

-- 
Alan Mackenzie (Nuremberg, Germany).