bug#70367: 30.0.50; Inconsistent Syntax Highlighting

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Amol Surati]

--text follows this line--

The problem is not found in terminal emacs built from the released 29.3.tar.gz,
or with emacs running under GUI (i.e. under PGTK).

The problem is seen with terminal emacs built from the master branch, at various
commit levels.

Problem: When a large file (for e.g. vulkan_core.h) is opened, certain
constructs have their syntax highlighting broken. The video found at [1] shows
the behaviour. At the end of the video, one can see one instance of the problem;
the syntax highlighting for the enum constant
'VK_STRUCTURE_TYPE_EVENT_CREATE_INFO = 10,' abruptly breaks. The entire
identifier VK_STRUCTURE_TYPE_EVENT_CREATE_INFO must be one colour. Instead,
'VK_STRUCTURE_TYPE_EVENT_CREA' is of the expected colour, while
'TE_INFO' is of the colour that is expected with '= 10,'. You may want to
download the video and then play it, if Google Drive plays it at a resolution
that is lower than the video's native resolution.

Within this same session, there were other such enum constants with broken
highlighting, though they have not been captured in the video.
The termscript is attached at [2].

The graphics session is Wayland with swaywm as its compositor; XWayland is
not enabled. The terminal emulator is 'foot'. Another terminal emulator,
'alacritty' was also tested; the problem occurred there too.

The problem doesn't seem to occur with small-sized files; After reducing the
vulkan_core.h to contain only around 235 lines, emacs was able to show the
(reduced) file with consistent highlighting.

Thank you,
Amol Surati

[1] https://drive.google.com/file/d/1C2pSlh3x1g91lUsErryLnLP5995Jcg6c/
[2] https://pastebin.com/UiKSZWm7
------------------------------------------------------------------------

In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu)
Repository revision: 8b210a636fe426f47bccdb111af61d6310755dde
Repository branch: master
System Description: Arch Linux

Configured using:
 'configure --prefix=/home/user/tools/emacs --without-all
 --with-native-compilation=aot --with-zlib --without-x --without-json
 --without-sound --with-small-ja-dic --disable-build-details
 --without-sqlite3 --with-compress-install 'CFLAGS=-O2 -mtune=native
 -march=native -fomit-frame-pointer''

Configured features:
GMP NATIVE_COMP PDUMPER SECCOMP XIM ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  global-display-fill-column-indicator-mode: t
  display-fill-column-indicator-mode: t
  save-place-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  minibuffer-regexp-mode: t
  column-number-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  auto-save-visited-mode: t

Load-path shadows:
None found.

Features:
(shadow sort regexp-opt mail-extr emacsbug message mailcap yank-media
puny dired dnd dired-loaddefs rfc822 mml mml-sec password-cache epa
derived epg rfc6068 epg-config gnus-util text-property-search time-date
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils term/xterm xterm byte-opt gv bytecomp byte-compile
modus-vivendi-theme modus-themes subr-x easy-mmode
display-fill-column-indicator saveplace cl-seq cl-loaddefs cl-lib rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 126813 13455) (symbols 48 8344 0) (strings 32 19368 1588)
 (string-bytes 1 634561) (vectors 16 7702) (vector-slots 8 94144 6327)
 (floats 8 50 8) (intervals 56 241 1) (buffers 984 11))

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Eli Zaretskii]

> From: Amol Surati <suratiamol@gmail.com>
> Date: Sat, 13 Apr 2024 18:12:54 +0530
> 
> The problem is not found in terminal emacs built from the released 29.3.tar.gz,
> or with emacs running under GUI (i.e. under PGTK).
> 
> The problem is seen with terminal emacs built from the master branch, at various
> commit levels.
> 
> Problem: When a large file (for e.g. vulkan_core.h) is opened, certain
> constructs have their syntax highlighting broken. The video found at [1] shows
> the behaviour. At the end of the video, one can see one instance of the problem;
> the syntax highlighting for the enum constant
> 'VK_STRUCTURE_TYPE_EVENT_CREATE_INFO = 10,' abruptly breaks. The entire
> identifier VK_STRUCTURE_TYPE_EVENT_CREATE_INFO must be one colour. Instead,
> 'VK_STRUCTURE_TYPE_EVENT_CREA' is of the expected colour, while
> 'TE_INFO' is of the colour that is expected with '= 10,'. You may want to
> download the video and then play it, if Google Drive plays it at a resolution
> that is lower than the video's native resolution.
> 
> Within this same session, there were other such enum constants with broken
> highlighting, though they have not been captured in the video.
> The termscript is attached at [2].
> 
> The graphics session is Wayland with swaywm as its compositor; XWayland is
> not enabled. The terminal emulator is 'foot'. Another terminal emulator,
> 'alacritty' was also tested; the problem occurred there too.
> 
> The problem doesn't seem to occur with small-sized files; After reducing the
> vulkan_core.h to contain only around 235 lines, emacs was able to show the
> (reduced) file with consistent highlighting.

FWIW, I cannot reproduce this with stock Emacs 29.3 and vulkan_core.h
file that I downloaded from this site:

  https://github.com/KhronosGroup/dfdutils/blob/main/vulkan/vulkan_core.h

I tried both the default cc-mode and c-ts-mode, and they both produce
correct display with fill syntax highlighting that does NOT break.

If the above is not the file where you see the problem, please post
the offending file, or tell where it can be downloaded.

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Eli Zaretskii]

> Cc: 70367@debbugs.gnu.org
> Date: Sat, 13 Apr 2024 20:44:37 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Amol Surati <suratiamol@gmail.com>
> > Date: Sat, 13 Apr 2024 18:12:54 +0530
> > 
> > The problem is not found in terminal emacs built from the released 29.3.tar.gz,
> > or with emacs running under GUI (i.e. under PGTK).
> > 
> > The problem is seen with terminal emacs built from the master branch, at various
> > commit levels.
> > 
> > Problem: When a large file (for e.g. vulkan_core.h) is opened, certain
> > constructs have their syntax highlighting broken. The video found at [1] shows
> > the behaviour. At the end of the video, one can see one instance of the problem;
> > the syntax highlighting for the enum constant
> > 'VK_STRUCTURE_TYPE_EVENT_CREATE_INFO = 10,' abruptly breaks. The entire
> > identifier VK_STRUCTURE_TYPE_EVENT_CREATE_INFO must be one colour. Instead,
> > 'VK_STRUCTURE_TYPE_EVENT_CREA' is of the expected colour, while
> > 'TE_INFO' is of the colour that is expected with '= 10,'. You may want to
> > download the video and then play it, if Google Drive plays it at a resolution
> > that is lower than the video's native resolution.
> > 
> > Within this same session, there were other such enum constants with broken
> > highlighting, though they have not been captured in the video.
> > The termscript is attached at [2].
> > 
> > The graphics session is Wayland with swaywm as its compositor; XWayland is
> > not enabled. The terminal emulator is 'foot'. Another terminal emulator,
> > 'alacritty' was also tested; the problem occurred there too.
> > 
> > The problem doesn't seem to occur with small-sized files; After reducing the
> > vulkan_core.h to contain only around 235 lines, emacs was able to show the
> > (reduced) file with consistent highlighting.
> 
> FWIW, I cannot reproduce this with stock Emacs 29.3 and vulkan_core.h
> file that I downloaded from this site:
> 
>   https://github.com/KhronosGroup/dfdutils/blob/main/vulkan/vulkan_core.h

I see now that you say you see this with the master branch, so I
tested that version as well, and I still don't see the problem.

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Stephen Berman via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Sat, 13 Apr 2024 20:48:28 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> Cc: 70367@debbugs.gnu.org
>> Date: Sat, 13 Apr 2024 20:44:37 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>>
>> > From: Amol Surati <suratiamol@gmail.com>
>> > Date: Sat, 13 Apr 2024 18:12:54 +0530
>> >
>> > The problem is not found in terminal emacs built from the released 29.3.tar.gz,
>> > or with emacs running under GUI (i.e. under PGTK).
>> >
>> > The problem is seen with terminal emacs built from the master branch, at various
>> > commit levels.
>> >
>> > Problem: When a large file (for e.g. vulkan_core.h) is opened, certain
>> > constructs have their syntax highlighting broken. The video found at [1] shows
>> > the behaviour. At the end of the video, one can see one instance of the problem;
>> > the syntax highlighting for the enum constant
>> > 'VK_STRUCTURE_TYPE_EVENT_CREATE_INFO = 10,' abruptly breaks. The entire
>> > identifier VK_STRUCTURE_TYPE_EVENT_CREATE_INFO must be one colour. Instead,
>> > 'VK_STRUCTURE_TYPE_EVENT_CREA' is of the expected colour, while
>> > 'TE_INFO' is of the colour that is expected with '= 10,'. You may want to
>> > download the video and then play it, if Google Drive plays it at a resolution
>> > that is lower than the video's native resolution.
>> >
>> > Within this same session, there were other such enum constants with broken
>> > highlighting, though they have not been captured in the video.
>> > The termscript is attached at [2].
>> >
>> > The graphics session is Wayland with swaywm as its compositor; XWayland is
>> > not enabled. The terminal emulator is 'foot'. Another terminal emulator,
>> > 'alacritty' was also tested; the problem occurred there too.
>> >
>> > The problem doesn't seem to occur with small-sized files; After reducing the
>> > vulkan_core.h to contain only around 235 lines, emacs was able to show the
>> > (reduced) file with consistent highlighting.
>>
>> FWIW, I cannot reproduce this with stock Emacs 29.3 and vulkan_core.h
>> file that I downloaded from this site:
>>
>>   https://github.com/KhronosGroup/dfdutils/blob/main/vulkan/vulkan_core.h
>
> I see now that you say you see this with the master branch, so I
> tested that version as well, and I still don't see the problem.

I see exactly the same misfontification as the OP in the same file
(which I happen to have on my system), as well as several more similar
misfontifications further down in that file -- but only with c-mode from
cc-mode.el.  With c-ts-mode I see no misfontifications in that file.
This is with GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+
Version 3.24.41, cairo version 1.18.0) of 2024-04-11.

Steve Berman

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: suratiamol@gmail.com,  70367@debbugs.gnu.org
> Date: Sat, 13 Apr 2024 21:00:16 +0200
> 
> On Sat, 13 Apr 2024 20:48:28 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >> Cc: 70367@debbugs.gnu.org
> >> Date: Sat, 13 Apr 2024 20:44:37 +0300
> >> From: Eli Zaretskii <eliz@gnu.org>
> >>
> >> > From: Amol Surati <suratiamol@gmail.com>
> >> > Date: Sat, 13 Apr 2024 18:12:54 +0530
> >> >
> >> > The problem is not found in terminal emacs built from the released 29.3.tar.gz,
> >> > or with emacs running under GUI (i.e. under PGTK).
> >> >
> >> > The problem is seen with terminal emacs built from the master branch, at various
> >> > commit levels.
> >> >
> >> > Problem: When a large file (for e.g. vulkan_core.h) is opened, certain
> >> > constructs have their syntax highlighting broken. The video found at [1] shows
> >> > the behaviour. At the end of the video, one can see one instance of the problem;
> >> > the syntax highlighting for the enum constant
> >> > 'VK_STRUCTURE_TYPE_EVENT_CREATE_INFO = 10,' abruptly breaks. The entire
> >> > identifier VK_STRUCTURE_TYPE_EVENT_CREATE_INFO must be one colour. Instead,
> >> > 'VK_STRUCTURE_TYPE_EVENT_CREA' is of the expected colour, while
> >> > 'TE_INFO' is of the colour that is expected with '= 10,'. You may want to
> >> > download the video and then play it, if Google Drive plays it at a resolution
> >> > that is lower than the video's native resolution.
> >> >
> >> > Within this same session, there were other such enum constants with broken
> >> > highlighting, though they have not been captured in the video.
> >> > The termscript is attached at [2].
> >> >
> >> > The graphics session is Wayland with swaywm as its compositor; XWayland is
> >> > not enabled. The terminal emulator is 'foot'. Another terminal emulator,
> >> > 'alacritty' was also tested; the problem occurred there too.
> >> >
> >> > The problem doesn't seem to occur with small-sized files; After reducing the
> >> > vulkan_core.h to contain only around 235 lines, emacs was able to show the
> >> > (reduced) file with consistent highlighting.
> >>
> >> FWIW, I cannot reproduce this with stock Emacs 29.3 and vulkan_core.h
> >> file that I downloaded from this site:
> >>
> >>   https://github.com/KhronosGroup/dfdutils/blob/main/vulkan/vulkan_core.h
> >
> > I see now that you say you see this with the master branch, so I
> > tested that version as well, and I still don't see the problem.
> 
> I see exactly the same misfontification as the OP in the same file
> (which I happen to have on my system), as well as several more similar
> misfontifications further down in that file -- but only with c-mode from
> cc-mode.el.  With c-ts-mode I see no misfontifications in that file.
> This is with GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+
> Version 3.24.41, cairo version 1.18.0) of 2024-04-11.

Strange.  I see no misfontifications with either mode.

Alan, would you please have a look?

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Amol Surati]

[-- Attachment #1: Type: text/plain, Size: 3756 bytes --]

On Sun, 14 Apr 2024 at 00:35, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Stephen Berman <stephen.berman@gmx.net>
> > Cc: suratiamol@gmail.com,  70367@debbugs.gnu.org
> > Date: Sat, 13 Apr 2024 21:00:16 +0200
> >
> > On Sat, 13 Apr 2024 20:48:28 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > >> Cc: 70367@debbugs.gnu.org
> > >> Date: Sat, 13 Apr 2024 20:44:37 +0300
> > >> From: Eli Zaretskii <eliz@gnu.org>
> > >>
> > >> > From: Amol Surati <suratiamol@gmail.com>
> > >> > Date: Sat, 13 Apr 2024 18:12:54 +0530
> > >> >
> > >> > The problem is not found in terminal emacs built from the released 29.3.tar.gz,
> > >> > or with emacs running under GUI (i.e. under PGTK).
> > >> >
> > >> > The problem is seen with terminal emacs built from the master branch, at various
> > >> > commit levels.
> > >> >
> > >> > Problem: When a large file (for e.g. vulkan_core.h) is opened, certain
> > >> > constructs have their syntax highlighting broken. The video found at [1] shows
> > >> > the behaviour. At the end of the video, one can see one instance of the problem;
> > >> > the syntax highlighting for the enum constant
> > >> > 'VK_STRUCTURE_TYPE_EVENT_CREATE_INFO = 10,' abruptly breaks. The entire
> > >> > identifier VK_STRUCTURE_TYPE_EVENT_CREATE_INFO must be one colour. Instead,
> > >> > 'VK_STRUCTURE_TYPE_EVENT_CREA' is of the expected colour, while
> > >> > 'TE_INFO' is of the colour that is expected with '= 10,'. You may want to
> > >> > download the video and then play it, if Google Drive plays it at a resolution
> > >> > that is lower than the video's native resolution.
> > >> >
> > >> > Within this same session, there were other such enum constants with broken
> > >> > highlighting, though they have not been captured in the video.
> > >> > The termscript is attached at [2].
> > >> >
> > >> > The graphics session is Wayland with swaywm as its compositor; XWayland is
> > >> > not enabled. The terminal emulator is 'foot'. Another terminal emulator,
> > >> > 'alacritty' was also tested; the problem occurred there too.
> > >> >
> > >> > The problem doesn't seem to occur with small-sized files; After reducing the
> > >> > vulkan_core.h to contain only around 235 lines, emacs was able to show the
> > >> > (reduced) file with consistent highlighting.
> > >>
> > >> FWIW, I cannot reproduce this with stock Emacs 29.3 and vulkan_core.h
> > >> file that I downloaded from this site:
> > >>
> > >>   https://github.com/KhronosGroup/dfdutils/blob/main/vulkan/vulkan_core.h
> > >
> > > I see now that you say you see this with the master branch, so I
> > > tested that version as well, and I still don't see the problem.
> >
> > I see exactly the same misfontification as the OP in the same file
> > (which I happen to have on my system), as well as several more similar
> > misfontifications further down in that file -- but only with c-mode from
> > cc-mode.el.  With c-ts-mode I see no misfontifications in that file.
> > This is with GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+
> > Version 3.24.41, cairo version 1.18.0) of 2024-04-11.
>
> Strange.  I see no misfontifications with either mode.

Apologies. I missed Eli's email about the C modes.

My emacs build is devoid of most of the settings and
features, including GUI and tree-sitter (the config command is in
the original report). So it is likely that only cc-mode is affected,
and not c-ts-mode.

Note also that vulkan_core.h isn't special. A C source/header file
with a long enough enum definition also works. Attached is a C
program that generates to stdout the contents of such a header
file. Opening the contents (after they are saved to a file by stdout
redirection, etc.) in emacs demonstrates the problem.

-Amol

>
> Alan, would you please have a look?

[-- Attachment #2: repro.c --]
[-- Type: text/x-csrc, Size: 336 bytes --]

#include <stdio.h>

int main ()
{
	static char str[1024];

	printf ("#ifndef REPRO_H\n");
	printf ("#define REPRO_H\n");
	printf ("enum numbers {\n");
	for (int i = 0; i < 20000; ++i) {
		snprintf (str, 1024, "\tA_LONG_LONG_NAME_FOR_THE_NUMBER_%d = %d,\n", i, i);
		printf (str);
	}
	printf ("};\n");
	printf ("#endif\n");
	return 0;
}

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Alan Mackenzie]

Hello, Amol.

Thanks for taking the trouble to report this bug, and thanks even more
for the convenient test file generator, which was extremely helpful.

On Sun, Apr 14, 2024 at 03:44:01 +0530, Amol Surati wrote:
> On Sun, 14 Apr 2024 at 00:35, Eli Zaretskii <eliz@gnu.org> wrote:

> > > From: Stephen Berman <stephen.berman@gmx.net>
> > > Cc: suratiamol@gmail.com,  70367@debbugs.gnu.org
> > > Date: Sat, 13 Apr 2024 21:00:16 +0200

> > > On Sat, 13 Apr 2024 20:48:28 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

> > > >> Cc: 70367@debbugs.gnu.org
> > > >> Date: Sat, 13 Apr 2024 20:44:37 +0300
> > > >> From: Eli Zaretskii <eliz@gnu.org>

> > > >> > From: Amol Surati <suratiamol@gmail.com>
> > > >> > Date: Sat, 13 Apr 2024 18:12:54 +0530

> > > >> > The problem is not found in terminal emacs built from the released 29.3.tar.gz,
> > > >> > or with emacs running under GUI (i.e. under PGTK).

> > > >> > The problem is seen with terminal emacs built from the master branch, at various
> > > >> > commit levels.

> > > >> > Problem: When a large file (for e.g. vulkan_core.h) is opened, certain
> > > >> > constructs have their syntax highlighting broken. The video found at [1] shows
> > > >> > the behaviour. At the end of the video, one can see one instance of the problem;
> > > >> > the syntax highlighting for the enum constant
> > > >> > 'VK_STRUCTURE_TYPE_EVENT_CREATE_INFO = 10,' abruptly breaks. The entire
> > > >> > identifier VK_STRUCTURE_TYPE_EVENT_CREATE_INFO must be one colour. Instead,
> > > >> > 'VK_STRUCTURE_TYPE_EVENT_CREA' is of the expected colour, while
> > > >> > 'TE_INFO' is of the colour that is expected with '= 10,'. You may want to
> > > >> > download the video and then play it, if Google Drive plays it at a resolution
> > > >> > that is lower than the video's native resolution.

> > > >> > Within this same session, there were other such enum constants with broken
> > > >> > highlighting, though they have not been captured in the video.
> > > >> > The termscript is attached at [2].

> > > >> > The graphics session is Wayland with swaywm as its compositor; XWayland is
> > > >> > not enabled. The terminal emulator is 'foot'. Another terminal emulator,
> > > >> > 'alacritty' was also tested; the problem occurred there too.

> > > >> > The problem doesn't seem to occur with small-sized files; After reducing the
> > > >> > vulkan_core.h to contain only around 235 lines, emacs was able to show the
> > > >> > (reduced) file with consistent highlighting.

> > > I see exactly the same misfontification as the OP in the same file
> > > (which I happen to have on my system), as well as several more similar
> > > misfontifications further down in that file -- but only with c-mode from
> > > cc-mode.el.  With c-ts-mode I see no misfontifications in that file.
> > > This is with GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+
> > > Version 3.24.41, cairo version 1.18.0) of 2024-04-11.

> > Strange.  I see no misfontifications with either mode.

> Apologies. I missed Eli's email about the C modes.

> My emacs build is devoid of most of the settings and
> features, including GUI and tree-sitter (the config command is in
> the original report). So it is likely that only cc-mode is affected,
> and not c-ts-mode.

This is indeed the case.

> Note also that vulkan_core.h isn't special. A C source/header file
> with a long enough enum definition also works. Attached is a C
> program that generates to stdout the contents of such a header
> file. Opening the contents (after they are saved to a file by stdout
> redirection, etc.) in emacs demonstrates the problem.

The problem is long stretches of code (>= 500 characters) where there're
no statement boundaries or braces.  These frequently occur in enums.  An
ad hoc limit to 500 characters backward search is there for speed.

However, this bit of code was not checking whether it found a
brace/statement or hit the 500 char limit, hence the mis-fontification.

The patch below tries to fix this.  Would you please apply it to
cc-mode.el (in .../lisp/progmodes), byte compile the result, and load it
into your Emacs (or restart Emacs).  Then please try it out on the real
files that showed the bug.  Please let me know if the bug really is
fixed.  (If you want any help with patching or byte compiling, feel free
to send me private email.)



diff -r 709b797bdef8 cc-mode.el
--- a/cc-mode.el	Tue Mar 26 20:26:16 2024 +0000
+++ b/cc-mode.el	Sun Apr 14 02:39:32 2024 +0000
@@ -2437,7 +2437,7 @@
 			   (backward-char)
 			   (setq pseudo (c-cheap-inside-bracelist-p (c-parse-state)))))))
 	       (goto-char pseudo))
-	     t)
+	     pseudo)
 	   ;; Move forward to the start of the next declaration.
 	   (progn (c-forward-syntactic-ws)
 		  ;; Have we got stuck in a comment at EOB?


> -Amol


> > Alan, would you please have a look?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Amol Surati]

Hello, Alan.

On Sun, 14 Apr 2024 at 08:16, Alan Mackenzie <acm@muc.de> wrote:
>
> Hello, Amol.
>
> Thanks for taking the trouble to report this bug, and thanks even more
> for the convenient test file generator, which was extremely helpful.

Thank you for the kind words.

>
> On Sun, Apr 14, 2024 at 03:44:01 +0530, Amol Surati wrote:
> > On Sun, 14 Apr 2024 at 00:35, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > > > From: Stephen Berman <stephen.berman@gmx.net>
> > > > Cc: suratiamol@gmail.com,  70367@debbugs.gnu.org
> > > > Date: Sat, 13 Apr 2024 21:00:16 +0200
>
> > > > On Sat, 13 Apr 2024 20:48:28 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
>
> > > > >> Cc: 70367@debbugs.gnu.org
> > > > >> Date: Sat, 13 Apr 2024 20:44:37 +0300
> > > > >> From: Eli Zaretskii <eliz@gnu.org>
>
> > > > >> > From: Amol Surati <suratiamol@gmail.com>
> > > > >> > Date: Sat, 13 Apr 2024 18:12:54 +0530
>
> > > > >> > The problem is not found in terminal emacs built from the released 29.3.tar.gz,
> > > > >> > or with emacs running under GUI (i.e. under PGTK).
>
> > > > >> > The problem is seen with terminal emacs built from the master branch, at various
> > > > >> > commit levels.
>
> > > > >> > Problem: When a large file (for e.g. vulkan_core.h) is opened, certain
> > > > >> > constructs have their syntax highlighting broken. The video found at [1] shows
> > > > >> > the behaviour. At the end of the video, one can see one instance of the problem;
> > > > >> > the syntax highlighting for the enum constant
> > > > >> > 'VK_STRUCTURE_TYPE_EVENT_CREATE_INFO = 10,' abruptly breaks. The entire
> > > > >> > identifier VK_STRUCTURE_TYPE_EVENT_CREATE_INFO must be one colour. Instead,
> > > > >> > 'VK_STRUCTURE_TYPE_EVENT_CREA' is of the expected colour, while
> > > > >> > 'TE_INFO' is of the colour that is expected with '= 10,'. You may want to
> > > > >> > download the video and then play it, if Google Drive plays it at a resolution
> > > > >> > that is lower than the video's native resolution.
>
> > > > >> > Within this same session, there were other such enum constants with broken
> > > > >> > highlighting, though they have not been captured in the video.
> > > > >> > The termscript is attached at [2].
>
> > > > >> > The graphics session is Wayland with swaywm as its compositor; XWayland is
> > > > >> > not enabled. The terminal emulator is 'foot'. Another terminal emulator,
> > > > >> > 'alacritty' was also tested; the problem occurred there too.
>
> > > > >> > The problem doesn't seem to occur with small-sized files; After reducing the
> > > > >> > vulkan_core.h to contain only around 235 lines, emacs was able to show the
> > > > >> > (reduced) file with consistent highlighting.
>
> > > > I see exactly the same misfontification as the OP in the same file
> > > > (which I happen to have on my system), as well as several more similar
> > > > misfontifications further down in that file -- but only with c-mode from
> > > > cc-mode.el.  With c-ts-mode I see no misfontifications in that file.
> > > > This is with GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+
> > > > Version 3.24.41, cairo version 1.18.0) of 2024-04-11.
>
> > > Strange.  I see no misfontifications with either mode.
>
> > Apologies. I missed Eli's email about the C modes.
>
> > My emacs build is devoid of most of the settings and
> > features, including GUI and tree-sitter (the config command is in
> > the original report). So it is likely that only cc-mode is affected,
> > and not c-ts-mode.
>
> This is indeed the case.

Understood.

>
> > Note also that vulkan_core.h isn't special. A C source/header file
> > with a long enough enum definition also works. Attached is a C
> > program that generates to stdout the contents of such a header
> > file. Opening the contents (after they are saved to a file by stdout
> > redirection, etc.) in emacs demonstrates the problem.
>
> The problem is long stretches of code (>= 500 characters) where there're
> no statement boundaries or braces.  These frequently occur in enums.  An
> ad hoc limit to 500 characters backward search is there for speed.

Consistent with the observed behaviour, that it is mostly enums that are
affected.

>
> However, this bit of code was not checking whether it found a
> brace/statement or hit the 500 char limit, hence the mis-fontification.
>
> The patch below tries to fix this.  Would you please apply it to
> cc-mode.el (in .../lisp/progmodes), byte compile the result, and load it
> into your Emacs (or restart Emacs).  Then please try it out on the real
> files that showed the bug.  Please let me know if the bug really is
> fixed.  (If you want any help with patching or byte compiling, feel free
> to send me private email.)

Thanks for the patch. It indeed fixes the highlighting problem on the
real file vulkan_core.h (I know about only this one real file that's affected),
as well as it does on the test file.

-Amol

>
>
>
> diff -r 709b797bdef8 cc-mode.el
> --- a/cc-mode.el        Tue Mar 26 20:26:16 2024 +0000
> +++ b/cc-mode.el        Sun Apr 14 02:39:32 2024 +0000
> @@ -2437,7 +2437,7 @@
>                            (backward-char)
>                            (setq pseudo (c-cheap-inside-bracelist-p (c-parse-state)))))))
>                (goto-char pseudo))
> -            t)
> +            pseudo)
>            ;; Move forward to the start of the next declaration.
>            (progn (c-forward-syntactic-ws)
>                   ;; Have we got stuck in a comment at EOB?
>
>
> > -Amol
>
>
> > > Alan, would you please have a look?
>
> --
> Alan Mackenzie (Nuremberg, Germany).

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Alan Mackenzie]

Hello, Amol.

On Sun, Apr 14, 2024 at 10:37:32 +0530, Amol Surati wrote:
> Hello, Alan.

> On Sun, 14 Apr 2024 at 08:16, Alan Mackenzie <acm@muc.de> wrote:

> > Thanks for taking the trouble to report this bug, and thanks even more
> > for the convenient test file generator, which was extremely helpful.

> Thank you for the kind words.


> > On Sun, Apr 14, 2024 at 03:44:01 +0530, Amol Surati wrote:

[ .... ]

> > > My emacs build is devoid of most of the settings and
> > > features, including GUI and tree-sitter (the config command is in
> > > the original report). So it is likely that only cc-mode is affected,
> > > and not c-ts-mode.

> > This is indeed the case.

> Understood.


> > > Note also that vulkan_core.h isn't special. A C source/header file
> > > with a long enough enum definition also works. Attached is a C
> > > program that generates to stdout the contents of such a header
> > > file. Opening the contents (after they are saved to a file by stdout
> > > redirection, etc.) in emacs demonstrates the problem.

> > The problem is long stretches of code (>= 500 characters) where there're
> > no statement boundaries or braces.  These frequently occur in enums.  An
> > ad hoc limit to 500 characters backward search is there for speed.

> Consistent with the observed behaviour, that it is mostly enums that are
> affected.


> > However, this bit of code was not checking whether it found a
> > brace/statement or hit the 500 char limit, hence the mis-fontification.

> > The patch below tries to fix this.  Would you please apply it to
> > cc-mode.el (in .../lisp/progmodes), byte compile the result, and load it
> > into your Emacs (or restart Emacs).  Then please try it out on the real
> > files that showed the bug.  Please let me know if the bug really is
> > fixed.  (If you want any help with patching or byte compiling, feel free
> > to send me private email.)

> Thanks for the patch. It indeed fixes the highlighting problem on the
> real file vulkan_core.h (I know about only this one real file that's affected),
> as well as it does on the test file.

Thanks for the rapid testing!  It would appear the bug has been fixed, so
I've committed the fix to Emacs, CC Mode, and XEmacs.  I'm closing the
bug with this post.

> -Amol

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#70367: 30.0.50; Inconsistent Syntax Highlighting

[Amol Surati]

On Sat, 13 Apr 2024 at 23:18, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > Cc: 70367@debbugs.gnu.org
> > Date: Sat, 13 Apr 2024 20:44:37 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> >
> > > From: Amol Surati <suratiamol@gmail.com>
> > > Date: Sat, 13 Apr 2024 18:12:54 +0530
> > >
> > > The problem is not found in terminal emacs built from the released 29.3.tar.gz,
> > > or with emacs running under GUI (i.e. under PGTK).
> > >
> > > The problem is seen with terminal emacs built from the master branch, at various
> > > commit levels.
> > >
> > > Problem: When a large file (for e.g. vulkan_core.h) is opened, certain
> > > constructs have their syntax highlighting broken. The video found at [1] shows
> > > the behaviour. At the end of the video, one can see one instance of the problem;
> > > the syntax highlighting for the enum constant
> > > 'VK_STRUCTURE_TYPE_EVENT_CREATE_INFO = 10,' abruptly breaks. The entire
> > > identifier VK_STRUCTURE_TYPE_EVENT_CREATE_INFO must be one colour. Instead,
> > > 'VK_STRUCTURE_TYPE_EVENT_CREA' is of the expected colour, while
> > > 'TE_INFO' is of the colour that is expected with '= 10,'. You may want to
> > > download the video and then play it, if Google Drive plays it at a resolution
> > > that is lower than the video's native resolution.
> > >
> > > Within this same session, there were other such enum constants with broken
> > > highlighting, though they have not been captured in the video.
> > > The termscript is attached at [2].
> > >
> > > The graphics session is Wayland with swaywm as its compositor; XWayland is
> > > not enabled. The terminal emulator is 'foot'. Another terminal emulator,
> > > 'alacritty' was also tested; the problem occurred there too.
> > >
> > > The problem doesn't seem to occur with small-sized files; After reducing the
> > > vulkan_core.h to contain only around 235 lines, emacs was able to show the
> > > (reduced) file with consistent highlighting.
> >
> > FWIW, I cannot reproduce this with stock Emacs 29.3 and vulkan_core.h
> > file that I downloaded from this site:
> >
> >   https://github.com/KhronosGroup/dfdutils/blob/main/vulkan/vulkan_core.h
>
> I see now that you say you see this with the master branch, so I
> tested that version as well, and I still don't see the problem.

Thank you for looking into this problem.

The file can be found at [3], though I was able to reproduce the problem
even with the link that you had downloaded.

You may have to scroll the file up-down (I use page-up/dn keys) in order
to trigger the problem, though it usually exhibits the problem within a few
(less 10) page-up/down scroll commands.

Within the file [3], it seems only enums are affected, though I haven't
checked the entire file for consistency of syntax highlighting.

Which particular enum constant gets affected may also vary at times, even
within the same session, if one scrolls out and away to another portion of the
file, and then returns back.

The video I had posted was with 'emacs -Q'. A screenshot with better
colour contrast is at [4]; the corresponding termscript is at [5]. This
time I was able to capture a corresponding breakage within the
termscript.

The break within the highlighting [4] can be clearly matched with a break
in the termscript contents. If the termscript file [5] is searched for
L_DEVICE_LINEAR_COLOR_ATTACHMENT_FEATURES_NV,
one can clearly see that the identifier is broken into two.
The unbroken identifier is:
VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_LINEAR_COLOR_ATTACHMENT_FEATURES_NV.
The highlight breaks between 'PHYSICA' and 'L_DEVICE...'. The
termscript [5] exhibits a corresponding break, *exactly* matching the
break in the syntax highlighting.

Thank you,
Amol Surati
-----------------------------------------------------------------
[3] https://raw.githubusercontent.com/KhronosGroup/Vulkan-Headers/main/include/vulkan/vulkan_core.h
[4] https://imgur.com/a/gqNZGDO
[5] https://pastebin.com/VQR76Gsy