bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Richard Copley]

[-- Attachment #1: Type: text/plain, Size: 4003 bytes --]

There seems to be an issue with compilation-mode highlighting in
recent Emacs master. Not all lines that match the regexp alist
are highlighted. See attached screenshot for an example.
The "checking" lines are highlighted more or less at random.
There are errors from jit-lock (see below).

In GNU Emacs 26.0.50 (build 2, x86_64-w64-mingw32)
 of 2017-09-02 built on MACHINE
Repository revision: f2a074830c588d2a1c240afbd709a029a4c1a42f
Windowing system distributor 'Microsoft Corp.', version 10.0.15063
Recent messages:
Error during redisplay: (jit-lock-function 2553) signaled
(void-function compilation-face)
Error during redisplay: (jit-lock-function 2616) signaled
(void-function compilation-face)
Error during redisplay: (jit-lock-function 2645) signaled
(void-function compilation-face)
Error during redisplay: (jit-lock-function 2674) signaled
(void-function compilation-face)
Error during redisplay: (jit-lock-function 2733) signaled
(void-function compilation-face) [2 times]
Error during redisplay: (jit-lock-function 2764) signaled
(void-function compilation-face)
Error during redisplay: (jit-lock-function 2793) signaled
(void-function compilation-face)
Error during redisplay: (jit-lock-function 2822) signaled
(void-function compilation-face)
Error during redisplay: (jit-lock-function 2862) signaled
(void-function compilation-face)
Error during redisplay: (jit-lock-function 2995) signaled
(void-function compilation-face)

Configured using:
 'configure --config-cache --with-modules --without-pop CFLAGS=-Ofast'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS MODULES

Important settings:
  value of $EMACSLOADPATH: c:\emacs-lisp;
  value of $LANG: ENG
  locale-coding-system: cp1252

Major mode: Compilation

Minor modes in effect:
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr misearch multi-isearch emacsbug message subr-x
puny dired dired-loaddefs format-spec rfc822 mml mml-sec password-cache
epa derived epg epg-config gnus-util rmail rmail-loaddefs mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils shell
pcomplete compile comint ansi-color ring vc-git diff-mode easy-mmode map
seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib bat-mode
easymenu elec-pair time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote w32notify w32 multi-tty make-network-process emacs)

Memory information:
((conses 16 110759 24525)
 (symbols 56 21338 1)
 (miscs 48 69 194)
 (strings 32 33412 1339)
 (string-bytes 1 872517)
 (vectors 16 16549)
 (vector-slots 8 506344 10373)
 (floats 8 61 181)
 (intervals 56 849 128)
 (buffers 992 14))

[-- Attachment #2: compile.png --]
[-- Type: image/png, Size: 36158 bytes --]

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Mon, 4 Sep 2017 20:21:32 +0100
> 
> There seems to be an issue with compilation-mode highlighting in
> recent Emacs master. Not all lines that match the regexp alist
> are highlighted. See attached screenshot for an example.
> The "checking" lines are highlighted more or less at random.
> There are errors from jit-lock (see below).
> 
> In GNU Emacs 26.0.50 (build 2, x86_64-w64-mingw32)
>  of 2017-09-02 built on MACHINE
> Repository revision: f2a074830c588d2a1c240afbd709a029a4c1a42f
> Windowing system distributor 'Microsoft Corp.', version 10.0.15063
> Recent messages:
> Error during redisplay: (jit-lock-function 2553) signaled
> (void-function compilation-face)

This is because Tom's changes in 846870e removed the function
compilation-face, which is still referenced in several places in
compile.el.

Tom, could you take a look, please?

Thanks.

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Eli Zaretskii]

> Date: Tue, 05 Sep 2017 18:26:57 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 28349@debbugs.gnu.org
> 
> > Error during redisplay: (jit-lock-function 2553) signaled
> > (void-function compilation-face)
> 
> This is because Tom's changes in 846870e removed the function
> compilation-face, which is still referenced in several places in
> compile.el.
> 
> Tom, could you take a look, please?

No comments, so I fixed this bug by resurrecting the lost function,
and I'm marking the bug done.

Thanks.

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Tom Tromey]

Eli> No comments, so I fixed this bug by resurrecting the lost function,
Eli> and I'm marking the bug done.

Thanks.  I think this was the right thing to do.

Tom

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Richard Copley]

On 10 September 2017 at 18:21, Tom Tromey <tom@tromey.com> wrote:
> Eli> No comments, so I fixed this bug by resurrecting the lost function,
> Eli> and I'm marking the bug done.
>
> Thanks.  I think this was the right thing to do.
>
> Tom

Thanks. There seems to be another similar issue remaining.
Font-locking still fails for some lines "at random". This time the errors are

Error during redisplay: (jit-lock-function 94586) signaled
(args-out-of-range 0 3)
Error during redisplay: (jit-lock-function 284681) signaled
(args-out-of-range 0 3)
Error during redisplay: (jit-lock-function 285181) signaled
(args-out-of-range 0 3)
etc.

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Sun, 10 Sep 2017 19:33:44 +0100
> Cc: Tom Tromey <tom@tromey.com>, 28349@debbugs.gnu.org
> 
> Thanks. There seems to be another similar issue remaining.
> Font-locking still fails for some lines "at random". This time the errors are
> 
> Error during redisplay: (jit-lock-function 94586) signaled
> (args-out-of-range 0 3)
> Error during redisplay: (jit-lock-function 284681) signaled
> (args-out-of-range 0 3)
> Error during redisplay: (jit-lock-function 285181) signaled
> (args-out-of-range 0 3)

Can you show a more informative Lisp backtrace?

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Richard Copley]

On 10 September 2017 at 20:42, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Richard Copley <rcopley@gmail.com>
>> Date: Sun, 10 Sep 2017 19:33:44 +0100
>> Cc: Tom Tromey <tom@tromey.com>, 28349@debbugs.gnu.org
>>
>> Thanks. There seems to be another similar issue remaining.
>> Font-locking still fails for some lines "at random". This time the errors are
>>
>> Error during redisplay: (jit-lock-function 94586) signaled
>> (args-out-of-range 0 3)
>> Error during redisplay: (jit-lock-function 284681) signaled
>> (args-out-of-range 0 3)
>> Error during redisplay: (jit-lock-function 285181) signaled
>> (args-out-of-range 0 3)
>
> Can you show a more informative Lisp backtrace?

If I disable jit-lock and enable debug-on-error, then re-run the
compilation (toggling font-lock-mode is not sufficient), I get:

Debugger entered--Lisp error: (args-out-of-range 0 3)
  add-text-properties(0 3 (compilation-message #s(compilation--message
:loc (nil 37 (("addpm.c" "c:/projects/emacs/nt") nil (37 #3)) nil nil)
:type 0 :end-loc nil) help-echo "mouse-2: visit this file and line"
keymap compilation-button-map mouse-face highlight))
  compilation-parse-errors(6217 #<marker at 6913 in *compilation*>)
  compilation--parse-region(6217 #<marker at 6913 in *compilation*>)
  compilation--ensure-parse(6913)
  font-lock-fontify-keywords-region(6217 6913 nil)
  font-lock-default-fontify-region(6217 6913 nil)
  font-lock-fontify-region(6217 6913)
  font-lock-after-change-function(6217 6913 0)
  compilation-filter(#<process compilation> "make[1]: Entering
directory '/c/projects/emacs/lib'\n  AR       libgnu.a\nmake[1]:
Leaving directory '/c/projects/emacs/lib'\nmake[1]: Entering directory
'/c/projects/emacs/nt'\n  CCLD     addpm.exe\naddpm.c:42:0: warning:
\"_WIN32_WINNT\" redefined\n #define _WIN32_WINNT _WIN32_WINNT_WIN7\n
\nIn file included from
C:/msys64/mingw64/x86_64-w64-mingw32/include/crtdefs.h:10:0,\n
        from C:/msys64/mingw64/x86_64-w64-mingw32/include/stdlib.h:9,\n
                from
addpm.c:37:\nC:/msys64/mingw64/x86_64-w64-mingw32/include/_mingw.h:225:0:
note: this is the location of the previous definition\n #define
_WIN32_WINNT 0x502\n \nmake[1]: Leaving directory
'/c/projects/emacs/nt'\nmake -C lib-src all\n")

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Richard Copley]

On 10 September 2017 at 20:57, Richard Copley <rcopley@gmail.com> wrote:
> On 10 September 2017 at 20:42, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Richard Copley <rcopley@gmail.com>
>>> Date: Sun, 10 Sep 2017 19:33:44 +0100
>>> Cc: Tom Tromey <tom@tromey.com>, 28349@debbugs.gnu.org
>>>
>>> Thanks. There seems to be another similar issue remaining.
>>> Font-locking still fails for some lines "at random". This time the errors are
>>>
>>> Error during redisplay: (jit-lock-function 94586) signaled
>>> (args-out-of-range 0 3)
>>> Error during redisplay: (jit-lock-function 284681) signaled
>>> (args-out-of-range 0 3)
>>> Error during redisplay: (jit-lock-function 285181) signaled
>>> (args-out-of-range 0 3)
>>
>> Can you show a more informative Lisp backtrace?
>
> If I disable jit-lock and enable debug-on-error, then re-run the
> compilation (toggling font-lock-mode is not sufficient), I get:
>
> Debugger entered--Lisp error: (args-out-of-range 0 3)
>   add-text-properties(0 3 (compilation-message #s(compilation--message
> :loc (nil 37 (("addpm.c" "c:/projects/emacs/nt") nil (37 #3)) nil nil)
> :type 0 :end-loc nil) help-echo "mouse-2: visit this file and line"
> keymap compilation-button-map mouse-face highlight))
>   compilation-parse-errors(6217 #<marker at 6913 in *compilation*>)
>   compilation--parse-region(6217 #<marker at 6913 in *compilation*>)
>   compilation--ensure-parse(6913)
>   font-lock-fontify-keywords-region(6217 6913 nil)
>   font-lock-default-fontify-region(6217 6913 nil)
>   font-lock-fontify-region(6217 6913)
>   font-lock-after-change-function(6217 6913 0)
>   compilation-filter(#<process compilation> "make[1]: Entering
> directory '/c/projects/emacs/lib'\n  AR       libgnu.a\nmake[1]:
> Leaving directory '/c/projects/emacs/lib'\nmake[1]: Entering directory
> '/c/projects/emacs/nt'\n  CCLD     addpm.exe\naddpm.c:42:0: warning:
> \"_WIN32_WINNT\" redefined\n #define _WIN32_WINNT _WIN32_WINNT_WIN7\n
> \nIn file included from
> C:/msys64/mingw64/x86_64-w64-mingw32/include/crtdefs.h:10:0,\n
>         from C:/msys64/mingw64/x86_64-w64-mingw32/include/stdlib.h:9,\n
>                 from
> addpm.c:37:\nC:/msys64/mingw64/x86_64-w64-mingw32/include/_mingw.h:225:0:
> note: this is the location of the previous definition\n #define
> _WIN32_WINNT 0x502\n \nmake[1]: Leaving directory
> '/c/projects/emacs/nt'\nmake -C lib-src all\n")

This is the backtrace when string-match sets the match data to (0 3):

  (file-truename (cdr file))
  (if (cdr file) (progn (file-truename (cdr file)) (debug)))
  (let ((filename (car file)) (spec-directory (if (cdr file)
(file-truename (cdr file))))) (if (and (boundp
'comint-file-name-prefix) (not (equal comint-file-name-prefix "")))
(progn (if (file-name-absolute-p filename) (setq filename (concat
comint-file-name-prefix filename)) (if spec-directory (setq
spec-directory (file-truename (concat comint-file-name-prefix
spec-directory))))))) (if compilation-parse-errors-filename-function
(progn (setq filename (funcall
compilation-parse-errors-filename-function filename)))) (setq filename
(command-line-normalize-file-name filename)) (puthash file (or
(gethash (cons filename spec-directory) compilation-locs) (puthash
(cons filename spec-directory) (cons (list filename spec-directory)
(cons fmt nil)) compilation-locs)) compilation-locs))
  (or (gethash file compilation-locs) (let ((filename (car file))
(spec-directory (if (cdr file) (file-truename (cdr file))))) (if (and
(boundp 'comint-file-name-prefix) (not (equal comint-file-name-prefix
""))) (progn (if (file-name-absolute-p filename) (setq filename
(concat comint-file-name-prefix filename)) (if spec-directory (setq
spec-directory (file-truename (concat comint-file-name-prefix
spec-directory))))))) (if compilation-parse-errors-filename-function
(progn (setq filename (funcall
compilation-parse-errors-filename-function filename)))) (setq filename
(command-line-normalize-file-name filename)) (puthash file (or
(gethash (cons filename spec-directory) compilation-locs) (puthash
(cons filename spec-directory) (cons (list filename spec-directory)
(cons fmt nil)) compilation-locs)) compilation-locs)))
  compilation-get-file-structure(("addpm.c" . "/c/projects/emacs/nt") nil)
  compilation-internal-error-properties(("addpm.c" .
"/c/projects/emacs/nt") 37 nil nil nil 0 nil)
  compilation-error-properties(1 2 nil 3 nil 0 nil)
  compilation-parse-errors(290 #<marker at 742 in *compilation*>)
  compilation--parse-region(290 #<marker at 742 in *compilation*>)
  compilation--ensure-parse(742)
  font-lock-fontify-keywords-region(290 742 nil)
  font-lock-default-fontify-region(290 742 nil)
  font-lock-fontify-region(290 742)
  font-lock-after-change-function(290 742 0)
  compilation-filter(#<process compilation> "  CCLD
addpm.exe\naddpm.c:42:0: warning: \"_WIN32_WINNT\" redefined\n #define
_WIN32_WINNT _WIN32_WINNT_WIN7\n \nIn file included from
C:/msys64/mingw64/x86_64-w64-mingw32/include/crtdefs.h:10:0,\n
        from C:/msys64/mingw64/x86_64-w64-mingw32/include/stdlib.h:9,\n
                from
addpm.c:37:\nC:/msys64/mingw64/x86_64-w64-mingw32/include/_mingw.h:225:0:
note: this is the location of the previous definition\n #define
_WIN32_WINNT 0x502\n \n")

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Sun, 10 Sep 2017 23:41:45 +0100
> Cc: Tom Tromey <tom@tromey.com>, 28349@debbugs.gnu.org
> 
> This is the backtrace when string-match sets the match data to (0 3):

So you are saying the problem is in the call to string-match?  Or does
some code clobber match-data between that call and when the data is
used?

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Richard Copley]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

On 11 Sep 2017 15:54, "Eli Zaretskii" <eliz@gnu.org> wrote:

> From: Richard Copley <rcopley@gmail.com>
> Date: Sun, 10 Sep 2017 23:41:45 +0100
> Cc: Tom Tromey <tom@tromey.com>, 28349@debbugs.gnu.org
>
> This is the backtrace when string-match sets the match data to (0 3):

So you are saying the problem is in the call to string-match?  Or does
some code clobber match-data between that call and when the data is
used?


The call to string-match clobbers the correct match data, which was
obtained just before by re-search-forward, in compilation-parse-errors.

FYI this is not new and not related to Tom's change as far as I can see.

In compilation-parse-errors (compile.el:1426) (upcase comments added):

        [...]
        (goto-char start)
        (while (re-search-forward pat end t)  ;; SET MATCH-DATA
          (when (setq props (compilation-error-properties ;; CLOBBER IT
                             file line end-line col end-col (or type 2)
fmt))
        [...]
        ;; USE IT

[-- Attachment #2: Type: text/html, Size: 2118 bytes --]

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Mon, 11 Sep 2017 16:33:05 +0100
> Cc: Tom Tromey <tom@tromey.com>, 28349@debbugs.gnu.org
> 
> The call to string-match clobbers the correct match data, which was
> obtained just before by re-search-forward, in compilation-parse-errors.
> 
> FYI this is not new and not related to Tom's change as far as I can see.
> 
> In compilation-parse-errors (compile.el:1426) (upcase comments added):
> 
> [...]
> (goto-char start)
> (while (re-search-forward pat end t) ;; SET MATCH-DATA
> (when (setq props (compilation-error-properties ;; CLOBBER IT
> file line end-line col end-col (or type 2) fmt))
> [...]
> ;; USE IT

Thanks.  Can you post a short file which exhibits the problem under
compilation-mode?  I tried to use your examples in this discussion,
but none of them triggers these errors, so I'm probably missing
something.

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Richard Copley]

On 11 September 2017 at 17:56, Eli Zaretskii <eliz@gnu.org> wrote:
> Thanks.  Can you post a short file which exhibits the problem under
> compilation-mode?  I tried to use your examples in this discussion,
> but none of them triggers these errors, so I'm probably missing
> something.

I'll keep trying to come up with an example.

Some of the key ingredients:
  * Native Windows Emacs
  * MSYS make
  * An "Entering directory /c/..." message
  * An error or warning

There might be another ingredient that's specific to my
environment, but I don't think so.

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Richard Copley]

On 11 September 2017 at 20:51, Richard Copley <rcopley@gmail.com> wrote:
> On 11 September 2017 at 17:56, Eli Zaretskii <eliz@gnu.org> wrote:
>> Thanks.  Can you post a short file which exhibits the problem under
>> compilation-mode?  I tried to use your examples in this discussion,
>> but none of them triggers these errors, so I'm probably missing
>> something.
>
> I'll keep trying to come up with an example.
>
> Some of the key ingredients:
>   * Native Windows Emacs
>   * MSYS make
>   * An "Entering directory /c/..." message
>   * An error or warning
>
> There might be another ingredient that's specific to my
> environment, but I don't think so.

I had my own file-name-handler-alist entry that clobbered the match
data by calling (unmsys--filename) from "subr.el". Not an Emacs bug.

Ignore my previous attribution of this to file-truename.

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Sat, 16 Sep 2017 19:30:36 +0100
> Cc: 28349@debbugs.gnu.org
> 
> I had my own file-name-handler-alist entry that clobbered the match
> data by calling (unmsys--filename) from "subr.el". Not an Emacs bug.
> 
> Ignore my previous attribution of this to file-truename.

OK, thanks for looking into this.

bug#28349: 26.0.50; compilation mode syntax highlighting incorrect

[Richard Copley]

On 16 September 2017 at 19:37, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Richard Copley <rcopley@gmail.com>
>> Date: Sat, 16 Sep 2017 19:30:36 +0100
>> Cc: 28349@debbugs.gnu.org
>>
>> I had my own file-name-handler-alist entry that clobbered the match
>> data by calling (unmsys--filename) from "subr.el". Not an Emacs bug.
>>
>> Ignore my previous attribution of this to file-truename.
>
> OK, thanks for looking into this.

Thank you for asking the question.