bug#28850: 26.0.90; Error running timer 'jit-lock-stealth-fontify': (error "Invalid search bound (wrong side of point)")

[Alan Mackenzie <acm@muc.de>]

Hello again, Eli.

On Tue, Oct 24, 2017 at 20:33:12 +0000, Alan Mackenzie wrote:
> On Tue, Oct 24, 2017 at 17:46:08 +0300, Eli Zaretskii wrote:
> > > Date: Sun, 22 Oct 2017 20:13:40 +0000
> > > Cc: 28850@debbugs.gnu.org
> > > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > Thanks, this fixes a very large part of the problem, so I think you
> > should install this on the release branch.

DONE.

> > It doesn't solve all of it, though, because I got that breakpoint hit
> > again.  This time it took much longer before that happened (a sign
> > that most of the problem is indeed solved), and the backtrace is
> > different.

> This is an entirely separate bug.

> > Here's the C and the Lisp backtraces, followed by some relevant
> > values:

> [ .... ].

> >   Lisp Backtrace:
> >   "re-search-forward" (0x88c050)
> >   0xad7f328 PVEC_COMPILED
> >   "font-lock-fontify-keywords-region" (0x88ca70)
> >   "font-lock-default-fontify-region" (0x88ce60)
> >   "c-font-lock-fontify-region" (0x88d230)
> >   "font-lock-fontify-region" (0x88d578)
> >   0x13810840 PVEC_COMPILED
> >   "run-hook-wrapped" (0x88daf0)
> >   "jit-lock--run-functions" (0x88dee0)
> >   "jit-lock-fontify-now" (0x88e3e0)
> >   "jit-lock-stealth-fontify" (0x88e9d0)
> >   "apply" (0x88e9c8)
> >   "timer-event-handler" (0x88edb8)
> >   (gdb) pp current_buffer->name_
> >   "platform.h"
> >   (gdb) pp current_buffer->directory_
> >   "d:/utils/lz4-1.7.5/programs/"
> >   (gdb) p PT
> >   $1 = 2645
> >   (gdb) p bound
> >   $2 = make_number(2425)
> >   (gdb) up
> >   #1  0x011cd2c9 in Fre_search_forward (regexp=XIL(0x800000000adcb1f0),
> >       bound=..., noerror=..., count=...) at search.c:2271
> >   2271      return search_command (regexp, bound, noerror, count, 1, 1, 0);
> >   (gdb) pp regexp
> >   "\\(\\=\\|\\(\\=\\|[^\\]\\)[
> >   ]\\)\\s *#\\s *\\(\\(?:\\(?:el\\)?if\\)\\)\\([^[:alnum:]_$]\\|$\\)\\(\\\\\\(.\\|
> >   [
> >   ]\\)\\|[^
> >   ]\\)*"

> I've tracked down that regexp.  It matches "#if" or "#elif" inside
> macros.  It's used in c-cpp-matchers in cc-fonts.el.

> > As you see, point is at 2645, whereas BOUND is at 2425.  This happens
> > in the file platform.h from the lz4-1.7.5 distribution.  Here's the
> > relevant part of platform.h with the two locations shown (I added an
> > empty line for each "^" marker):

> The reason for this is that a generated lambda form with the argument
> LIMIT (which would be the end of a jit-lock chunk or similar) internally
> binds LIMIT to the end of the current macro.  Inside this binding, which
> searches for "defined" repeatedly, we go forward to after the last
> "defined", as indicated in your source excerpt below.  Unfortunately,
> this point is beyond the original LIMIT supplied to the lambda, so in the
> next re-search-forward, point is the wrong side of this original LIMIT.

> This particular bit of CC Mode is "write only" code, and it could take
> me some while to disentangle the stack of macros and function generators
> which has produced it.  I think I'm going to extract this form and
> rewrite it more by hand, making it simpler to debug in the future.

Actually, it wasn't that difficult to amend that form generator.  Would
you please try out the patch below, which should apply cleanly to
master.

> > /* **************************************
> > *  Detect 64-bit OS
> > *  http://nadeausoftware.com/articles/2012/02/c_c_tip_how_detect_processor_type_using_compiler_predefined_macros
> > ****************************************/
> > #if defined __ia64 || defined _M_IA64                                                                               /* Intel Itanium */ \
> >   || defined __powerpc64__ || defined __ppc64__ || defined __PPC64__                                                /* POWER 64-bit */  \
> >   || (defined __sparc && (defined __sparcv9 || defined __sparc_v9__ || defined __arch64__)) || defined __sparc64__  /* SPARC 64-bit */  \
> >   || defined __x86_64__s || defined _M_X64                                                                          /* x86 64-bit */    \
> >   || defined __arm64__ || defined __aarch64__ || defined __ARM64_ARCH_8__                                           /* ARM 64-bit */    \
> >   || (defined __mips  && (__mips == 64 || __mips == 4 || __mips == 3))                                              /* MIPS 64-bit */   \
> >                                                                       ^
> >   || defined _LP64 || defined __LP64__ /* NetBSD, OpenBSD */ || defined __64BIT__ /* AIX */ || defined _ADDR64 /* Cray */               \
> >   || (defined __SIZEOF_POINTER__ && __SIZEOF_POINTER__ == 8) /* gcc */
> >               ^
> > #  if !defined(__64BIT__)
> > #    define __64BIT__  1
> > #  endif
> > #endif

> > As you see, BOUND is after the closing paren, after "3))", and point
> > is at the beginning of "__SIZEOF_POINTER__" a couple of lines further.

> Yes.  BOUND would have been the end of a jit-lock-chunk.

[ .... ]


diff -r c2f277772ea2 cc-fonts.el
--- a/cc-fonts.el	Wed Oct 25 18:00:13 2017 +0000
+++ b/cc-fonts.el	Wed Oct 25 18:57:20 2017 +0000
@@ -286,12 +286,17 @@
 			      nil)))))
 	  res))))
 
-  (defun c-make-font-lock-search-form (regexp highlights)
+  (defun c-make-font-lock-search-form (regexp highlights &optional check-point)
     ;; Return a lisp form which will fontify every occurrence of REGEXP
     ;; (a regular expression, NOT a function) between POINT and `limit'
     ;; with HIGHLIGHTS, a list of highlighters as specified on page
-    ;; "Search-based Fontification" in the elisp manual.
-    `(while (re-search-forward ,regexp limit t)
+    ;; "Search-based Fontification" in the elisp manual.  If CHECK-POINT
+    ;; is non-nil, we will check (< (point) limit) in the main loop.
+    `(while
+	 ,(if check-point
+	      `(and (< (point) limit)
+		    (re-search-forward ,regexp limit t))
+	    `(re-search-forward ,regexp limit t))
        (unless (progn
 		 (goto-char (match-beginning 0))
 		 (c-skip-comments-and-strings limit))
@@ -470,7 +475,9 @@
 			,(c-make-font-lock-search-form
 			  regexp highlights)))))
 	     state-stanzas)
-	  ,(c-make-font-lock-search-form (car normal) (cdr normal))
+	  ;; In the next form, check that point hasn't been moved beyond
+	  ;; `limit' in any of the above stanzas.
+	  ,(c-make-font-lock-search-form (car normal) (cdr normal) t)
 	  nil))))
 
 ;  (eval-after-load "edebug" ; 2006-07-09: def-edebug-spec is now in subr.el.

-- 
Alan Mackenzie (Nuremberg, Germany).