Somwhat subtle issues with raw string literals in C++

Somwhat subtle issues with raw string literals in C++

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

Hi,

there still seem to be some subtle issues with detection of raw string
literals. Unfortunately they are hard to reproduce. One example that fails
(for me) with a recent master build is:

cd /tmp
wget
https://raw.githubusercontent.com/google/protobuf/ef7894e2dc6d287419e42a4fdc52cdfedd386d16/conformance/conformance_test.cc
/path/to/emacs -Q +686 conformance_test.cc

Around that line the fontification of the raw string literals is wrong
(quote characters are treated as string terminators), in other parts of the
file the fontification is correct. This typically happens with files that
contain many large raw string literals that contain quote characters.

[-- Attachment #2: Type: text/html, Size: 918 bytes --]

Re: Somwhat subtle issues with raw string literals in C++

[Alan Mackenzie]

Hello, Philipp.

On Tue, Jun 28, 2016 at 03:33:30PM +0000, Philipp Stephani wrote:
> Hi,

> there still seem to be some subtle issues with detection of raw string
> literals. Unfortunately they are hard to reproduce. One example that fails
> (for me) with a recent master build is:

> cd /tmp
> wget
> https://raw.githubusercontent.com/google/protobuf/ef7894e2dc6d287419e42a4fdc52cdfedd386d16/conformance/conformance_test.cc
> /path/to/emacs -Q +686 conformance_test.cc

> Around that line the fontification of the raw string literals is wrong
> (quote characters are treated as string terminators), in other parts of the
> file the fontification is correct. This typically happens with files that
> contain many large raw string literals that contain quote characters.

Thanks for the report.  I've downloaded that file, and I can reproduce
the error.  I'll look into it.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Somwhat subtle issues with raw string literals in C++

[Alan Mackenzie]

Hello again, Philipp,

On Tue, Jun 28, 2016 at 03:33:30PM +0000, Philipp Stephani wrote:
> Hi,

> there still seem to be some subtle issues with detection of raw string
> literals. Unfortunately they are hard to reproduce. One example that fails
> (for me) with a recent master build is:

> cd /tmp
> wget
> https://raw.githubusercontent.com/google/protobuf/ef7894e2dc6d287419e42a4fdc52cdfedd386d16/conformance/conformance_test.cc
> /path/to/emacs -Q +686 conformance_test.cc

> Around that line the fontification of the raw string literals is wrong
> (quote characters are treated as string terminators), in other parts of the
> file the fontification is correct. This typically happens with files that
> contain many large raw string literals that contain quote characters.

Again, thanks for the report.

The problem was a mishandling of a cache, with the result that the raw
string handling code mistakenly believed it was within a string when it
wasn't, at a critical point in the file.

Would you please try out the following patch and confirm that it fixes
the problem, or let me know what's still not working properly.

Thanks in advance!




diff -r 68a956fb5f55 cc-engine.el
--- a/cc-engine.el	Mon Jun 27 10:18:54 2016 +0000
+++ b/cc-engine.el	Wed Jun 29 13:43:27 2016 +0000
@@ -2292,6 +2292,12 @@
 ;; is reduced by buffer changes, and increased by invocations of
 ;; `c-parse-ps-state-below'.
 
+(defsubst c-truncate-semi-nonlit-pos-cache (pos)
+  ;; Truncate the upper bound of the cache `c-state-semi-nonlit-pos-cache' to
+  ;; POS, if it is higher than that position.
+  (setq c-state-semi-nonlit-pos-cache-limit
+	(min c-state-semi-nonlit-pos-cache-limit pos)))
+
 (defun c-state-semi-pp-to-literal (here &optional not-in-delimiter)
   ;; Do a parse-partial-sexp from a position in the buffer before HERE which
   ;; isn't in a literal, and return information about HERE, either:
@@ -2495,7 +2501,7 @@
       (let ((c c-state-semi-nonlit-pos-cache)
 	    elt state pos npos high-elt)
 	;; Trim the cache to take account of buffer changes.
-	(while (and c (> (c-ps-state-cache-pos (c-ps-state-cache-pos (car c)))
+	(while (and c (> (c-ps-state-cache-pos (car c))
 			 c-state-semi-nonlit-pos-cache-limit))
 	  (setq c (cdr c)))
 	(setq c-state-semi-nonlit-pos-cache c)
@@ -3488,8 +3494,7 @@
   ;; HERE.
   (if (<= here c-state-nonlit-pos-cache-limit)
       (setq c-state-nonlit-pos-cache-limit (1- here)))
-  (if (<= here c-state-semi-nonlit-pos-cache-limit)
-      (setq c-state-semi-nonlit-pos-cache-limit (1- here)))
+  (c-truncate-semi-nonlit-pos-cache here)
 
   ;; `c-state-cache':
   ;; Case 1: if `here' is in a literal containing point-min, everything
@@ -6082,19 +6085,32 @@
     (cond
      ((null open-paren-prop)
       ;; A terminated raw string
-      (if (search-forward (concat ")" id "\"") nil t)
-	  (c-clear-char-property-with-value
-	   (1+ open-paren) (match-beginning 0) 'syntax-table '(1))))
+      (when (search-forward (concat ")" id "\"") nil t)
+	(let* ((closing-paren (match-beginning 0))
+	       (first-punctuation
+		(save-match-data
+		  (goto-char (1+ open-paren))
+		  (and (c-search-forward-char-property 'syntax-table '(1)
+						       closing-paren)
+		       (1- (point)))))
+	       )
+	  (when first-punctuation
+	    (c-clear-char-property-with-value
+	     first-punctuation (match-beginning 0) 'syntax-table '(1))
+	    (c-truncate-semi-nonlit-pos-cache first-punctuation)
+	    ))))
      ((or (and (equal open-paren-prop '(15)) (null bound))
 	  (equal open-paren-prop '(1)))
       ;; An unterminated raw string either not in a macro, or in a macro with
       ;; the open parenthesis right up against the end of macro
       (c-clear-char-property open-quote 'syntax-table)
+      (c-truncate-semi-nonlit-pos-cache open-quote)
       (c-clear-char-property open-paren 'syntax-table))
      (t
       ;; An unterminated string in a macro, with at least one char after the
       ;; open paren
       (c-clear-char-property open-quote 'syntax-table)
+      (c-truncate-semi-nonlit-pos-cache open-quote)
       (c-clear-char-property open-paren 'syntax-table)
       (let ((after-string-fence-pos
 	     (save-excursion
@@ -6194,9 +6210,11 @@
 	(while (progn (skip-syntax-forward "^\"" end-string)
 		      (< (point) end-string))
 	  (c-put-char-property (point) 'syntax-table '(1)) ; punctuation
+	  (c-truncate-semi-nonlit-pos-cache (point))
 	  (forward-char))
 	(goto-char after-quote))
     (c-put-char-property open-quote 'syntax-table '(1))	     ; punctuation
+    (c-truncate-semi-nonlit-pos-cache open-quote)
     (c-put-char-property open-paren 'syntax-table '(15))     ; generic string
     (when bound
       ;; In a CPP construct, we try to apply a generic-string `syntax-table'
@@ -6223,9 +6241,12 @@
 		       "\\(\\\\\n\\)*\\=")) ; 11
              (1+ open-paren) t))
 	  (if (match-beginning 10)
-	      (c-put-char-property (match-beginning 10) 'syntax-table '(15))
+	      (progn
+		(c-put-char-property (match-beginning 10) 'syntax-table '(15))
+		(c-truncate-semi-nonlit-pos-cache (match-beginning 10)))
 	    (c-put-char-property (match-beginning 5) 'syntax-table '(1))
-	    (c-put-char-property (1+ (match-beginning 5)) 'syntax-table '(15)))
+	    (c-put-char-property (1+ (match-beginning 5)) 'syntax-table '(15))
+	    (c-truncate-semi-nonlit-pos-cache (1+ (match-beginning 5))))
 	(c-put-char-property open-paren 'syntax-table '(1)))
       (goto-char bound))))
 


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Somwhat subtle issues with raw string literals in C++

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1358 bytes --]

Alan Mackenzie <acm@muc.de> schrieb am Do., 30. Juni 2016 um 07:59 Uhr:

> Hello again, Philipp,
>
> On Tue, Jun 28, 2016 at 03:33:30PM +0000, Philipp Stephani wrote:
> > Hi,
>
> > there still seem to be some subtle issues with detection of raw string
> > literals. Unfortunately they are hard to reproduce. One example that
> fails
> > (for me) with a recent master build is:
>
> > cd /tmp
> > wget
> >
> https://raw.githubusercontent.com/google/protobuf/ef7894e2dc6d287419e42a4fdc52cdfedd386d16/conformance/conformance_test.cc
> > /path/to/emacs -Q +686 conformance_test.cc
>
> > Around that line the fontification of the raw string literals is wrong
> > (quote characters are treated as string terminators), in other parts of
> the
> > file the fontification is correct. This typically happens with files that
> > contain many large raw string literals that contain quote characters.
>
> Again, thanks for the report.
>
> The problem was a mishandling of a cache, with the result that the raw
> string handling code mistakenly believed it was within a string when it
> wasn't, at a critical point in the file.
>
> Would you please try out the following patch and confirm that it fixes
> the problem, or let me know what's still not working properly.
>
>
At least in the case I tested, the patch appears to fix the problem. Thanks
a lot for the quick fix.

[-- Attachment #2: Type: text/html, Size: 1920 bytes --]

Re: Somwhat subtle issues with raw string literals in C++

[Alan Mackenzie]

Hello, Philipp

On Thu, Jun 30, 2016 at 11:35:27AM +0000, Philipp Stephani wrote:
> Alan Mackenzie <acm@muc.de> schrieb am Do., 30. Juni 2016 um 07:59 Uhr:

> > Hello again, Philipp,

> > On Tue, Jun 28, 2016 at 03:33:30PM +0000, Philipp Stephani wrote:
> > > Hi,

> > > there still seem to be some subtle issues with detection of raw string
> > > literals. Unfortunately they are hard to reproduce. One example that
> > fails
> > > (for me) with a recent master build is:

> > > cd /tmp
> > > wget

> > https://raw.githubusercontent.com/google/protobuf/ef7894e2dc6d287419e42a4fdc52cdfedd386d16/conformance/conformance_test.cc
> > > /path/to/emacs -Q +686 conformance_test.cc

> > > Around that line the fontification of the raw string literals is wrong
> > > (quote characters are treated as string terminators), in other parts of
> > the
> > > file the fontification is correct. This typically happens with files that
> > > contain many large raw string literals that contain quote characters.

> > Again, thanks for the report.

> > The problem was a mishandling of a cache, with the result that the raw
> > string handling code mistakenly believed it was within a string when it
> > wasn't, at a critical point in the file.

> > Would you please try out the following patch and confirm that it fixes
> > the problem, or let me know what's still not working properly.


> At least in the case I tested, the patch appears to fix the problem. Thanks
> a lot for the quick fix.

Thank you again for the report.  I've committed the patch to the Emacs
master branch.

-- 
Alan Mackenzie (Nuremberg, Germany).