bug#36347: 26.2.90; fontification mistakes contraction for a string in HTML

bug#36347: 26.2.90; fontification mistakes contraction for a string in HTML

[Mike Kupfer]

[-- Attachment #1: Type: text/plain, Size: 377 bytes --]

To reproduce:

1. start emacs with -Q
2. Visit (C-x C-f) the attached file (foo.html).

When I do this with 26.2.90, the text starting with the apostrophe in
"doesn't", and ending at the end of file, is fontified as a string.

When I do this with 26.2-rc1, the text is fontified as I would expect
(HTML tags are in blue, everything else is in the default black or dark
gray).


[-- Attachment #2: test case --]
[-- Type: text/html, Size: 935 bytes --]

[-- Attachment #3: Type: text/plain, Size: 3206 bytes --]




In GNU Emacs 26.2.90 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw scroll bars)
 of 2019-06-23 built on alto
Windowing system distributor 'The X.Org Foundation', version 11.0.11902000
System Description:	Debian GNU/Linux 9.9 (stretch)

Recent messages:
Mark set [3 times]
Auto-saving...done
Saving file /home/kupfer/private_html/computers.html...
Wrote /home/kupfer/private_html/computers.html
When done with this frame, type C-x 5 0
GNU Emacs 26.2.90 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw scroll bars) of 2019-06-23
Quit
When done with this frame, type C-x 5 0
Quit
Mark saved where search started

Configured features:
XPM JPEG TIFF GIF PNG SOUND NOTIFY GNUTLS LIBXML2 FREETYPE LIBOTF XFT
ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM THREADS

Important settings:
  value of $LC_TIME: C
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  delete-selection-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny format-spec rfc822 mml
mml-sec password-cache epa derived epg epg-config gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils dired-x dired dired-loaddefs add-log
skeleton misearch multi-isearch sgml-mode seq gv dom elec-pair server
noutline outline easy-mmode cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs byte-opt bytecomp
byte-compile cconv shell pcomplete comint ansi-color ring xcscope
easymenu advice delsel vc cl-loaddefs cl-lib vc-dispatcher timeclock
mdk-hacks time-date mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core term/tty-colors frame cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads inotify
dynamic-setting font-render-setting x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 152389 9893)
 (symbols 48 26251 1)
 (miscs 40 136 313)
 (strings 32 47063 956)
 (string-bytes 1 1299530)
 (vectors 16 20032)
 (vector-slots 8 595254 13062)
 (floats 8 89 321)
 (intervals 56 1199 0)
 (buffers 992 17))

bug#36347: 26.2.90; fontification mistakes contraction for a string in HTML

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

Mike Kupfer <mkupfer@alum.berkeley.edu> writes:

> To reproduce:
>
> 1. start emacs with -Q
> 2. Visit (C-x C-f) the attached file (foo.html).
>
> When I do this with 26.2.90, the text starting with the apostrophe in
> "doesn't", and ending at the end of file, is fontified as a string.
>
> When I do this with 26.2-rc1, the text is fontified as I would expect
> (HTML tags are in blue, everything else is in the default black or dark
> gray).

The problem is the parens around the quote, sgml-mode uses the wrong
syntax table during propertizing.  The only reason it doesn't show up in
26.2 is because sgml-mode fails to fontify single quoted strings at all.
The same problem exists in 26.2 and earlier with double quote, although
it's less likely to have an unmatched double quote in text.

<html>(")</html>

<html>(')</html>

So the question is what to do about this for the release branch.

1. Revert the fix which adds handling of single quotes.  This means
single quotes never cause highlighting, even when they should.  This is
a regression in nxml-mode relative to Emacs 25, but sgml-mode worked
like that for a while already (in Emacs 26, nxml-mode was changed to
rely on some of sgml-mode's code).

2. Fix this bug with the patch below, it's fairly small and
straightforward.  And also, can be disabled by setting
syntax-ppss-table to nil in sgml-mode-hook.

3. Do nothing (for 26.3, that is).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 3302 bytes --]

From cb2f5380e3e78b7d956c604c03c21122c8ced36d Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Sun, 23 Jun 2019 21:27:43 -0400
Subject: [PATCH] Fix sgml-mode handling of quotes within parens (Bug#36347)

* lisp/textmodes/sgml-mode.el (sgml-syntax-propertize): Use
syntax-ppss-table if set.  This is only needed on the release branch,
on master the caller (syntax-propertize) already does this.
(sgml-mode): Set syntax-ppss-table to sgml-tag-syntax-table.  This
correctly classifies parens as punctuation, so they won't confuse the
parser.
* test/lisp/textmodes/sgml-mode-tests.el (sgml-tests--quotes-syntax):
New test copied from master, with two cases added for this bug.
---
 lisp/textmodes/sgml-mode.el            | 10 ++++++----
 test/lisp/textmodes/sgml-mode-tests.el | 22 ++++++++++++++++++++++
 2 files changed, 28 insertions(+), 4 deletions(-)

diff --git a/lisp/textmodes/sgml-mode.el b/lisp/textmodes/sgml-mode.el
index 128e58810e..895ce844e9 100644
--- a/lisp/textmodes/sgml-mode.el
+++ b/lisp/textmodes/sgml-mode.el
@@ -357,10 +357,11 @@ (eval-and-compile
 (defun sgml-syntax-propertize (start end)
   "Syntactic keywords for `sgml-mode'."
   (goto-char start)
-  (sgml-syntax-propertize-inside end)
-  (funcall
-   (syntax-propertize-rules sgml-syntax-propertize-rules)
-   start end))
+  (with-syntax-table (or syntax-ppss-table (syntax-table)) ; Not needed on master.
+    (sgml-syntax-propertize-inside end)
+    (funcall
+     (syntax-propertize-rules sgml-syntax-propertize-rules)
+     start end)))
 
 (defun sgml-syntax-propertize-inside (end)
   (let ((ppss (syntax-ppss)))
@@ -568,6 +569,7 @@ (define-derived-mode sgml-mode text-mode '(sgml-xml-mode "XML" "SGML")
 			      sgml-font-lock-keywords-2)
 			     nil t))
   (setq-local syntax-propertize-function #'sgml-syntax-propertize)
+  (setq-local syntax-ppss-table sgml-tag-syntax-table)
   (setq-local facemenu-add-face-function 'sgml-mode-facemenu-add-face-function)
   (setq-local sgml-xml-mode (sgml-xml-guess))
   (unless sgml-xml-mode
diff --git a/test/lisp/textmodes/sgml-mode-tests.el b/test/lisp/textmodes/sgml-mode-tests.el
index 7318a667b3..0000b352ff 100644
--- a/test/lisp/textmodes/sgml-mode-tests.el
+++ b/test/lisp/textmodes/sgml-mode-tests.el
@@ -130,5 +130,27 @@ (ert-deftest sgml-delete-tag-bug-8203-should-not-delete-apostrophe ()
    (sgml-delete-tag 1)
    (should (string= "Winter is comin'" (buffer-string)))))
 
+(ert-deftest sgml-tests--quotes-syntax ()
+  (dolist (str '("a\"b <t>c'd</t>"
+                 "a'b <t>c\"d</t>"
+                 "<t>\"a'</t>"
+                 "<t>'a\"</t>"
+                 "<t>\"a'\"</t>"
+                 "<t>'a\"'</t>"
+                 "a\"b <tag>c'd</tag>"
+                 ;;"<tag>c>'d</tag>" Fixed in master.
+                 "<t><!-- \" --></t>"
+                 "<t><!-- ' --></t>"
+                 "<t>(')</t>"
+                 "<t>(\")</t>"
+                 ))
+   (with-temp-buffer
+     (sgml-mode)
+     (insert str)
+     (ert-info ((format "%S" str) :prefix "Test case: ")
+       ;; Check that last tag is parsed as a tag.
+       (should (= 1 (car (syntax-ppss (1- (point-max))))))
+       (should (= 0 (car (syntax-ppss (point-max)))))))))
+
 (provide 'sgml-mode-tests)
 ;;; sgml-mode-tests.el ends here
-- 
2.11.0

bug#36347: 26.2.90; fontification mistakes contraction for a string in HTML

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Date: Sun, 23 Jun 2019 21:47:00 -0400
> Cc: 36347@debbugs.gnu.org
> 
> So the question is what to do about this for the release branch.
> 
> 1. Revert the fix which adds handling of single quotes.  This means
> single quotes never cause highlighting, even when they should.  This is
> a regression in nxml-mode relative to Emacs 25, but sgml-mode worked
> like that for a while already (in Emacs 26, nxml-mode was changed to
> rely on some of sgml-mode's code).
> 
> 2. Fix this bug with the patch below, it's fairly small and
> straightforward.  And also, can be disabled by setting
> syntax-ppss-table to nil in sgml-mode-hook.
> 
> 3. Do nothing (for 26.3, that is).

Let's do 2).  I'd also like to ask Mike to please apply the change
locally and run with it, so we collect as much experience with it
before RC as possible.

Thanks.

bug#36347: 26.2.90; fontification mistakes contraction for a string in HTML

[Noam Postavsky]

tags 36347 fixed
close 36347 26.3
quit

Eli Zaretskii <eliz@gnu.org> writes:
>
> Let's do 2).  I'd also like to ask Mike to please apply the change
> locally and run with it, so we collect as much experience with it
> before RC as possible.

Okay, pushed to emacs-26, and merged to master.

e62ad04963 2019-06-25T18:58:23-04:00 "Fix sgml-mode handling of quotes within parens (Bug#36347)"
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=e62ad04963982ea9cc7622b32484778845bc2ec1

bug#36347: 26.2.90; fontification mistakes contraction for a string in HTML

[Mike Kupfer]

Eli Zaretskii wrote:

> I'd also like to ask Mike to please apply the change
> locally and run with it, so we collect as much experience with it
> before RC as possible.

Okay, it's installed locally.  I can confirm that it fixes the problem
that I originally saw.

Thanks!

mike

bug#36347: 26.2.90; fontification mistakes contraction for a string in HTML

[Eli Zaretskii]

> From: Mike Kupfer <mkupfer@alum.berkeley.edu>
> cc: 36347@debbugs.gnu.org
> Date: Tue, 25 Jun 2019 20:12:14 -0700
> 
> Eli Zaretskii wrote:
> 
> > I'd also like to ask Mike to please apply the change
> > locally and run with it, so we collect as much experience with it
> > before RC as possible.
> 
> Okay, it's installed locally.  I can confirm that it fixes the problem
> that I originally saw.

Thanks, please be sure to report any problems you see, as I'm
currently planning to have the next pretest be the first release
candidate.

bug#36347: 26.2.90; fontification mistakes contraction for a string in HTML

[Mike Kupfer]

Eli Zaretskii wrote:

> please be sure to report any problems you see

Will do.

mike