bug#2967: smart quotes in .emacs

bug#2967: smart quotes in .emacs

[Kit O'Connell]

Hi,

I was troubleshooting a friend's .emacs issues with the help of some
internet assistance. When I copy/pasted people's suggested .emacs
entries, some smart quotes got into the file, causing error messages.
You can see the whole process of how this happened here:
http://dreamcafe.com/words/2009/04/11/help-us-dr-internet-emacs-edition/#comment-5222

I think it would be good if emacs looked for smart quotes in .emacs
files and gave a warning or notice if it detected them. This would
help troubleshooting.

Kit

bug#2967: smart quotes in .emacs

[Stefan Monnier]

> I was troubleshooting a friend's .emacs issues with the help of some
> internet assistance.  When I copy/pasted people's suggested .emacs
> entries, some smart quotes got into the file, causing error messages.

Please explain what you mean by "smart quotes" and describe the error
messages you got (and when you got them, and if they seemed to have any
actual effect).


        Stefan

bug#2967: smart quotes in .emacs

[Glenn Morris]

Stefan Monnier wrote:

>> I was troubleshooting a friend's .emacs issues with the help of some
>> internet assistance.  When I copy/pasted people's suggested .emacs
>> entries, some smart quotes got into the file, causing error messages.
>
> Please explain what you mean by "smart quotes" and describe the error
> messages you got (and when you got them, and if they seemed to have any
> actual effect).

I assume the OP means something like:

(set 'foo t)

where the quote character is actually U+2019 "RIGHT SINGLE QUOTATION MARK"
rather than a plain old-ASCII apostrophe.

This leads to errors like:

  Symbol's value as variable is void: 'foo

Recent example:

http://lists.gnu.org/archive/html/help-gnu-emacs/2012-08/msg00117.html

bug#2967: smart quotes in .emacs

[npostavs]

Glenn Morris <rgm@gnu.org> writes:

> I assume the OP means something like:
>
> (set 'foo t)
>
> where the quote character is actually U+2019 "RIGHT SINGLE QUOTATION MARK"
> rather than a plain old-ASCII apostrophe.
>
> This leads to errors like:
>
>   Symbol's value as variable is void: 'foo

Adding Alan to Cc, I think this is relevant also to the scenario in
Bug#23425.

I propose making these funny quotes give a read syntax error in symbol
name contexts.  Theoretically this breaks backwards compability with
code that uses variables containing them, but probably no such code
exists.

Something like this (not sure if this is the best way to test for them):

--- i/src/lread.c
+++ w/src/lread.c
@@ -3349,6 +3349,15 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 	     ? multibyte_chars_in_text ((unsigned char *) read_buffer,
 					nbytes)
 	     : nbytes);
+        if (!quoted && multibyte)
+          {
+            switch (STRING_CHAR ((unsigned char *) read_buffer))
+              {
+              case 0x2018:      /* LEFT SINGLE QUOTATION MARK */
+              case 0x2019:      /* RIGHT SINGLE QUOTATION MARK */
+                invalid_syntax ("strange quote");
+              }
+          }
 	Lisp_Object name = ((uninterned_symbol && ! NILP (Vpurify_flag)
 			     ? make_pure_string : make_specified_string)
 			    (read_buffer, nchars, nbytes, multibyte));

We then get the following error from evaluating (read (format-message "'foo")):

Debugger entered--Lisp error: (invalid-read-syntax "strange quote")
  read("’foo")
  eval((read (format-message "'foo")) nil)

bug#2967: smart quotes in .emacs

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Date: Tue, 06 Jun 2017 23:46:48 -0400
> Cc: 2967@debbugs.gnu.org, Alan Mackenzie <acm@muc.de>,
> 	Stefan Monnier <monnier@iro.umontreal.ca>
> 
> --- i/src/lread.c
> +++ w/src/lread.c
> @@ -3349,6 +3349,15 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
>  	     ? multibyte_chars_in_text ((unsigned char *) read_buffer,
>  					nbytes)
>  	     : nbytes);
> +        if (!quoted && multibyte)
> +          {
> +            switch (STRING_CHAR ((unsigned char *) read_buffer))
> +              {
> +              case 0x2018:      /* LEFT SINGLE QUOTATION MARK */
> +              case 0x2019:      /* RIGHT SINGLE QUOTATION MARK */
> +                invalid_syntax ("strange quote");
> +              }
> +          }
>  	Lisp_Object name = ((uninterned_symbol && ! NILP (Vpurify_flag)
>  			     ? make_pure_string : make_specified_string)
>  			    (read_buffer, nchars, nbytes, multibyte));
> 
> We then get the following error from evaluating (read (format-message "'foo")):
> 
> Debugger entered--Lisp error: (invalid-read-syntax "strange quote")
>   read("’foo")
>   eval((read (format-message "'foo")) nil)

Please include the character itself in the error message text.

Also, are we sure these two are the only ones we want to catch in this
manner?

Finally, if this is accepted, let's have a test for this, and maybe
also some documentation, at least in NEWS.

Thanks.

bug#2967: smart quotes in .emacs

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

tags 2967 + patch
quit

Eli Zaretskii <eliz@gnu.org> writes:

> Please include the character itself in the error message text.
>
> Also, are we sure these two are the only ones we want to catch in this
> manner?

I added some more likely looking ones from
https://en.wikipedia.org/wiki/Quotation_mark_glyphs.

> Finally, if this is accepted, let's have a test for this, and maybe
> also some documentation, at least in NEWS.


[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 3825 bytes --]

From 438030f42698f347e38a9dea1561eedadd7e2edc Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Wed, 7 Jun 2017 19:59:09 -0400
Subject: [PATCH v2] Signal error for symbol names with strange quotes
 (Bug#2967)

* src/lread.c (read1): Signal an error when a symbol starts with a
non-escaped quote-like character.
* test/src/lread-tests.el (lread-tests--funny-quote-symbols): New
test.
* etc/NEWS: Announce change.
---
 etc/NEWS                |  4 ++++
 src/lread.c             | 18 ++++++++++++++++++
 test/src/lread-tests.el | 17 +++++++++++++++++
 3 files changed, 39 insertions(+)

diff --git a/etc/NEWS b/etc/NEWS
index edb71118ef..d297d049d1 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1110,6 +1110,10 @@ instead of its first.
 renamed to 'lread--old-style-backquotes'.  No user code should use
 this variable.
 
+** To avoid confusion caused by "smart quotes", the reader no longer
+accepts Lisp symbols which begin with the following quotation
+characters: ‘’‛“”‟〞＂＇, unless they are escaped with backslash.
+
 +++
 ** Module functions are now implemented slightly differently; in
 particular, the function 'internal--module-call' has been removed.
diff --git a/src/lread.c b/src/lread.c
index 901e40b348..dbaadce4b4 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -3479,6 +3479,24 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 	    if (! NILP (result))
 	      return unbind_to (count, result);
 	  }
+        if (!quoted && multibyte)
+          {
+            int ch = STRING_CHAR ((unsigned char *) read_buffer);
+            switch (ch)
+              {
+              case 0x2018: /* LEFT SINGLE QUOTATION MARK */
+              case 0x2019: /* RIGHT SINGLE QUOTATION MARK */
+              case 0x201B: /* SINGLE HIGH-REVERSED-9 QUOTATION MARK */
+              case 0x201C: /* LEFT DOUBLE QUOTATION MARK */
+              case 0x201D: /* RIGHT DOUBLE QUOTATION MARK */
+              case 0x201F: /* DOUBLE HIGH-REVERSED-9 QUOTATION MARK */
+              case 0x301E: /* DOUBLE PRIME QUOTATION MARK */
+              case 0xFF02: /* FULLWIDTH QUOTATION MARK */
+              case 0xFF07: /* FULLWIDTH APOSTROPHE */
+                xsignal2 (Qinvalid_read_syntax, build_string ("strange quote"),
+                          CALLN (Fstring, make_number (ch)));
+              }
+          }
 	{
 	  Lisp_Object result;
 	  ptrdiff_t nbytes = p - read_buffer;
diff --git a/test/src/lread-tests.el b/test/src/lread-tests.el
index a0a317feee..dd5a2003b4 100644
--- a/test/src/lread-tests.el
+++ b/test/src/lread-tests.el
@@ -142,6 +142,23 @@ (ert-deftest lread-tests--unescaped-char-literals ()
                            "unescaped character literals "
                            "`?\"', `?(', `?)', `?;', `?[', `?]' detected!")))))
 
+(ert-deftest lread-tests--funny-quote-symbols ()
+  "Check that 'smart quotes' or similar trigger errors in symbol names."
+  (dolist (quote-char
+           '(#x2018 ;; LEFT SINGLE QUOTATION MARK
+             #x2019 ;; RIGHT SINGLE QUOTATION MARK
+             #x201B ;; SINGLE HIGH-REVERSED-9 QUOTATION MARK
+             #x201C ;; LEFT DOUBLE QUOTATION MARK
+             #x201D ;; RIGHT DOUBLE QUOTATION MARK
+             #x201F ;; DOUBLE HIGH-REVERSED-9 QUOTATION MARK
+             #x301E ;; DOUBLE PRIME QUOTATION MARK
+             #xFF02 ;; FULLWIDTH QUOTATION MARK
+             #xFF07 ;; FULLWIDTH APOSTROPHE
+             ))
+    (let ((str (format "%cfoo" quote-char)))
+     (should-error (read str) :type 'invalid-read-syntax)
+     (should (eq (read (concat "\\" str)) (intern str))))))
+
 (ert-deftest lread-test-bug26837 ()
   "Test for http://debbugs.gnu.org/26837 ."
   (let ((load-path (cons
-- 
2.11.1

bug#2967: smart quotes in .emacs

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 2967@debbugs.gnu.org,  rgm@gnu.org,  monnier@iro.umontreal.ca,  acm@muc.de
> Date: Sun, 16 Jul 2017 20:49:39 -0400
> 
> > Please include the character itself in the error message text.
> >
> > Also, are we sure these two are the only ones we want to catch in this
> > manner?
> 
> I added some more likely looking ones from
> https://en.wikipedia.org/wiki/Quotation_mark_glyphs.
> 
> > Finally, if this is accepted, let's have a test for this, and maybe
> > also some documentation, at least in NEWS.
> 
> 
> >From 438030f42698f347e38a9dea1561eedadd7e2edc Mon Sep 17 00:00:00 2001
> From: Noam Postavsky <npostavs@gmail.com>
> Date: Wed, 7 Jun 2017 19:59:09 -0400
> Subject: [PATCH v2] Signal error for symbol names with strange quotes
>  (Bug#2967)
> 
> * src/lread.c (read1): Signal an error when a symbol starts with a
> non-escaped quote-like character.
> * test/src/lread-tests.el (lread-tests--funny-quote-symbols): New
> test.
> * etc/NEWS: Announce change.

Thanks, this LGTM.  Please push.

bug#2967: smart quotes in .emacs

[npostavs]

tags 2967 fixed
close 2967 26.1
quit

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, this LGTM.  Please push.

Done.

[1: 8b18911a5c]: 2017-07-22 08:47:31 -0400
  Signal error for symbol names with strange quotes (Bug#2967)
  http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=8b18911a5c7c6c8a15b3cff12a4376ba68205e1c