Re: Change of Lisp syntax for "fancy" quotes in Emacs 27?

[Paul Eggert <eggert@cs.ucla.edu>]

[-- Attachment #1: Type: text/plain, Size: 1230 bytes --]

Aaron Ecay wrote:
> I was surprised to learn that this is the case, in light of what is
> said in the Elisp reference about symbol names

Good point; thanks. In the spirit of "be strict about what you generate", the 
Emacs printer should escape any character that is not in the list of characters 
documented in the Elisp manual as being safe (i.e., as not requiring escaping). 
This is elementary future-proofing, and is independent of whether we want Emacs 
to warn about or disallow confusable chars in symbols.

Proposed patches against 'master' attached. The first merely simplifes the code 
without changing its effect. The second fixes a bug in the manual, which 
incorrectly states that '?' never needs escaping in symbol names. These two 
patches are routine. (I assume the second one should be applied to emacs26 
instead of to master.)

The third patch changes the Lisp printer to escape characters as suggested above.

The fourth patch changes the Lisp printer to escape '?' only at the start of a 
symbol. This is nicer for programs using Scheme-style naming conventions in 
Emacs Lisp, e.g., 'fooish?' rather than 'fooishp'. I discovered the need for 
this patch when I wrote the second patch.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Simplify-print_object-a-bit.patch --]
[-- Type: text/x-patch; name="0001-Simplify-print_object-a-bit.patch", Size: 2780 bytes --]

From c03b816016f8cc2f15d275e7ad23448366489277 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 3 Feb 2018 20:29:00 -0800
Subject: [PATCH 1/4] Simplify print_object a bit

* src/print.c (print_object): Simplify by using C99
constructs, and by taking advantage of the fact that Lisp
strings are are followed by null bytes.
---
 src/print.c | 40 ++++++++++++++++------------------------
 1 file changed, 16 insertions(+), 24 deletions(-)

diff --git a/src/print.c b/src/print.c
index b3c0f6f..d3eb49d 100644
--- a/src/print.c
+++ b/src/print.c
@@ -1916,38 +1916,29 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
     case Lisp_Symbol:
       {
 	bool confusing;
-	unsigned char *p = SDATA (SYMBOL_NAME (obj));
-	unsigned char *end = p + SBYTES (SYMBOL_NAME (obj));
-	int c;
-	ptrdiff_t i, i_byte;
-	ptrdiff_t size_byte;
-	Lisp_Object name;
-
-	name = SYMBOL_NAME (obj);
-
-	if (p != end && (*p == '-' || *p == '+')) p++;
-	if (p == end)
-	  confusing = 0;
+	Lisp_Object name = SYMBOL_NAME (obj);
+	ptrdiff_t size_byte = SBYTES (name);
+	unsigned char *p = SDATA (name);
+	unsigned char *end = p + size_byte;
+
 	/* If symbol name begins with a digit, and ends with a digit,
 	   and contains nothing but digits and `e', it could be treated
 	   as a number.  So set CONFUSING.
 
-	   Symbols that contain periods could also be taken as numbers,
-	   but periods are always escaped, so we don't have to worry
-	   about them here.  */
-	else if (*p >= '0' && *p <= '9'
-		 && end[-1] >= '0' && end[-1] <= '9')
+	   Symbols that contain '.' or '#' could also be taken as
+	   numbers, but these are always escaped so don't worry about
+	   them here.  */
+	if (c_isdigit (p[*p == '-' || *p == '+']) && c_isdigit (end[-1]))
 	  {
-	    while (p != end && ((*p >= '0' && *p <= '9')
-				/* Needed for \2e10.  */
-				|| *p == 'e' || *p == 'E'))
+	    /* Check for 'e' too; needed for \2e10.  */
+	    do
 	      p++;
+	    while (c_isdigit (*p) || *p == 'e' || *p == 'E');
+
 	    confusing = (end == p);
 	  }
 	else
-	  confusing = 0;
-
-	size_byte = SBYTES (name);
+	  confusing = false;
 
 	if (! NILP (Vprint_gensym)
             && !SYMBOL_INTERNED_IN_INITIAL_OBARRAY_P (obj))
@@ -1958,10 +1949,11 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 	    break;
 	  }
 
-	for (i = 0, i_byte = 0; i_byte < size_byte;)
+	for (ptrdiff_t i = 0, i_byte = 0; i_byte < size_byte;)
 	  {
 	    /* Here, we must convert each multi-byte form to the
 	       corresponding character code before handing it to PRINTCHAR.  */
+	    int c;
 	    FETCH_STRING_CHAR_ADVANCE (c, name, i, i_byte);
 	    maybe_quit ();
 
-- 
2.7.4


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Say-needs-escaping-at-start-of-symbol.patch --]
[-- Type: text/x-patch; name="0002-Say-needs-escaping-at-start-of-symbol.patch", Size: 1210 bytes --]

From 4b945a3fcbf6ff2bde4595dd8b8f472d1b3d17af Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 3 Feb 2018 20:30:21 -0800
Subject: [PATCH 2/4] Say ? needs escaping at start of symbol.

* doc/lispref/objects.texi: ? is also special.
---
 doc/lispref/objects.texi | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/doc/lispref/objects.texi b/doc/lispref/objects.texi
index af74062..f0420e6 100644
--- a/doc/lispref/objects.texi
+++ b/doc/lispref/objects.texi
@@ -557,7 +557,8 @@ Symbol Type
 of the name suffice as long as the name does not look like a number.
 (If it does, write a @samp{\} at the beginning of the name to force
 interpretation as a symbol.)  The characters @samp{_~!@@$%^&:<>@{@}?} are
-less often used but also require no special punctuation.  Any other
+less often used but also require no special punctuation, except that
+@samp{\} must precede @samp{?} at the start of a symbol.  Any other
 characters may be included in a symbol's name by escaping them with a
 backslash.  In contrast to its use in strings, however, a backslash in
 the name of a symbol simply quotes the single character that follows the
-- 
2.7.4


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-prin1-etc.-now-escape-more-chars-in-symbols.patch --]
[-- Type: text/x-patch; name="0003-prin1-etc.-now-escape-more-chars-in-symbols.patch", Size: 3147 bytes --]

From 2add3a1595f709bb071e2b775970038470b2fab2 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 3 Feb 2018 20:30:48 -0800
Subject: [PATCH 3/4] prin1 etc. now escape more chars in symbols

Inspired by email from Aaron Ecay in:
https://lists.gnu.org/r/emacs-devel/2018-02/msg00125.html
* etc/NEWS: Mention this.
* src/print.c (print_object): Escape any character that is not
documented to not require escaping.
---
 etc/NEWS    |  7 +++++++
 src/print.c | 37 +++++++++++++++++++++++++++++++------
 2 files changed, 38 insertions(+), 6 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index afd0fba..2a46002 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -87,6 +87,13 @@ regular expression was previously invalid, but is now accepted:
 
    x\{32768\}
 
+** 'print' and related functions now escape more chars in symbols.
+They now escape any symbol character that is outside the documented
+set of characters that do not need escaping.  For example, (print
+(intern "n\u0456l")) now outputs "n\іl" instead of "nіl", as a hint to
+the reader that the "і" is not the usual U+0069 LATIN SMALL LETTER I,
+but is instead U+0456 CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I.
+
 \f
 * Editing Changes in Emacs 27.1
 
diff --git a/src/print.c b/src/print.c
index d3eb49d..7eca36a 100644
--- a/src/print.c
+++ b/src/print.c
@@ -1959,12 +1959,37 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 
 	    if (escapeflag)
 	      {
-		if (c == '\"' || c == '\\' || c == '\''
-		    || c == ';' || c == '#' || c == '(' || c == ')'
-		    || c == ',' || c == '.' || c == '`'
-		    || c == '[' || c == ']' || c == '?' || c <= 040
-                    || confusing
-		    || (i == 1 && confusable_symbol_character_p (c)))
+		switch (c)
+		  {
+		    /* The Emacs Lisp manual lists these characters as
+		       not requiring escaping in symbols.  Although some
+		       other characters might also work, play it safe
+		       and escape all but these characters.  */
+		  case '!': case '$': case '%': case '&':
+		  case '*': case '-': case '+': case '/':
+		  case '0': case '1': case '2': case '3': case '4':
+		  case '5': case '6': case '7': case '8': case '9':
+		  case ':': case '<': case '=': case '>': case '@':
+		  case 'A': case 'B': case 'C': case 'D': case 'E': case 'F':
+		  case 'G': case 'H': case 'I': case 'J': case 'K': case 'L':
+		  case 'M': case 'N': case 'O': case 'P': case 'Q': case 'R':
+		  case 'S': case 'T': case 'U': case 'V': case 'W': case 'X':
+		  case 'Y': case 'Z':
+		  case '^': case '_':
+		  case 'a': case 'b': case 'c': case 'd': case 'e': case 'f':
+		  case 'g': case 'h': case 'i': case 'j': case 'k': case 'l':
+		  case 'm': case 'n': case 'o': case 'p': case 'q': case 'r':
+		  case 's': case 't': case 'u': case 'v': case 'w': case 'x':
+		  case 'y': case 'z':
+		  case '{': case '}': case '~':
+		    break;
+
+		  default:
+		    confusing = true;
+		    break;
+		  }
+
+		if (confusing)
 		  {
 		    printchar ('\\', printcharfun);
 		    confusing = false;
-- 
2.7.4


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-Escape-only-at-start-of-symbol.patch --]
[-- Type: text/x-patch; name="0004-Escape-only-at-start-of-symbol.patch", Size: 1913 bytes --]

From 4289ea136de4876b5dfc20d83b5a2556d1b5d8e6 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 3 Feb 2018 20:39:48 -0800
Subject: [PATCH 4/4] Escape ? only at start of symbol

* src/print.c (print_object): Do it.
---
 etc/NEWS    | 4 ++++
 src/print.c | 4 ++--
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 2a46002..c435136 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -94,6 +94,10 @@ set of characters that do not need escaping.  For example, (print
 the reader that the "і" is not the usual U+0069 LATIN SMALL LETTER I,
 but is instead U+0456 CYRILLIC SMALL LETTER BYELORUSSIAN-UKRAINIAN I.
 
+** 'print' etc. no longer escape non-initial '?' in symbols.
+For example, the symbol 'list?' is now printed as-is.  Initial '?'
+is still escaped, e.g., (print (intern "?x")) still outputs "\?x".
+
 \f
 * Editing Changes in Emacs 27.1
 
diff --git a/src/print.c b/src/print.c
index 7eca36a..dfd6c50 100644
--- a/src/print.c
+++ b/src/print.c
@@ -1938,7 +1938,7 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 	    confusing = (end == p);
 	  }
 	else
-	  confusing = false;
+	  confusing = *p == '?';
 
 	if (! NILP (Vprint_gensym)
             && !SYMBOL_INTERNED_IN_INITIAL_OBARRAY_P (obj))
@@ -1969,7 +1969,7 @@ print_object (Lisp_Object obj, Lisp_Object printcharfun, bool escapeflag)
 		  case '*': case '-': case '+': case '/':
 		  case '0': case '1': case '2': case '3': case '4':
 		  case '5': case '6': case '7': case '8': case '9':
-		  case ':': case '<': case '=': case '>': case '@':
+		  case ':': case '<': case '=': case '>': case '?': case '@':
 		  case 'A': case 'B': case 'C': case 'D': case 'E': case 'F':
 		  case 'G': case 'H': case 'I': case 'J': case 'K': case 'L':
 		  case 'M': case 'N': case 'O': case 'P': case 'Q': case 'R':
-- 
2.7.4