[PATCH] Unicode Lisp reader escapes

[Aidan Kehoe <kehoea@parhasard.net>]

I realise you are all focused on the release with an intensity that would
scare small children, were any of them let near, but if any of you have a
minute free, I’d love to hear philosophical and technical objections to the
below.

The background is that it hasn’t ever been possible to consistently specify
a non-Latin-1 character by means of a general escape sequence, since what
character a given integer represents varies from release to release and even
from invocation to invocation. The below allows you to specify a backslash
escape with exactly four or exactly eight hexadecimal digits in a character
or string, and have the editor interpret them as the corresponding Unicode
code point. So, ?\u20AC would be interpreted as the Euro sign, "\u0448" as
Cyrillic sha, ?\U001D0ED as Byzantine musical symbol arktiko ke. 

Why not wait until the Unicode branch is merged? Well, that won’t solve the
problem either; people naturally want their code to be as compatible as
possible, so they will avoid the assumption that the integer-to-character
mapping is Unicode compatible as long as there are editors in the wild for
which that is not true. If this is integrated a good bit before the Unicode
branch is (which is what I would like), it will mean people can use this
syntax (which most modern programming languages have already, and which
people use) and be sure it’s compatible years before what would otherwise be
the case. 

lispref/ChangeLog addition:

2006-04-29  Aidan Kehoe  <kehoea@parhasard.net>

	* objects.texi (Character Type):
	Describe the Unicode character escape syntax; \uABCD or \U00ABCDEF
	specifies Unicode characters U+ABCD and U+ABCDEF respectively. 
	

src/ChangeLog addition:

2006-04-29  Aidan Kehoe  <kehoea@parhasard.net>

	* lread.c (read_escape):
	Provide a Unicode character escape syntax; \u followed by exactly
	four or \U followed by exactly eight hex digits in a comment or
	string is read as a Unicode character with that code point. 


GNU Emacs Trunk source patch:
Diff command:   cvs -q diff -u
Files affected: src/lread.c lispref/objects.texi

Index: lispref/objects.texi
===================================================================
RCS file: /sources/emacs/emacs/lispref/objects.texi,v
retrieving revision 1.51
diff -u -u -r1.51 objects.texi
--- lispref/objects.texi	6 Feb 2006 11:55:10 -0000	1.51
+++ lispref/objects.texi	29 Apr 2006 15:15:09 -0000
@@ -431,6 +431,20 @@
 bit values are 2**22 for alt, 2**23 for super and 2**24 for hyper.
 @end ifnottex
 
+@cindex unicode character escape
+  Emacs provides a syntax for specifying characters by their Unicode code
+points.  @samp{?\uABCD} will give you an Emacs character that maps to
+the code point @samp{U+ABCD} in Unicode-based representations (UTF-8
+text files, Unicode-oriented fonts, etc.) There is a slightly different
+syntax for specifying characters with code points above @samp{#xFFFF};
+@samp{\U00ABCDEF} will give you an Emacs character that maps to the code
+point @samp{U+ABCDEF} in Unicode-based representations, if such an Emacs
+character exists.
+
+  Unlike in some other languages, while this syntax is available for
+character literals, and (see later) in strings, it is not available
+elsewhere in your Lisp source code.
+
 @cindex @samp{\} in character constant
 @cindex backslash in character constant
 @cindex octal character code
Index: src/lread.c
===================================================================
RCS file: /sources/emacs/emacs/src/lread.c,v
retrieving revision 1.350
diff -u -u -r1.350 lread.c
--- src/lread.c	27 Feb 2006 02:04:35 -0000	1.350
+++ src/lread.c	29 Apr 2006 15:15:10 -0000
@@ -1743,6 +1743,9 @@
      int *byterep;
 {
   register int c = READCHAR;
+  /* \u allows up to four hex digits, \U up to eight. Default to the
+     behaviour for \u, and change this value in the case that \U is seen. */
+  int unicode_hex_count = 4;
 
   *byterep = 0;
 
@@ -1907,6 +1910,48 @@
 	return i;
       }
 
+    case 'U':
+      /* Post-Unicode-2.0: Up to eight hex chars */
+      unicode_hex_count = 8;
+    case 'u':
+
+      /* A Unicode escape. We only permit them in strings and characters,
+	 not arbitrarily in the source code as in some other languages. */
+      {
+	int i = 0;
+	int count = 0;
+	Lisp_Object lisp_char;
+	while (++count <= unicode_hex_count)
+	  {
+	    c = READCHAR;
+	    /* isdigit(), isalpha() may be locale-specific, which we don't
+	       want. */
+	    if      (c >= '0' && c <= '9')  i = (i << 4) + (c - '0');
+	    else if (c >= 'a' && c <= 'f')  i = (i << 4) + (c - 'a') + 10;
+            else if (c >= 'A' && c <= 'F')  i = (i << 4) + (c - 'A') + 10;
+	    else
+	      {
+		error ("Non-hex digit used for Unicode escape");
+		break;
+	      }
+	  }
+
+	lisp_char = call2(intern("decode-char"), intern("ucs"),
+			  make_number(i));
+
+	if (EQ(Qnil, lisp_char))
+	  {
+	    /* This is ugly and horrible and trashes the user's data. */
+	    XSETFASTINT (i, MAKE_CHAR (charset_katakana_jisx0201, 
+				       34 + 128, 46 + 128));
+            return i;
+	  }
+	else
+	  {
+	    return XFASTINT (lisp_char);
+	  }
+      }
+
     default:
       if (BASE_LEADING_CODE_P (c))
 	c = read_multibyte (c, readcharfun);

-- 
In the beginning God created the heavens and the earth. And God was a
bug-eyed, hexagonal smurf with a head of electrified hair; and God said:
“Si, mi chiamano Mimi...”