bug#36507: 27.0.50; Crash on evaluating invalid UTF-8 byte sequence on MacOS

[YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>]

On Fri, 05 Jul 2019 11:04:21 +0900,
Stefan Kangas wrote:
> 
> When evaluating the following expression, I get a crash under "emacs -Q"
> compiled from current master.
> 
> (decode-coding-string "\xE3\x32\x9A\x36" 'chinese-gb18030)
> 
> This expression is tested in batch mode with no problems on the same
> system, now on master in test/lisp/bookmark-tests.el:281.
> 
> The expression was suggested in Bug#36452, where
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > Please add to that text something that doesn't yield valid
> > UTF-8 byte sequence.  For example, these two strings:
> >
> >   (decode-coding-string "\xE3\x32\x9A\x36" 'chinese-gb18030)
> 
> I think the issue as such is beyond me, but I can reproduce this every time.
> Please let me know if you need help testing or more information.
> 
> Before crash, I get this output:
> Thread 1 received signal SIGSEGV, Segmentation fault.
> 0x00007fff8ddbd326 in CFCharacterSetIsLongCharacterMember () from
> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation

Please try the patch below.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

diff --git a/src/macfont.m b/src/macfont.m
index f736fbf0e1e..2b7f963fd61 100644
--- a/src/macfont.m
+++ b/src/macfont.m
@@ -2076,7 +2076,7 @@ static int macfont_variation_glyphs (struct font *, int c,
               ptrdiff_t j;
 
               for (j = 0; j < ASIZE (chars); j++)
-                if (TYPE_RANGED_FIXNUMP (UTF32Char, AREF (chars, j))
+                if (RANGED_FIXNUMP (0, AREF (chars, j), MAX_UNICODE_CHAR)
                     && CFCharacterSetIsLongCharacterMember (desc_charset,
                                                             XFIXNAT (AREF (chars, j))))
                   break;
@@ -2710,6 +2710,9 @@ So we use CTFontDescriptorCreateMatchingFontDescriptor (no
   int result;
   CFCharacterSetRef charset;
 
+  if (c < 0 || c > MAX_UNICODE_CHAR)
+    return false;
+
   block_input ();
   if (FONT_ENTITY_P (font))
     {