bug#38564: 27.0.50; macOS "emacs -nw" git-gutter-mode segfault

[Robert Pluim <rpluim@gmail.com>]

>>>>> On Tue, 10 Dec 2019 17:33:37 -0800, Mike Hamrick <mikeh@muppetlabs.com> said:

    Mike> Hi there,
    Mike> Here is how I got emacs to segfault and drop be back to the shell:
    Mike> - compile emacs 27.0.50 under macOS
    Mike> - install the git-gutter package
    Mike> - set up an after-init-hook for global-git-gutter-mode
    Mike> - emacs -nw /some/file/under/version/control

Iʼm surprised nobody's seen this before.

    Mike> The relevant part of my init.el looks like:
    >> (use-package git-gutter
    >> :ensure t
    >> :init
    >> (add-hook 'after-init-hook 'global-git-gutter-mode))

    Mike>   * frame #0: 0x000000010416ab3e emacs`ns_lookup_indexed_color(idx=18446744073709551613, f=0x00007facab80e210) at nsterm.m:2097:64
    Mike>     frame #1: 0x000000010416b5fc emacs`ns_color_index_to_rgba(idx=-3, f=0x00007facab80e210) at nsterm.m:2299:9

    Mike> diff --git a/src/nsterm.m b/src/nsterm.m
    Mike> index c415159890..ea2b141d95 100644
    Mike> --- a/src/nsterm.m
    Mike> +++ b/src/nsterm.m
    Mike> @@ -2094,6 +2094,8 @@ so some key presses (TAB) are swallowed by the system.  */
    Mike>  NSColor *
    Mike>  ns_lookup_indexed_color (unsigned long idx, struct frame *f)
    Mike>  {
    Mike> +  if (FRAME_DISPLAY_INFO (f) == nil)
    Mike> +    return nil;
    Mike>    struct ns_color_table *color_table = FRAME_DISPLAY_INFO (f)->color_table;
    Mike>    if (idx < 1 || idx >= color_table->avail)
    Mike>      return nil;

Right idea, but one layer lower than needed. ns_color_index_to_rgba
should not be calling ns_lookup_indexed_color in non-gui mode. Try
this instead:

diff --git a/src/nsterm.m b/src/nsterm.m
index 52a9830be8..814a090370 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -2290,19 +2290,24 @@ so some key presses (TAB) are swallowed by the system.  */
 
 /* Convert an index into the color table into an RGBA value.  Used in
    xdisp.c:extend_face_to_end_of_line when comparing faces and frame
-   color values.  */
+   color values.  No-op on non-gui frames*/
 
 unsigned long
 ns_color_index_to_rgba(int idx, struct frame *f)
 {
-  NSColor *col;
-  col = ns_lookup_indexed_color (idx, f);
+  if (FRAME_DISPLAY_INFO (f))
+    {
+      NSColor *col;
+      col = ns_lookup_indexed_color (idx, f);
 
-  EmacsCGFloat r, g, b, a;
-  [col getRed: &r green: &g blue: &b alpha: &a];
+      EmacsCGFloat r, g, b, a;
+      [col getRed: &r green: &g blue: &b alpha: &a];
 
-  return ARGB_TO_ULONG((int)(a*255),
-                       (int)(r*255), (int)(g*255), (int)(b*255));
+      return ARGB_TO_ULONG((int)(a*255),
+                           (int)(r*255), (int)(g*255), (int)(b*255));
+    }
+  else
+    return idx;
 }
 
 void