bug#32338: 26.1; term.el broken on macOS

[Eli Zaretskii <eliz@gnu.org>]

> From: Constantine Vetoshev <vetoshev@gmail.com>
> Date: Tue, 2 Oct 2018 15:51:11 -0700
> Cc: Alan Third <alan@idiocy.org>, Noam Postavsky <npostavs@gmail.com>, 32338@debbugs.gnu.org
> 
> One change from my past reports: after compiling Emacs with -g flags,
> I have now managed to reproduce the crash under lldb, including
> attaching to the forked process which eats CPU after the crash.
> Backtrace from that process is attached.

Great, thanks.

> The highlights (as far as I noticed) are:
> - emacs_re_max_failures and the older re_max_failures are not
> initialized at this point

I believe this is incorrect.  re_max_failures is statically assigned a
value of 40000 in regex-emacs.c, and should be initialized at link
time.  Your build is with optimizations, isn't it?  I think the
optimizer reordered instructions, which creates the illusion that
re_max_failures has a garbled value at that point.  The value of
newlim, 10022912, is correct, you can confirm that by calculating it
by hand assuming that re_max_failures is 40000 and using the other
values your debugging session shows.

> - in the working branch, newlim is reset to rlim.rlim_max; in the
> broken branch, it is not
> - in the working branch, setrlimit does not get called; in the broken
> branch, it does

Right.

> I'm guessing the problem is with the uninitialized values for
> *_re_max_failures and the resulting values being assigned to lim and
> newlim. It seems to only work on the working branch by accident
> because, for whatever reason, newlim always gets reset to
> rlim.rlim_max and setrlimit doesn't get called.

No, I think the problem is with this line:

   > 884            if (pagesize <= newlim - lim)

In your case newlim is smaller than lim, but rlim_t is an unsigned
data type on your system, so the subtraction wraps around and produces
a large positive value, which then tricks Emacs into thinking it needs
to enlarge the stack, whereas in reality the stack space already
available, 67MB, is large enough.  (Btw, that value sounds too large,
I wonder if it's some problem with getrlimit on your system.)

So please try the patch below with the emacs-26 branch, and see if the
problem goes away.

diff --git a/src/emacs.c b/src/emacs.c
index 483e848..c0b4bd9 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -875,7 +875,8 @@ main (int argc, char **argv)
 	    newlim = rlim.rlim_max;
 	  newlim -= newlim % pagesize;
 
-	  if (pagesize <= newlim - lim)
+	  if (newlim > lim	/* in case rlim_t is an unsigned type */
+	      && pagesize <= newlim - lim)
 	    {
 	      rlim.rlim_cur = newlim;
 	      if (setrlimit (RLIMIT_STACK, &rlim) == 0)
@@ -884,9 +885,9 @@ main (int argc, char **argv)
 	}
       /* If the stack is big enough, let regex.c more of it before
          falling back to heap allocation.  */
-      emacs_re_safe_alloca = max
-        (min (lim - extra, SIZE_MAX) * (min_ratio / ratio),
-         MAX_ALLOCA);
+      emacs_re_safe_alloca =
+	max (min (min (0, lim - extra), SIZE_MAX) * (min_ratio / ratio),
+	     MAX_ALLOCA);
     }
 #endif /* HAVE_SETRLIMIT and RLIMIT_STACK and not CYGWIN */