bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[Keith David Bershatsky]

I have modified versions of undo-tree.el and the cl family of functions (with different names [e.g., "lcl-..."] that use the old-style defstruct), which may be able to still take advantage of the vector method of dealing with structs.  I am receiving a message "Re-entering top level after C stack overflow".  It appears that `read` can no longer handle the following type of structure, perhaps because there is no built-in backwards compatibility -- this is a small example of the what `read` is able to handle in earlier versions of Emacs, but not the current master branch:

[cl-struct-undo-tree [nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil ([nil (#1=[nil nil ((26 . 27)) nil (22874 59645 117315 0) 0 nil (((22874 59645 117309 0) . t)) nil nil]) ((25 . 26)) nil (22874 59645 117331 0) 0 nil (((22874 59645 107309 0))) nil nil]) ((24 . 25)) nil (22874 59645 117335 0) 0 nil (((22874 59645 97309 0))) nil nil]) ((23 . 24)) nil (22874 59645 117339 0) 0 nil (((22874 59645 87309 0))) nil nil]) ((22 . 23)) nil (22874 59645 117343 0) 0 nil (((22874 59645 77309 0))) nil nil]) ((21 . 22)) nil (22874 59645 117347 0) 0 nil (((22874 59645 67309 0))) nil nil]) ((20 . 21)) nil (22874 59645 117351 0) 0 nil (((22874 59645 57309 0))) nil nil]) ((19 . 20)) nil (22874 59645 117354 0)
  0 nil (((22874 59645 47309 0))) nil nil]) ((18 . 19)) nil (22874 59645 117358 0) 0 nil (((22874 59645 37309 0))) nil nil]) ((17 . 18)) nil (22874 59645 117363 0) 0 nil (((22874 59645 27309 0))) nil
  nil]) ((16 . 17)) nil (22874 59645 117366 0) 0 nil (((22874 59645 17309 0))) nil nil]) ((15 . 16)) nil (22874 59645 117370 0) 0 nil (((22874 59645 7309 0))) nil nil]) ((14 . 15)) nil (22874 59645 117374 0) 0 nil (((22874 59644 997309 0))) nil nil]) ((13 . 14)) nil (22874 59645 117378 0) 0 nil (((22874 59644 987309 0))) nil nil]) ((12 . 13)) nil (22874 59645 117382 0) 0 nil (((22874 59644 977309 0))) nil nil]) ((11 . 12)) nil (22874 59645 117386 0) 0 nil (((22874 59644 967309 0))) nil nil]) ((10 . 11)) nil (22874 59645 117390 0) 0 nil (((22874 59644 957309 0))) nil nil]) ((9 . 10)) nil (22874 59645 117394 0) 0 nil (((22874 59644 947309 0))) nil nil]) ((8 . 9)) nil (22874 59645 117398 0) 0 nil (((22874 59644 937309 0))) nil nil]) ((7 . 8)) nil (22874 59645 117402 0) 0 nil (((22874 59644 92
 7309 0))) nil nil]) ((6 . 7)) nil (22874 59645 117405 0) 0 nil (((22874 59644 917309 0))) nil nil]) ((5 . 6)) nil (22874 59645 117409 0) 0 nil (((22874 59644 907309 0))) nil nil]) ((4 . 5)) nil (228
 74 59645 117413 0) 0 nil (((22874 59644 897309 0))) nil nil]) ((3 . 4)) nil (22874 59645 117417 0) 0 nil (((22874 59644 887309 0))) nil nil]) ((2 . 3)) nil (22874 59645 117420 0) 0 nil (((22874 59644 877309 0))) nil nil]) ((1 . 2) (t 22874 59561 0 0)) nil (22874 59645 117425 0) 0 nil (((22874 59644 867309 0))) nil nil]) nil nil (22874 59632 379899 0) 0 nil (((0 0))) nil nil] #1# 216 26 nil #1#]

Here is the backtrace for the first 20 frames:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00007fff5f26ef08
0x000000010023ae38 in read1 (readcharfun=Cannot access memory at address 0x7fff5f26ef08
) at lread.c:2674
2674	{
(gdb) bt
#0  0x000000010023ae38 in read1 (readcharfun=Cannot access memory at address 0x7fff5f26ef08
) at lread.c:2674
#1  0x000000010023eba5 in read_list (flag=true, readcharfun={i = 4773552677}) at lread.c:3866
#2  0x000000010023e7d6 in read_vector (readcharfun={i = 4773552677}, bytecodeflag=false) at lread.c:3776
#3  0x000000010023b01e in read1 (readcharfun={i = 4773552677}, pch=0x7fff5f2777a4, first_in_list=true) at lread.c:2695
#4  0x000000010023eba5 in read_list (flag=false, readcharfun={i = 4773552677}) at lread.c:3866
#5  0x000000010023b001 in read1 (readcharfun={i = 4773552677}, pch=0x7fff5f27bc24, first_in_list=false) at lread.c:2692
#6  0x000000010023eba5 in read_list (flag=true, readcharfun={i = 4773552677}) at lread.c:3866
#7  0x000000010023e7d6 in read_vector (readcharfun={i = 4773552677}, bytecodeflag=false) at lread.c:3776
#8  0x000000010023b01e in read1 (readcharfun={i = 4773552677}, pch=0x7fff5f280134, first_in_list=true) at lread.c:2695
#9  0x000000010023eba5 in read_list (flag=false, readcharfun={i = 4773552677}) at lread.c:3866
#10 0x000000010023b001 in read1 (readcharfun={i = 4773552677}, pch=0x7fff5f2845b4, first_in_list=false) at lread.c:2692
#11 0x000000010023eba5 in read_list (flag=true, readcharfun={i = 4773552677}) at lread.c:3866
#12 0x000000010023e7d6 in read_vector (readcharfun={i = 4773552677}, bytecodeflag=false) at lread.c:3776
#13 0x000000010023b01e in read1 (readcharfun={i = 4773552677}, pch=0x7fff5f288ac4, first_in_list=true) at lread.c:2695
#14 0x000000010023eba5 in read_list (flag=false, readcharfun={i = 4773552677}) at lread.c:3866
#15 0x000000010023b001 in read1 (readcharfun={i = 4773552677}, pch=0x7fff5f28cf44, first_in_list=false) at lread.c:2692
#16 0x000000010023eba5 in read_list (flag=true, readcharfun={i = 4773552677}) at lread.c:3866
#17 0x000000010023e7d6 in read_vector (readcharfun={i = 4773552677}, bytecodeflag=false) at lread.c:3776
#18 0x000000010023b01e in read1 (readcharfun={i = 4773552677}, pch=0x7fff5f291454, first_in_list=true) at lread.c:2695
#19 0x000000010023eba5 in read_list (flag=false, readcharfun={i = 4773552677}) at lread.c:3866
#20 0x000000010023b001 in read1 (readcharfun={i = 4773552677}, pch=0x7fff5f2958d4, first_in_list=false) at lread.c:2692

***

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[Keith David Bershatsky]

Perhaps we can use a prior function written by @npostavs -- (make-deep-object DEPTH) -- to create a large enough example of nested vectors and lists such that someone on the Emacs development team can duplicate the C stack overflow with `read`:

    (require 'cl-lib)

    (defun make-deep-object (depth)
      (let ((obj 1))
        (while (> (cl-decf depth) 0)
          (setq obj (vector (list obj))))
        obj))

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[npostavs]

tags 27779 + confirmed
quit

Keith David Bershatsky <esq@lawlist.com> writes:

> Perhaps we can use a prior function written by @npostavs --
> (make-deep-object DEPTH) -- to create a large enough example of nested
> vectors and lists such that someone on the Emacs development team can
> duplicate the C stack overflow with `read`:
>
>     (require 'cl-lib)
>
>     (defun make-deep-object (depth)
>       (let ((obj 1))
>         (while (> (cl-decf depth) 0)
>           (setq obj (vector (list obj))))
>         obj))

Hah, yes, it seems that `read' uses up more stack than `print', e.g.,
(make-deep-object 4600) succeeds, but (read (prin1-to-string
(make-deep-object 4600))) overflows.

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[Keith David Bershatsky]

Thank you for coming up with a test to reproduce this issue.

I was not able to workaround this problem in Emacs 26.0.50 by issuing the following command from the terminal prior to launching Emacs from the same terminal session:

ulimit -S -s unlimited

Keith

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[npostavs]

Keith David Bershatsky <esq@lawlist.com> writes:

> I was not able to workaround this problem in Emacs 26.0.50 by issuing the following command from the terminal prior to launching Emacs from the same terminal session:

s/not able/able/ ? (it worked for me)

> ulimit -S -s unlimited

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[Keith David Bershatsky]

Setting `ulimit -S -s unlimited` from the terminal prior to launching the GUI version of Emacs 26.0.50 from the same terminal session on OSX 10.6.8 does NOT resolve/fix the "Re-entering top level after C stack overflow" that appears in the *Messages* buffer.  :(

Emacs 25.2.1, on the other hand, IS able to successfully handle the following example:

(require 'cl-lib)

(defun make-deep-object (depth)
    (let ((obj 1))
      (while (> (cl-decf depth) 0)
        (setq obj (vector (list obj))))
      obj))

(let* ((print-circle t)
       (max-lisp-eval-depth most-positive-fixnum)
       (max-specpdl-size most-positive-fixnum)
       (deep-object (make-deep-object 4600))
       (string (prin1-to-string deep-object))
       (result (read string)))
  (when result
    (message "Success!")))

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[npostavs]

Keith David Bershatsky <esq@lawlist.com> writes:

> Setting `ulimit -S -s unlimited` from the terminal prior to launching
> the GUI version of Emacs 26.0.50 from the same terminal session on OSX
> 10.6.8 does NOT resolve/fix the "Re-entering top level after C stack
> overflow" that appears in the *Messages* buffer.  :(

Hmm, the trick works on GNU/Linux.

>
> Emacs 25.2.1, on the other hand, IS able to successfully handle the following example:

Yes, it's a bit strange that on 26.0.50 prin1-to-string can go up to
almost 60000, while `read' gives up at 4600.  By the way, for future
reference, I suggest adding a length count to the "Success!" output; I
got confused for a bit when seeing C-M-x easily succeed up to very large
depths (which is because it binds print-level to a small value).

(let* ((print-circle t)
       (max-lisp-eval-depth most-positive-fixnum)
       (max-specpdl-size most-positive-fixnum)
       (deep-object (make-deep-object 6000))
       (string (prin1-to-string deep-object))
       (result (read string)))
  (when result
    (message "Success! (%d)" (length string))))

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[Keith David Bershatsky]

By process of elimination via the lo-tech method of trying different nightly builds from https://emacsformacosx.com/builds/all , I have determined that December 8, 2016 was the last build where I am able to successfully run from the terminal:

ulimit -S -s unlimited

/path/to/gui/version/emacs/built/12_08_2016

and then from the *scratch* buffer, evaluate the following:

(require 'cl-lib)

(defun make-deep-object (depth)
    (let ((obj 1))
      (while (> (cl-decf depth) 0)
        (setq obj (vector (list obj))))
      obj))

(let* ((print-circle t)
       (max-lisp-eval-depth most-positive-fixnum)
       (max-specpdl-size most-positive-fixnum)
       (deep-object (make-deep-object 6000))
       (string (prin1-to-string deep-object))
       (result (read string)))
  (when result
    (message "Success! (%d)" (length string))))

The builds that were made AFTER December 8, 2016 all crash through June 11, 2017.  At some point AFTER June 11, 2017, Emacs changed from crashing to just throwing an error message "Re-entering top level after C stack overflow".

The build of December 8, 2016 does not have `emacs-repository-version` defined, so it will be necessary to look at the commits on December 7, 2016 and December 8, 2016 to see what changed that eliminated the prior ability to use `ulimit -S -s unlimited` as a workaround to the stack overflow problem.

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[npostavs]

Keith David Bershatsky <esq@lawlist.com> writes:

> By process of elimination via the lo-tech method of trying different
> nightly builds from https://emacsformacosx.com/builds/all , I have
> determined that December 8, 2016 was the last build where I am able to
> successfully run from the terminal:

Likely candidate: [1: f0a1e9ec3f], specifically the addition of stackbuf
in this hunk means read1 will use a lot more stack per call.

@@ -2541,8 +2556,9 @@ read_integer (Lisp_Object readcharfun, EMACS_INT radix)
 read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 {
   int c;
-  bool uninterned_symbol = 0;
+  bool uninterned_symbol = false;
   bool multibyte;
+  char stackbuf[MAX_ALLOCA];
 
   *pch = 0;

> At some point AFTER June 11, 2017, Emacs changed from
> crashing to just throwing an error message "Re-entering top level
> after C stack overflow".

I presume that's from [2: 9dee1c884e].

[1: f0a1e9ec3f]: 2016-12-08 13:00:32 -0800
  Make read1 more reentrant
  http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=f0a1e9ec3fba3d5bea5bd62f525dba3fb005d1b1

[2: 9dee1c884e]: 2017-07-14 04:57:18 -0700
  Improve stack-overflow heuristic on GNU/Linux
  http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=9dee1c884eb50ba282eb9dd2495c5269add25963

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

This is a work in progress, but here is a partial:

The first time I see a break in the prior expected behavior was on 11/19/2017 (c61ee94959ba96b2a327df0684593f7e569e30be) when portions of Emacs 25 were merged with the master branch and `emacs.c` was modified.  Here is a partial reversal of the code that broke the ability to avoid a stack overflow by setting `ulimit -S -s unlimited`.  The patch applies to the above-mentioned head -- i.e., the patch will apply following a hard reset.

I have not yet been able to follow the subsequent breaks to avoid a stack overflow with the current master branch.


[-- Attachment #2: patch_5e2ae74df54d4090c591c79ab13e7713c6654b9c.diff --]
[-- Type: application/diff, Size: 3552 bytes --]

bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

Dear Paul and Noam:

I have determined that bug #27779 was introduced on December 8, 2016 with commit f0a1e9ec3fba3d5bea5bd62f525dba3fb005d1b1.  The following patch to the Emacs 26 branch as of today (01/08/2018) reverses the commit and enables the test below to be completed successfully.

[FYI:  I am on OSX 10.6.8 and am manually increasing the stack limit with `ulimit -S -s unlimited` so that I can have rather large custom undo-tree histories.]

(require 'cl-lib)

(defun make-deep-object (depth)
    (let ((obj 1))
      (while (> (cl-decf depth) 0)
        (setq obj (vector (list obj))))
      obj))

;;; STACK OVERFLOW:  problem with `read'
;;; The bug was introduced on December 8, 2016 with
;;; commit:  f0a1e9ec3fba3d5bea5bd62f525dba3fb005d1b1
(let* ((print-circle t)
       (max-lisp-eval-depth most-positive-fixnum)
       (max-specpdl-size most-positive-fixnum)
       (deep-object (make-deep-object 6000))
       (string (prin1-to-string deep-object))
       (result (read string)))
  (when result
    (message "Bug #27779:  Success! (%d)" (length string))))


[-- Attachment #2: patch.diff --]
[-- Type: application/diff, Size: 9713 bytes --]

bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 494 bytes --]

Hello all,

I've run into this issue today when loading up undo-tree on Emacs 26.1.
The last patch fixes it for me, I only had to delete the first context
line of hunk #13 for it to apply with some fuzz.  I expect plenty more
people to run into this as undo-tree is a dependency for Evil and
therefore keeps Emacs users from properly booting their editor (it's
particularly fatal for the daemon, I couldn't get `emacsclient` to
successfully connect to it after it ran into the issue).

Vasilij

[-- Attachment #2: stack-overflow.patch --]
[-- Type: text/plain, Size: 9696 bytes --]

diff --git a/src/lread.c b/src/lread.c
index 45d6064..1219077 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -2272,28 +2272,16 @@ read0 (Lisp_Object readcharfun)
 	    Fmake_string (make_number (1), make_number (c)));
 }
 \f
-/* Grow a read buffer BUF that contains OFFSET useful bytes of data,
-   by at least MAX_MULTIBYTE_LENGTH bytes.  Update *BUF_ADDR and
-   *BUF_SIZE accordingly; 0 <= OFFSET <= *BUF_SIZE.  If *BUF_ADDR is
-   initially null, BUF is on the stack: copy its data to the new heap
-   buffer.  Otherwise, BUF must equal *BUF_ADDR and can simply be
-   reallocated.  Either way, remember the heap allocation (which is at
-   pdl slot COUNT) so that it can be freed when unwinding the stack.*/
-
-static char *
-grow_read_buffer (char *buf, ptrdiff_t offset,
-		  char **buf_addr, ptrdiff_t *buf_size, ptrdiff_t count)
+static ptrdiff_t read_buffer_size;
+static char *read_buffer;
+ 
+/* Grow the read buffer by at least MAX_MULTIBYTE_LENGTH bytes.  */
+ 
+static void
+grow_read_buffer (void)
 {
-  char *p = xpalloc (*buf_addr, buf_size, MAX_MULTIBYTE_LENGTH, -1, 1);
-  if (!*buf_addr)
-    {
-      memcpy (p, buf, offset);
-      record_unwind_protect_ptr (xfree, p);
-    }
-  else
-    set_unwind_protect_ptr (count, xfree, p);
-  *buf_addr = p;
-  return p;
+  read_buffer = xpalloc (read_buffer, &read_buffer_size,
+			 MAX_MULTIBYTE_LENGTH, -1, 1);
 }
 
 /* Return the scalar value that has the Unicode character name NAME.
@@ -2673,10 +2661,8 @@ static Lisp_Object
 read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 {
   int c;
-  bool uninterned_symbol = false;
+  bool uninterned_symbol = 0;
   bool multibyte;
-  char stackbuf[MAX_ALLOCA];
-  current_thread->stack_top = stackbuf;
 
   *pch = 0;
 
@@ -3028,7 +3014,7 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
       /* #:foo is the uninterned symbol named foo.  */
       if (c == ':')
 	{
-	  uninterned_symbol = true;
+	  uninterned_symbol = 1;
 	  c = READCHAR;
 	  if (!(c > 040
 		&& c != NO_BREAK_SPACE
@@ -3285,20 +3271,16 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 
     case '"':
       {
-	ptrdiff_t count = SPECPDL_INDEX ();
-	char *read_buffer = stackbuf;
-	ptrdiff_t read_buffer_size = sizeof stackbuf;
-	char *heapbuf = NULL;
 	char *p = read_buffer;
 	char *end = read_buffer + read_buffer_size;
 	int ch;
 	/* True if we saw an escape sequence specifying
 	   a multibyte character.  */
-	bool force_multibyte = false;
+	bool force_multibyte = 0;
 	/* True if we saw an escape sequence specifying
 	   a single-byte character.  */
-	bool force_singlebyte = false;
-	bool cancel = false;
+	bool force_singlebyte = 0;
+	bool cancel = 0;
 	ptrdiff_t nchars = 0;
 
 	while ((ch = READCHAR) >= 0
@@ -3307,9 +3289,7 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 	    if (end - p < MAX_MULTIBYTE_LENGTH)
 	      {
 		ptrdiff_t offset = p - read_buffer;
-		read_buffer = grow_read_buffer (read_buffer, offset,
-						&heapbuf, &read_buffer_size,
-						count);
+		grow_read_buffer ();
 		p = read_buffer + offset;
 		end = read_buffer + read_buffer_size;
 	      }
@@ -3324,7 +3304,7 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 		if (ch == -1)
 		  {
 		    if (p == read_buffer)
-		      cancel = true;
+		      cancel = 1;
 		    continue;
 		  }
 
@@ -3332,9 +3312,9 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 		ch = ch & ~CHAR_MODIFIER_MASK;
 
 		if (CHAR_BYTE8_P (ch))
-		  force_singlebyte = true;
+		  force_singlebyte = 1;
 		else if (! ASCII_CHAR_P (ch))
-		  force_multibyte = true;
+		  force_multibyte = 1;
 		else		/* I.e. ASCII_CHAR_P (ch).  */
 		  {
 		    /* Allow `\C- ' and `\C-?'.  */
@@ -3360,7 +3340,7 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 			   string.  */
 			modifiers &= ~CHAR_META;
 			ch = BYTE8_TO_CHAR (ch | 0x80);
-			force_singlebyte = true;
+			force_singlebyte = 1;
 		      }
 		  }
 
@@ -3373,9 +3353,9 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 	      {
 		p += CHAR_STRING (ch, (unsigned char *) p);
 		if (CHAR_BYTE8_P (ch))
-		  force_singlebyte = true;
+		  force_singlebyte = 1;
 		else if (! ASCII_CHAR_P (ch))
-		  force_multibyte = true;
+		  force_multibyte = 1;
 	      }
 	    nchars++;
 	  }
@@ -3387,7 +3367,7 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 	   return zero instead.  This is for doc strings
 	   that we are really going to find in etc/DOC.nn.nn.  */
 	if (!NILP (Vpurify_flag) && NILP (Vdoc_file_name) && cancel)
-	  return unbind_to (count, make_number (0));
+	  return make_number (0);
 
 	if (! force_multibyte && force_singlebyte)
 	  {
@@ -3398,11 +3378,9 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 	    p = read_buffer + nchars;
 	  }
 
-	Lisp_Object result
-	  = make_specified_string (read_buffer, nchars, p - read_buffer,
-				   (force_multibyte
-				    || (p - read_buffer != nchars)));
-	return unbind_to (count, result);
+	return make_specified_string (read_buffer, nchars, p - read_buffer,
+				      (force_multibyte
+				       || (p - read_buffer != nchars)));
       }
 
     case '.':
@@ -3430,54 +3408,58 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 
     read_symbol:
       {
-	ptrdiff_t count = SPECPDL_INDEX ();
-	char *read_buffer = stackbuf;
-	ptrdiff_t read_buffer_size = sizeof stackbuf;
-	char *heapbuf = NULL;
 	char *p = read_buffer;
-	char *end = read_buffer + read_buffer_size;
-	bool quoted = false;
+	bool quoted = 0;
 	EMACS_INT start_position = readchar_count - 1;
 
-	do
-	  {
-	    if (end - p < MAX_MULTIBYTE_LENGTH + 1)
-	      {
-		ptrdiff_t offset = p - read_buffer;
-		read_buffer = grow_read_buffer (read_buffer, offset,
-						&heapbuf, &read_buffer_size,
-						count);
-		p = read_buffer + offset;
-		end = read_buffer + read_buffer_size;
-	      }
+	{
+	  char *end = read_buffer + read_buffer_size;
+ 
+	  do
+	    {
+	      if (end - p < MAX_MULTIBYTE_LENGTH)
+		{
+		  ptrdiff_t offset = p - read_buffer;
+		  grow_read_buffer ();
+		  p = read_buffer + offset;
+		  end = read_buffer + read_buffer_size;
+		}
 
-	    if (c == '\\')
-	      {
-		c = READCHAR;
-		if (c == -1)
-		  end_of_file_error ();
-		quoted = true;
-	      }
+	      if (c == '\\')
+		{
+		  c = READCHAR;
+		  if (c == -1)
+		    end_of_file_error ();
+		  quoted = 1;
+		}
 
-	    if (multibyte)
-	      p += CHAR_STRING (c, (unsigned char *) p);
-	    else
-	      *p++ = c;
-	    c = READCHAR;
-	  }
-	while (c > 040
-	       && c != NO_BREAK_SPACE
-	       && (c >= 0200
-		   || strchr ("\"';()[]#`,", c) == NULL));
+	      if (multibyte)
+		p += CHAR_STRING (c, (unsigned char *) p);
+	      else
+		*p++ = c;
+	      c = READCHAR;
+	    }
+	  while (c > 040
+		 && c != NO_BREAK_SPACE
+		 && (c >= 0200
+		     || strchr ("\"';()[]#`,", c) == NULL));
 
-	*p = 0;
-	UNREAD (c);
+	  if (p == end)
+	    {
+	      ptrdiff_t offset = p - read_buffer;
+	      grow_read_buffer ();
+	      p = read_buffer + offset;
+	      end = read_buffer + read_buffer_size;
+	    }
+	  *p = 0;
+	  UNREAD (c);
+	}
 
 	if (!quoted && !uninterned_symbol)
 	  {
 	    Lisp_Object result = string_to_number (read_buffer, 10, 0);
 	    if (! NILP (result))
-	      return unbind_to (count, result);
+	      return result;
 	  }
         if (!quoted && multibyte)
           {
@@ -3498,48 +3480,25 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
           }
 	{
-	  Lisp_Object result;
+	  Lisp_Object name, result;
 	  ptrdiff_t nbytes = p - read_buffer;
 	  ptrdiff_t nchars
 	    = (multibyte
 	       ? multibyte_chars_in_text ((unsigned char *) read_buffer,
 					  nbytes)
 	       : nbytes);
-
-	  if (uninterned_symbol)
-	    {
-	      Lisp_Object name
-		= ((! NILP (Vpurify_flag)
-		    ? make_pure_string : make_specified_string)
-		   (read_buffer, nchars, nbytes, multibyte));
-	      result = Fmake_symbol (name);
-	    }
-	  else
-	    {
-	      /* Don't create the string object for the name unless
-		 we're going to retain it in a new symbol.
-
-		 Like intern_1 but supports multibyte names.  */
-	      Lisp_Object obarray = check_obarray (Vobarray);
-	      Lisp_Object tem = oblookup (obarray, read_buffer,
-					  nchars, nbytes);
-
-	      if (SYMBOLP (tem))
-		result = tem;
-	      else
-		{
-		  Lisp_Object name
-		    = make_specified_string (read_buffer, nchars, nbytes,
-					     multibyte);
-		  result = intern_driver (name, obarray, tem);
-		}
-	    }
-
+ 
+	  name = ((uninterned_symbol && ! NILP (Vpurify_flag)
+		   ? make_pure_string : make_specified_string)
+		  (read_buffer, nchars, nbytes, multibyte));
+	  result = (uninterned_symbol ? Fmake_symbol (name)
+		    : Fintern (name, Qnil));
+ 
 	  if (EQ (Vread_with_symbol_positions, Qt)
 	      || EQ (Vread_with_symbol_positions, readcharfun))
 	    Vread_symbol_positions_list
 	      = Fcons (Fcons (result, make_number (start_position)),
 		       Vread_symbol_positions_list);
-	  return unbind_to (count, result);
+	  return result;
 	}
       }
     }
@@ -4320,7 +4279,12 @@ OBARRAY defaults to the value of `obarray'.  */)
 void
 init_obarray (void)
 {
-  Vobarray = Fmake_vector (make_number (OBARRAY_SIZE), make_number (0));
+  Lisp_Object oblength;
+  ptrdiff_t size = 100 + MAX_MULTIBYTE_LENGTH;
+ 
+  XSETFASTINT (oblength, OBARRAY_SIZE);
+ 
+  Vobarray = Fmake_vector (oblength, make_number (0));
   initial_obarray = Vobarray;
   staticpro (&initial_obarray);
 
@@ -4343,6 +4307,9 @@ init_obarray (void)
   Vpurify_flag = Qt;
 
   DEFSYM (Qvariable_documentation, "variable-documentation");
+
+  read_buffer = xmalloc (size);
+  read_buffer_size = size;
 }
 \f
 void

bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.

[Noam Postavsky]

Vasilij Schneidermann <mail@vasilij.de> writes:

> I expect plenty more people to run into this as undo-tree is a
> dependency for Evil and therefore keeps Emacs users from properly
> booting their editor

Only if they've set undo-tree-auto-save-history, right?  (which, for all
I know, might be a really popular setting)

bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.

[Vasilij Schneidermann]

> Only if they've set undo-tree-auto-save-history, right?  (which, for all
> I know, might be a really popular setting)

Hm, you're right.  This bug effectively neuters that feature, if I
disable it I can load up Emacs just fine.  The setting isn't nearly as
popular as Evil.

I can only wonder how many more packages rely on reading deeply nested
trees...

bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.

[Noam Postavsky]

Vasilij Schneidermann <mail@vasilij.de> writes:

> I've run into this issue today when loading up undo-tree on Emacs 26.1.

I'm wondering if Paul's patch at https://debbugs.gnu.org/31995#21 (which
is in Emacs 26.2) effectively fixes this for practical use of undo-tree
history?  That is, we know stack overflow is still possible for a
sufficiently deep nested object, but that patch should make it possible
for Emacs to read much deeper objects without overflowing.

bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.

[Keith David Bershatsky]

I performed the following tests on OSX 10.6.8 (Snow Leopard):

SOURCE:  git clone -b emacs-26 git://git.sv.gnu.org/emacs.git

CONFIGURE:  CFLAGS='-Wall -O0 -g3' ./configure \
            --with-ns \
            --enable-checking='yes,glyphs' \
            --enable-check-lisp-object-type \
            --without-compress-install \
            --without-makeinfo \
            --with-gnutls=no \
            --with-mailutils \
            --without-makeinfo

LISP:

(require 'cl-lib)

(defun make-deep-object (depth)
    (let ((obj 1))
      (while (> (cl-decf depth) 0)
        (setq obj (vector (list obj))))
      obj))

;;; STACK OVERFLOW:  problem with `read'
;;; The bug was introduced on December 8, 2016 with
;;; commit:  f0a1e9ec3fba3d5bea5bd62f525dba3fb005d1b1
(let* ((print-circle t)
       (max-lisp-eval-depth most-positive-fixnum)
       (max-specpdl-size most-positive-fixnum)
       (deep-object (make-deep-object 6000))
       (string (prin1-to-string deep-object))
       (result (read string)))
  (when result
    (message "Bug #27779:  Success! (%d)" (length string))))

RESULT _WITH_ TERMINAL (ulimit -S -s unlimited):  "Bug #27779:  Success! (23997)"

RESULT _WITHOUT_ TERMINAL (ulimit -S -s unlimited):  "Re-entering top level after C stack overflow"

I use a custom fork of undo-tree with a previous version of the CL Lisp libraries and have not done any testing to see how Emacs 26.2 would fair with my custom fork, or with the stock version of undo-tree.  I have a list of known issues affecting my fork, including, but not limited to:  Emacs Bug #27571 and a need for the old CL Lisp libraries ....  I have not done any testing to see whether #27571 still exists.

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

> Date: [05-01-2019 19:30:17] <01 May 2019 22:30:17 -0400>
> From: Noam Postavsky <npostavs@gmail.com>
> To: Vasilij Schneidermann <mail@vasilij.de>
> Cc: Keith David Bershatsky <esq@lawlist.com>, Noam Postavsky <npostavs@users.sourceforge.net>,
> Paul Eggert <eggert@cs.ucla.edu>, 27779@debbugs.gnu.org
> Subject: Re: bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.
> 
> Vasilij Schneidermann <mail@vasilij.de> writes:
> 
> > I've run into this issue today when loading up undo-tree on Emacs 26.1.
> 
> I'm wondering if Paul's patch at https://debbugs.gnu.org/31995#21 (which
> is in Emacs 26.2) effectively fixes this for practical use of undo-tree
> history?  That is, we know stack overflow is still possible for a
> sufficiently deep nested object, but that patch should make it possible
> for Emacs to read much deeper objects without overflowing.

bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.

[Paul Eggert]

Keith David Bershatsky wrote:
> CONFIGURE:  CFLAGS='-Wall -O0 -g3' ./configure \
>              --with-ns \
>              --enable-checking='yes,glyphs' \
>              --enable-check-lisp-object-type \
>              --without-compress-install \
>              --without-makeinfo \
>              --with-gnutls=no \
>              --with-mailutils \
>              --without-makeinfo

What happens if you try it with -O2 rather than -O0?

Also, what happens if you try it without --enable-checking='yes,glyphs'?

I have a suspicion that the debugging options are blowing the stack.

bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.

[Keith David Bershatsky]

TWO (2) TESTS:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; TEST # 1.

SOURCE:  git clone -b emacs-26 git://git.sv.gnu.org/emacs.git

CONFIGURE:

CFLAGS='-Wall -O2 -g3' ./configure \
--with-ns \
--enable-checking='yes,glyphs' \
--enable-check-lisp-object-type \
--without-compress-install \
--without-makeinfo \
--with-gnutls=no \
--with-mailutils \
--without-makeinfo

LISP:

(require 'cl-lib)

(defun make-deep-object (depth)
    (let ((obj 1))
      (while (> (cl-decf depth) 0)
        (setq obj (vector (list obj))))
      obj))

;;; STACK OVERFLOW:  problem with `read'
;;; The bug was introduced on December 8, 2016 with
;;; commit:  f0a1e9ec3fba3d5bea5bd62f525dba3fb005d1b1
(let* ((print-circle t)
       (max-lisp-eval-depth most-positive-fixnum)
       (max-specpdl-size most-positive-fixnum)
       (deep-object (make-deep-object 6000))
       (string (prin1-to-string deep-object))
       (result (read string)))
  (when result
    (message "Bug #27779:  Success! (%d)" (length string))))

TEST:  Lanched without the terminal by opening Emacs.app.

RESULT:  Bug #27779:  Success! (23997)

No need to set the terminal with "ulimit -S -s unlimited" before launching Emacs from the terminal.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; TEST # 2.

SOURCE:  git clone -b emacs-26 git://git.sv.gnu.org/emacs.git

CONFIGURE:

CFLAGS='-Wall -O0 -g3' ./configure \
--with-ns \
--enable-check-lisp-object-type \
--without-compress-install \
--without-makeinfo \
--with-gnutls=no \
--with-mailutils \
--without-makeinfo

LISP:

(require 'cl-lib)

(defun make-deep-object (depth)
    (let ((obj 1))
      (while (> (cl-decf depth) 0)
        (setq obj (vector (list obj))))
      obj))

;;; STACK OVERFLOW:  problem with `read'
;;; The bug was introduced on December 8, 2016 with
;;; commit:  f0a1e9ec3fba3d5bea5bd62f525dba3fb005d1b1
(let* ((print-circle t)
       (max-lisp-eval-depth most-positive-fixnum)
       (max-specpdl-size most-positive-fixnum)
       (deep-object (make-deep-object 6000))
       (string (prin1-to-string deep-object))
       (result (read string)))
  (when result
    (message "Bug #27779:  Success! (%d)" (length string))))

TEST:  Lanched without the terminal by opening Emacs.app.

RESULT:  Re-entering top level after C stack overflow

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

> Date: [05-02-2019 00:26:45] <2 May 2019 00:26:45 -0700>
> From: Paul Eggert <eggert@cs.ucla.edu>
> To: Keith David Bershatsky <esq@lawlist.com>, Noam Postavsky <npostavs@gmail.com>
> Cc: 27779@debbugs.gnu.org, mail@vasilij.de
> Subject: Re: bug#27779: #27779; C stack overflow from `read' on deeply nested lisp object.
> 
> Keith David Bershatsky wrote:
> > CONFIGURE:  CFLAGS='-Wall -O0 -g3' ./configure \
> >              --with-ns \
> >              --enable-checking='yes,glyphs' \
> >              --enable-check-lisp-object-type \
> >              --without-compress-install \
> >              --without-makeinfo \
> >              --with-gnutls=no \
> >              --with-mailutils \
> >              --without-makeinfo
> 
> What happens if you try it with -O2 rather than -O0?
> 
> Also, what happens if you try it without --enable-checking='yes,glyphs'?
> 
> I have a suspicion that the debugging options are blowing the stack.

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[Lars Ingebrigtsen]

Keith David Bershatsky <esq@lawlist.com> writes:

> TEST:  Lanched without the terminal by opening Emacs.app.
>
> RESULT:  Re-entering top level after C stack overflow

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

I've tried reproducing this issue, but with the test cases in the final
message here, I get "Success" in the current Emacs.

Are you still seeing this issue in recent Emacs versions?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[Keith David Bershatsky]

As to the most recent versions of Emacs 28 and the Master Branch, built on El Capitan, the issue is not present when using the test case:

(require 'cl-lib)

(defun make-deep-object (depth)
    (let ((obj 1))
      (while (> (cl-decf depth) 0)
        (setq obj (vector (list obj))))
      obj))

(let* ((print-circle t)
       (max-lisp-eval-depth most-positive-fixnum)
       (max-specpdl-size most-positive-fixnum)
       (deep-object (make-deep-object 4600))
       (string (prin1-to-string deep-object))
       (result (read string)))
  (when result
    (message "Success!")))

I do not have any computers running a higher version of the MacOS operating system.

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

> Date: [04-21-2022 06:33:29] <21 Apr 2022 15:33:29 +0200>
> From: Lars Ingebrigtsen <larsi@gnus.org>
> To: Keith David Bershatsky <esq@lawlist.com>
> Cc: Paul Eggert <eggert@cs.ucla.edu>, mail@vasilij.de, npostavs@gmail.com, 27779@debbugs.gnu.org
> Subject: Re: bug#27779: 26.0.50:  read -- Re-entering top level after C stack overflow
> 
> Keith David Bershatsky <esq@lawlist.com> writes:
> 
> > TEST:  Lanched without the terminal by opening Emacs.app.
> >
> > RESULT:  Re-entering top level after C stack overflow
> 
> (I'm going through old bug reports that unfortunately weren't resolved
> at the time.)
> 
> I've tried reproducing this issue, but with the test cases in the final
> message here, I get "Success" in the current Emacs.
> 
> Are you still seeing this issue in recent Emacs versions?

bug#27779: 26.0.50: read -- Re-entering top level after C stack overflow

[Paul Eggert]

On 4/21/22 11:56, Keith David Bershatsky wrote:
> As to the most recent versions of Emacs 28 and the Master Branch, built on El Capitan, the issue is not present when using the test case:

Thanks for checking. I think this bug was addressed in 2018, here:

https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=b38b91a83491b6812e8267d0247355f0e8e3e189

so I am closing the bug report.