bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello, Emacs.

When parse-partial-sexp finishes a parse, it fails to record whether or
not its end point is just after the first character of a two character
comment starter or ender.  When the resulting state is used as an
argument to resume the parse, p-p-s will be unaware that the comment has
started or ended and produce false results.

Proposed solution: Add an extra element to the parser state, recording the
syntax of the last character passed over before the end of the parse.
This would be used by parse-partial-sexp to initialise its parse.

Also: the existing element 9 (the list of currently open parens) and the
new element should be explicitly documented in the Elisp manual, together
with a statement that there may be further elements in the parse state
used internally by parse-partial-sexp (for future expansion).

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Andreas Röhler]

On 15.03.2016 10:13, Alan Mackenzie wrote:
> Hello, Emacs.
>
> When parse-partial-sexp finishes a parse, it fails to record whether or
> not its end point is just after the first character of a two character
> comment starter or ender.  When the resulting state is used as an
> argument to resume the parse, p-p-s will be unaware that the comment has
> started or ended and produce false results.
>
> Proposed solution: Add an extra element to the parser state, recording the
> syntax of the last character passed over before the end of the parse.
> This would be used by parse-partial-sexp to initialise its parse.
>
> Also: the existing element 9 (the list of currently open parens) and the
> new element should be explicitly documented in the Elisp manual, together
> with a statement that there may be further elements in the parse state
> used internally by parse-partial-sexp (for future expansion).
>

Hi Alan,

a comment start might be composed not just by two characters, but by 
three or more. What then?

Cheers,

Andreas

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello, Andreas.

On Tue, Mar 15, 2016 at 10:35:08AM +0100, Andreas Röhler wrote:


> On 15.03.2016 10:13, Alan Mackenzie wrote:
> > Hello, Emacs.

> > When parse-partial-sexp finishes a parse, it fails to record whether or
> > not its end point is just after the first character of a two character
> > comment starter or ender.  When the resulting state is used as an
> > argument to resume the parse, p-p-s will be unaware that the comment has
> > started or ended and produce false results.

> > Proposed solution: Add an extra element to the parser state, recording the
> > syntax of the last character passed over before the end of the parse.
> > This would be used by parse-partial-sexp to initialise its parse.

> > Also: the existing element 9 (the list of currently open parens) and the
> > new element should be explicitly documented in the Elisp manual, together
> > with a statement that there may be further elements in the parse state
> > used internally by parse-partial-sexp (for future expansion).


> a comment start might be composed not just by two characters, but by 
> three or more. What then?

We'd have to start thinking about extending parse-partial-sexp, or
invent some workaround.  Maybe.  There must be some languages (?html)
where this is the case.  What is done in these?

> Cheers,

> Andreas

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Andreas Röhler]

On 15.03.2016 11:15, Alan Mackenzie wrote:
> Hello, Andreas.
>
> On Tue, Mar 15, 2016 at 10:35:08AM +0100, Andreas Röhler wrote:
>
>
>> On 15.03.2016 10:13, Alan Mackenzie wrote:
>>> Hello, Emacs.
>>> When parse-partial-sexp finishes a parse, it fails to record whether or
>>> not its end point is just after the first character of a two character
>>> comment starter or ender.  When the resulting state is used as an
>>> argument to resume the parse, p-p-s will be unaware that the comment has
>>> started or ended and produce false results.
>>> Proposed solution: Add an extra element to the parser state, recording the
>>> syntax of the last character passed over before the end of the parse.
>>> This would be used by parse-partial-sexp to initialise its parse.
>>> Also: the existing element 9 (the list of currently open parens) and the
>>> new element should be explicitly documented in the Elisp manual, together
>>> with a statement that there may be further elements in the parse state
>>> used internally by parse-partial-sexp (for future expansion).
>
>> a comment start might be composed not just by two characters, but by
>> three or more. What then?
> We'd have to start thinking about extending parse-partial-sexp, or
> invent some workaround.  Maybe.  There must be some languages (?html)
> where this is the case.  What is done in these?

May you send me this (or more) example use-cases? Couldn't find the one 
already given, sorry.

Addressed this issue in my generic beg-end.el

https://github.com/andreas-roehler/werkstatt/blob/master/subroutines/beg-end.el

In case beg-end forms used a start-string, look if the char-at-point 
would match this string.
Then look if the char-before is before in string, etc.

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Stefan Monnier]

> Proposed solution: Add an extra element to the parser state, recording the
> syntax of the last character passed over before the end of the parse.
> This would be used by parse-partial-sexp to initialise its parse.

Another option is to record "the start of current element" (in case we
were in the middle of an element).  This could potentially reuse (nth
5 ppss) by generalizing it, or it could use a new entry.

The choice probably doesn't matter much and will probably be more
a question of "what's easier to implement".

> Also: the existing element 9 (the list of currently open parens) and the
> new element should be explicitly documented in the Elisp manual, together
> with a statement that there may be further elements in the parse state
> used internally by parse-partial-sexp (for future expansion).

Indeed.

Andreas Röhler added:
> a comment start might be composed not just by two characters, but by three
> or more. What then?

Andreas, I suggest that you go back and take a closer look at
parse-partial-sexp, syntax-ppss, and syntax-tables in general because
lately you've made several comments like the one here which show you're
just not familiar with the topic at all.  Syntax tables do not support
comment markers longer than 2 characters (currently).  Emacs supports
those via the `syntax-table' text-property only (which typically marks
the first char of each "long comment starter" as being "the comment
starter").


        Stefan

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

On Thu, Mar 17, 2016 at 08:58:27AM -0400, Stefan Monnier wrote:
> > Proposed solution: Add an extra element to the parser state, recording the
> > syntax of the last character passed over before the end of the parse.
> > This would be used by parse-partial-sexp to initialise its parse.

> Another option is to record "the start of current element" (in case we
> were in the middle of an element).  This could potentially reuse (nth
> 5 ppss) by generalizing it, or it could use a new entry.

> The choice probably doesn't matter much and will probably be more
> a question of "what's easier to implement".

> > Also: the existing element 9 (the list of currently open parens) and the
> > new element should be explicitly documented in the Elisp manual, together
> > with a statement that there may be further elements in the parse state
> > used internally by parse-partial-sexp (for future expansion).

> Indeed.

OK, I've got a patch ready.  It's bigger than anticipated, purely
because it also does some refactoring.  It actually adds two elements to
the parser state, and I believe that makes the parser state complete.

Here's the patch:


Enhance parse-partial-sexp correctly to handle two character commit delimiters

Do this by adding two new fields to the parser state: the syntax of the last
character scanned, and the last end of comment scanned.  This should make the
parser state complete.
Also document element 9 of the parser state.  Also refactor the code a bit.

* src/syntax.c (struct lisp_parse_state): Add two new fields.
(internalize_parse_state): New function, extracted from scan_sexps_forward.
(back_comment): Call internalize_parse_state.
(forw_comment): Return the syntax of the last character scanned to the caller.
(Fforward_comment, scan_lists): New dummy variables, passed to forw_comment.
(scan_sexps_forward): Remove a redundant state parameter.  Access all `state'
information via the address parameter `state'.  Remove the code which converts
from external to internal form of `state'.  Access buffer contents only from
`from' onwards.  Reformulate code at the top of the main loop correctly to
recognize comment openers when starting in the middle of one.  Call
forw_comment with extra argument (for return of final syntax value).
(Fparse_partial_sexp): Document elements 9, 10, 11 of the parser state in the
doc string.  Clarify the doc string in general.  Call
internalize_parse_state.  Take account of the new elements when consing up the
output parser state.

* doc/lispref/syntax.texi: (Parser State): Document element 9 and the new
elements 10 and 11.  Minor wording corrections (remove reference to "trivial
cases").
(Low Level Parsing): Minor corrections


diff --git a/doc/lispref/syntax.texi b/doc/lispref/syntax.texi
index d5a7eba..67a00d7 100644
--- a/doc/lispref/syntax.texi
+++ b/doc/lispref/syntax.texi
@@ -791,10 +791,10 @@ Parser State
 @subsection Parser State
 @cindex parser state
 
-  A @dfn{parser state} is a list of ten elements describing the state
-of the syntactic parser, after it parses the text between a specified
-starting point and a specified end point in the buffer.  Parsing
-functions such as @code{syntax-ppss}
+  A @dfn{parser state} is a list of (currently) twelve elements
+describing the state of the syntactic parser, after it parses the text
+between a specified starting point and a specified end point in the
+buffer.  Parsing functions such as @code{syntax-ppss}
 @ifnottex
 (@pxref{Position Parse})
 @end ifnottex
@@ -851,15 +851,21 @@ Parser State
 this element is @code{nil}.
 
 @item
-Internal data for continuing the parsing.  The meaning of this
-data is subject to change; it is used if you pass this list
-as the @var{state} argument to another call.
+The list of the positions of the currently open parentheses, starting
+with the outermost.
+
+@item
+The @var{syntax-code} (@pxref{Syntax Table Internals}) of the last
+buffer position scanned, or @code{nil} if no scanning has happened.
+
+@item
+The position after the previous end of comment, or @code{nil} if the
+scanning has not passed a comment end.
 @end enumerate
 
   Elements 1, 2, and 6 are ignored in a state which you pass as an
-argument to continue parsing, and elements 8 and 9 are used only in
-trivial cases.  Those elements are mainly used internally by the
-parser code.
+argument to continue parsing.  Elements 9 to 11 are mainly used
+internally by the parser code.
 
   One additional piece of useful information is available from a
 parser state using this function:
@@ -898,11 +904,11 @@ Low-Level Parsing
 
 If the fourth argument @var{stop-before} is non-@code{nil}, parsing
 stops when it comes to any character that starts a sexp.  If
-@var{stop-comment} is non-@code{nil}, parsing stops when it comes to the
-start of an unnested comment.  If @var{stop-comment} is the symbol
+@var{stop-comment} is non-@code{nil}, parsing stops after the start of
+an unnested comment.  If @var{stop-comment} is the symbol
 @code{syntax-table}, parsing stops after the start of an unnested
-comment or a string, or the end of an unnested comment or a string,
-whichever comes first.
+comment or a string, or after the end of an unnested comment or a
+string, whichever comes first.
 
 If @var{state} is @code{nil}, @var{start} is assumed to be at the top
 level of parenthesis structure, such as the beginning of a function
diff --git a/src/syntax.c b/src/syntax.c
index 249d0d5..11b1ff0 100644
--- a/src/syntax.c
+++ b/src/syntax.c
@@ -153,6 +153,9 @@ struct lisp_parse_state
     ptrdiff_t comstr_start;  /* Position of last comment/string starter.  */
     Lisp_Object levelstarts; /* Char numbers of starts-of-expression
 				of levels (starting from outermost).  */
+    int prev_syntax; /* Syntax of previous character scanned, or Smax. */
+    ptrdiff_t prev_comment_end; /* Position after end of last closed
+                                   comment, or -1. */
   };
 \f
 /* These variables are a cache for finding the start of a defun.
@@ -176,7 +179,8 @@ static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
 static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool);
 static void scan_sexps_forward (struct lisp_parse_state *,
                                 ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
-                                bool, Lisp_Object, int);
+                                bool, int);
+static void internalize_parse_state (Lisp_Object, struct lisp_parse_state *);
 static bool in_classes (int, Lisp_Object);
 static void parse_sexp_propertize (ptrdiff_t charpos);
 
@@ -911,10 +915,11 @@ back_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
 	}
       do
 	{
+          internalize_parse_state (Qnil, &state);
 	  scan_sexps_forward (&state,
 			      defun_start, defun_start_byte,
 			      comment_end, TYPE_MINIMUM (EMACS_INT),
-			      0, Qnil, 0);
+			      0, 0);
 	  defun_start = comment_end;
 	  if (!adjusted)
 	    {
@@ -2314,7 +2319,9 @@ in_classes (int c, Lisp_Object iso_classes)
    into *CHARPOS_PTR and the corresponding bytepos into *BYTEPOS_PTR.
    Else, return false and store the charpos STOP into *CHARPOS_PTR, the
    corresponding bytepos into *BYTEPOS_PTR and the current nesting
-   (as defined for state.incomment) in *INCOMMENT_PTR.
+   (as defined for state->incomment) in *INCOMMENT_PTR.  The
+   SYNTAX_WITH_FLAGS of the last character scanned in the comment is
+   stored into *last_syntax_ptr.
 
    The comment end is the last character of the comment rather than the
    character just after the comment.
@@ -2326,7 +2333,7 @@ static bool
 forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
 	      EMACS_INT nesting, int style, int prev_syntax,
 	      ptrdiff_t *charpos_ptr, ptrdiff_t *bytepos_ptr,
-	      EMACS_INT *incomment_ptr)
+	      EMACS_INT *incomment_ptr, int *last_syntax_ptr)
 {
   register int c, c1;
   register enum syntaxcode code;
@@ -2346,6 +2353,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
 	  *incomment_ptr = nesting;
 	  *charpos_ptr = from;
 	  *bytepos_ptr = from_byte;
+          *last_syntax_ptr = syntax;
 	  return 0;
 	}
       c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
@@ -2415,6 +2423,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
     }
   *charpos_ptr = from;
   *bytepos_ptr = from_byte;
+  *last_syntax_ptr = syntax;
   return 1;
 }
 
@@ -2436,6 +2445,7 @@ between them, return t; otherwise return nil.  */)
   EMACS_INT count1;
   ptrdiff_t out_charpos, out_bytepos;
   EMACS_INT dummy;
+  int dummy2;
 
   CHECK_NUMBER (count);
   count1 = XINT (count);
@@ -2499,7 +2509,7 @@ between them, return t; otherwise return nil.  */)
 	}
       /* We're at the start of a comment.  */
       found = forw_comment (from, from_byte, stop, comnested, comstyle, 0,
-			    &out_charpos, &out_bytepos, &dummy);
+			    &out_charpos, &out_bytepos, &dummy, &dummy2);
       from = out_charpos; from_byte = out_bytepos;
       if (!found)
 	{
@@ -2659,6 +2669,7 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
   ptrdiff_t from_byte;
   ptrdiff_t out_bytepos, out_charpos;
   EMACS_INT dummy;
+  int dummy2;
   bool multibyte_symbol_p = sexpflag && multibyte_syntax_as_symbol;
 
   if (depth > 0) min_depth = 0;
@@ -2755,7 +2766,8 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
 	      UPDATE_SYNTAX_TABLE_FORWARD (from);
 	      found = forw_comment (from, from_byte, stop,
 				    comnested, comstyle, 0,
-				    &out_charpos, &out_bytepos, &dummy);
+				    &out_charpos, &out_bytepos, &dummy,
+                                    &dummy2);
 	      from = out_charpos, from_byte = out_bytepos;
 	      if (!found)
 		{
@@ -3119,7 +3131,7 @@ the prefix syntax flag (p).  */)
 }
 \f
 /* Parse forward from FROM / FROM_BYTE to END,
-   assuming that FROM has state OLDSTATE (nil means FROM is start of function),
+   assuming that FROM has state STATE (nil means FROM is start of function),
    and return a description of the state of the parse at END.
    If STOPBEFORE, stop at the start of an atom.
    If COMMENTSTOP is 1, stop at the start of a comment.
@@ -3127,12 +3139,11 @@ the prefix syntax flag (p).  */)
    after the beginning of a string, or after the end of a string.  */
 
 static void
-scan_sexps_forward (struct lisp_parse_state *stateptr,
+scan_sexps_forward (struct lisp_parse_state *state,
 		    ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t end,
 		    EMACS_INT targetdepth, bool stopbefore,
-		    Lisp_Object oldstate, int commentstop)
+		    int commentstop)
 {
-  struct lisp_parse_state state;
   enum syntaxcode code;
   int c1;
   bool comnested;
@@ -3148,7 +3159,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
   Lisp_Object tem;
   ptrdiff_t prev_from;		/* Keep one character before FROM.  */
   ptrdiff_t prev_from_byte;
-  int prev_from_syntax;
+  int prev_from_syntax, prev_prev_from_syntax;
   bool boundary_stop = commentstop == -1;
   bool nofence;
   bool found;
@@ -3165,6 +3176,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
 do { prev_from = from;				\
      prev_from_byte = from_byte; 		\
      temp = FETCH_CHAR_AS_MULTIBYTE (prev_from_byte);	\
+     prev_prev_from_syntax = prev_from_syntax;  \
      prev_from_syntax = SYNTAX_WITH_FLAGS (temp); \
      INC_BOTH (from, from_byte);		\
      if (from < end)				\
@@ -3174,88 +3186,38 @@ do { prev_from = from;				\
   immediate_quit = 1;
   QUIT;
 
-  if (NILP (oldstate))
-    {
-      depth = 0;
-      state.instring = -1;
-      state.incomment = 0;
-      state.comstyle = 0;	/* comment style a by default.  */
-      state.comstr_start = -1;	/* no comment/string seen.  */
-    }
-  else
-    {
-      tem = Fcar (oldstate);
-      if (!NILP (tem))
-	depth = XINT (tem);
-      else
-	depth = 0;
-
-      oldstate = Fcdr (oldstate);
-      oldstate = Fcdr (oldstate);
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      /* Check whether we are inside string_fence-style string: */
-      state.instring = (!NILP (tem)
-			? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
-			: -1);
-
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      state.incomment = (!NILP (tem)
-			 ? (INTEGERP (tem) ? XINT (tem) : -1)
-			 : 0);
-
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      start_quoted = !NILP (tem);
+  depth = state->depth;
+  start_quoted = state->quoted;
+  prev_prev_from_syntax = Smax;
+  prev_from_syntax = state->prev_syntax;
 
-      /* if the eighth element of the list is nil, we are in comment
-	 style a.  If it is non-nil, we are in comment style b */
-      oldstate = Fcdr (oldstate);
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      state.comstyle = (NILP (tem)
-			? 0
-			: (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
-			   ? XINT (tem)
-			   : ST_COMMENT_STYLE));
-
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      state.comstr_start =
-	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      while (!NILP (tem))		/* >= second enclosing sexps.  */
-	{
-	  Lisp_Object temhd = Fcar (tem);
-	  if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
-	    curlevel->last = XINT (temhd);
-	  if (++curlevel == endlevel)
-	    curlevel--; /* error ("Nesting too deep for parser"); */
-	  curlevel->prev = -1;
-	  curlevel->last = -1;
-	  tem = Fcdr (tem);
-	}
+  tem = state->levelstarts;
+  while (!NILP (tem))		/* >= second enclosing sexps.  */
+    {
+      Lisp_Object temhd = Fcar (tem);
+      if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
+        curlevel->last = XINT (temhd);
+      if (++curlevel == endlevel)
+        curlevel--; /* error ("Nesting too deep for parser"); */
+      curlevel->prev = -1;
+      curlevel->last = -1;
+      tem = Fcdr (tem);
     }
-  state.quoted = 0;
-  mindepth = depth;
-
   curlevel->prev = -1;
   curlevel->last = -1;
 
-  SETUP_SYNTAX_TABLE (prev_from, 1);
-  temp = FETCH_CHAR (prev_from_byte);
-  prev_from_syntax = SYNTAX_WITH_FLAGS (temp);
-  UPDATE_SYNTAX_TABLE_FORWARD (from);
+  state->quoted = 0;
+  mindepth = depth;
+
+  SETUP_SYNTAX_TABLE (from, 1);
 
   /* Enter the loop at a place appropriate for initial state.  */
 
-  if (state.incomment)
+  if (state->incomment)
     goto startincomment;
-  if (state.instring >= 0)
+  if (state->instring >= 0)
     {
-      nofence = state.instring != ST_STRING_STYLE;
+      nofence = state->instring != ST_STRING_STYLE;
       if (start_quoted)
 	goto startquotedinstring;
       goto startinstring;
@@ -3266,10 +3228,10 @@ do { prev_from = from;				\
   while (from < end)
     {
       int syntax;
-      INC_FROM;
-      code = prev_from_syntax & 0xff;
 
       if (from < end
+          && (state->prev_comment_end == -1
+              || prev_from >= state->prev_comment_end)
 	  && SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
 	  && (c1 = FETCH_CHAR (from_byte),
 	      syntax = SYNTAX_WITH_FLAGS (c1),
@@ -3280,32 +3242,37 @@ do { prev_from = from;				\
 	  /* Record the comment style we have entered so that only
 	     the comment-end sequence of the same style actually
 	     terminates the comment section.  */
-	  state.comstyle
+	  state->comstyle
 	    = SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax);
 	  comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax)
 		       | SYNTAX_FLAGS_COMMENT_NESTED (syntax));
-	  state.incomment = comnested ? 1 : -1;
-	  state.comstr_start = prev_from;
+	  state->incomment = comnested ? 1 : -1;
+	  state->comstr_start = prev_from;
 	  INC_FROM;
 	  code = Scomment;
 	}
-      else if (code == Scomment_fence)
-	{
-	  /* Record the comment style we have entered so that only
-	     the comment-end sequence of the same style actually
-	     terminates the comment section.  */
-	  state.comstyle = ST_COMMENT_STYLE;
-	  state.incomment = -1;
-	  state.comstr_start = prev_from;
-	  code = Scomment;
-	}
-      else if (code == Scomment)
-	{
-	  state.comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
-	  state.incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
-			     1 : -1);
-	  state.comstr_start = prev_from;
-	}
+      else
+        {
+          INC_FROM;
+          code = prev_from_syntax & 0xff;
+          if (code == Scomment_fence)
+            {
+              /* Record the comment style we have entered so that only
+                 the comment-end sequence of the same style actually
+                 terminates the comment section.  */
+              state->comstyle = ST_COMMENT_STYLE;
+              state->incomment = -1;
+              state->comstr_start = prev_from;
+              code = Scomment;
+            }
+          else if (code == Scomment)
+            {
+              state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
+              state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
+                                 1 : -1);
+              state->comstr_start = prev_from;
+            }
+        }
 
       if (SYNTAX_FLAGS_PREFIX (prev_from_syntax))
 	continue;
@@ -3357,18 +3324,21 @@ do { prev_from = from;				\
 	     middle of it.  We don't want to do that if we're just at the
 	     beginning of the comment (think of (*) ... (*)).  */
 	  found = forw_comment (from, from_byte, end,
-				state.incomment, state.comstyle,
-				(from == BEGV || from < state.comstr_start + 3)
+				state->incomment, state->comstyle,
+				(from == BEGV || from < state->comstr_start + 3)
 				? 0 : prev_from_syntax,
-				&out_charpos, &out_bytepos, &state.incomment);
+				&out_charpos, &out_bytepos, &state->incomment,
+                                &prev_from_syntax);
 	  from = out_charpos; from_byte = out_bytepos;
-	  /* Beware!  prev_from and friends are invalid now.
-	     Luckily, the `done' doesn't use them and the INC_FROM
-	     sets them to a sane value without looking at them. */
+	  /* Beware!  prev_from and friends (except prev_from_syntax)
+	     are invalid now.  Luckily, the `done' doesn't use them
+	     and the INC_FROM sets them to a sane value without
+	     looking at them. */
 	  if (!found) goto done;
 	  INC_FROM;
-	  state.incomment = 0;
-	  state.comstyle = 0;	/* reset the comment style */
+	  state->incomment = 0;
+	  state->comstyle = 0;	/* reset the comment style */
+          state->prev_comment_end = from;
 	  if (boundary_stop) goto done;
 	  break;
 
@@ -3396,16 +3366,16 @@ do { prev_from = from;				\
 
 	case Sstring:
 	case Sstring_fence:
-	  state.comstr_start = from - 1;
+	  state->comstr_start = from - 1;
 	  if (stopbefore) goto stop;  /* this arg means stop at sexp start */
 	  curlevel->last = prev_from;
-	  state.instring = (code == Sstring
+	  state->instring = (code == Sstring
 			    ? (FETCH_CHAR_AS_MULTIBYTE (prev_from_byte))
 			    : ST_STRING_STYLE);
 	  if (boundary_stop) goto done;
 	startinstring:
 	  {
-	    nofence = state.instring != ST_STRING_STYLE;
+	    nofence = state->instring != ST_STRING_STYLE;
 
 	    while (1)
 	      {
@@ -3419,7 +3389,7 @@ do { prev_from = from;				\
 		/* Check C_CODE here so that if the char has
 		   a syntax-table property which says it is NOT
 		   a string character, it does not end the string.  */
-		if (nofence && c == state.instring && c_code == Sstring)
+		if (nofence && c == state->instring && c_code == Sstring)
 		  break;
 
 		switch (c_code)
@@ -3442,7 +3412,7 @@ do { prev_from = from;				\
 	      }
 	  }
 	string_end:
-	  state.instring = -1;
+	  state->instring = -1;
 	  curlevel->prev = curlevel->last;
 	  INC_FROM;
 	  if (boundary_stop) goto done;
@@ -3461,25 +3431,99 @@ do { prev_from = from;				\
  stop:   /* Here if stopping before start of sexp. */
   from = prev_from;    /* We have just fetched the char that starts it; */
   from_byte = prev_from_byte;
+  prev_from_syntax = prev_prev_from_syntax;
   goto done; /* but return the position before it. */
 
  endquoted:
-  state.quoted = 1;
+  state->quoted = 1;
  done:
-  state.depth = depth;
-  state.mindepth = mindepth;
-  state.thislevelstart = curlevel->prev;
-  state.prevlevelstart
+  state->depth = depth;
+  state->mindepth = mindepth;
+  state->thislevelstart = curlevel->prev;
+  state->prevlevelstart
     = (curlevel == levelstart) ? -1 : (curlevel - 1)->last;
-  state.location = from;
-  state.location_byte = from_byte;
-  state.levelstarts = Qnil;
+  state->location = from;
+  state->location_byte = from_byte;
+  state->levelstarts = Qnil;
   while (curlevel > levelstart)
-    state.levelstarts = Fcons (make_number ((--curlevel)->last),
-			       state.levelstarts);
+    state->levelstarts = Fcons (make_number ((--curlevel)->last),
+                                state->levelstarts);
+  state->prev_syntax = prev_from_syntax;
   immediate_quit = 0;
+}
+
+/* Convert a (lisp) parse state to the internal form used in
+   scan_sexps_forward.  */
+static void
+internalize_parse_state (Lisp_Object external, struct lisp_parse_state *state)
+{
+  Lisp_Object tem;
+
+  if (NILP (external))
+    {
+      state->depth = 0;
+      state->instring = -1;
+      state->incomment = 0;
+      state->quoted = 0;
+      state->comstyle = 0;	/* comment style a by default.  */
+      state->comstr_start = -1;	/* no comment/string seen.  */
+      state->levelstarts = Qnil;
+      state->prev_syntax = Smax;
+      state->prev_comment_end = -1;
+    }
+  else
+    {
+      tem = Fcar (external);
+      if (!NILP (tem))
+	state->depth = XINT (tem);
+      else
+	state->depth = 0;
+
+      external = Fcdr (external);
+      external = Fcdr (external);
+      external = Fcdr (external);
+      tem = Fcar (external);
+      /* Check whether we are inside string_fence-style string: */
+      state->instring = (!NILP (tem)
+                         ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
+                         : -1);
+
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->incomment = (!NILP (tem)
+                          ? (INTEGERP (tem) ? XINT (tem) : -1)
+                          : 0);
 
-  *stateptr = state;
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->quoted = !NILP (tem);
+
+      /* if the eighth element of the list is nil, we are in comment
+	 style a.  If it is non-nil, we are in comment style b */
+      external = Fcdr (external);
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->comstyle = (NILP (tem)
+                         ? 0
+                         : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
+                            ? XINT (tem)
+                            : ST_COMMENT_STYLE));
+
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->comstr_start =
+	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->levelstarts = tem;
+
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->prev_syntax = NILP (tem) ? Smax : XINT (tem);
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->prev_comment_end = NILP (tem) ? -1 : XINT (tem);
+    }
 }
 
 DEFUN ("parse-partial-sexp", Fparse_partial_sexp, Sparse_partial_sexp, 2, 6, 0,
@@ -3488,6 +3532,7 @@ Parsing stops at TO or when certain criteria are met;
  point is set to where parsing stops.
 If fifth arg OLDSTATE is omitted or nil,
  parsing assumes that FROM is the beginning of a function.
+
 Value is a list of elements describing final state of parsing:
  0. depth in parens.
  1. character address of start of innermost containing list; nil if none.
@@ -3501,16 +3546,20 @@ Value is a list of elements describing final state of parsing:
  6. the minimum paren-depth encountered during this scan.
  7. style of comment, if any.
  8. character address of start of comment or string; nil if not in one.
- 9. Intermediate data for continuation of parsing (subject to change).
+ 9. List of positions of currently open parens, outermost first.
+10. Syntax of last character scanned, or nil if no scanning has happened.
+11. Position after end of previous comment scanned, or nil.
+12..... Possible further internal information used by `parse-partial-sexp'.
+
 If third arg TARGETDEPTH is non-nil, parsing stops if the depth
 in parentheses becomes equal to TARGETDEPTH.
-Fourth arg STOPBEFORE non-nil means stop when come to
+Fourth arg STOPBEFORE non-nil means stop when we come to
  any character that starts a sexp.
 Fifth arg OLDSTATE is a list like what this function returns.
  It is used to initialize the state of the parse.  Elements number 1, 2, 6
  are ignored.
-Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
- If it is symbol `syntax-table', stop after the start of a comment or a
+Sixth arg COMMENTSTOP non-nil means stop after the start of a comment.
+ If it is the symbol `syntax-table', stop after the start of a comment or a
  string, or after end of a comment or a string.  */)
   (Lisp_Object from, Lisp_Object to, Lisp_Object targetdepth,
    Lisp_Object stopbefore, Lisp_Object oldstate, Lisp_Object commentstop)
@@ -3527,15 +3576,17 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
     target = TYPE_MINIMUM (EMACS_INT);	/* We won't reach this depth.  */
 
   validate_region (&from, &to);
+  internalize_parse_state (oldstate, &state);
   scan_sexps_forward (&state, XINT (from), CHAR_TO_BYTE (XINT (from)),
 		      XINT (to),
-		      target, !NILP (stopbefore), oldstate,
+		      target, !NILP (stopbefore),
 		      (NILP (commentstop)
 		       ? 0 : (EQ (commentstop, Qsyntax_table) ? -1 : 1)));
 
   SET_PT_BOTH (state.location, state.location_byte);
 
-  return Fcons (make_number (state.depth),
+  return
+    Fcons (make_number (state.depth),
 	   Fcons (state.prevlevelstart < 0
 		  ? Qnil : make_number (state.prevlevelstart),
 	     Fcons (state.thislevelstart < 0
@@ -3553,11 +3604,18 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
 				  ? Qsyntax_table
 				  : make_number (state.comstyle))
 			       : Qnil),
-			      Fcons (((state.incomment
-				       || (state.instring >= 0))
-				      ? make_number (state.comstr_start)
-				      : Qnil),
-				     Fcons (state.levelstarts, Qnil))))))))));
+		         Fcons (((state.incomment
+                                  || (state.instring >= 0))
+                                 ? make_number (state.comstr_start)
+                                 : Qnil),
+			   Fcons (state.levelstarts,
+                             Fcons (state.prev_syntax == Smax
+                                    ? Qnil
+                                    : make_number (state.prev_syntax),
+                               Fcons (state.prev_comment_end == -1
+                                      ? Qnil
+                                      : make_number (state.prev_comment_end),
+                                      Qnil))))))))))));
 }
 \f
 void


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Stefan Monnier]

> Do this by adding two new fields to the parser state: the syntax of the last
> character scanned, and the last end of comment scanned.  This should make the
> parser state complete.

Thanks.  I like the "syntax of the last character scanned", but I don't
understand the reasoning behind "last end of comment scanned".  Why is
this relevant?  Is it in case the "last character scanned" was a "slash
ending a comment" so as to avoid treating "*/*" as both a comment closer and
a subsequent opener?

If so, I'm not sure I like it.  It sounds to me like there's a chance
it's actually incomplete (e.g. it doesn't address the similar problem
when the "last character scanned" is an end of a string which also
happens to be a valid first-char of a comment-starter), and even if it
isn't, it "feels ad-hoc" to me.

Would it be difficult to do the following instead:
- get rid of element 11.
- change element 10 so it's nil if the last char was an "end of
  something".  Another way to look at it, is that the element 10 should
  only be non-nil if the "next lexeme" might start on that
  previous character.

I also have a side question: IIUC your patch makes the 5th element
redundant (can be replaced with a test whether "last char syntax" was
"escape"), is that right?


        Stefan


> Also document element 9 of the parser state.  Also refactor the code a bit.

> * src/syntax.c (struct lisp_parse_state): Add two new fields.
> (internalize_parse_state): New function, extracted from scan_sexps_forward.
> (back_comment): Call internalize_parse_state.
> (forw_comment): Return the syntax of the last character scanned to the caller.
> (Fforward_comment, scan_lists): New dummy variables, passed to forw_comment.
> (scan_sexps_forward): Remove a redundant state parameter.  Access all `state'
> information via the address parameter `state'.  Remove the code which converts
> from external to internal form of `state'.  Access buffer contents only from
> `from' onwards.  Reformulate code at the top of the main loop correctly to
> recognize comment openers when starting in the middle of one.  Call
> forw_comment with extra argument (for return of final syntax value).
> (Fparse_partial_sexp): Document elements 9, 10, 11 of the parser state in the
> doc string.  Clarify the doc string in general.  Call
> internalize_parse_state.  Take account of the new elements when consing up the
> output parser state.

> * doc/lispref/syntax.texi: (Parser State): Document element 9 and the new
> elements 10 and 11.  Minor wording corrections (remove reference to "trivial
> cases").
> (Low Level Parsing): Minor corrections


> diff --git a/doc/lispref/syntax.texi b/doc/lispref/syntax.texi
> index d5a7eba..67a00d7 100644
> --- a/doc/lispref/syntax.texi
> +++ b/doc/lispref/syntax.texi
> @@ -791,10 +791,10 @@ Parser State
>  @subsection Parser State
>  @cindex parser state
 
> -  A @dfn{parser state} is a list of ten elements describing the state
> -of the syntactic parser, after it parses the text between a specified
> -starting point and a specified end point in the buffer.  Parsing
> -functions such as @code{syntax-ppss}
> +  A @dfn{parser state} is a list of (currently) twelve elements
> +describing the state of the syntactic parser, after it parses the text
> +between a specified starting point and a specified end point in the
> +buffer.  Parsing functions such as @code{syntax-ppss}
>  @ifnottex
>  (@pxref{Position Parse})
>  @end ifnottex
> @@ -851,15 +851,21 @@ Parser State
>  this element is @code{nil}.
 
>  @item
> -Internal data for continuing the parsing.  The meaning of this
> -data is subject to change; it is used if you pass this list
> -as the @var{state} argument to another call.
> +The list of the positions of the currently open parentheses, starting
> +with the outermost.
> +
> +@item
> +The @var{syntax-code} (@pxref{Syntax Table Internals}) of the last
> +buffer position scanned, or @code{nil} if no scanning has happened.
> +
> +@item
> +The position after the previous end of comment, or @code{nil} if the
> +scanning has not passed a comment end.
>  @end enumerate
 
>    Elements 1, 2, and 6 are ignored in a state which you pass as an
> -argument to continue parsing, and elements 8 and 9 are used only in
> -trivial cases.  Those elements are mainly used internally by the
> -parser code.
> +argument to continue parsing.  Elements 9 to 11 are mainly used
> +internally by the parser code.
 
>    One additional piece of useful information is available from a
>  parser state using this function:
> @@ -898,11 +904,11 @@ Low-Level Parsing
 
>  If the fourth argument @var{stop-before} is non-@code{nil}, parsing
>  stops when it comes to any character that starts a sexp.  If
> -@var{stop-comment} is non-@code{nil}, parsing stops when it comes to the
> -start of an unnested comment.  If @var{stop-comment} is the symbol
> +@var{stop-comment} is non-@code{nil}, parsing stops after the start of
> +an unnested comment.  If @var{stop-comment} is the symbol
>  @code{syntax-table}, parsing stops after the start of an unnested
> -comment or a string, or the end of an unnested comment or a string,
> -whichever comes first.
> +comment or a string, or after the end of an unnested comment or a
> +string, whichever comes first.
 
>  If @var{state} is @code{nil}, @var{start} is assumed to be at the top
>  level of parenthesis structure, such as the beginning of a function
> diff --git a/src/syntax.c b/src/syntax.c
> index 249d0d5..11b1ff0 100644
> --- a/src/syntax.c
> +++ b/src/syntax.c
> @@ -153,6 +153,9 @@ struct lisp_parse_state
>      ptrdiff_t comstr_start;  /* Position of last comment/string starter.  */
>      Lisp_Object levelstarts; /* Char numbers of starts-of-expression
>  				of levels (starting from outermost).  */
> +    int prev_syntax; /* Syntax of previous character scanned, or Smax. */
> +    ptrdiff_t prev_comment_end; /* Position after end of last closed
> +                                   comment, or -1. */
>    };
>  \f
>  /* These variables are a cache for finding the start of a defun.
> @@ -176,7 +179,8 @@ static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
>  static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool);
>  static void scan_sexps_forward (struct lisp_parse_state *,
>                                  ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
> -                                bool, Lisp_Object, int);
> +                                bool, int);
> +static void internalize_parse_state (Lisp_Object, struct lisp_parse_state *);
>  static bool in_classes (int, Lisp_Object);
>  static void parse_sexp_propertize (ptrdiff_t charpos);
 
> @@ -911,10 +915,11 @@ back_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	}
>        do
>  	{
> +          internalize_parse_state (Qnil, &state);
>  	  scan_sexps_forward (&state,
>  			      defun_start, defun_start_byte,
>  			      comment_end, TYPE_MINIMUM (EMACS_INT),
> -			      0, Qnil, 0);
> +			      0, 0);
>  	  defun_start = comment_end;
>  	  if (!adjusted)
>  	    {
> @@ -2314,7 +2319,9 @@ in_classes (int c, Lisp_Object iso_classes)
>     into *CHARPOS_PTR and the corresponding bytepos into *BYTEPOS_PTR.
>     Else, return false and store the charpos STOP into *CHARPOS_PTR, the
>     corresponding bytepos into *BYTEPOS_PTR and the current nesting
> -   (as defined for state.incomment) in *INCOMMENT_PTR.
> +   (as defined for state->incomment) in *INCOMMENT_PTR.  The
> +   SYNTAX_WITH_FLAGS of the last character scanned in the comment is
> +   stored into *last_syntax_ptr.
 
>     The comment end is the last character of the comment rather than the
>     character just after the comment.
> @@ -2326,7 +2333,7 @@ static bool
>  forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	      EMACS_INT nesting, int style, int prev_syntax,
>  	      ptrdiff_t *charpos_ptr, ptrdiff_t *bytepos_ptr,
> -	      EMACS_INT *incomment_ptr)
> +	      EMACS_INT *incomment_ptr, int *last_syntax_ptr)
>  {
>    register int c, c1;
>    register enum syntaxcode code;
> @@ -2346,6 +2353,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	  *incomment_ptr = nesting;
>  	  *charpos_ptr = from;
>  	  *bytepos_ptr = from_byte;
> +          *last_syntax_ptr = syntax;
>  	  return 0;
>  	}
>        c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
> @@ -2415,6 +2423,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>      }
>    *charpos_ptr = from;
>    *bytepos_ptr = from_byte;
> +  *last_syntax_ptr = syntax;
>    return 1;
>  }
 
> @@ -2436,6 +2445,7 @@ between them, return t; otherwise return nil.  */)
>    EMACS_INT count1;
>    ptrdiff_t out_charpos, out_bytepos;
>    EMACS_INT dummy;
> +  int dummy2;
 
>    CHECK_NUMBER (count);
>    count1 = XINT (count);
> @@ -2499,7 +2509,7 @@ between them, return t; otherwise return nil.  */)
>  	}
>        /* We're at the start of a comment.  */
>        found = forw_comment (from, from_byte, stop, comnested, comstyle, 0,
> -			    &out_charpos, &out_bytepos, &dummy);
> +			    &out_charpos, &out_bytepos, &dummy, &dummy2);
>        from = out_charpos; from_byte = out_bytepos;
>        if (!found)
>  	{
> @@ -2659,6 +2669,7 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
>    ptrdiff_t from_byte;
>    ptrdiff_t out_bytepos, out_charpos;
>    EMACS_INT dummy;
> +  int dummy2;
>    bool multibyte_symbol_p = sexpflag && multibyte_syntax_as_symbol;
 
>    if (depth > 0) min_depth = 0;
> @@ -2755,7 +2766,8 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
>  	      UPDATE_SYNTAX_TABLE_FORWARD (from);
>  	      found = forw_comment (from, from_byte, stop,
>  				    comnested, comstyle, 0,
> -				    &out_charpos, &out_bytepos, &dummy);
> +				    &out_charpos, &out_bytepos, &dummy,
> +                                    &dummy2);
>  	      from = out_charpos, from_byte = out_bytepos;
>  	      if (!found)
>  		{
> @@ -3119,7 +3131,7 @@ the prefix syntax flag (p).  */)
>  }
>  \f
>  /* Parse forward from FROM / FROM_BYTE to END,
> -   assuming that FROM has state OLDSTATE (nil means FROM is start of function),
> +   assuming that FROM has state STATE (nil means FROM is start of function),
>     and return a description of the state of the parse at END.
>     If STOPBEFORE, stop at the start of an atom.
>     If COMMENTSTOP is 1, stop at the start of a comment.
> @@ -3127,12 +3139,11 @@ the prefix syntax flag (p).  */)
>     after the beginning of a string, or after the end of a string.  */
 
>  static void
> -scan_sexps_forward (struct lisp_parse_state *stateptr,
> +scan_sexps_forward (struct lisp_parse_state *state,
>  		    ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t end,
>  		    EMACS_INT targetdepth, bool stopbefore,
> -		    Lisp_Object oldstate, int commentstop)
> +		    int commentstop)
>  {
> -  struct lisp_parse_state state;
>    enum syntaxcode code;
>    int c1;
>    bool comnested;
> @@ -3148,7 +3159,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
>    Lisp_Object tem;
>    ptrdiff_t prev_from;		/* Keep one character before FROM.  */
>    ptrdiff_t prev_from_byte;
> -  int prev_from_syntax;
> +  int prev_from_syntax, prev_prev_from_syntax;
>    bool boundary_stop = commentstop == -1;
>    bool nofence;
>    bool found;
> @@ -3165,6 +3176,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
>  do { prev_from = from;				\
>       prev_from_byte = from_byte; 		\
>       temp = FETCH_CHAR_AS_MULTIBYTE (prev_from_byte);	\
> +     prev_prev_from_syntax = prev_from_syntax;  \
>       prev_from_syntax = SYNTAX_WITH_FLAGS (temp); \
>       INC_BOTH (from, from_byte);		\
>       if (from < end)				\
> @@ -3174,88 +3186,38 @@ do { prev_from = from;				\
>    immediate_quit = 1;
>    QUIT;
 
> -  if (NILP (oldstate))
> -    {
> -      depth = 0;
> -      state.instring = -1;
> -      state.incomment = 0;
> -      state.comstyle = 0;	/* comment style a by default.  */
> -      state.comstr_start = -1;	/* no comment/string seen.  */
> -    }
> -  else
> -    {
> -      tem = Fcar (oldstate);
> -      if (!NILP (tem))
> -	depth = XINT (tem);
> -      else
> -	depth = 0;
> -
> -      oldstate = Fcdr (oldstate);
> -      oldstate = Fcdr (oldstate);
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      /* Check whether we are inside string_fence-style string: */
> -      state.instring = (!NILP (tem)
> -			? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
> -			: -1);
> -
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      state.incomment = (!NILP (tem)
> -			 ? (INTEGERP (tem) ? XINT (tem) : -1)
> -			 : 0);
> -
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      start_quoted = !NILP (tem);
> +  depth = state->depth;
> +  start_quoted = state->quoted;
> +  prev_prev_from_syntax = Smax;
> +  prev_from_syntax = state->prev_syntax;
 
> -      /* if the eighth element of the list is nil, we are in comment
> -	 style a.  If it is non-nil, we are in comment style b */
> -      oldstate = Fcdr (oldstate);
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      state.comstyle = (NILP (tem)
> -			? 0
> -			: (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
> -			   ? XINT (tem)
> -			   : ST_COMMENT_STYLE));
> -
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      state.comstr_start =
> -	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      while (!NILP (tem))		/* >= second enclosing sexps.  */
> -	{
> -	  Lisp_Object temhd = Fcar (tem);
> -	  if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
> -	    curlevel->last = XINT (temhd);
> -	  if (++curlevel == endlevel)
> -	    curlevel--; /* error ("Nesting too deep for parser"); */
> -	  curlevel->prev = -1;
> -	  curlevel->last = -1;
> -	  tem = Fcdr (tem);
> -	}
> +  tem = state->levelstarts;
> +  while (!NILP (tem))		/* >= second enclosing sexps.  */
> +    {
> +      Lisp_Object temhd = Fcar (tem);
> +      if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
> +        curlevel->last = XINT (temhd);
> +      if (++curlevel == endlevel)
> +        curlevel--; /* error ("Nesting too deep for parser"); */
> +      curlevel->prev = -1;
> +      curlevel->last = -1;
> +      tem = Fcdr (tem);
>      }
> -  state.quoted = 0;
> -  mindepth = depth;
> -
curlevel-> prev = -1;
curlevel-> last = -1;
 
> -  SETUP_SYNTAX_TABLE (prev_from, 1);
> -  temp = FETCH_CHAR (prev_from_byte);
> -  prev_from_syntax = SYNTAX_WITH_FLAGS (temp);
> -  UPDATE_SYNTAX_TABLE_FORWARD (from);
> +  state->quoted = 0;
> +  mindepth = depth;
> +
> +  SETUP_SYNTAX_TABLE (from, 1);
 
>    /* Enter the loop at a place appropriate for initial state.  */
 
> -  if (state.incomment)
> +  if (state->incomment)
>      goto startincomment;
> -  if (state.instring >= 0)
> +  if (state->instring >= 0)
>      {
> -      nofence = state.instring != ST_STRING_STYLE;
> +      nofence = state->instring != ST_STRING_STYLE;
>        if (start_quoted)
>  	goto startquotedinstring;
>        goto startinstring;
> @@ -3266,10 +3228,10 @@ do { prev_from = from;				\
>    while (from < end)
>      {
>        int syntax;
> -      INC_FROM;
> -      code = prev_from_syntax & 0xff;
 
>        if (from < end
> +          && (state->prev_comment_end == -1
> +              || prev_from >= state->prev_comment_end)
>  	  && SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
>  	  && (c1 = FETCH_CHAR (from_byte),
>  	      syntax = SYNTAX_WITH_FLAGS (c1),
> @@ -3280,32 +3242,37 @@ do { prev_from = from;				\
>  	  /* Record the comment style we have entered so that only
>  	     the comment-end sequence of the same style actually
>  	     terminates the comment section.  */
> -	  state.comstyle
> +	  state->comstyle
>  	    = SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax);
>  	  comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax)
>  		       | SYNTAX_FLAGS_COMMENT_NESTED (syntax));
> -	  state.incomment = comnested ? 1 : -1;
> -	  state.comstr_start = prev_from;
> +	  state->incomment = comnested ? 1 : -1;
> +	  state->comstr_start = prev_from;
>  	  INC_FROM;
>  	  code = Scomment;
>  	}
> -      else if (code == Scomment_fence)
> -	{
> -	  /* Record the comment style we have entered so that only
> -	     the comment-end sequence of the same style actually
> -	     terminates the comment section.  */
> -	  state.comstyle = ST_COMMENT_STYLE;
> -	  state.incomment = -1;
> -	  state.comstr_start = prev_from;
> -	  code = Scomment;
> -	}
> -      else if (code == Scomment)
> -	{
> -	  state.comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
> -	  state.incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
> -			     1 : -1);
> -	  state.comstr_start = prev_from;
> -	}
> +      else
> +        {
> +          INC_FROM;
> +          code = prev_from_syntax & 0xff;
> +          if (code == Scomment_fence)
> +            {
> +              /* Record the comment style we have entered so that only
> +                 the comment-end sequence of the same style actually
> +                 terminates the comment section.  */
> +              state->comstyle = ST_COMMENT_STYLE;
> +              state->incomment = -1;
> +              state->comstr_start = prev_from;
> +              code = Scomment;
> +            }
> +          else if (code == Scomment)
> +            {
> +              state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
> +              state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
> +                                 1 : -1);
> +              state->comstr_start = prev_from;
> +            }
> +        }
 
>        if (SYNTAX_FLAGS_PREFIX (prev_from_syntax))
>  	continue;
> @@ -3357,18 +3324,21 @@ do { prev_from = from;				\
>  	     middle of it.  We don't want to do that if we're just at the
>  	     beginning of the comment (think of (*) ... (*)).  */
>  	  found = forw_comment (from, from_byte, end,
> -				state.incomment, state.comstyle,
> -				(from == BEGV || from < state.comstr_start + 3)
> +				state->incomment, state->comstyle,
> +				(from == BEGV || from < state->comstr_start + 3)
>  				? 0 : prev_from_syntax,
> -				&out_charpos, &out_bytepos, &state.incomment);
> +				&out_charpos, &out_bytepos, &state->incomment,
> +                                &prev_from_syntax);
>  	  from = out_charpos; from_byte = out_bytepos;
> -	  /* Beware!  prev_from and friends are invalid now.
> -	     Luckily, the `done' doesn't use them and the INC_FROM
> -	     sets them to a sane value without looking at them. */
> +	  /* Beware!  prev_from and friends (except prev_from_syntax)
> +	     are invalid now.  Luckily, the `done' doesn't use them
> +	     and the INC_FROM sets them to a sane value without
> +	     looking at them. */
>  	  if (!found) goto done;
>  	  INC_FROM;
> -	  state.incomment = 0;
> -	  state.comstyle = 0;	/* reset the comment style */
> +	  state->incomment = 0;
> +	  state->comstyle = 0;	/* reset the comment style */
> +          state->prev_comment_end = from;
>  	  if (boundary_stop) goto done;
>  	  break;
 
> @@ -3396,16 +3366,16 @@ do { prev_from = from;				\
 
>  	case Sstring:
>  	case Sstring_fence:
> -	  state.comstr_start = from - 1;
> +	  state->comstr_start = from - 1;
>  	  if (stopbefore) goto stop;  /* this arg means stop at sexp start */
curlevel-> last = prev_from;
> -	  state.instring = (code == Sstring
> +	  state->instring = (code == Sstring
>  			    ? (FETCH_CHAR_AS_MULTIBYTE (prev_from_byte))
>  			    : ST_STRING_STYLE);
>  	  if (boundary_stop) goto done;
>  	startinstring:
>  	  {
> -	    nofence = state.instring != ST_STRING_STYLE;
> +	    nofence = state->instring != ST_STRING_STYLE;
 
>  	    while (1)
>  	      {
> @@ -3419,7 +3389,7 @@ do { prev_from = from;				\
>  		/* Check C_CODE here so that if the char has
>  		   a syntax-table property which says it is NOT
>  		   a string character, it does not end the string.  */
> -		if (nofence && c == state.instring && c_code == Sstring)
> +		if (nofence && c == state->instring && c_code == Sstring)
>  		  break;
 
>  		switch (c_code)
> @@ -3442,7 +3412,7 @@ do { prev_from = from;				\
>  	      }
>  	  }
>  	string_end:
> -	  state.instring = -1;
> +	  state->instring = -1;
curlevel-> prev = curlevel->last;
>  	  INC_FROM;
>  	  if (boundary_stop) goto done;
> @@ -3461,25 +3431,99 @@ do { prev_from = from;				\
>   stop:   /* Here if stopping before start of sexp. */
>    from = prev_from;    /* We have just fetched the char that starts it; */
>    from_byte = prev_from_byte;
> +  prev_from_syntax = prev_prev_from_syntax;
>    goto done; /* but return the position before it. */
 
>   endquoted:
> -  state.quoted = 1;
> +  state->quoted = 1;
>   done:
> -  state.depth = depth;
> -  state.mindepth = mindepth;
> -  state.thislevelstart = curlevel->prev;
> -  state.prevlevelstart
> +  state->depth = depth;
> +  state->mindepth = mindepth;
> +  state->thislevelstart = curlevel->prev;
> +  state->prevlevelstart
>      = (curlevel == levelstart) ? -1 : (curlevel - 1)->last;
> -  state.location = from;
> -  state.location_byte = from_byte;
> -  state.levelstarts = Qnil;
> +  state->location = from;
> +  state->location_byte = from_byte;
> +  state->levelstarts = Qnil;
>    while (curlevel > levelstart)
> -    state.levelstarts = Fcons (make_number ((--curlevel)->last),
> -			       state.levelstarts);
> +    state->levelstarts = Fcons (make_number ((--curlevel)->last),
> +                                state->levelstarts);
> +  state->prev_syntax = prev_from_syntax;
>    immediate_quit = 0;
> +}
> +
> +/* Convert a (lisp) parse state to the internal form used in
> +   scan_sexps_forward.  */
> +static void
> +internalize_parse_state (Lisp_Object external, struct lisp_parse_state *state)
> +{
> +  Lisp_Object tem;
> +
> +  if (NILP (external))
> +    {
> +      state->depth = 0;
> +      state->instring = -1;
> +      state->incomment = 0;
> +      state->quoted = 0;
> +      state->comstyle = 0;	/* comment style a by default.  */
> +      state->comstr_start = -1;	/* no comment/string seen.  */
> +      state->levelstarts = Qnil;
> +      state->prev_syntax = Smax;
> +      state->prev_comment_end = -1;
> +    }
> +  else
> +    {
> +      tem = Fcar (external);
> +      if (!NILP (tem))
> +	state->depth = XINT (tem);
> +      else
> +	state->depth = 0;
> +
> +      external = Fcdr (external);
> +      external = Fcdr (external);
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      /* Check whether we are inside string_fence-style string: */
> +      state->instring = (!NILP (tem)
> +                         ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
> +                         : -1);
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->incomment = (!NILP (tem)
> +                          ? (INTEGERP (tem) ? XINT (tem) : -1)
> +                          : 0);
 
> -  *stateptr = state;
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->quoted = !NILP (tem);
> +
> +      /* if the eighth element of the list is nil, we are in comment
> +	 style a.  If it is non-nil, we are in comment style b */
> +      external = Fcdr (external);
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->comstyle = (NILP (tem)
> +                         ? 0
> +                         : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
> +                            ? XINT (tem)
> +                            : ST_COMMENT_STYLE));
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->comstr_start =
> +	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->levelstarts = tem;
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->prev_syntax = NILP (tem) ? Smax : XINT (tem);
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->prev_comment_end = NILP (tem) ? -1 : XINT (tem);
> +    }
>  }
 
>  DEFUN ("parse-partial-sexp", Fparse_partial_sexp, Sparse_partial_sexp, 2, 6, 0,
> @@ -3488,6 +3532,7 @@ Parsing stops at TO or when certain criteria are met;
>   point is set to where parsing stops.
>  If fifth arg OLDSTATE is omitted or nil,
>   parsing assumes that FROM is the beginning of a function.
> +
>  Value is a list of elements describing final state of parsing:
>   0. depth in parens.
>   1. character address of start of innermost containing list; nil if none.
> @@ -3501,16 +3546,20 @@ Value is a list of elements describing final state of parsing:
>   6. the minimum paren-depth encountered during this scan.
>   7. style of comment, if any.
>   8. character address of start of comment or string; nil if not in one.
> - 9. Intermediate data for continuation of parsing (subject to change).
> + 9. List of positions of currently open parens, outermost first.
> +10. Syntax of last character scanned, or nil if no scanning has happened.
> +11. Position after end of previous comment scanned, or nil.
> +12..... Possible further internal information used by `parse-partial-sexp'.
> +
>  If third arg TARGETDEPTH is non-nil, parsing stops if the depth
>  in parentheses becomes equal to TARGETDEPTH.
> -Fourth arg STOPBEFORE non-nil means stop when come to
> +Fourth arg STOPBEFORE non-nil means stop when we come to
>   any character that starts a sexp.
>  Fifth arg OLDSTATE is a list like what this function returns.
>   It is used to initialize the state of the parse.  Elements number 1, 2, 6
>   are ignored.
> -Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
> - If it is symbol `syntax-table', stop after the start of a comment or a
> +Sixth arg COMMENTSTOP non-nil means stop after the start of a comment.
> + If it is the symbol `syntax-table', stop after the start of a comment or a
>   string, or after end of a comment or a string.  */)
>    (Lisp_Object from, Lisp_Object to, Lisp_Object targetdepth,
>     Lisp_Object stopbefore, Lisp_Object oldstate, Lisp_Object commentstop)
> @@ -3527,15 +3576,17 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
>      target = TYPE_MINIMUM (EMACS_INT);	/* We won't reach this depth.  */
 
>    validate_region (&from, &to);
> +  internalize_parse_state (oldstate, &state);
>    scan_sexps_forward (&state, XINT (from), CHAR_TO_BYTE (XINT (from)),
>  		      XINT (to),
> -		      target, !NILP (stopbefore), oldstate,
> +		      target, !NILP (stopbefore),
>  		      (NILP (commentstop)
>  		       ? 0 : (EQ (commentstop, Qsyntax_table) ? -1 : 1)));
 
>    SET_PT_BOTH (state.location, state.location_byte);
 
> -  return Fcons (make_number (state.depth),
> +  return
> +    Fcons (make_number (state.depth),
>  	   Fcons (state.prevlevelstart < 0
>  		  ? Qnil : make_number (state.prevlevelstart),
>  	     Fcons (state.thislevelstart < 0
> @@ -3553,11 +3604,18 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
>  				  ? Qsyntax_table
>  				  : make_number (state.comstyle))
>  			       : Qnil),
> -			      Fcons (((state.incomment
> -				       || (state.instring >= 0))
> -				      ? make_number (state.comstr_start)
> -				      : Qnil),
> -				     Fcons (state.levelstarts, Qnil))))))))));
> +		         Fcons (((state.incomment
> +                                  || (state.instring >= 0))
> +                                 ? make_number (state.comstr_start)
> +                                 : Qnil),
> +			   Fcons (state.levelstarts,
> +                             Fcons (state.prev_syntax == Smax
> +                                    ? Qnil
> +                                    : make_number (state.prev_syntax),
> +                               Fcons (state.prev_comment_end == -1
> +                                      ? Qnil
> +                                      : make_number (state.prev_comment_end),
> +                                      Qnil))))))))))));
>  }
>  \f
>  void


>> Stefan

> -- 
> Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello, Stefan.

On Fri, Mar 18, 2016 at 12:49:07AM -0400, Stefan Monnier wrote:
> > Do this by adding two new fields to the parser state: the syntax of the last
> > character scanned, and the last end of comment scanned.  This should make the
> > parser state complete.

> Thanks.  I like the "syntax of the last character scanned", but I don't
> understand the reasoning behind "last end of comment scanned".  Why is
> this relevant?  Is it in case the "last character scanned" was a "slash
> ending a comment" so as to avoid treating "*/*" as both a comment closer and
> a subsequent opener?

That's exactly the reason.

> If so, I'm not sure I like it.

I don't really like it either.

> It sounds to me like there's a chance it's actually incomplete (e.g.
> it doesn't address the similar problem when the "last character
> scanned" is an end of a string which also happens to be a valid
> first-char of a comment-starter), and even if it isn't, it "feels
> ad-hoc" to me.

Now even I wouldn't have come up with that end-of-string scenario.  ;-)
Such a scenario is presumably one reason why, in scan_sexps_forward, two
character comment delimiters are handled before strings.

> Would it be difficult to do the following instead:
> - get rid of element 11.

Done.

> - change element 10 so it's nil if the last char was an "end of
>   something".  Another way to look at it, is that the element 10 should
>   only be non-nil if the "next lexeme" might start on that
>   previous character.

I've tried this, and it's somewhat ugly.  Setting the "previous_syntax"
to nil is also needed for the asterisk in "/*".  The nil would appear to
mean "the syntactic value of the last character has already been used
up".  So the "previous_syntax" is nil in the most interesting cases.  It
also feels somewhat ad-hoc.

How about this idea: element 10 will record the syntax of the previous
character ONLY when it is potentially the first character of a two
character comment delimiter, otherwise it'll be nil.  At least that's
being honest about what the thing's being used for.

> I also have a side question: IIUC your patch makes the 5th element
> redundant (can be replaced with a test whether "last char syntax" was
> "escape"), is that right?

It would appear to be, yes.  We really can't get rid of element 5,
though, because there will surely be code out there that uses it.  But
if I change element 10 as outlined above, element 5 will no longer be
redundant.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello again, Stefan.

On Fri, Mar 18, 2016 at 03:11:55PM +0000, Alan Mackenzie wrote:

> Done.

> > - change element 10 so it's nil if the last char was an "end of
> >   something".  Another way to look at it, is that the element 10 should
> >   only be non-nil if the "next lexeme" might start on that
> >   previous character.

[ .... ]

> How about this idea: element 10 will record the syntax of the previous
> character ONLY when it is potentially the first character of a two
> character comment delimiter, otherwise it'll be nil.  At least that's
> being honest about what the thing's being used for.

That's exactly what you suggested.  Apologies for not reading your post
a bit more carefully.  I think we're agreed, then.  I'll implement it.

> >         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Stefan Monnier]

>> It sounds to me like there's a chance it's actually incomplete (e.g.
>> it doesn't address the similar problem when the "last character
>> scanned" is an end of a string which also happens to be a valid
>> first-char of a comment-starter), and even if it isn't, it "feels
>> ad-hoc" to me.
> Now even I wouldn't have come up with that end-of-string scenario.  ;-)

I don't work in embedded systems, but Coq/Agda's total functions force
you to consider all possible cases.

> Such a scenario is presumably one reason why, in scan_sexps_forward, two
> character comment delimiters are handled before strings.

It doesn't handle the exact same situation, but it's closely related
indeed.

>> - change element 10 so it's nil if the last char was an "end of
>> something".  Another way to look at it, is that the element 10 should
>> only be non-nil if the "next lexeme" might start on that
>> previous character.

> I've tried this, and it's somewhat ugly.  Setting the "previous_syntax"
> to nil is also needed for the asterisk in "/*".  The nil would appear to
> mean "the syntactic value of the last character has already been used
> up".  So the "previous_syntax" is nil in the most interesting cases.  It
> also feels somewhat ad-hoc.

> How about this idea: element 10 will record the syntax of the previous
> character ONLY when it is potentially the first character of a two
> character comment delimiter, otherwise it'll be nil.  At least that's
> being honest about what the thing's being used for.

IIUC the only difference between what I (think I) suggested and what
you're proposing is that you want to return nil for the "prev is
backslash" whereas I was suggesting to return non-nil in that case.
[ AFAIK the only two-char elements we handle so far as the comment
delimiters and the backslash escapes.  ]
Do I understand this right?

> It would appear to be, yes.  We really can't get rid of element 5,
> though, because there will surely be code out there that uses it.  But
> if I change element 10 as outlined above, element 5 will no longer be
> redundant.

I'd even be tempted to re-use element 5, although it might
conceivably break some code out there.

But even if we don't re-use element 5, I would actually much prefer to
render element 5 redundant.


        Stefan

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello, Stefan.

On Fri, Mar 18, 2016 at 12:23:02PM -0400, Stefan Monnier wrote:

> >> - change element 10 so it's nil if the last char was an "end of
> >> something".  Another way to look at it, is that the element 10 should
> >> only be non-nil if the "next lexeme" might start on that
> >> previous character.

> > I've tried this, and it's somewhat ugly.  Setting the "previous_syntax"
> > to nil is also needed for the asterisk in "/*".  The nil would appear to
> > mean "the syntactic value of the last character has already been used
> > up".  So the "previous_syntax" is nil in the most interesting cases.  It
> > also feels somewhat ad-hoc.

> > How about this idea: element 10 will record the syntax of the previous
> > character ONLY when it is potentially the first character of a two
> > character comment delimiter, otherwise it'll be nil.  At least that's
> > being honest about what the thing's being used for.

> IIUC the only difference between what I (think I) suggested and what
> you're proposing is that you want to return nil for the "prev is
> backslash" whereas I was suggesting to return non-nil in that case.
> [ AFAIK the only two-char elements we handle so far as the comment
> delimiters and the backslash escapes.  ]

We also have Scharquote, which scan_sexps_forward handles identically to
Sescape.

> Do I understand this right?

Yes, but I've no strong feelings on the matter.

> > It would appear to be, yes.  We really can't get rid of element 5,
> > though, because there will surely be code out there that uses it.  But
> > if I change element 10 as outlined above, element 5 will no longer be
> > redundant.

> I'd even be tempted to re-use element 5, although it might
> conceivably break some code out there.

I have bad feelings about that.  Is it really worth the risk, just to
save one cons cell on a list that not that many instances of exist at
any time?

> But even if we don't re-use element 5, I would actually much prefer to
> render element 5 redundant.

OK.  Here's an updated patch which does just that.  Comments would be
welcome.

>         Stefan


Amend parse-partial-sexp correctly to handle two character comment delimiters

Do this by adding a new field to the parser state: the syntax of the last
character scanned, should that be the first char of a (potential) two char
construct, nil otherwise.
This should make the parser state complete.
Also document element 9 of the parser state.  Also refactor the code a bit.

* src/syntax.c (struct lisp_parse_state): Add a new field.
(SYNTAX_FLAGS_COMSTARTEND_FIRST): New function.
(internalize_parse_state): New function, extracted from scan_sexps_forward.
(back_comment): Call internalize_parse_state.
(forw_comment): Return the syntax of the last character scanned to the caller.
(Fforward_comment, scan_lists): New dummy variables, passed to forw_comment.
(scan_sexps_forward): Remove a redundant state parameter.  Access all `state'
information via the address parameter `state'.  Remove the code which converts
from external to internal form of `state'.  Access buffer contents only from
`from' onwards.  Reformulate code at the top of the main loop correctly to
recognize comment openers when starting in the middle of one.  Call
forw_comment with extra argument (for return of final syntax value).
(Fparse_partial_sexp): Document elements 9, 10 of the parser state in the
doc string.  Clarify the doc string in general.  Call
internalize_parse_state.  Take account of the new elements when consing up the
output parser state.

* doc/lispref/syntax.texi: (Parser State): Document element 9 and the new
element 10.  Minor wording corrections (remove reference to "trivial cases").
(Low Level Parsing): Minor corrections.




diff --git a/doc/lispref/syntax.texi b/doc/lispref/syntax.texi
index d5a7eba..f81c164 100644
--- a/doc/lispref/syntax.texi
+++ b/doc/lispref/syntax.texi
@@ -791,10 +791,10 @@ Parser State
 @subsection Parser State
 @cindex parser state
 
-  A @dfn{parser state} is a list of ten elements describing the state
-of the syntactic parser, after it parses the text between a specified
-starting point and a specified end point in the buffer.  Parsing
-functions such as @code{syntax-ppss}
+  A @dfn{parser state} is a list of (currently) eleven elements
+describing the state of the syntactic parser, after it parses the text
+between a specified starting point and a specified end point in the
+buffer.  Parsing functions such as @code{syntax-ppss}
 @ifnottex
 (@pxref{Position Parse})
 @end ifnottex
@@ -851,15 +851,20 @@ Parser State
 this element is @code{nil}.
 
 @item
-Internal data for continuing the parsing.  The meaning of this
-data is subject to change; it is used if you pass this list
-as the @var{state} argument to another call.
+The list of the positions of the currently open parentheses, starting
+with the outermost.
+
+@item
+When the last buffer position scanned was the (potential) first
+character of a two character construct (comment delimiter or
+escaped/char-quoted character pair), the @var{syntax-code}
+(@pxref{Syntax Table Internals}) of that position.  Otherwise
+@code{nil}.
 @end enumerate
 
   Elements 1, 2, and 6 are ignored in a state which you pass as an
-argument to continue parsing, and elements 8 and 9 are used only in
-trivial cases.  Those elements are mainly used internally by the
-parser code.
+argument to continue parsing.  Elements 9 and 10 are mainly used
+internally by the parser code.
 
   One additional piece of useful information is available from a
 parser state using this function:
@@ -898,11 +903,11 @@ Low-Level Parsing
 
 If the fourth argument @var{stop-before} is non-@code{nil}, parsing
 stops when it comes to any character that starts a sexp.  If
-@var{stop-comment} is non-@code{nil}, parsing stops when it comes to the
-start of an unnested comment.  If @var{stop-comment} is the symbol
+@var{stop-comment} is non-@code{nil}, parsing stops after the start of
+an unnested comment.  If @var{stop-comment} is the symbol
 @code{syntax-table}, parsing stops after the start of an unnested
-comment or a string, or the end of an unnested comment or a string,
-whichever comes first.
+comment or a string, or after the end of an unnested comment or a
+string, whichever comes first.
 
 If @var{state} is @code{nil}, @var{start} is assumed to be at the top
 level of parenthesis structure, such as the beginning of a function
diff --git a/src/syntax.c b/src/syntax.c
index 249d0d5..e6a1942 100644
--- a/src/syntax.c
+++ b/src/syntax.c
@@ -81,6 +81,11 @@ SYNTAX_FLAGS_COMEND_SECOND (int flags)
   return (flags >> 19) & 1;
 }
 static bool
+SYNTAX_FLAGS_COMSTARTEND_FIRST (int flags)
+{
+  return (flags & 0x50000) != 0;
+}
+static bool
 SYNTAX_FLAGS_PREFIX (int flags)
 {
   return (flags >> 20) & 1;
@@ -153,6 +158,10 @@ struct lisp_parse_state
     ptrdiff_t comstr_start;  /* Position of last comment/string starter.  */
     Lisp_Object levelstarts; /* Char numbers of starts-of-expression
 				of levels (starting from outermost).  */
+    int prev_syntax; /* Syntax of previous position scanned, when
+                        that position (potentially) holds the first char
+                        of a 2-char construct, i.e. comment delimiter
+                        or Sescape, etc.  Smax otherwise. */
   };
 \f
 /* These variables are a cache for finding the start of a defun.
@@ -176,7 +185,8 @@ static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
 static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool);
 static void scan_sexps_forward (struct lisp_parse_state *,
                                 ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
-                                bool, Lisp_Object, int);
+                                bool, int);
+static void internalize_parse_state (Lisp_Object, struct lisp_parse_state *);
 static bool in_classes (int, Lisp_Object);
 static void parse_sexp_propertize (ptrdiff_t charpos);
 
@@ -911,10 +921,11 @@ back_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
 	}
       do
 	{
+          internalize_parse_state (Qnil, &state);
 	  scan_sexps_forward (&state,
 			      defun_start, defun_start_byte,
 			      comment_end, TYPE_MINIMUM (EMACS_INT),
-			      0, Qnil, 0);
+			      0, 0);
 	  defun_start = comment_end;
 	  if (!adjusted)
 	    {
@@ -2314,7 +2325,9 @@ in_classes (int c, Lisp_Object iso_classes)
    into *CHARPOS_PTR and the corresponding bytepos into *BYTEPOS_PTR.
    Else, return false and store the charpos STOP into *CHARPOS_PTR, the
    corresponding bytepos into *BYTEPOS_PTR and the current nesting
-   (as defined for state.incomment) in *INCOMMENT_PTR.
+   (as defined for state->incomment) in *INCOMMENT_PTR.  The
+   SYNTAX_WITH_FLAGS of the last character scanned in the comment is
+   stored into *last_syntax_ptr.
 
    The comment end is the last character of the comment rather than the
    character just after the comment.
@@ -2326,7 +2339,7 @@ static bool
 forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
 	      EMACS_INT nesting, int style, int prev_syntax,
 	      ptrdiff_t *charpos_ptr, ptrdiff_t *bytepos_ptr,
-	      EMACS_INT *incomment_ptr)
+	      EMACS_INT *incomment_ptr, int *last_syntax_ptr)
 {
   register int c, c1;
   register enum syntaxcode code;
@@ -2346,6 +2359,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
 	  *incomment_ptr = nesting;
 	  *charpos_ptr = from;
 	  *bytepos_ptr = from_byte;
+          *last_syntax_ptr = syntax;
 	  return 0;
 	}
       c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
@@ -2415,6 +2429,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
     }
   *charpos_ptr = from;
   *bytepos_ptr = from_byte;
+  *last_syntax_ptr = syntax;
   return 1;
 }
 
@@ -2436,6 +2451,7 @@ between them, return t; otherwise return nil.  */)
   EMACS_INT count1;
   ptrdiff_t out_charpos, out_bytepos;
   EMACS_INT dummy;
+  int dummy2;
 
   CHECK_NUMBER (count);
   count1 = XINT (count);
@@ -2499,7 +2515,7 @@ between them, return t; otherwise return nil.  */)
 	}
       /* We're at the start of a comment.  */
       found = forw_comment (from, from_byte, stop, comnested, comstyle, 0,
-			    &out_charpos, &out_bytepos, &dummy);
+			    &out_charpos, &out_bytepos, &dummy, &dummy2);
       from = out_charpos; from_byte = out_bytepos;
       if (!found)
 	{
@@ -2659,6 +2675,7 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
   ptrdiff_t from_byte;
   ptrdiff_t out_bytepos, out_charpos;
   EMACS_INT dummy;
+  int dummy2;
   bool multibyte_symbol_p = sexpflag && multibyte_syntax_as_symbol;
 
   if (depth > 0) min_depth = 0;
@@ -2755,7 +2772,8 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
 	      UPDATE_SYNTAX_TABLE_FORWARD (from);
 	      found = forw_comment (from, from_byte, stop,
 				    comnested, comstyle, 0,
-				    &out_charpos, &out_bytepos, &dummy);
+				    &out_charpos, &out_bytepos, &dummy,
+                                    &dummy2);
 	      from = out_charpos, from_byte = out_bytepos;
 	      if (!found)
 		{
@@ -3119,7 +3137,7 @@ the prefix syntax flag (p).  */)
 }
 \f
 /* Parse forward from FROM / FROM_BYTE to END,
-   assuming that FROM has state OLDSTATE (nil means FROM is start of function),
+   assuming that FROM has state STATE,
    and return a description of the state of the parse at END.
    If STOPBEFORE, stop at the start of an atom.
    If COMMENTSTOP is 1, stop at the start of a comment.
@@ -3127,12 +3145,11 @@ the prefix syntax flag (p).  */)
    after the beginning of a string, or after the end of a string.  */
 
 static void
-scan_sexps_forward (struct lisp_parse_state *stateptr,
+scan_sexps_forward (struct lisp_parse_state *state,
 		    ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t end,
 		    EMACS_INT targetdepth, bool stopbefore,
-		    Lisp_Object oldstate, int commentstop)
+		    int commentstop)
 {
-  struct lisp_parse_state state;
   enum syntaxcode code;
   int c1;
   bool comnested;
@@ -3148,7 +3165,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
   Lisp_Object tem;
   ptrdiff_t prev_from;		/* Keep one character before FROM.  */
   ptrdiff_t prev_from_byte;
-  int prev_from_syntax;
+  int prev_from_syntax, prev_prev_from_syntax;
   bool boundary_stop = commentstop == -1;
   bool nofence;
   bool found;
@@ -3165,6 +3182,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
 do { prev_from = from;				\
      prev_from_byte = from_byte; 		\
      temp = FETCH_CHAR_AS_MULTIBYTE (prev_from_byte);	\
+     prev_prev_from_syntax = prev_from_syntax;  \
      prev_from_syntax = SYNTAX_WITH_FLAGS (temp); \
      INC_BOTH (from, from_byte);		\
      if (from < end)				\
@@ -3174,88 +3192,38 @@ do { prev_from = from;				\
   immediate_quit = 1;
   QUIT;
 
-  if (NILP (oldstate))
-    {
-      depth = 0;
-      state.instring = -1;
-      state.incomment = 0;
-      state.comstyle = 0;	/* comment style a by default.  */
-      state.comstr_start = -1;	/* no comment/string seen.  */
-    }
-  else
-    {
-      tem = Fcar (oldstate);
-      if (!NILP (tem))
-	depth = XINT (tem);
-      else
-	depth = 0;
-
-      oldstate = Fcdr (oldstate);
-      oldstate = Fcdr (oldstate);
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      /* Check whether we are inside string_fence-style string: */
-      state.instring = (!NILP (tem)
-			? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
-			: -1);
+  depth = state->depth;
+  start_quoted = state->quoted;
+  prev_prev_from_syntax = Smax;
+  prev_from_syntax = state->prev_syntax;
 
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      state.incomment = (!NILP (tem)
-			 ? (INTEGERP (tem) ? XINT (tem) : -1)
-			 : 0);
-
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      start_quoted = !NILP (tem);
-
-      /* if the eighth element of the list is nil, we are in comment
-	 style a.  If it is non-nil, we are in comment style b */
-      oldstate = Fcdr (oldstate);
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      state.comstyle = (NILP (tem)
-			? 0
-			: (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
-			   ? XINT (tem)
-			   : ST_COMMENT_STYLE));
-
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      state.comstr_start =
-	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
-      oldstate = Fcdr (oldstate);
-      tem = Fcar (oldstate);
-      while (!NILP (tem))		/* >= second enclosing sexps.  */
-	{
-	  Lisp_Object temhd = Fcar (tem);
-	  if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
-	    curlevel->last = XINT (temhd);
-	  if (++curlevel == endlevel)
-	    curlevel--; /* error ("Nesting too deep for parser"); */
-	  curlevel->prev = -1;
-	  curlevel->last = -1;
-	  tem = Fcdr (tem);
-	}
+  tem = state->levelstarts;
+  while (!NILP (tem))		/* >= second enclosing sexps.  */
+    {
+      Lisp_Object temhd = Fcar (tem);
+      if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
+        curlevel->last = XINT (temhd);
+      if (++curlevel == endlevel)
+        curlevel--; /* error ("Nesting too deep for parser"); */
+      curlevel->prev = -1;
+      curlevel->last = -1;
+      tem = Fcdr (tem);
     }
-  state.quoted = 0;
-  mindepth = depth;
-
   curlevel->prev = -1;
   curlevel->last = -1;
 
-  SETUP_SYNTAX_TABLE (prev_from, 1);
-  temp = FETCH_CHAR (prev_from_byte);
-  prev_from_syntax = SYNTAX_WITH_FLAGS (temp);
-  UPDATE_SYNTAX_TABLE_FORWARD (from);
+  state->quoted = 0;
+  mindepth = depth;
+
+  SETUP_SYNTAX_TABLE (from, 1);
 
   /* Enter the loop at a place appropriate for initial state.  */
 
-  if (state.incomment)
+  if (state->incomment)
     goto startincomment;
-  if (state.instring >= 0)
+  if (state->instring >= 0)
     {
-      nofence = state.instring != ST_STRING_STYLE;
+      nofence = state->instring != ST_STRING_STYLE;
       if (start_quoted)
 	goto startquotedinstring;
       goto startinstring;
@@ -3266,11 +3234,8 @@ do { prev_from = from;				\
   while (from < end)
     {
       int syntax;
-      INC_FROM;
-      code = prev_from_syntax & 0xff;
 
-      if (from < end
-	  && SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
+      if (SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
 	  && (c1 = FETCH_CHAR (from_byte),
 	      syntax = SYNTAX_WITH_FLAGS (c1),
 	      SYNTAX_FLAGS_COMSTART_SECOND (syntax)))
@@ -3280,32 +3245,39 @@ do { prev_from = from;				\
 	  /* Record the comment style we have entered so that only
 	     the comment-end sequence of the same style actually
 	     terminates the comment section.  */
-	  state.comstyle
+	  state->comstyle
 	    = SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax);
 	  comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax)
 		       | SYNTAX_FLAGS_COMMENT_NESTED (syntax));
-	  state.incomment = comnested ? 1 : -1;
-	  state.comstr_start = prev_from;
+	  state->incomment = comnested ? 1 : -1;
+	  state->comstr_start = prev_from;
 	  INC_FROM;
+          prev_from_syntax = Smax; /* the syntax has already been
+                                      "used up". */
 	  code = Scomment;
 	}
-      else if (code == Scomment_fence)
-	{
-	  /* Record the comment style we have entered so that only
-	     the comment-end sequence of the same style actually
-	     terminates the comment section.  */
-	  state.comstyle = ST_COMMENT_STYLE;
-	  state.incomment = -1;
-	  state.comstr_start = prev_from;
-	  code = Scomment;
-	}
-      else if (code == Scomment)
-	{
-	  state.comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
-	  state.incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
-			     1 : -1);
-	  state.comstr_start = prev_from;
-	}
+      else
+        {
+          INC_FROM;
+          code = prev_from_syntax & 0xff;
+          if (code == Scomment_fence)
+            {
+              /* Record the comment style we have entered so that only
+                 the comment-end sequence of the same style actually
+                 terminates the comment section.  */
+              state->comstyle = ST_COMMENT_STYLE;
+              state->incomment = -1;
+              state->comstr_start = prev_from;
+              code = Scomment;
+            }
+          else if (code == Scomment)
+            {
+              state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
+              state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
+                                 1 : -1);
+              state->comstr_start = prev_from;
+            }
+        }
 
       if (SYNTAX_FLAGS_PREFIX (prev_from_syntax))
 	continue;
@@ -3350,25 +3322,28 @@ do { prev_from = from;				\
 
 	case Scomment_fence: /* Can't happen because it's handled above.  */
 	case Scomment:
-	  if (commentstop || boundary_stop) goto done;
+          if (commentstop || boundary_stop) goto done;
 	startincomment:
 	  /* The (from == BEGV) test was to enter the loop in the middle so
 	     that we find a 2-char comment ender even if we start in the
 	     middle of it.  We don't want to do that if we're just at the
 	     beginning of the comment (think of (*) ... (*)).  */
 	  found = forw_comment (from, from_byte, end,
-				state.incomment, state.comstyle,
-				(from == BEGV || from < state.comstr_start + 3)
-				? 0 : prev_from_syntax,
-				&out_charpos, &out_bytepos, &state.incomment);
+				state->incomment, state->comstyle,
+				from == BEGV ? 0 : prev_from_syntax,
+				&out_charpos, &out_bytepos, &state->incomment,
+                                &prev_from_syntax);
 	  from = out_charpos; from_byte = out_bytepos;
-	  /* Beware!  prev_from and friends are invalid now.
-	     Luckily, the `done' doesn't use them and the INC_FROM
-	     sets them to a sane value without looking at them. */
+	  /* Beware!  prev_from and friends (except prev_from_syntax)
+	     are invalid now.  Luckily, the `done' doesn't use them
+	     and the INC_FROM sets them to a sane value without
+	     looking at them. */
 	  if (!found) goto done;
 	  INC_FROM;
-	  state.incomment = 0;
-	  state.comstyle = 0;	/* reset the comment style */
+	  state->incomment = 0;
+	  state->comstyle = 0;	/* reset the comment style */
+          prev_from_syntax = Smax; /* Ensure "*|*" can't open a spurious new
+                                      comment. */
 	  if (boundary_stop) goto done;
 	  break;
 
@@ -3396,16 +3371,16 @@ do { prev_from = from;				\
 
 	case Sstring:
 	case Sstring_fence:
-	  state.comstr_start = from - 1;
+	  state->comstr_start = from - 1;
 	  if (stopbefore) goto stop;  /* this arg means stop at sexp start */
 	  curlevel->last = prev_from;
-	  state.instring = (code == Sstring
+	  state->instring = (code == Sstring
 			    ? (FETCH_CHAR_AS_MULTIBYTE (prev_from_byte))
 			    : ST_STRING_STYLE);
 	  if (boundary_stop) goto done;
 	startinstring:
 	  {
-	    nofence = state.instring != ST_STRING_STYLE;
+	    nofence = state->instring != ST_STRING_STYLE;
 
 	    while (1)
 	      {
@@ -3419,7 +3394,7 @@ do { prev_from = from;				\
 		/* Check C_CODE here so that if the char has
 		   a syntax-table property which says it is NOT
 		   a string character, it does not end the string.  */
-		if (nofence && c == state.instring && c_code == Sstring)
+		if (nofence && c == state->instring && c_code == Sstring)
 		  break;
 
 		switch (c_code)
@@ -3442,7 +3417,7 @@ do { prev_from = from;				\
 	      }
 	  }
 	string_end:
-	  state.instring = -1;
+	  state->instring = -1;
 	  curlevel->prev = curlevel->last;
 	  INC_FROM;
 	  if (boundary_stop) goto done;
@@ -3461,25 +3436,96 @@ do { prev_from = from;				\
  stop:   /* Here if stopping before start of sexp. */
   from = prev_from;    /* We have just fetched the char that starts it; */
   from_byte = prev_from_byte;
+  prev_from_syntax = prev_prev_from_syntax;
   goto done; /* but return the position before it. */
 
  endquoted:
-  state.quoted = 1;
+  state->quoted = 1;
  done:
-  state.depth = depth;
-  state.mindepth = mindepth;
-  state.thislevelstart = curlevel->prev;
-  state.prevlevelstart
+  state->depth = depth;
+  state->mindepth = mindepth;
+  state->thislevelstart = curlevel->prev;
+  state->prevlevelstart
     = (curlevel == levelstart) ? -1 : (curlevel - 1)->last;
-  state.location = from;
-  state.location_byte = from_byte;
-  state.levelstarts = Qnil;
+  state->location = from;
+  state->location_byte = from_byte;
+  state->levelstarts = Qnil;
   while (curlevel > levelstart)
-    state.levelstarts = Fcons (make_number ((--curlevel)->last),
-			       state.levelstarts);
+    state->levelstarts = Fcons (make_number ((--curlevel)->last),
+                                state->levelstarts);
+  state->prev_syntax = (SYNTAX_FLAGS_COMSTARTEND_FIRST (prev_from_syntax)
+                        || state->quoted) ? prev_from_syntax : Smax;
   immediate_quit = 0;
+}
+
+/* Convert a (lisp) parse state to the internal form used in
+   scan_sexps_forward.  */
+static void
+internalize_parse_state (Lisp_Object external, struct lisp_parse_state *state)
+{
+  Lisp_Object tem;
+
+  if (NILP (external))
+    {
+      state->depth = 0;
+      state->instring = -1;
+      state->incomment = 0;
+      state->quoted = 0;
+      state->comstyle = 0;	/* comment style a by default.  */
+      state->comstr_start = -1;	/* no comment/string seen.  */
+      state->levelstarts = Qnil;
+      state->prev_syntax = Smax;
+    }
+  else
+    {
+      tem = Fcar (external);
+      if (!NILP (tem))
+	state->depth = XINT (tem);
+      else
+	state->depth = 0;
+
+      external = Fcdr (external);
+      external = Fcdr (external);
+      external = Fcdr (external);
+      tem = Fcar (external);
+      /* Check whether we are inside string_fence-style string: */
+      state->instring = (!NILP (tem)
+                         ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
+                         : -1);
+
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->incomment = (!NILP (tem)
+                          ? (INTEGERP (tem) ? XINT (tem) : -1)
+                          : 0);
+
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->quoted = !NILP (tem);
 
-  *stateptr = state;
+      /* if the eighth element of the list is nil, we are in comment
+	 style a.  If it is non-nil, we are in comment style b */
+      external = Fcdr (external);
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->comstyle = (NILP (tem)
+                         ? 0
+                         : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
+                            ? XINT (tem)
+                            : ST_COMMENT_STYLE));
+
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->comstr_start =
+	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->levelstarts = tem;
+
+      external = Fcdr (external);
+      tem = Fcar (external);
+      state->prev_syntax = NILP (tem) ? Smax : XINT (tem);
+    }
 }
 
 DEFUN ("parse-partial-sexp", Fparse_partial_sexp, Sparse_partial_sexp, 2, 6, 0,
@@ -3488,6 +3534,7 @@ Parsing stops at TO or when certain criteria are met;
  point is set to where parsing stops.
 If fifth arg OLDSTATE is omitted or nil,
  parsing assumes that FROM is the beginning of a function.
+
 Value is a list of elements describing final state of parsing:
  0. depth in parens.
  1. character address of start of innermost containing list; nil if none.
@@ -3501,16 +3548,22 @@ Value is a list of elements describing final state of parsing:
  6. the minimum paren-depth encountered during this scan.
  7. style of comment, if any.
  8. character address of start of comment or string; nil if not in one.
- 9. Intermediate data for continuation of parsing (subject to change).
+ 9. List of positions of currently open parens, outermost first.
+10. When the last position scanned holds the first character of a
+    (potential) two character construct, the syntax of that position,
+    otherwise nil.  That construct can be a two character comment
+    delimiter or an Escaped or Char-quoted character.
+11..... Possible further internal information used by `parse-partial-sexp'.
+
 If third arg TARGETDEPTH is non-nil, parsing stops if the depth
 in parentheses becomes equal to TARGETDEPTH.
-Fourth arg STOPBEFORE non-nil means stop when come to
+Fourth arg STOPBEFORE non-nil means stop when we come to
  any character that starts a sexp.
 Fifth arg OLDSTATE is a list like what this function returns.
  It is used to initialize the state of the parse.  Elements number 1, 2, 6
  are ignored.
-Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
- If it is symbol `syntax-table', stop after the start of a comment or a
+Sixth arg COMMENTSTOP non-nil means stop after the start of a comment.
+ If it is the symbol `syntax-table', stop after the start of a comment or a
  string, or after end of a comment or a string.  */)
   (Lisp_Object from, Lisp_Object to, Lisp_Object targetdepth,
    Lisp_Object stopbefore, Lisp_Object oldstate, Lisp_Object commentstop)
@@ -3527,15 +3580,17 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
     target = TYPE_MINIMUM (EMACS_INT);	/* We won't reach this depth.  */
 
   validate_region (&from, &to);
+  internalize_parse_state (oldstate, &state);
   scan_sexps_forward (&state, XINT (from), CHAR_TO_BYTE (XINT (from)),
 		      XINT (to),
-		      target, !NILP (stopbefore), oldstate,
+		      target, !NILP (stopbefore),
 		      (NILP (commentstop)
 		       ? 0 : (EQ (commentstop, Qsyntax_table) ? -1 : 1)));
 
   SET_PT_BOTH (state.location, state.location_byte);
 
-  return Fcons (make_number (state.depth),
+  return
+    Fcons (make_number (state.depth),
 	   Fcons (state.prevlevelstart < 0
 		  ? Qnil : make_number (state.prevlevelstart),
 	     Fcons (state.thislevelstart < 0
@@ -3553,11 +3608,15 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
 				  ? Qsyntax_table
 				  : make_number (state.comstyle))
 			       : Qnil),
-			      Fcons (((state.incomment
-				       || (state.instring >= 0))
-				      ? make_number (state.comstr_start)
-				      : Qnil),
-				     Fcons (state.levelstarts, Qnil))))))))));
+		         Fcons (((state.incomment
+                                  || (state.instring >= 0))
+                                 ? make_number (state.comstr_start)
+                                 : Qnil),
+			   Fcons (state.levelstarts,
+                             Fcons (state.prev_syntax == Smax
+                                    ? Qnil
+                                    : make_number (state.prev_syntax),
+                                Qnil)))))))))));
 }
 \f
 void



-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Stefan Monnier]

> We also have Scharquote, which scan_sexps_forward handles identically to
> Sescape.

Yes, it's two syntax codes which are 100% equivalent.  An accident of
history I guess.

> I have bad feelings about that.  Is it really worth the risk, just to
> save one cons cell on a list that not that many instances of exist at
> any time?

As you know, I like to take short term risks for long term benefits.

>> But even if we don't re-use element 5, I would actually much prefer to
>> render element 5 redundant.
> OK.  Here's an updated patch which does just that.  Comments would be
> welcome.

I'll take a closer look later, thanks.


        Stefan

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello, Stefan.

On Fri, Mar 18, 2016 at 03:36:40PM -0400, Stefan Monnier wrote:

> > OK.  Here's an updated patch which does just that.  Comments would be
> > welcome.

> I'll take a closer look later, thanks.

I found some problems at ends of comments.  The upshot is that
forw_comment must inform scan_sexps_forward, on a failed search, whether
the last character it scanned is still "syntactically live", or whether
that last character's syntax was "used up" in closing or opening a
comment.  On a successful search, that character's syntax is always
"used up" in closing the comment.

Would you like to see the patch again, or should I just commit it?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Stefan Monnier]

> Would you like to see the patch again, or should I just commit it?

I'd like to hear what John thinks about the idea of re-using "nth 5"
instead of adding a new entry, but other than that, I think it's OK
to commit, thanks.


        Stefan

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Bug fixed.

On Sat, Mar 19, 2016 at 09:30:32PM -0400, Stefan Monnier wrote:

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[John Wiegley]

>>>>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> I'd like to hear what John thinks about the idea of re-using "nth 5" instead
> of adding a new entry, but other than that, I think it's OK to commit,
> thanks.

How long has this stuff been out in the field?  Do you think it's well known
enough that anyone is depending on the earlier behavior of the nth 5 value?  I
have a feeling it's OK to re-use it.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Stefan Monnier]

>> I'd like to hear what John thinks about the idea of re-using "nth 5" instead
>> of adding a new entry, but other than that, I think it's OK to commit,
>> thanks.
> How long has this stuff been out in the field?

Many many years.

> Do you think it's well known enough that anyone is depending on the
> earlier behavior of the nth 5 value?

There are definitely packages which use the (nth 5 ..) value returned
from parse-partial-sexp.  E.g. cperl-mode does:

			    state (parse-partial-sexp pre-B p))
		      (or (nth 3 state)
			  (nth 4 state)
			  (nth 5 state)
			  (error "`%s' inside `%s' BLOCK" A if-string))
as well as

	     (let ((pps (parse-partial-sexp (point) found)))
	       (or (nth 3 pps) (nth 4 pps) (nth 5 pps)))))

and verilog-mode does:

		 (setq state (parse-partial-sexp (point) end-mod-point 0 t nil))
		 (or (> (car state) 0)	; in parens
		     (nth 5 state)		; comment
		     ))

sh-script also uses it, along with perl-mode.

> I have a feeling it's OK to re-use it.

That's also my feeling.  All the uses I've found would be unaffected
(e.g. because they're in modes where there are no 2-char comment
markers, so there is really no change in behavior; or because it's only
used at positions which can't be in the middle of a 2-char comment
marker).
It's a "natural extension" of the previous meaning of "nth 5".

But admittedly, it's hard/impossible to find all uses, so I can't claim
with confidence that it won't break some code somewhere.


        Stefan

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello, John.

On Sun, Apr 03, 2016 at 03:53:02PM -0700, John Wiegley wrote:
> >>>>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> > I'd like to hear what John thinks about the idea of re-using "nth 5" instead
> > of adding a new entry, but other than that, I think it's OK to commit,
> > thanks.

> How long has this stuff been out in the field?  Do you think it's well known
> enough that anyone is depending on the earlier behavior of the nth 5 value?  I
> have a feeling it's OK to re-use it.

My feeling is that it would be better not to change the definition of
the fifth element, but it's not a strong feeling.

One concern I have is that there is code out there which compensates for
the previous inadequate behaviour (I know there is in CC Mode), and it
may be more difficult to switch off this compensation if there isn't an
easy way to distinguish new from old, such as (> (length state) 10).

> -- 
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Stefan Monnier]

> One concern I have is that there is code out there which compensates for
> the previous inadequate behaviour (I know there is in CC Mode), and it
> may be more difficult to switch off this compensation if there isn't an
> easy way to distinguish new from old, such as (> (length state) 10).

I'd be very surprised if other packages went to that trouble, but if
needed you can still distinguish the new from the old with something like:

   (defconst pps-is-new
     (let ((st (make-syntax-table)))
       (modify-syntax-entry ?/ ". 14" st)
       (with-temp-buffer
         (with-syntax-table st
           (insert "/")
           (nth 5 (parse-partial-sexp (point-min) (point-max)))))))


-- Stefan

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello, Stefan.

On Tue, Apr 05, 2016 at 09:50:41AM -0400, Stefan Monnier wrote:
> > One concern I have is that there is code out there which compensates for
> > the previous inadequate behaviour (I know there is in CC Mode), and it
> > may be more difficult to switch off this compensation if there isn't an
> > easy way to distinguish new from old, such as (> (length state) 10).

> I'd be very surprised if other packages went to that trouble, but if
> needed you can still distinguish the new from the old with something like:

>    (defconst pps-is-new
>      (let ((st (make-syntax-table)))
>        (modify-syntax-entry ?/ ". 14" st)
>        (with-temp-buffer
>          (with-syntax-table st
>            (insert "/")
>            (nth 5 (parse-partial-sexp (point-min) (point-max)))))))

It can certainly be done, yes, but that way it can only really be done
at set up time, wherease (> (length state) 10) could be done more or
less at any time.

It was just a small point, really.

> -- Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Stefan Monnier]

> (scan_sexps_forward): Remove a redundant state parameter.  Access all `state'
> information via the address parameter `state'.

Have you taken a look at the performance impact of this part of the change?
I don't expect it will make much difference, but I'm actually wondering
whether it makes things slower or faster.


        Stefan

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello, Stefan.

On Fri, Mar 18, 2016 at 12:27:36PM -0400, Stefan Monnier wrote:
> > (scan_sexps_forward): Remove a redundant state parameter.  Access all `state'
> > information via the address parameter `state'.

> Have you taken a look at the performance impact of this part of the change?
> I don't expect it will make much difference, but I'm actually wondering
> whether it makes things slower or faster.

I didn't give all that much thought to it.  With a "local" state,
state.field will be addressed as a constant offset from the stack frame
base register.  With a "remote" state, state->field will be addressed as
a constant offset from some address register.  Provided the processor
has enough registers available, it shouldn't make a difference.  But on
an architecture with a restricted set of registers (?old 80x86), it might
make things slower if an address register needs to be repeatedly loaded,
or even repeatedly stacked around function calls.

I'm going to try timing it both ways:  (parse-partial-sexp (point-min)
(point-max)) on xdisp.c (what else?):

Code with "->": 0.03793740272521973 seconds.
Code with "." : 0.03828787803649902 seconds.

So, at least on my machine, the "indirect" version is faster, by around
1%.  Not a great difference, but I'm surprised by the way it went.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Stefan Monnier]

> I didn't give all that much thought to it.  With a "local" state,
> state.field will be addressed as a constant offset from the stack frame
> base register.  With a "remote" state, state->field will be addressed as
> a constant offset from some address register.  Provided the processor
> has enough registers available, it shouldn't make a difference.  But on
> an architecture with a restricted set of registers (?old 80x86), it might
> make things slower if an address register needs to be repeatedly loaded,
> or even repeatedly stacked around function calls.

That was my first reaction as well.  But my other self was telling me "I
can't say why, but my gut feeling says that this code is "cleaner"
and should hence be easier to optimize".

> So, at least on my machine, the "indirect" version is faster, by
> around 1%.  Not a great difference, but I'm surprised by the way
> it went.

Thanks for the test.  As expected, it's a wash, but it's good to confirm
that the cleaner version is at least no slower,


        Stefan