Re: [Emacs-diffs] master 9dcf599: Amend parse-partial-sexp correctly to handle two character comment delimiters

* Re: [Emacs-diffs] master 9dcf599: Amend parse-partial-sexp correctly to handle two character comment delimiters
       [not found] ` <E1ahdIL-0001Ul-Uz@vcs.savannah.gnu.org>
@ 2016-03-20 13:47   ` Stefan Monnier
  2016-03-20 14:22     ` Alan Mackenzie
  2016-03-20 14:40     ` Alan Mackenzie
  0 siblings, 2 replies; 4+ messages in thread
From: Stefan Monnier @ 2016-03-20 13:47 UTC (permalink / raw
  To: emacs-devel; +Cc: Alan Mackenzie

What was John's opinion on reusing nth 5?

        Stefan

>>>>> "Alan" == Alan Mackenzie <acm@muc.de> writes:

> branch: master
> commit 9dcf5998935c8aaa846d7585b81f0dcfe1935b3d
> Author: Alan Mackenzie <acm@muc.de>
> Commit: Alan Mackenzie <acm@muc.de>

>     Amend parse-partial-sexp correctly to handle two character comment delimiters

>     Do this by adding a new field to the parser state: the syntax of the last
>     character scanned, should that be the first char of a (potential) two char
>     construct, nil otherwise.
>     This should make the parser state complete.
>     Also document element 9 of the parser state.  Also refactor the code a bit.

>     * src/syntax.c (struct lisp_parse_state): Add a new field.
>     (SYNTAX_FLAGS_COMSTARTEND_FIRST): New function.
>     (internalize_parse_state): New function, extracted from scan_sexps_forward.
>     (back_comment): Call internalize_parse_state.
>     (forw_comment): Return the syntax of the last character scanned to the caller
>     when that character might be the first of a two character construct.
>     (Fforward_comment, scan_lists): New dummy variables, passed to forw_comment.
>     (scan_sexps_forward): Remove a redundant state parameter.  Access all `state'
>     information via the address parameter `state'.  Remove the code which converts
>     from external to internal form of `state'.  Access buffer contents only from
>     `from' onwards.  Reformulate code at the top of the main loop correctly to
>     recognize comment openers when starting in the middle of one.  Call
>     forw_comment with extra argument (for return of syntax value of possible first
>     char of a two char construct).
>     (Fparse_partial_sexp): Document elements 9, 10 of the parser state in the
>     doc string.  Clarify the doc string in general.  Call
>     internalize_parse_state.  Take account of the new elements when consing up the
>     output parser state.

>     * doc/lispref/syntax.texi: (Parser State): Document element 9 and the new
>     element 10.  Minor wording corrections (remove reference to "trivial
>     cases").
>     (Low Level Parsing): Minor corrections.

>     * etc/NEWS: Note new element 10, and documentation of element 9 of parser
>     state.
> ---
>  doc/lispref/syntax.texi |   33 +++--
>  etc/NEWS                |   12 ++
>  src/syntax.c            |  372 ++++++++++++++++++++++++++++-------------------
>  3 files changed, 252 insertions(+), 165 deletions(-)

> diff --git a/doc/lispref/syntax.texi b/doc/lispref/syntax.texi
> index d5a7eba..f81c164 100644
> --- a/doc/lispref/syntax.texi
> +++ b/doc/lispref/syntax.texi
> @@ -791,10 +791,10 @@ Hooks}).
>  @subsection Parser State
>  @cindex parser state

> -  A @dfn{parser state} is a list of ten elements describing the state
> -of the syntactic parser, after it parses the text between a specified
> -starting point and a specified end point in the buffer.  Parsing
> -functions such as @code{syntax-ppss}
> +  A @dfn{parser state} is a list of (currently) eleven elements
> +describing the state of the syntactic parser, after it parses the text
> +between a specified starting point and a specified end point in the
> +buffer.  Parsing functions such as @code{syntax-ppss}
>  @ifnottex
>  (@pxref{Position Parse})
>  @end ifnottex
> @@ -851,15 +851,20 @@ position where the string began.  When outside of strings and comments,
>  this element is @code{nil}.

>  @item
> -Internal data for continuing the parsing.  The meaning of this
> -data is subject to change; it is used if you pass this list
> -as the @var{state} argument to another call.
> +The list of the positions of the currently open parentheses, starting
> +with the outermost.
> +
> +@item
> +When the last buffer position scanned was the (potential) first
> +character of a two character construct (comment delimiter or
> +escaped/char-quoted character pair), the @var{syntax-code}
> +(@pxref{Syntax Table Internals}) of that position.  Otherwise
> +@code{nil}.
>  @end enumerate

>    Elements 1, 2, and 6 are ignored in a state which you pass as an
> -argument to continue parsing, and elements 8 and 9 are used only in
> -trivial cases.  Those elements are mainly used internally by the
> -parser code.
> +argument to continue parsing.  Elements 9 and 10 are mainly used
> +internally by the parser code.

>    One additional piece of useful information is available from a
>  parser state using this function:
> @@ -898,11 +903,11 @@ The depth starts at 0, or at whatever is given in @var{state}.

>  If the fourth argument @var{stop-before} is non-@code{nil}, parsing
>  stops when it comes to any character that starts a sexp.  If
> -@var{stop-comment} is non-@code{nil}, parsing stops when it comes to the
> -start of an unnested comment.  If @var{stop-comment} is the symbol
> +@var{stop-comment} is non-@code{nil}, parsing stops after the start of
> +an unnested comment.  If @var{stop-comment} is the symbol
>  @code{syntax-table}, parsing stops after the start of an unnested
> -comment or a string, or the end of an unnested comment or a string,
> -whichever comes first.
> +comment or a string, or after the end of an unnested comment or a
> +string, whichever comes first.

>  If @var{state} is @code{nil}, @var{start} is assumed to be at the top
>  level of parenthesis structure, such as the beginning of a function
> diff --git a/etc/NEWS b/etc/NEWS
> index d963dee..ea32153 100644
> --- a/etc/NEWS
> +++ b/etc/NEWS
> @@ -175,6 +175,18 @@ a new window when opening man pages when there's already one, use
>          (inhibit-same-window . nil)
>          (mode . Man-mode))))

> ++++
> +** `parse-partial-sexp' state has a new element.  Element 10 is
> +non-nil when the last character scanned might be the first character
> +of a two character construct, i.e. a comment delimiter or escaped
> +character.  Its value is the syntax of that last character.
> +
> ++++
> +** `parse-partial-sexp''s state, element 9, has now been confirmed as
> +permanent and documented, and may be used by Lisp programs.  Its value
> +is a list of currently open parenthesis positions, starting with the
> +outermost parenthesis.
> +
>  \f
>  * Changes in Emacs 25.2 on Non-Free Operating Systems

> diff --git a/src/syntax.c b/src/syntax.c
> index fdcfdfc..ffe0ea5 100644
> --- a/src/syntax.c
> +++ b/src/syntax.c
> @@ -81,6 +81,11 @@ SYNTAX_FLAGS_COMEND_SECOND (int flags)
>    return (flags >> 19) & 1;
>  }
>  static bool
> +SYNTAX_FLAGS_COMSTARTEND_FIRST (int flags)
> +{
> +  return (flags & 0x50000) != 0;
> +}
> +static bool
>  SYNTAX_FLAGS_PREFIX (int flags)
>  {
>    return (flags >> 20) & 1;
> @@ -153,6 +158,10 @@ struct lisp_parse_state
>      ptrdiff_t comstr_start;  /* Position of last comment/string starter.  */
>      Lisp_Object levelstarts; /* Char numbers of starts-of-expression
>  				of levels (starting from outermost).  */
> +    int prev_syntax; /* Syntax of previous position scanned, when
> +                        that position (potentially) holds the first char
> +                        of a 2-char construct, i.e. comment delimiter
> +                        or Sescape, etc.  Smax otherwise. */
>    };
>  \f
>  /* These variables are a cache for finding the start of a defun.
> @@ -176,7 +185,8 @@ static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
>  static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool);
>  static void scan_sexps_forward (struct lisp_parse_state *,
>                                  ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
> -                                bool, Lisp_Object, int);
> +                                bool, int);
> +static void internalize_parse_state (Lisp_Object, struct lisp_parse_state *);
>  static bool in_classes (int, Lisp_Object);
>  static void parse_sexp_propertize (ptrdiff_t charpos);

> @@ -911,10 +921,11 @@ back_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	}
>        do
>  	{
> +          internalize_parse_state (Qnil, &state);
>  	  scan_sexps_forward (&state,
>  			      defun_start, defun_start_byte,
>  			      comment_end, TYPE_MINIMUM (EMACS_INT),
> -			      0, Qnil, 0);
> +			      0, 0);
>  	  defun_start = comment_end;
>  	  if (!adjusted)
>  	    {
> @@ -2310,11 +2321,15 @@ in_classes (int c, Lisp_Object iso_classes)
>     PREV_SYNTAX is the SYNTAX_WITH_FLAGS of the previous character
>       (or 0 If the search cannot start in the middle of a two-character).

> -   If successful, return true and store the charpos of the comment's end
> -   into *CHARPOS_PTR and the corresponding bytepos into *BYTEPOS_PTR.
> -   Else, return false and store the charpos STOP into *CHARPOS_PTR, the
> -   corresponding bytepos into *BYTEPOS_PTR and the current nesting
> -   (as defined for state.incomment) in *INCOMMENT_PTR.
> +   If successful, return true and store the charpos of the comment's
> +   end into *CHARPOS_PTR and the corresponding bytepos into
> +   *BYTEPOS_PTR.  Else, return false and store the charpos STOP into
> +   *CHARPOS_PTR, the corresponding bytepos into *BYTEPOS_PTR and the
> +   current nesting (as defined for state->incomment) in
> +   *INCOMMENT_PTR.  Should the last character scanned in an incomplete
> +   comment be a possible first character of a two character construct,
> +   we store its SYNTAX_WITH_FLAGS into *last_syntax_ptr.  Otherwise,
> +   we store Smax into *last_syntax_ptr.

>     The comment end is the last character of the comment rather than the
>     character just after the comment.
> @@ -2326,7 +2341,7 @@ static bool
>  forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	      EMACS_INT nesting, int style, int prev_syntax,
>  	      ptrdiff_t *charpos_ptr, ptrdiff_t *bytepos_ptr,
> -	      EMACS_INT *incomment_ptr)
> +	      EMACS_INT *incomment_ptr, int *last_syntax_ptr)
>  {
>    register int c, c1;
>    register enum syntaxcode code;
> @@ -2337,7 +2352,8 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>    /* Enter the loop in the middle so that we find
>       a 2-char comment ender if we start in the middle of it.  */
>    syntax = prev_syntax;
> -  if (syntax != 0) goto forw_incomment;
> +  code = syntax & 0xff;
> +  if (syntax != 0 && from < stop) goto forw_incomment;

>    while (1)
>      {
> @@ -2346,6 +2362,12 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	  *incomment_ptr = nesting;
>  	  *charpos_ptr = from;
>  	  *bytepos_ptr = from_byte;
> +          *last_syntax_ptr =
> +            (code == Sescape || code == Scharquote
> +             || SYNTAX_FLAGS_COMEND_FIRST (syntax)
> +             || (nesting > 0
> +                 && SYNTAX_FLAGS_COMSTART_FIRST (syntax)))
> +            ? syntax : Smax ;
>  	  return 0;
>  	}
>        c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
> @@ -2386,7 +2408,9 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	       SYNTAX_FLAGS_COMMENT_NESTED (other_syntax))
>  	      ? nesting > 0 : nesting < 0))
>  	{
> -	  if (--nesting <= 0)
> +	  syntax = Smax;        /* So that "|#" (lisp) can not return
> +                                   the syntax of "#" in *last_syntax_ptr. */
> +          if (--nesting <= 0)
>  	    /* We have encountered a comment end of the same style
>  	       as the comment sequence which began this comment section.  */
>  	    break;
> @@ -2408,6 +2432,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	/* We have encountered a nested comment of the same style
>  	   as the comment sequence which began this comment section.  */
>  	{
> +          syntax = Smax; /* So that "#|#" isn't also a comment ender. */
>  	  INC_BOTH (from, from_byte);
>  	  UPDATE_SYNTAX_TABLE_FORWARD (from);
>  	  nesting++;
> @@ -2415,6 +2440,8 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>      }
>    *charpos_ptr = from;
>    *bytepos_ptr = from_byte;
> +  *last_syntax_ptr = Smax; /* Any syntactic power the last byte had is
> +                              used up. */
>    return 1;
>  }

> @@ -2436,6 +2463,7 @@ between them, return t; otherwise return nil.  */)
>    EMACS_INT count1;
>    ptrdiff_t out_charpos, out_bytepos;
>    EMACS_INT dummy;
> +  int dummy2;

>    CHECK_NUMBER (count);
>    count1 = XINT (count);
> @@ -2499,7 +2527,7 @@ between them, return t; otherwise return nil.  */)
>  	}
>        /* We're at the start of a comment.  */
>        found = forw_comment (from, from_byte, stop, comnested, comstyle, 0,
> -			    &out_charpos, &out_bytepos, &dummy);
> +			    &out_charpos, &out_bytepos, &dummy, &dummy2);
>        from = out_charpos; from_byte = out_bytepos;
>        if (!found)
>  	{
> @@ -2659,6 +2687,7 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
>    ptrdiff_t from_byte;
>    ptrdiff_t out_bytepos, out_charpos;
>    EMACS_INT dummy;
> +  int dummy2;
>    bool multibyte_symbol_p = sexpflag && multibyte_syntax_as_symbol;

>    if (depth > 0) min_depth = 0;
> @@ -2755,7 +2784,8 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
>  	      UPDATE_SYNTAX_TABLE_FORWARD (from);
>  	      found = forw_comment (from, from_byte, stop,
>  				    comnested, comstyle, 0,
> -				    &out_charpos, &out_bytepos, &dummy);
> +				    &out_charpos, &out_bytepos, &dummy,
> +                                    &dummy2);
>  	      from = out_charpos, from_byte = out_bytepos;
>  	      if (!found)
>  		{
> @@ -3119,7 +3149,7 @@ the prefix syntax flag (p).  */)
>  }
>  \f
>  /* Parse forward from FROM / FROM_BYTE to END,
> -   assuming that FROM has state OLDSTATE (nil means FROM is start of function),
> +   assuming that FROM has state STATE,
>     and return a description of the state of the parse at END.
>     If STOPBEFORE, stop at the start of an atom.
>     If COMMENTSTOP is 1, stop at the start of a comment.
> @@ -3127,12 +3157,11 @@ the prefix syntax flag (p).  */)
>     after the beginning of a string, or after the end of a string.  */

>  static void
> -scan_sexps_forward (struct lisp_parse_state *stateptr,
> +scan_sexps_forward (struct lisp_parse_state *state,
>  		    ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t end,
>  		    EMACS_INT targetdepth, bool stopbefore,
> -		    Lisp_Object oldstate, int commentstop)
> +		    int commentstop)
>  {
> -  struct lisp_parse_state state;
>    enum syntaxcode code;
>    int c1;
>    bool comnested;
> @@ -3148,7 +3177,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
>    Lisp_Object tem;
>    ptrdiff_t prev_from;		/* Keep one character before FROM.  */
>    ptrdiff_t prev_from_byte;
> -  int prev_from_syntax;
> +  int prev_from_syntax, prev_prev_from_syntax;
>    bool boundary_stop = commentstop == -1;
>    bool nofence;
>    bool found;
> @@ -3165,6 +3194,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
>  do { prev_from = from;				\
>       prev_from_byte = from_byte; 		\
>       temp = FETCH_CHAR_AS_MULTIBYTE (prev_from_byte);	\
> +     prev_prev_from_syntax = prev_from_syntax;  \
>       prev_from_syntax = SYNTAX_WITH_FLAGS (temp); \
>       INC_BOTH (from, from_byte);		\
>       if (from < end)				\
> @@ -3174,88 +3204,38 @@ do { prev_from = from;				\
>    immediate_quit = 1;
>    QUIT;

> -  if (NILP (oldstate))
> -    {
> -      depth = 0;
> -      state.instring = -1;
> -      state.incomment = 0;
> -      state.comstyle = 0;	/* comment style a by default.  */
> -      state.comstr_start = -1;	/* no comment/string seen.  */
> -    }
> -  else
> -    {
> -      tem = Fcar (oldstate);
> -      if (!NILP (tem))
> -	depth = XINT (tem);
> -      else
> -	depth = 0;
> -
> -      oldstate = Fcdr (oldstate);
> -      oldstate = Fcdr (oldstate);
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      /* Check whether we are inside string_fence-style string: */
> -      state.instring = (!NILP (tem)
> -			? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
> -			: -1);
> -
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      state.incomment = (!NILP (tem)
> -			 ? (INTEGERP (tem) ? XINT (tem) : -1)
> -			 : 0);
> -
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      start_quoted = !NILP (tem);
> +  depth = state->depth;
> +  start_quoted = state->quoted;
> +  prev_prev_from_syntax = Smax;
> +  prev_from_syntax = state->prev_syntax;

> -      /* if the eighth element of the list is nil, we are in comment
> -	 style a.  If it is non-nil, we are in comment style b */
> -      oldstate = Fcdr (oldstate);
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      state.comstyle = (NILP (tem)
> -			? 0
> -			: (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
> -			   ? XINT (tem)
> -			   : ST_COMMENT_STYLE));
> -
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      state.comstr_start =
> -	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      while (!NILP (tem))		/* >= second enclosing sexps.  */
> -	{
> -	  Lisp_Object temhd = Fcar (tem);
> -	  if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
> -	    curlevel->last = XINT (temhd);
> -	  if (++curlevel == endlevel)
> -	    curlevel--; /* error ("Nesting too deep for parser"); */
> -	  curlevel->prev = -1;
> -	  curlevel->last = -1;
> -	  tem = Fcdr (tem);
> -	}
> +  tem = state->levelstarts;
> +  while (!NILP (tem))		/* >= second enclosing sexps.  */
> +    {
> +      Lisp_Object temhd = Fcar (tem);
> +      if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
> +        curlevel->last = XINT (temhd);
> +      if (++curlevel == endlevel)
> +        curlevel--; /* error ("Nesting too deep for parser"); */
> +      curlevel->prev = -1;
> +      curlevel->last = -1;
> +      tem = Fcdr (tem);
>      }
> -  state.quoted = 0;
> -  mindepth = depth;
> -
curlevel-> prev = -1;
curlevel-> last = -1;

> -  SETUP_SYNTAX_TABLE (prev_from, 1);
> -  temp = FETCH_CHAR (prev_from_byte);
> -  prev_from_syntax = SYNTAX_WITH_FLAGS (temp);
> -  UPDATE_SYNTAX_TABLE_FORWARD (from);
> +  state->quoted = 0;
> +  mindepth = depth;
> +
> +  SETUP_SYNTAX_TABLE (from, 1);

>    /* Enter the loop at a place appropriate for initial state.  */

> -  if (state.incomment)
> +  if (state->incomment)
>      goto startincomment;
> -  if (state.instring >= 0)
> +  if (state->instring >= 0)
>      {
> -      nofence = state.instring != ST_STRING_STYLE;
> +      nofence = state->instring != ST_STRING_STYLE;
>        if (start_quoted)
>  	goto startquotedinstring;
>        goto startinstring;
> @@ -3266,11 +3246,8 @@ do { prev_from = from;				\
>    while (from < end)
>      {
>        int syntax;
> -      INC_FROM;
> -      code = prev_from_syntax & 0xff;

> -      if (from < end
> -	  && SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
> +      if (SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
>  	  && (c1 = FETCH_CHAR (from_byte),
>  	      syntax = SYNTAX_WITH_FLAGS (c1),
>  	      SYNTAX_FLAGS_COMSTART_SECOND (syntax)))
> @@ -3280,32 +3257,39 @@ do { prev_from = from;				\
>  	  /* Record the comment style we have entered so that only
>  	     the comment-end sequence of the same style actually
>  	     terminates the comment section.  */
> -	  state.comstyle
> +	  state->comstyle
>  	    = SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax);
>  	  comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax)
>  		       | SYNTAX_FLAGS_COMMENT_NESTED (syntax));
> -	  state.incomment = comnested ? 1 : -1;
> -	  state.comstr_start = prev_from;
> +	  state->incomment = comnested ? 1 : -1;
> +	  state->comstr_start = prev_from;
>  	  INC_FROM;
> +          prev_from_syntax = Smax; /* the syntax has already been
> +                                      "used up". */
>  	  code = Scomment;
>  	}
> -      else if (code == Scomment_fence)
> -	{
> -	  /* Record the comment style we have entered so that only
> -	     the comment-end sequence of the same style actually
> -	     terminates the comment section.  */
> -	  state.comstyle = ST_COMMENT_STYLE;
> -	  state.incomment = -1;
> -	  state.comstr_start = prev_from;
> -	  code = Scomment;
> -	}
> -      else if (code == Scomment)
> -	{
> -	  state.comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
> -	  state.incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
> -			     1 : -1);
> -	  state.comstr_start = prev_from;
> -	}
> +      else
> +        {
> +          INC_FROM;
> +          code = prev_from_syntax & 0xff;
> +          if (code == Scomment_fence)
> +            {
> +              /* Record the comment style we have entered so that only
> +                 the comment-end sequence of the same style actually
> +                 terminates the comment section.  */
> +              state->comstyle = ST_COMMENT_STYLE;
> +              state->incomment = -1;
> +              state->comstr_start = prev_from;
> +              code = Scomment;
> +            }
> +          else if (code == Scomment)
> +            {
> +              state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
> +              state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
> +                                 1 : -1);
> +              state->comstr_start = prev_from;
> +            }
> +        }

>        if (SYNTAX_FLAGS_PREFIX (prev_from_syntax))
>  	continue;
> @@ -3350,26 +3334,28 @@ do { prev_from = from;				\

>  	case Scomment_fence: /* Can't happen because it's handled above.  */
>  	case Scomment:
> -	  if (commentstop || boundary_stop) goto done;
> +          if (commentstop || boundary_stop) goto done;
>  	startincomment:
>  	  /* The (from == BEGV) test was to enter the loop in the middle so
>  	     that we find a 2-char comment ender even if we start in the
>  	     middle of it.  We don't want to do that if we're just at the
>  	     beginning of the comment (think of (*) ... (*)).  */
>  	  found = forw_comment (from, from_byte, end,
> -				state.incomment, state.comstyle,
> -				(from == BEGV || from < state.comstr_start + 3)
> -				? 0 : prev_from_syntax,
> -				&out_charpos, &out_bytepos, &state.incomment);
> +				state->incomment, state->comstyle,
> +				from == BEGV ? 0 : prev_from_syntax,
> +				&out_charpos, &out_bytepos, &state->incomment,
> +                                &prev_from_syntax);
>  	  from = out_charpos; from_byte = out_bytepos;
> -	  /* Beware!  prev_from and friends are invalid now.
> -	     Luckily, the `done' doesn't use them and the INC_FROM
> -	     sets them to a sane value without looking at them. */
> +	  /* Beware!  prev_from and friends (except prev_from_syntax)
> +	     are invalid now.  Luckily, the `done' doesn't use them
> +	     and the INC_FROM sets them to a sane value without
> +	     looking at them. */
>  	  if (!found) goto done;
>  	  INC_FROM;
> -	  state.incomment = 0;
> -	  state.comstyle = 0;	/* reset the comment style */
> -	  if (boundary_stop) goto done;
> +	  state->incomment = 0;
> +	  state->comstyle = 0;	/* reset the comment style */
> +	  prev_from_syntax = Smax; /* For the comment closer */
> +          if (boundary_stop) goto done;
>  	  break;

>  	case Sopen:
> @@ -3396,16 +3382,16 @@ do { prev_from = from;				\

>  	case Sstring:
>  	case Sstring_fence:
> -	  state.comstr_start = from - 1;
> +	  state->comstr_start = from - 1;
>  	  if (stopbefore) goto stop;  /* this arg means stop at sexp start */
curlevel-> last = prev_from;
> -	  state.instring = (code == Sstring
> +	  state->instring = (code == Sstring
>  			    ? (FETCH_CHAR_AS_MULTIBYTE (prev_from_byte))
>  			    : ST_STRING_STYLE);
>  	  if (boundary_stop) goto done;
>  	startinstring:
>  	  {
> -	    nofence = state.instring != ST_STRING_STYLE;
> +	    nofence = state->instring != ST_STRING_STYLE;

>  	    while (1)
>  	      {
> @@ -3419,7 +3405,7 @@ do { prev_from = from;				\
>  		/* Check C_CODE here so that if the char has
>  		   a syntax-table property which says it is NOT
>  		   a string character, it does not end the string.  */
> -		if (nofence && c == state.instring && c_code == Sstring)
> +		if (nofence && c == state->instring && c_code == Sstring)
>  		  break;

>  		switch (c_code)
> @@ -3442,7 +3428,7 @@ do { prev_from = from;				\
>  	      }
>  	  }
>  	string_end:
> -	  state.instring = -1;
> +	  state->instring = -1;
curlevel-> prev = curlevel->last;
>  	  INC_FROM;
>  	  if (boundary_stop) goto done;
> @@ -3461,25 +3447,96 @@ do { prev_from = from;				\
>   stop:   /* Here if stopping before start of sexp. */
>    from = prev_from;    /* We have just fetched the char that starts it; */
>    from_byte = prev_from_byte;
> +  prev_from_syntax = prev_prev_from_syntax;
>    goto done; /* but return the position before it. */

>   endquoted:
> -  state.quoted = 1;
> +  state->quoted = 1;
>   done:
> -  state.depth = depth;
> -  state.mindepth = mindepth;
> -  state.thislevelstart = curlevel->prev;
> -  state.prevlevelstart
> +  state->depth = depth;
> +  state->mindepth = mindepth;
> +  state->thislevelstart = curlevel->prev;
> +  state->prevlevelstart
>      = (curlevel == levelstart) ? -1 : (curlevel - 1)->last;
> -  state.location = from;
> -  state.location_byte = from_byte;
> -  state.levelstarts = Qnil;
> +  state->location = from;
> +  state->location_byte = from_byte;
> +  state->levelstarts = Qnil;
>    while (curlevel > levelstart)
> -    state.levelstarts = Fcons (make_number ((--curlevel)->last),
> -			       state.levelstarts);
> +    state->levelstarts = Fcons (make_number ((--curlevel)->last),
> +                                state->levelstarts);
> +  state->prev_syntax = (SYNTAX_FLAGS_COMSTARTEND_FIRST (prev_from_syntax)
> +                        || state->quoted) ? prev_from_syntax : Smax;
>    immediate_quit = 0;
> +}
> +
> +/* Convert a (lisp) parse state to the internal form used in
> +   scan_sexps_forward.  */
> +static void
> +internalize_parse_state (Lisp_Object external, struct lisp_parse_state *state)
> +{
> +  Lisp_Object tem;
> +
> +  if (NILP (external))
> +    {
> +      state->depth = 0;
> +      state->instring = -1;
> +      state->incomment = 0;
> +      state->quoted = 0;
> +      state->comstyle = 0;	/* comment style a by default.  */
> +      state->comstr_start = -1;	/* no comment/string seen.  */
> +      state->levelstarts = Qnil;
> +      state->prev_syntax = Smax;
> +    }
> +  else
> +    {
> +      tem = Fcar (external);
> +      if (!NILP (tem))
> +	state->depth = XINT (tem);
> +      else
> +	state->depth = 0;
> +
> +      external = Fcdr (external);
> +      external = Fcdr (external);
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      /* Check whether we are inside string_fence-style string: */
> +      state->instring = (!NILP (tem)
> +                         ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
> +                         : -1);
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->incomment = (!NILP (tem)
> +                          ? (INTEGERP (tem) ? XINT (tem) : -1)
> +                          : 0);
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->quoted = !NILP (tem);

> -  *stateptr = state;
> +      /* if the eighth element of the list is nil, we are in comment
> +	 style a.  If it is non-nil, we are in comment style b */
> +      external = Fcdr (external);
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->comstyle = (NILP (tem)
> +                         ? 0
> +                         : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
> +                            ? XINT (tem)
> +                            : ST_COMMENT_STYLE));
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->comstr_start =
> +	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->levelstarts = tem;
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->prev_syntax = NILP (tem) ? Smax : XINT (tem);
> +    }
>  }

>  DEFUN ("parse-partial-sexp", Fparse_partial_sexp, Sparse_partial_sexp, 2, 6, 0,
> @@ -3488,6 +3545,7 @@ Parsing stops at TO or when certain criteria are met;
>   point is set to where parsing stops.
>  If fifth arg OLDSTATE is omitted or nil,
>   parsing assumes that FROM is the beginning of a function.
> +
>  Value is a list of elements describing final state of parsing:
>   0. depth in parens.
>   1. character address of start of innermost containing list; nil if none.
> @@ -3501,16 +3559,22 @@ Value is a list of elements describing final state of parsing:
>   6. the minimum paren-depth encountered during this scan.
>   7. style of comment, if any.
>   8. character address of start of comment or string; nil if not in one.
> - 9. Intermediate data for continuation of parsing (subject to change).
> + 9. List of positions of currently open parens, outermost first.
> +10. When the last position scanned holds the first character of a
> +    (potential) two character construct, the syntax of that position,
> +    otherwise nil.  That construct can be a two character comment
> +    delimiter or an Escaped or Char-quoted character.
> +11..... Possible further internal information used by `parse-partial-sexp'.
> +
>  If third arg TARGETDEPTH is non-nil, parsing stops if the depth
>  in parentheses becomes equal to TARGETDEPTH.
> -Fourth arg STOPBEFORE non-nil means stop when come to
> +Fourth arg STOPBEFORE non-nil means stop when we come to
>   any character that starts a sexp.
>  Fifth arg OLDSTATE is a list like what this function returns.
>   It is used to initialize the state of the parse.  Elements number 1, 2, 6
>   are ignored.
> -Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
> - If it is symbol `syntax-table', stop after the start of a comment or a
> +Sixth arg COMMENTSTOP non-nil means stop after the start of a comment.
> + If it is the symbol `syntax-table', stop after the start of a comment or a
>   string, or after end of a comment or a string.  */)
>    (Lisp_Object from, Lisp_Object to, Lisp_Object targetdepth,
>     Lisp_Object stopbefore, Lisp_Object oldstate, Lisp_Object commentstop)
> @@ -3527,15 +3591,17 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
>      target = TYPE_MINIMUM (EMACS_INT);	/* We won't reach this depth.  */

>    validate_region (&from, &to);
> +  internalize_parse_state (oldstate, &state);
>    scan_sexps_forward (&state, XINT (from), CHAR_TO_BYTE (XINT (from)),
>  		      XINT (to),
> -		      target, !NILP (stopbefore), oldstate,
> +		      target, !NILP (stopbefore),
>  		      (NILP (commentstop)
>  		       ? 0 : (EQ (commentstop, Qsyntax_table) ? -1 : 1)));

>    SET_PT_BOTH (state.location, state.location_byte);

> -  return Fcons (make_number (state.depth),
> +  return
> +    Fcons (make_number (state.depth),
>  	   Fcons (state.prevlevelstart < 0
>  		  ? Qnil : make_number (state.prevlevelstart),
>  	     Fcons (state.thislevelstart < 0
> @@ -3553,11 +3619,15 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
>  				  ? Qsyntax_table
>  				  : make_number (state.comstyle))
>  			       : Qnil),
> -			      Fcons (((state.incomment
> -				       || (state.instring >= 0))
> -				      ? make_number (state.comstr_start)
> -				      : Qnil),
> -				     Fcons (state.levelstarts, Qnil))))))))));
> +		         Fcons (((state.incomment
> +                                  || (state.instring >= 0))
> +                                 ? make_number (state.comstr_start)
> +                                 : Qnil),
> +			   Fcons (state.levelstarts,
> +                             Fcons (state.prev_syntax == Smax
> +                                    ? Qnil
> +                                    : make_number (state.prev_syntax),
> +                                Qnil)))))))))));
>  }
>  \f
>  void

> _______________________________________________
> Emacs-diffs mailing list
> Emacs-diffs@gnu.org
> https://lists.gnu.org/mailman/listinfo/emacs-diffs

^ permalink raw reply	[flat|nested] 4+ messages in thread