bug#24870: 26.0.50; parse-partial-sexp ignores comment-end

[Alan Mackenzie <acm@muc.de>]

Hello again, Noam.

On Tue, Dec 13, 2016 at 11:04:45PM -0500, npostavs@users.sourceforge.net wrote:
> npostavs@users.sourceforge.net writes:


> > I have tracked the issue down to scan_sexps_forward in syntax.c

> Applying this change which reverts part of [1] seems to fix the issue:


> --- i/src/syntax.c
> +++ w/src/syntax.c
> @@ -3192,7 +3192,11 @@ scan_sexps_forward (struct lisp_parse_state *state,

>    while (from < end)
>      {
> -      if (SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
> +      INC_FROM;
> +      code = prev_from_syntax & 0xff;
> +
> +      if (from < end
> +          && SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
>  	  && (c1 = FETCH_CHAR (from_byte),
>  	      syntax = SYNTAX_WITH_FLAGS (c1),
>  	      SYNTAX_FLAGS_COMSTART_SECOND (syntax)))
> @@ -3213,8 +3217,6 @@ scan_sexps_forward (struct lisp_parse_state *state,
>  	}
>        else
>          {
> -          INC_FROM;
> -          code = prev_from_syntax & 0xff;
>            if (code == Scomment_fence)
>              {
>                /* Record the comment style we have entered so that only

Alas, that patch won't do.  The very first thing that must be done in the
loop is to check for a two-character comment delimiter, of which the
first character might have been recorded in OLDSTATE element 9 rather
than having been scanned by scan_sexps_forward on the previous loop
iteration.

My analysis of what's happening in the bug recipe you posted one or two
posts previously, here in scan_sexps_forward is as follows.  (In that
recipe, "{-C-}\nX" was parse-partial-sexp'd over, and the syntax table
had been set to recognise "{-" and "-}" as matching comment delimiters.)
(i) On the first iteration of the main loop, the "{" is read.  It is
  recognised as an opening paren, and causes the "current paren depth" to
  be incremented.
(ii) On the second iteration of the loop, the "-" is read.  The function
  now recognises the two-character comment opener, and proceeds to read
  the innards of the comment together with its closing delimiter (the
  "-}").
(iii) On the third and fourth iterations, the function reads "\n" and
  "X".  It then terminates, having reached point-max.
(iv) The paren depth counter remains at 1.

What is new here is characters with paren syntax also being components of
2-char comment delimiters.  I recently fixed a similar problem when
characters with word syntax were also flagged as 2-char comment delimiter
parts.  I think a similar patch at case label Sopen: (Line ~3322), which
would peek ahead at the next character to check for "{-" before
recognising the "{" as an open paren would be the best fix.

Do you want to make this fix, or should I do it?  If you want to do it,
I'm willing (indeed, eager) to review it for you.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).