* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
@ 2016-03-15 9:13 Alan Mackenzie
2016-03-15 9:35 ` Andreas Röhler
2016-03-17 12:58 ` Stefan Monnier
0 siblings, 2 replies; 23+ messages in thread
From: Alan Mackenzie @ 2016-03-15 9:13 UTC (permalink / raw)
To: 23019
Hello, Emacs.
When parse-partial-sexp finishes a parse, it fails to record whether or
not its end point is just after the first character of a two character
comment starter or ender. When the resulting state is used as an
argument to resume the parse, p-p-s will be unaware that the comment has
started or ended and produce false results.
Proposed solution: Add an extra element to the parser state, recording the
syntax of the last character passed over before the end of the parse.
This would be used by parse-partial-sexp to initialise its parse.
Also: the existing element 9 (the list of currently open parens) and the
new element should be explicitly documented in the Elisp manual, together
with a statement that there may be further elements in the parse state
used internally by parse-partial-sexp (for future expansion).
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-15 9:13 bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance Alan Mackenzie
@ 2016-03-15 9:35 ` Andreas Röhler
2016-03-15 10:15 ` Alan Mackenzie
2016-03-17 12:58 ` Stefan Monnier
1 sibling, 1 reply; 23+ messages in thread
From: Andreas Röhler @ 2016-03-15 9:35 UTC (permalink / raw)
To: 23019; +Cc: Alan Mackenzie
On 15.03.2016 10:13, Alan Mackenzie wrote:
> Hello, Emacs.
>
> When parse-partial-sexp finishes a parse, it fails to record whether or
> not its end point is just after the first character of a two character
> comment starter or ender. When the resulting state is used as an
> argument to resume the parse, p-p-s will be unaware that the comment has
> started or ended and produce false results.
>
> Proposed solution: Add an extra element to the parser state, recording the
> syntax of the last character passed over before the end of the parse.
> This would be used by parse-partial-sexp to initialise its parse.
>
> Also: the existing element 9 (the list of currently open parens) and the
> new element should be explicitly documented in the Elisp manual, together
> with a statement that there may be further elements in the parse state
> used internally by parse-partial-sexp (for future expansion).
>
Hi Alan,
a comment start might be composed not just by two characters, but by
three or more. What then?
Cheers,
Andreas
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-15 9:35 ` Andreas Röhler
@ 2016-03-15 10:15 ` Alan Mackenzie
2016-03-15 13:38 ` Andreas Röhler
0 siblings, 1 reply; 23+ messages in thread
From: Alan Mackenzie @ 2016-03-15 10:15 UTC (permalink / raw)
To: Andreas Röhler; +Cc: 23019
Hello, Andreas.
On Tue, Mar 15, 2016 at 10:35:08AM +0100, Andreas Röhler wrote:
> On 15.03.2016 10:13, Alan Mackenzie wrote:
> > Hello, Emacs.
> > When parse-partial-sexp finishes a parse, it fails to record whether or
> > not its end point is just after the first character of a two character
> > comment starter or ender. When the resulting state is used as an
> > argument to resume the parse, p-p-s will be unaware that the comment has
> > started or ended and produce false results.
> > Proposed solution: Add an extra element to the parser state, recording the
> > syntax of the last character passed over before the end of the parse.
> > This would be used by parse-partial-sexp to initialise its parse.
> > Also: the existing element 9 (the list of currently open parens) and the
> > new element should be explicitly documented in the Elisp manual, together
> > with a statement that there may be further elements in the parse state
> > used internally by parse-partial-sexp (for future expansion).
> a comment start might be composed not just by two characters, but by
> three or more. What then?
We'd have to start thinking about extending parse-partial-sexp, or
invent some workaround. Maybe. There must be some languages (?html)
where this is the case. What is done in these?
> Cheers,
> Andreas
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-15 10:15 ` Alan Mackenzie
@ 2016-03-15 13:38 ` Andreas Röhler
0 siblings, 0 replies; 23+ messages in thread
From: Andreas Röhler @ 2016-03-15 13:38 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: 23019
On 15.03.2016 11:15, Alan Mackenzie wrote:
> Hello, Andreas.
>
> On Tue, Mar 15, 2016 at 10:35:08AM +0100, Andreas Röhler wrote:
>
>
>> On 15.03.2016 10:13, Alan Mackenzie wrote:
>>> Hello, Emacs.
>>> When parse-partial-sexp finishes a parse, it fails to record whether or
>>> not its end point is just after the first character of a two character
>>> comment starter or ender. When the resulting state is used as an
>>> argument to resume the parse, p-p-s will be unaware that the comment has
>>> started or ended and produce false results.
>>> Proposed solution: Add an extra element to the parser state, recording the
>>> syntax of the last character passed over before the end of the parse.
>>> This would be used by parse-partial-sexp to initialise its parse.
>>> Also: the existing element 9 (the list of currently open parens) and the
>>> new element should be explicitly documented in the Elisp manual, together
>>> with a statement that there may be further elements in the parse state
>>> used internally by parse-partial-sexp (for future expansion).
>
>> a comment start might be composed not just by two characters, but by
>> three or more. What then?
> We'd have to start thinking about extending parse-partial-sexp, or
> invent some workaround. Maybe. There must be some languages (?html)
> where this is the case. What is done in these?
May you send me this (or more) example use-cases? Couldn't find the one
already given, sorry.
Addressed this issue in my generic beg-end.el
https://github.com/andreas-roehler/werkstatt/blob/master/subroutines/beg-end.el
In case beg-end forms used a start-string, look if the char-at-point
would match this string.
Then look if the char-before is before in string, etc.
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-15 9:13 bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance Alan Mackenzie
2016-03-15 9:35 ` Andreas Röhler
@ 2016-03-17 12:58 ` Stefan Monnier
2016-03-17 21:49 ` Alan Mackenzie
1 sibling, 1 reply; 23+ messages in thread
From: Stefan Monnier @ 2016-03-17 12:58 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: 23019
> Proposed solution: Add an extra element to the parser state, recording the
> syntax of the last character passed over before the end of the parse.
> This would be used by parse-partial-sexp to initialise its parse.
Another option is to record "the start of current element" (in case we
were in the middle of an element). This could potentially reuse (nth
5 ppss) by generalizing it, or it could use a new entry.
The choice probably doesn't matter much and will probably be more
a question of "what's easier to implement".
> Also: the existing element 9 (the list of currently open parens) and the
> new element should be explicitly documented in the Elisp manual, together
> with a statement that there may be further elements in the parse state
> used internally by parse-partial-sexp (for future expansion).
Indeed.
Andreas Röhler added:
> a comment start might be composed not just by two characters, but by three
> or more. What then?
Andreas, I suggest that you go back and take a closer look at
parse-partial-sexp, syntax-ppss, and syntax-tables in general because
lately you've made several comments like the one here which show you're
just not familiar with the topic at all. Syntax tables do not support
comment markers longer than 2 characters (currently). Emacs supports
those via the `syntax-table' text-property only (which typically marks
the first char of each "long comment starter" as being "the comment
starter").
Stefan
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-17 12:58 ` Stefan Monnier
@ 2016-03-17 21:49 ` Alan Mackenzie
2016-03-18 4:49 ` Stefan Monnier
2016-03-18 16:27 ` Stefan Monnier
0 siblings, 2 replies; 23+ messages in thread
From: Alan Mackenzie @ 2016-03-17 21:49 UTC (permalink / raw)
To: Stefan Monnier; +Cc: 23019
On Thu, Mar 17, 2016 at 08:58:27AM -0400, Stefan Monnier wrote:
> > Proposed solution: Add an extra element to the parser state, recording the
> > syntax of the last character passed over before the end of the parse.
> > This would be used by parse-partial-sexp to initialise its parse.
> Another option is to record "the start of current element" (in case we
> were in the middle of an element). This could potentially reuse (nth
> 5 ppss) by generalizing it, or it could use a new entry.
> The choice probably doesn't matter much and will probably be more
> a question of "what's easier to implement".
> > Also: the existing element 9 (the list of currently open parens) and the
> > new element should be explicitly documented in the Elisp manual, together
> > with a statement that there may be further elements in the parse state
> > used internally by parse-partial-sexp (for future expansion).
> Indeed.
OK, I've got a patch ready. It's bigger than anticipated, purely
because it also does some refactoring. It actually adds two elements to
the parser state, and I believe that makes the parser state complete.
Here's the patch:
Enhance parse-partial-sexp correctly to handle two character commit delimiters
Do this by adding two new fields to the parser state: the syntax of the last
character scanned, and the last end of comment scanned. This should make the
parser state complete.
Also document element 9 of the parser state. Also refactor the code a bit.
* src/syntax.c (struct lisp_parse_state): Add two new fields.
(internalize_parse_state): New function, extracted from scan_sexps_forward.
(back_comment): Call internalize_parse_state.
(forw_comment): Return the syntax of the last character scanned to the caller.
(Fforward_comment, scan_lists): New dummy variables, passed to forw_comment.
(scan_sexps_forward): Remove a redundant state parameter. Access all `state'
information via the address parameter `state'. Remove the code which converts
from external to internal form of `state'. Access buffer contents only from
`from' onwards. Reformulate code at the top of the main loop correctly to
recognize comment openers when starting in the middle of one. Call
forw_comment with extra argument (for return of final syntax value).
(Fparse_partial_sexp): Document elements 9, 10, 11 of the parser state in the
doc string. Clarify the doc string in general. Call
internalize_parse_state. Take account of the new elements when consing up the
output parser state.
* doc/lispref/syntax.texi: (Parser State): Document element 9 and the new
elements 10 and 11. Minor wording corrections (remove reference to "trivial
cases").
(Low Level Parsing): Minor corrections
diff --git a/doc/lispref/syntax.texi b/doc/lispref/syntax.texi
index d5a7eba..67a00d7 100644
--- a/doc/lispref/syntax.texi
+++ b/doc/lispref/syntax.texi
@@ -791,10 +791,10 @@ Parser State
@subsection Parser State
@cindex parser state
- A @dfn{parser state} is a list of ten elements describing the state
-of the syntactic parser, after it parses the text between a specified
-starting point and a specified end point in the buffer. Parsing
-functions such as @code{syntax-ppss}
+ A @dfn{parser state} is a list of (currently) twelve elements
+describing the state of the syntactic parser, after it parses the text
+between a specified starting point and a specified end point in the
+buffer. Parsing functions such as @code{syntax-ppss}
@ifnottex
(@pxref{Position Parse})
@end ifnottex
@@ -851,15 +851,21 @@ Parser State
this element is @code{nil}.
@item
-Internal data for continuing the parsing. The meaning of this
-data is subject to change; it is used if you pass this list
-as the @var{state} argument to another call.
+The list of the positions of the currently open parentheses, starting
+with the outermost.
+
+@item
+The @var{syntax-code} (@pxref{Syntax Table Internals}) of the last
+buffer position scanned, or @code{nil} if no scanning has happened.
+
+@item
+The position after the previous end of comment, or @code{nil} if the
+scanning has not passed a comment end.
@end enumerate
Elements 1, 2, and 6 are ignored in a state which you pass as an
-argument to continue parsing, and elements 8 and 9 are used only in
-trivial cases. Those elements are mainly used internally by the
-parser code.
+argument to continue parsing. Elements 9 to 11 are mainly used
+internally by the parser code.
One additional piece of useful information is available from a
parser state using this function:
@@ -898,11 +904,11 @@ Low-Level Parsing
If the fourth argument @var{stop-before} is non-@code{nil}, parsing
stops when it comes to any character that starts a sexp. If
-@var{stop-comment} is non-@code{nil}, parsing stops when it comes to the
-start of an unnested comment. If @var{stop-comment} is the symbol
+@var{stop-comment} is non-@code{nil}, parsing stops after the start of
+an unnested comment. If @var{stop-comment} is the symbol
@code{syntax-table}, parsing stops after the start of an unnested
-comment or a string, or the end of an unnested comment or a string,
-whichever comes first.
+comment or a string, or after the end of an unnested comment or a
+string, whichever comes first.
If @var{state} is @code{nil}, @var{start} is assumed to be at the top
level of parenthesis structure, such as the beginning of a function
diff --git a/src/syntax.c b/src/syntax.c
index 249d0d5..11b1ff0 100644
--- a/src/syntax.c
+++ b/src/syntax.c
@@ -153,6 +153,9 @@ struct lisp_parse_state
ptrdiff_t comstr_start; /* Position of last comment/string starter. */
Lisp_Object levelstarts; /* Char numbers of starts-of-expression
of levels (starting from outermost). */
+ int prev_syntax; /* Syntax of previous character scanned, or Smax. */
+ ptrdiff_t prev_comment_end; /* Position after end of last closed
+ comment, or -1. */
};
\f
/* These variables are a cache for finding the start of a defun.
@@ -176,7 +179,8 @@ static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool);
static void scan_sexps_forward (struct lisp_parse_state *,
ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
- bool, Lisp_Object, int);
+ bool, int);
+static void internalize_parse_state (Lisp_Object, struct lisp_parse_state *);
static bool in_classes (int, Lisp_Object);
static void parse_sexp_propertize (ptrdiff_t charpos);
@@ -911,10 +915,11 @@ back_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
}
do
{
+ internalize_parse_state (Qnil, &state);
scan_sexps_forward (&state,
defun_start, defun_start_byte,
comment_end, TYPE_MINIMUM (EMACS_INT),
- 0, Qnil, 0);
+ 0, 0);
defun_start = comment_end;
if (!adjusted)
{
@@ -2314,7 +2319,9 @@ in_classes (int c, Lisp_Object iso_classes)
into *CHARPOS_PTR and the corresponding bytepos into *BYTEPOS_PTR.
Else, return false and store the charpos STOP into *CHARPOS_PTR, the
corresponding bytepos into *BYTEPOS_PTR and the current nesting
- (as defined for state.incomment) in *INCOMMENT_PTR.
+ (as defined for state->incomment) in *INCOMMENT_PTR. The
+ SYNTAX_WITH_FLAGS of the last character scanned in the comment is
+ stored into *last_syntax_ptr.
The comment end is the last character of the comment rather than the
character just after the comment.
@@ -2326,7 +2333,7 @@ static bool
forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
EMACS_INT nesting, int style, int prev_syntax,
ptrdiff_t *charpos_ptr, ptrdiff_t *bytepos_ptr,
- EMACS_INT *incomment_ptr)
+ EMACS_INT *incomment_ptr, int *last_syntax_ptr)
{
register int c, c1;
register enum syntaxcode code;
@@ -2346,6 +2353,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
*incomment_ptr = nesting;
*charpos_ptr = from;
*bytepos_ptr = from_byte;
+ *last_syntax_ptr = syntax;
return 0;
}
c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
@@ -2415,6 +2423,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
}
*charpos_ptr = from;
*bytepos_ptr = from_byte;
+ *last_syntax_ptr = syntax;
return 1;
}
@@ -2436,6 +2445,7 @@ between them, return t; otherwise return nil. */)
EMACS_INT count1;
ptrdiff_t out_charpos, out_bytepos;
EMACS_INT dummy;
+ int dummy2;
CHECK_NUMBER (count);
count1 = XINT (count);
@@ -2499,7 +2509,7 @@ between them, return t; otherwise return nil. */)
}
/* We're at the start of a comment. */
found = forw_comment (from, from_byte, stop, comnested, comstyle, 0,
- &out_charpos, &out_bytepos, &dummy);
+ &out_charpos, &out_bytepos, &dummy, &dummy2);
from = out_charpos; from_byte = out_bytepos;
if (!found)
{
@@ -2659,6 +2669,7 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
ptrdiff_t from_byte;
ptrdiff_t out_bytepos, out_charpos;
EMACS_INT dummy;
+ int dummy2;
bool multibyte_symbol_p = sexpflag && multibyte_syntax_as_symbol;
if (depth > 0) min_depth = 0;
@@ -2755,7 +2766,8 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
UPDATE_SYNTAX_TABLE_FORWARD (from);
found = forw_comment (from, from_byte, stop,
comnested, comstyle, 0,
- &out_charpos, &out_bytepos, &dummy);
+ &out_charpos, &out_bytepos, &dummy,
+ &dummy2);
from = out_charpos, from_byte = out_bytepos;
if (!found)
{
@@ -3119,7 +3131,7 @@ the prefix syntax flag (p). */)
}
\f
/* Parse forward from FROM / FROM_BYTE to END,
- assuming that FROM has state OLDSTATE (nil means FROM is start of function),
+ assuming that FROM has state STATE (nil means FROM is start of function),
and return a description of the state of the parse at END.
If STOPBEFORE, stop at the start of an atom.
If COMMENTSTOP is 1, stop at the start of a comment.
@@ -3127,12 +3139,11 @@ the prefix syntax flag (p). */)
after the beginning of a string, or after the end of a string. */
static void
-scan_sexps_forward (struct lisp_parse_state *stateptr,
+scan_sexps_forward (struct lisp_parse_state *state,
ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t end,
EMACS_INT targetdepth, bool stopbefore,
- Lisp_Object oldstate, int commentstop)
+ int commentstop)
{
- struct lisp_parse_state state;
enum syntaxcode code;
int c1;
bool comnested;
@@ -3148,7 +3159,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
Lisp_Object tem;
ptrdiff_t prev_from; /* Keep one character before FROM. */
ptrdiff_t prev_from_byte;
- int prev_from_syntax;
+ int prev_from_syntax, prev_prev_from_syntax;
bool boundary_stop = commentstop == -1;
bool nofence;
bool found;
@@ -3165,6 +3176,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
do { prev_from = from; \
prev_from_byte = from_byte; \
temp = FETCH_CHAR_AS_MULTIBYTE (prev_from_byte); \
+ prev_prev_from_syntax = prev_from_syntax; \
prev_from_syntax = SYNTAX_WITH_FLAGS (temp); \
INC_BOTH (from, from_byte); \
if (from < end) \
@@ -3174,88 +3186,38 @@ do { prev_from = from; \
immediate_quit = 1;
QUIT;
- if (NILP (oldstate))
- {
- depth = 0;
- state.instring = -1;
- state.incomment = 0;
- state.comstyle = 0; /* comment style a by default. */
- state.comstr_start = -1; /* no comment/string seen. */
- }
- else
- {
- tem = Fcar (oldstate);
- if (!NILP (tem))
- depth = XINT (tem);
- else
- depth = 0;
-
- oldstate = Fcdr (oldstate);
- oldstate = Fcdr (oldstate);
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- /* Check whether we are inside string_fence-style string: */
- state.instring = (!NILP (tem)
- ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
- : -1);
-
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- state.incomment = (!NILP (tem)
- ? (INTEGERP (tem) ? XINT (tem) : -1)
- : 0);
-
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- start_quoted = !NILP (tem);
+ depth = state->depth;
+ start_quoted = state->quoted;
+ prev_prev_from_syntax = Smax;
+ prev_from_syntax = state->prev_syntax;
- /* if the eighth element of the list is nil, we are in comment
- style a. If it is non-nil, we are in comment style b */
- oldstate = Fcdr (oldstate);
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- state.comstyle = (NILP (tem)
- ? 0
- : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
- ? XINT (tem)
- : ST_COMMENT_STYLE));
-
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- state.comstr_start =
- RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- while (!NILP (tem)) /* >= second enclosing sexps. */
- {
- Lisp_Object temhd = Fcar (tem);
- if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
- curlevel->last = XINT (temhd);
- if (++curlevel == endlevel)
- curlevel--; /* error ("Nesting too deep for parser"); */
- curlevel->prev = -1;
- curlevel->last = -1;
- tem = Fcdr (tem);
- }
+ tem = state->levelstarts;
+ while (!NILP (tem)) /* >= second enclosing sexps. */
+ {
+ Lisp_Object temhd = Fcar (tem);
+ if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
+ curlevel->last = XINT (temhd);
+ if (++curlevel == endlevel)
+ curlevel--; /* error ("Nesting too deep for parser"); */
+ curlevel->prev = -1;
+ curlevel->last = -1;
+ tem = Fcdr (tem);
}
- state.quoted = 0;
- mindepth = depth;
-
curlevel->prev = -1;
curlevel->last = -1;
- SETUP_SYNTAX_TABLE (prev_from, 1);
- temp = FETCH_CHAR (prev_from_byte);
- prev_from_syntax = SYNTAX_WITH_FLAGS (temp);
- UPDATE_SYNTAX_TABLE_FORWARD (from);
+ state->quoted = 0;
+ mindepth = depth;
+
+ SETUP_SYNTAX_TABLE (from, 1);
/* Enter the loop at a place appropriate for initial state. */
- if (state.incomment)
+ if (state->incomment)
goto startincomment;
- if (state.instring >= 0)
+ if (state->instring >= 0)
{
- nofence = state.instring != ST_STRING_STYLE;
+ nofence = state->instring != ST_STRING_STYLE;
if (start_quoted)
goto startquotedinstring;
goto startinstring;
@@ -3266,10 +3228,10 @@ do { prev_from = from; \
while (from < end)
{
int syntax;
- INC_FROM;
- code = prev_from_syntax & 0xff;
if (from < end
+ && (state->prev_comment_end == -1
+ || prev_from >= state->prev_comment_end)
&& SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
&& (c1 = FETCH_CHAR (from_byte),
syntax = SYNTAX_WITH_FLAGS (c1),
@@ -3280,32 +3242,37 @@ do { prev_from = from; \
/* Record the comment style we have entered so that only
the comment-end sequence of the same style actually
terminates the comment section. */
- state.comstyle
+ state->comstyle
= SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax);
comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax)
| SYNTAX_FLAGS_COMMENT_NESTED (syntax));
- state.incomment = comnested ? 1 : -1;
- state.comstr_start = prev_from;
+ state->incomment = comnested ? 1 : -1;
+ state->comstr_start = prev_from;
INC_FROM;
code = Scomment;
}
- else if (code == Scomment_fence)
- {
- /* Record the comment style we have entered so that only
- the comment-end sequence of the same style actually
- terminates the comment section. */
- state.comstyle = ST_COMMENT_STYLE;
- state.incomment = -1;
- state.comstr_start = prev_from;
- code = Scomment;
- }
- else if (code == Scomment)
- {
- state.comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
- state.incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
- 1 : -1);
- state.comstr_start = prev_from;
- }
+ else
+ {
+ INC_FROM;
+ code = prev_from_syntax & 0xff;
+ if (code == Scomment_fence)
+ {
+ /* Record the comment style we have entered so that only
+ the comment-end sequence of the same style actually
+ terminates the comment section. */
+ state->comstyle = ST_COMMENT_STYLE;
+ state->incomment = -1;
+ state->comstr_start = prev_from;
+ code = Scomment;
+ }
+ else if (code == Scomment)
+ {
+ state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
+ state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
+ 1 : -1);
+ state->comstr_start = prev_from;
+ }
+ }
if (SYNTAX_FLAGS_PREFIX (prev_from_syntax))
continue;
@@ -3357,18 +3324,21 @@ do { prev_from = from; \
middle of it. We don't want to do that if we're just at the
beginning of the comment (think of (*) ... (*)). */
found = forw_comment (from, from_byte, end,
- state.incomment, state.comstyle,
- (from == BEGV || from < state.comstr_start + 3)
+ state->incomment, state->comstyle,
+ (from == BEGV || from < state->comstr_start + 3)
? 0 : prev_from_syntax,
- &out_charpos, &out_bytepos, &state.incomment);
+ &out_charpos, &out_bytepos, &state->incomment,
+ &prev_from_syntax);
from = out_charpos; from_byte = out_bytepos;
- /* Beware! prev_from and friends are invalid now.
- Luckily, the `done' doesn't use them and the INC_FROM
- sets them to a sane value without looking at them. */
+ /* Beware! prev_from and friends (except prev_from_syntax)
+ are invalid now. Luckily, the `done' doesn't use them
+ and the INC_FROM sets them to a sane value without
+ looking at them. */
if (!found) goto done;
INC_FROM;
- state.incomment = 0;
- state.comstyle = 0; /* reset the comment style */
+ state->incomment = 0;
+ state->comstyle = 0; /* reset the comment style */
+ state->prev_comment_end = from;
if (boundary_stop) goto done;
break;
@@ -3396,16 +3366,16 @@ do { prev_from = from; \
case Sstring:
case Sstring_fence:
- state.comstr_start = from - 1;
+ state->comstr_start = from - 1;
if (stopbefore) goto stop; /* this arg means stop at sexp start */
curlevel->last = prev_from;
- state.instring = (code == Sstring
+ state->instring = (code == Sstring
? (FETCH_CHAR_AS_MULTIBYTE (prev_from_byte))
: ST_STRING_STYLE);
if (boundary_stop) goto done;
startinstring:
{
- nofence = state.instring != ST_STRING_STYLE;
+ nofence = state->instring != ST_STRING_STYLE;
while (1)
{
@@ -3419,7 +3389,7 @@ do { prev_from = from; \
/* Check C_CODE here so that if the char has
a syntax-table property which says it is NOT
a string character, it does not end the string. */
- if (nofence && c == state.instring && c_code == Sstring)
+ if (nofence && c == state->instring && c_code == Sstring)
break;
switch (c_code)
@@ -3442,7 +3412,7 @@ do { prev_from = from; \
}
}
string_end:
- state.instring = -1;
+ state->instring = -1;
curlevel->prev = curlevel->last;
INC_FROM;
if (boundary_stop) goto done;
@@ -3461,25 +3431,99 @@ do { prev_from = from; \
stop: /* Here if stopping before start of sexp. */
from = prev_from; /* We have just fetched the char that starts it; */
from_byte = prev_from_byte;
+ prev_from_syntax = prev_prev_from_syntax;
goto done; /* but return the position before it. */
endquoted:
- state.quoted = 1;
+ state->quoted = 1;
done:
- state.depth = depth;
- state.mindepth = mindepth;
- state.thislevelstart = curlevel->prev;
- state.prevlevelstart
+ state->depth = depth;
+ state->mindepth = mindepth;
+ state->thislevelstart = curlevel->prev;
+ state->prevlevelstart
= (curlevel == levelstart) ? -1 : (curlevel - 1)->last;
- state.location = from;
- state.location_byte = from_byte;
- state.levelstarts = Qnil;
+ state->location = from;
+ state->location_byte = from_byte;
+ state->levelstarts = Qnil;
while (curlevel > levelstart)
- state.levelstarts = Fcons (make_number ((--curlevel)->last),
- state.levelstarts);
+ state->levelstarts = Fcons (make_number ((--curlevel)->last),
+ state->levelstarts);
+ state->prev_syntax = prev_from_syntax;
immediate_quit = 0;
+}
+
+/* Convert a (lisp) parse state to the internal form used in
+ scan_sexps_forward. */
+static void
+internalize_parse_state (Lisp_Object external, struct lisp_parse_state *state)
+{
+ Lisp_Object tem;
+
+ if (NILP (external))
+ {
+ state->depth = 0;
+ state->instring = -1;
+ state->incomment = 0;
+ state->quoted = 0;
+ state->comstyle = 0; /* comment style a by default. */
+ state->comstr_start = -1; /* no comment/string seen. */
+ state->levelstarts = Qnil;
+ state->prev_syntax = Smax;
+ state->prev_comment_end = -1;
+ }
+ else
+ {
+ tem = Fcar (external);
+ if (!NILP (tem))
+ state->depth = XINT (tem);
+ else
+ state->depth = 0;
+
+ external = Fcdr (external);
+ external = Fcdr (external);
+ external = Fcdr (external);
+ tem = Fcar (external);
+ /* Check whether we are inside string_fence-style string: */
+ state->instring = (!NILP (tem)
+ ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
+ : -1);
+
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->incomment = (!NILP (tem)
+ ? (INTEGERP (tem) ? XINT (tem) : -1)
+ : 0);
- *stateptr = state;
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->quoted = !NILP (tem);
+
+ /* if the eighth element of the list is nil, we are in comment
+ style a. If it is non-nil, we are in comment style b */
+ external = Fcdr (external);
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->comstyle = (NILP (tem)
+ ? 0
+ : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
+ ? XINT (tem)
+ : ST_COMMENT_STYLE));
+
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->comstr_start =
+ RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->levelstarts = tem;
+
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->prev_syntax = NILP (tem) ? Smax : XINT (tem);
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->prev_comment_end = NILP (tem) ? -1 : XINT (tem);
+ }
}
DEFUN ("parse-partial-sexp", Fparse_partial_sexp, Sparse_partial_sexp, 2, 6, 0,
@@ -3488,6 +3532,7 @@ Parsing stops at TO or when certain criteria are met;
point is set to where parsing stops.
If fifth arg OLDSTATE is omitted or nil,
parsing assumes that FROM is the beginning of a function.
+
Value is a list of elements describing final state of parsing:
0. depth in parens.
1. character address of start of innermost containing list; nil if none.
@@ -3501,16 +3546,20 @@ Value is a list of elements describing final state of parsing:
6. the minimum paren-depth encountered during this scan.
7. style of comment, if any.
8. character address of start of comment or string; nil if not in one.
- 9. Intermediate data for continuation of parsing (subject to change).
+ 9. List of positions of currently open parens, outermost first.
+10. Syntax of last character scanned, or nil if no scanning has happened.
+11. Position after end of previous comment scanned, or nil.
+12..... Possible further internal information used by `parse-partial-sexp'.
+
If third arg TARGETDEPTH is non-nil, parsing stops if the depth
in parentheses becomes equal to TARGETDEPTH.
-Fourth arg STOPBEFORE non-nil means stop when come to
+Fourth arg STOPBEFORE non-nil means stop when we come to
any character that starts a sexp.
Fifth arg OLDSTATE is a list like what this function returns.
It is used to initialize the state of the parse. Elements number 1, 2, 6
are ignored.
-Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
- If it is symbol `syntax-table', stop after the start of a comment or a
+Sixth arg COMMENTSTOP non-nil means stop after the start of a comment.
+ If it is the symbol `syntax-table', stop after the start of a comment or a
string, or after end of a comment or a string. */)
(Lisp_Object from, Lisp_Object to, Lisp_Object targetdepth,
Lisp_Object stopbefore, Lisp_Object oldstate, Lisp_Object commentstop)
@@ -3527,15 +3576,17 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
target = TYPE_MINIMUM (EMACS_INT); /* We won't reach this depth. */
validate_region (&from, &to);
+ internalize_parse_state (oldstate, &state);
scan_sexps_forward (&state, XINT (from), CHAR_TO_BYTE (XINT (from)),
XINT (to),
- target, !NILP (stopbefore), oldstate,
+ target, !NILP (stopbefore),
(NILP (commentstop)
? 0 : (EQ (commentstop, Qsyntax_table) ? -1 : 1)));
SET_PT_BOTH (state.location, state.location_byte);
- return Fcons (make_number (state.depth),
+ return
+ Fcons (make_number (state.depth),
Fcons (state.prevlevelstart < 0
? Qnil : make_number (state.prevlevelstart),
Fcons (state.thislevelstart < 0
@@ -3553,11 +3604,18 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
? Qsyntax_table
: make_number (state.comstyle))
: Qnil),
- Fcons (((state.incomment
- || (state.instring >= 0))
- ? make_number (state.comstr_start)
- : Qnil),
- Fcons (state.levelstarts, Qnil))))))))));
+ Fcons (((state.incomment
+ || (state.instring >= 0))
+ ? make_number (state.comstr_start)
+ : Qnil),
+ Fcons (state.levelstarts,
+ Fcons (state.prev_syntax == Smax
+ ? Qnil
+ : make_number (state.prev_syntax),
+ Fcons (state.prev_comment_end == -1
+ ? Qnil
+ : make_number (state.prev_comment_end),
+ Qnil))))))))))));
}
\f
void
> Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply related [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-17 21:49 ` Alan Mackenzie
@ 2016-03-18 4:49 ` Stefan Monnier
2016-03-18 15:11 ` Alan Mackenzie
2016-03-18 16:27 ` Stefan Monnier
1 sibling, 1 reply; 23+ messages in thread
From: Stefan Monnier @ 2016-03-18 4:49 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: 23019
> Do this by adding two new fields to the parser state: the syntax of the last
> character scanned, and the last end of comment scanned. This should make the
> parser state complete.
Thanks. I like the "syntax of the last character scanned", but I don't
understand the reasoning behind "last end of comment scanned". Why is
this relevant? Is it in case the "last character scanned" was a "slash
ending a comment" so as to avoid treating "*/*" as both a comment closer and
a subsequent opener?
If so, I'm not sure I like it. It sounds to me like there's a chance
it's actually incomplete (e.g. it doesn't address the similar problem
when the "last character scanned" is an end of a string which also
happens to be a valid first-char of a comment-starter), and even if it
isn't, it "feels ad-hoc" to me.
Would it be difficult to do the following instead:
- get rid of element 11.
- change element 10 so it's nil if the last char was an "end of
something". Another way to look at it, is that the element 10 should
only be non-nil if the "next lexeme" might start on that
previous character.
I also have a side question: IIUC your patch makes the 5th element
redundant (can be replaced with a test whether "last char syntax" was
"escape"), is that right?
Stefan
> Also document element 9 of the parser state. Also refactor the code a bit.
> * src/syntax.c (struct lisp_parse_state): Add two new fields.
> (internalize_parse_state): New function, extracted from scan_sexps_forward.
> (back_comment): Call internalize_parse_state.
> (forw_comment): Return the syntax of the last character scanned to the caller.
> (Fforward_comment, scan_lists): New dummy variables, passed to forw_comment.
> (scan_sexps_forward): Remove a redundant state parameter. Access all `state'
> information via the address parameter `state'. Remove the code which converts
> from external to internal form of `state'. Access buffer contents only from
> `from' onwards. Reformulate code at the top of the main loop correctly to
> recognize comment openers when starting in the middle of one. Call
> forw_comment with extra argument (for return of final syntax value).
> (Fparse_partial_sexp): Document elements 9, 10, 11 of the parser state in the
> doc string. Clarify the doc string in general. Call
> internalize_parse_state. Take account of the new elements when consing up the
> output parser state.
> * doc/lispref/syntax.texi: (Parser State): Document element 9 and the new
> elements 10 and 11. Minor wording corrections (remove reference to "trivial
> cases").
> (Low Level Parsing): Minor corrections
> diff --git a/doc/lispref/syntax.texi b/doc/lispref/syntax.texi
> index d5a7eba..67a00d7 100644
> --- a/doc/lispref/syntax.texi
> +++ b/doc/lispref/syntax.texi
> @@ -791,10 +791,10 @@ Parser State
> @subsection Parser State
> @cindex parser state
> - A @dfn{parser state} is a list of ten elements describing the state
> -of the syntactic parser, after it parses the text between a specified
> -starting point and a specified end point in the buffer. Parsing
> -functions such as @code{syntax-ppss}
> + A @dfn{parser state} is a list of (currently) twelve elements
> +describing the state of the syntactic parser, after it parses the text
> +between a specified starting point and a specified end point in the
> +buffer. Parsing functions such as @code{syntax-ppss}
> @ifnottex
> (@pxref{Position Parse})
> @end ifnottex
> @@ -851,15 +851,21 @@ Parser State
> this element is @code{nil}.
> @item
> -Internal data for continuing the parsing. The meaning of this
> -data is subject to change; it is used if you pass this list
> -as the @var{state} argument to another call.
> +The list of the positions of the currently open parentheses, starting
> +with the outermost.
> +
> +@item
> +The @var{syntax-code} (@pxref{Syntax Table Internals}) of the last
> +buffer position scanned, or @code{nil} if no scanning has happened.
> +
> +@item
> +The position after the previous end of comment, or @code{nil} if the
> +scanning has not passed a comment end.
> @end enumerate
> Elements 1, 2, and 6 are ignored in a state which you pass as an
> -argument to continue parsing, and elements 8 and 9 are used only in
> -trivial cases. Those elements are mainly used internally by the
> -parser code.
> +argument to continue parsing. Elements 9 to 11 are mainly used
> +internally by the parser code.
> One additional piece of useful information is available from a
> parser state using this function:
> @@ -898,11 +904,11 @@ Low-Level Parsing
> If the fourth argument @var{stop-before} is non-@code{nil}, parsing
> stops when it comes to any character that starts a sexp. If
> -@var{stop-comment} is non-@code{nil}, parsing stops when it comes to the
> -start of an unnested comment. If @var{stop-comment} is the symbol
> +@var{stop-comment} is non-@code{nil}, parsing stops after the start of
> +an unnested comment. If @var{stop-comment} is the symbol
> @code{syntax-table}, parsing stops after the start of an unnested
> -comment or a string, or the end of an unnested comment or a string,
> -whichever comes first.
> +comment or a string, or after the end of an unnested comment or a
> +string, whichever comes first.
> If @var{state} is @code{nil}, @var{start} is assumed to be at the top
> level of parenthesis structure, such as the beginning of a function
> diff --git a/src/syntax.c b/src/syntax.c
> index 249d0d5..11b1ff0 100644
> --- a/src/syntax.c
> +++ b/src/syntax.c
> @@ -153,6 +153,9 @@ struct lisp_parse_state
> ptrdiff_t comstr_start; /* Position of last comment/string starter. */
> Lisp_Object levelstarts; /* Char numbers of starts-of-expression
> of levels (starting from outermost). */
> + int prev_syntax; /* Syntax of previous character scanned, or Smax. */
> + ptrdiff_t prev_comment_end; /* Position after end of last closed
> + comment, or -1. */
> };
> \f
> /* These variables are a cache for finding the start of a defun.
> @@ -176,7 +179,8 @@ static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
> static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool);
> static void scan_sexps_forward (struct lisp_parse_state *,
> ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
> - bool, Lisp_Object, int);
> + bool, int);
> +static void internalize_parse_state (Lisp_Object, struct lisp_parse_state *);
> static bool in_classes (int, Lisp_Object);
> static void parse_sexp_propertize (ptrdiff_t charpos);
> @@ -911,10 +915,11 @@ back_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
> }
> do
> {
> + internalize_parse_state (Qnil, &state);
> scan_sexps_forward (&state,
> defun_start, defun_start_byte,
> comment_end, TYPE_MINIMUM (EMACS_INT),
> - 0, Qnil, 0);
> + 0, 0);
> defun_start = comment_end;
> if (!adjusted)
> {
> @@ -2314,7 +2319,9 @@ in_classes (int c, Lisp_Object iso_classes)
> into *CHARPOS_PTR and the corresponding bytepos into *BYTEPOS_PTR.
> Else, return false and store the charpos STOP into *CHARPOS_PTR, the
> corresponding bytepos into *BYTEPOS_PTR and the current nesting
> - (as defined for state.incomment) in *INCOMMENT_PTR.
> + (as defined for state->incomment) in *INCOMMENT_PTR. The
> + SYNTAX_WITH_FLAGS of the last character scanned in the comment is
> + stored into *last_syntax_ptr.
> The comment end is the last character of the comment rather than the
> character just after the comment.
> @@ -2326,7 +2333,7 @@ static bool
> forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
> EMACS_INT nesting, int style, int prev_syntax,
> ptrdiff_t *charpos_ptr, ptrdiff_t *bytepos_ptr,
> - EMACS_INT *incomment_ptr)
> + EMACS_INT *incomment_ptr, int *last_syntax_ptr)
> {
> register int c, c1;
> register enum syntaxcode code;
> @@ -2346,6 +2353,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
> *incomment_ptr = nesting;
> *charpos_ptr = from;
> *bytepos_ptr = from_byte;
> + *last_syntax_ptr = syntax;
> return 0;
> }
> c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
> @@ -2415,6 +2423,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
> }
> *charpos_ptr = from;
> *bytepos_ptr = from_byte;
> + *last_syntax_ptr = syntax;
> return 1;
> }
> @@ -2436,6 +2445,7 @@ between them, return t; otherwise return nil. */)
> EMACS_INT count1;
> ptrdiff_t out_charpos, out_bytepos;
> EMACS_INT dummy;
> + int dummy2;
> CHECK_NUMBER (count);
> count1 = XINT (count);
> @@ -2499,7 +2509,7 @@ between them, return t; otherwise return nil. */)
> }
> /* We're at the start of a comment. */
> found = forw_comment (from, from_byte, stop, comnested, comstyle, 0,
> - &out_charpos, &out_bytepos, &dummy);
> + &out_charpos, &out_bytepos, &dummy, &dummy2);
> from = out_charpos; from_byte = out_bytepos;
> if (!found)
> {
> @@ -2659,6 +2669,7 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
> ptrdiff_t from_byte;
> ptrdiff_t out_bytepos, out_charpos;
> EMACS_INT dummy;
> + int dummy2;
> bool multibyte_symbol_p = sexpflag && multibyte_syntax_as_symbol;
> if (depth > 0) min_depth = 0;
> @@ -2755,7 +2766,8 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
> UPDATE_SYNTAX_TABLE_FORWARD (from);
> found = forw_comment (from, from_byte, stop,
> comnested, comstyle, 0,
> - &out_charpos, &out_bytepos, &dummy);
> + &out_charpos, &out_bytepos, &dummy,
> + &dummy2);
> from = out_charpos, from_byte = out_bytepos;
> if (!found)
> {
> @@ -3119,7 +3131,7 @@ the prefix syntax flag (p). */)
> }
> \f
> /* Parse forward from FROM / FROM_BYTE to END,
> - assuming that FROM has state OLDSTATE (nil means FROM is start of function),
> + assuming that FROM has state STATE (nil means FROM is start of function),
> and return a description of the state of the parse at END.
> If STOPBEFORE, stop at the start of an atom.
> If COMMENTSTOP is 1, stop at the start of a comment.
> @@ -3127,12 +3139,11 @@ the prefix syntax flag (p). */)
> after the beginning of a string, or after the end of a string. */
> static void
> -scan_sexps_forward (struct lisp_parse_state *stateptr,
> +scan_sexps_forward (struct lisp_parse_state *state,
> ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t end,
> EMACS_INT targetdepth, bool stopbefore,
> - Lisp_Object oldstate, int commentstop)
> + int commentstop)
> {
> - struct lisp_parse_state state;
> enum syntaxcode code;
> int c1;
> bool comnested;
> @@ -3148,7 +3159,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
> Lisp_Object tem;
> ptrdiff_t prev_from; /* Keep one character before FROM. */
> ptrdiff_t prev_from_byte;
> - int prev_from_syntax;
> + int prev_from_syntax, prev_prev_from_syntax;
> bool boundary_stop = commentstop == -1;
> bool nofence;
> bool found;
> @@ -3165,6 +3176,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
> do { prev_from = from; \
> prev_from_byte = from_byte; \
> temp = FETCH_CHAR_AS_MULTIBYTE (prev_from_byte); \
> + prev_prev_from_syntax = prev_from_syntax; \
> prev_from_syntax = SYNTAX_WITH_FLAGS (temp); \
> INC_BOTH (from, from_byte); \
> if (from < end) \
> @@ -3174,88 +3186,38 @@ do { prev_from = from; \
> immediate_quit = 1;
> QUIT;
> - if (NILP (oldstate))
> - {
> - depth = 0;
> - state.instring = -1;
> - state.incomment = 0;
> - state.comstyle = 0; /* comment style a by default. */
> - state.comstr_start = -1; /* no comment/string seen. */
> - }
> - else
> - {
> - tem = Fcar (oldstate);
> - if (!NILP (tem))
> - depth = XINT (tem);
> - else
> - depth = 0;
> -
> - oldstate = Fcdr (oldstate);
> - oldstate = Fcdr (oldstate);
> - oldstate = Fcdr (oldstate);
> - tem = Fcar (oldstate);
> - /* Check whether we are inside string_fence-style string: */
> - state.instring = (!NILP (tem)
> - ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
> - : -1);
> -
> - oldstate = Fcdr (oldstate);
> - tem = Fcar (oldstate);
> - state.incomment = (!NILP (tem)
> - ? (INTEGERP (tem) ? XINT (tem) : -1)
> - : 0);
> -
> - oldstate = Fcdr (oldstate);
> - tem = Fcar (oldstate);
> - start_quoted = !NILP (tem);
> + depth = state->depth;
> + start_quoted = state->quoted;
> + prev_prev_from_syntax = Smax;
> + prev_from_syntax = state->prev_syntax;
> - /* if the eighth element of the list is nil, we are in comment
> - style a. If it is non-nil, we are in comment style b */
> - oldstate = Fcdr (oldstate);
> - oldstate = Fcdr (oldstate);
> - tem = Fcar (oldstate);
> - state.comstyle = (NILP (tem)
> - ? 0
> - : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
> - ? XINT (tem)
> - : ST_COMMENT_STYLE));
> -
> - oldstate = Fcdr (oldstate);
> - tem = Fcar (oldstate);
> - state.comstr_start =
> - RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
> - oldstate = Fcdr (oldstate);
> - tem = Fcar (oldstate);
> - while (!NILP (tem)) /* >= second enclosing sexps. */
> - {
> - Lisp_Object temhd = Fcar (tem);
> - if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
> - curlevel->last = XINT (temhd);
> - if (++curlevel == endlevel)
> - curlevel--; /* error ("Nesting too deep for parser"); */
> - curlevel->prev = -1;
> - curlevel->last = -1;
> - tem = Fcdr (tem);
> - }
> + tem = state->levelstarts;
> + while (!NILP (tem)) /* >= second enclosing sexps. */
> + {
> + Lisp_Object temhd = Fcar (tem);
> + if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
> + curlevel->last = XINT (temhd);
> + if (++curlevel == endlevel)
> + curlevel--; /* error ("Nesting too deep for parser"); */
> + curlevel->prev = -1;
> + curlevel->last = -1;
> + tem = Fcdr (tem);
> }
> - state.quoted = 0;
> - mindepth = depth;
> -
curlevel-> prev = -1;
curlevel-> last = -1;
> - SETUP_SYNTAX_TABLE (prev_from, 1);
> - temp = FETCH_CHAR (prev_from_byte);
> - prev_from_syntax = SYNTAX_WITH_FLAGS (temp);
> - UPDATE_SYNTAX_TABLE_FORWARD (from);
> + state->quoted = 0;
> + mindepth = depth;
> +
> + SETUP_SYNTAX_TABLE (from, 1);
> /* Enter the loop at a place appropriate for initial state. */
> - if (state.incomment)
> + if (state->incomment)
> goto startincomment;
> - if (state.instring >= 0)
> + if (state->instring >= 0)
> {
> - nofence = state.instring != ST_STRING_STYLE;
> + nofence = state->instring != ST_STRING_STYLE;
> if (start_quoted)
> goto startquotedinstring;
> goto startinstring;
> @@ -3266,10 +3228,10 @@ do { prev_from = from; \
> while (from < end)
> {
> int syntax;
> - INC_FROM;
> - code = prev_from_syntax & 0xff;
> if (from < end
> + && (state->prev_comment_end == -1
> + || prev_from >= state->prev_comment_end)
> && SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
> && (c1 = FETCH_CHAR (from_byte),
> syntax = SYNTAX_WITH_FLAGS (c1),
> @@ -3280,32 +3242,37 @@ do { prev_from = from; \
> /* Record the comment style we have entered so that only
> the comment-end sequence of the same style actually
> terminates the comment section. */
> - state.comstyle
> + state->comstyle
> = SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax);
> comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax)
> | SYNTAX_FLAGS_COMMENT_NESTED (syntax));
> - state.incomment = comnested ? 1 : -1;
> - state.comstr_start = prev_from;
> + state->incomment = comnested ? 1 : -1;
> + state->comstr_start = prev_from;
> INC_FROM;
> code = Scomment;
> }
> - else if (code == Scomment_fence)
> - {
> - /* Record the comment style we have entered so that only
> - the comment-end sequence of the same style actually
> - terminates the comment section. */
> - state.comstyle = ST_COMMENT_STYLE;
> - state.incomment = -1;
> - state.comstr_start = prev_from;
> - code = Scomment;
> - }
> - else if (code == Scomment)
> - {
> - state.comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
> - state.incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
> - 1 : -1);
> - state.comstr_start = prev_from;
> - }
> + else
> + {
> + INC_FROM;
> + code = prev_from_syntax & 0xff;
> + if (code == Scomment_fence)
> + {
> + /* Record the comment style we have entered so that only
> + the comment-end sequence of the same style actually
> + terminates the comment section. */
> + state->comstyle = ST_COMMENT_STYLE;
> + state->incomment = -1;
> + state->comstr_start = prev_from;
> + code = Scomment;
> + }
> + else if (code == Scomment)
> + {
> + state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
> + state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
> + 1 : -1);
> + state->comstr_start = prev_from;
> + }
> + }
> if (SYNTAX_FLAGS_PREFIX (prev_from_syntax))
> continue;
> @@ -3357,18 +3324,21 @@ do { prev_from = from; \
> middle of it. We don't want to do that if we're just at the
> beginning of the comment (think of (*) ... (*)). */
> found = forw_comment (from, from_byte, end,
> - state.incomment, state.comstyle,
> - (from == BEGV || from < state.comstr_start + 3)
> + state->incomment, state->comstyle,
> + (from == BEGV || from < state->comstr_start + 3)
> ? 0 : prev_from_syntax,
> - &out_charpos, &out_bytepos, &state.incomment);
> + &out_charpos, &out_bytepos, &state->incomment,
> + &prev_from_syntax);
> from = out_charpos; from_byte = out_bytepos;
> - /* Beware! prev_from and friends are invalid now.
> - Luckily, the `done' doesn't use them and the INC_FROM
> - sets them to a sane value without looking at them. */
> + /* Beware! prev_from and friends (except prev_from_syntax)
> + are invalid now. Luckily, the `done' doesn't use them
> + and the INC_FROM sets them to a sane value without
> + looking at them. */
> if (!found) goto done;
> INC_FROM;
> - state.incomment = 0;
> - state.comstyle = 0; /* reset the comment style */
> + state->incomment = 0;
> + state->comstyle = 0; /* reset the comment style */
> + state->prev_comment_end = from;
> if (boundary_stop) goto done;
> break;
> @@ -3396,16 +3366,16 @@ do { prev_from = from; \
> case Sstring:
> case Sstring_fence:
> - state.comstr_start = from - 1;
> + state->comstr_start = from - 1;
> if (stopbefore) goto stop; /* this arg means stop at sexp start */
curlevel-> last = prev_from;
> - state.instring = (code == Sstring
> + state->instring = (code == Sstring
> ? (FETCH_CHAR_AS_MULTIBYTE (prev_from_byte))
> : ST_STRING_STYLE);
> if (boundary_stop) goto done;
> startinstring:
> {
> - nofence = state.instring != ST_STRING_STYLE;
> + nofence = state->instring != ST_STRING_STYLE;
> while (1)
> {
> @@ -3419,7 +3389,7 @@ do { prev_from = from; \
> /* Check C_CODE here so that if the char has
> a syntax-table property which says it is NOT
> a string character, it does not end the string. */
> - if (nofence && c == state.instring && c_code == Sstring)
> + if (nofence && c == state->instring && c_code == Sstring)
> break;
> switch (c_code)
> @@ -3442,7 +3412,7 @@ do { prev_from = from; \
> }
> }
> string_end:
> - state.instring = -1;
> + state->instring = -1;
curlevel-> prev = curlevel->last;
> INC_FROM;
> if (boundary_stop) goto done;
> @@ -3461,25 +3431,99 @@ do { prev_from = from; \
> stop: /* Here if stopping before start of sexp. */
> from = prev_from; /* We have just fetched the char that starts it; */
> from_byte = prev_from_byte;
> + prev_from_syntax = prev_prev_from_syntax;
> goto done; /* but return the position before it. */
> endquoted:
> - state.quoted = 1;
> + state->quoted = 1;
> done:
> - state.depth = depth;
> - state.mindepth = mindepth;
> - state.thislevelstart = curlevel->prev;
> - state.prevlevelstart
> + state->depth = depth;
> + state->mindepth = mindepth;
> + state->thislevelstart = curlevel->prev;
> + state->prevlevelstart
> = (curlevel == levelstart) ? -1 : (curlevel - 1)->last;
> - state.location = from;
> - state.location_byte = from_byte;
> - state.levelstarts = Qnil;
> + state->location = from;
> + state->location_byte = from_byte;
> + state->levelstarts = Qnil;
> while (curlevel > levelstart)
> - state.levelstarts = Fcons (make_number ((--curlevel)->last),
> - state.levelstarts);
> + state->levelstarts = Fcons (make_number ((--curlevel)->last),
> + state->levelstarts);
> + state->prev_syntax = prev_from_syntax;
> immediate_quit = 0;
> +}
> +
> +/* Convert a (lisp) parse state to the internal form used in
> + scan_sexps_forward. */
> +static void
> +internalize_parse_state (Lisp_Object external, struct lisp_parse_state *state)
> +{
> + Lisp_Object tem;
> +
> + if (NILP (external))
> + {
> + state->depth = 0;
> + state->instring = -1;
> + state->incomment = 0;
> + state->quoted = 0;
> + state->comstyle = 0; /* comment style a by default. */
> + state->comstr_start = -1; /* no comment/string seen. */
> + state->levelstarts = Qnil;
> + state->prev_syntax = Smax;
> + state->prev_comment_end = -1;
> + }
> + else
> + {
> + tem = Fcar (external);
> + if (!NILP (tem))
> + state->depth = XINT (tem);
> + else
> + state->depth = 0;
> +
> + external = Fcdr (external);
> + external = Fcdr (external);
> + external = Fcdr (external);
> + tem = Fcar (external);
> + /* Check whether we are inside string_fence-style string: */
> + state->instring = (!NILP (tem)
> + ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
> + : -1);
> +
> + external = Fcdr (external);
> + tem = Fcar (external);
> + state->incomment = (!NILP (tem)
> + ? (INTEGERP (tem) ? XINT (tem) : -1)
> + : 0);
> - *stateptr = state;
> + external = Fcdr (external);
> + tem = Fcar (external);
> + state->quoted = !NILP (tem);
> +
> + /* if the eighth element of the list is nil, we are in comment
> + style a. If it is non-nil, we are in comment style b */
> + external = Fcdr (external);
> + external = Fcdr (external);
> + tem = Fcar (external);
> + state->comstyle = (NILP (tem)
> + ? 0
> + : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
> + ? XINT (tem)
> + : ST_COMMENT_STYLE));
> +
> + external = Fcdr (external);
> + tem = Fcar (external);
> + state->comstr_start =
> + RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
> + external = Fcdr (external);
> + tem = Fcar (external);
> + state->levelstarts = tem;
> +
> + external = Fcdr (external);
> + tem = Fcar (external);
> + state->prev_syntax = NILP (tem) ? Smax : XINT (tem);
> + external = Fcdr (external);
> + tem = Fcar (external);
> + state->prev_comment_end = NILP (tem) ? -1 : XINT (tem);
> + }
> }
> DEFUN ("parse-partial-sexp", Fparse_partial_sexp, Sparse_partial_sexp, 2, 6, 0,
> @@ -3488,6 +3532,7 @@ Parsing stops at TO or when certain criteria are met;
> point is set to where parsing stops.
> If fifth arg OLDSTATE is omitted or nil,
> parsing assumes that FROM is the beginning of a function.
> +
> Value is a list of elements describing final state of parsing:
> 0. depth in parens.
> 1. character address of start of innermost containing list; nil if none.
> @@ -3501,16 +3546,20 @@ Value is a list of elements describing final state of parsing:
> 6. the minimum paren-depth encountered during this scan.
> 7. style of comment, if any.
> 8. character address of start of comment or string; nil if not in one.
> - 9. Intermediate data for continuation of parsing (subject to change).
> + 9. List of positions of currently open parens, outermost first.
> +10. Syntax of last character scanned, or nil if no scanning has happened.
> +11. Position after end of previous comment scanned, or nil.
> +12..... Possible further internal information used by `parse-partial-sexp'.
> +
> If third arg TARGETDEPTH is non-nil, parsing stops if the depth
> in parentheses becomes equal to TARGETDEPTH.
> -Fourth arg STOPBEFORE non-nil means stop when come to
> +Fourth arg STOPBEFORE non-nil means stop when we come to
> any character that starts a sexp.
> Fifth arg OLDSTATE is a list like what this function returns.
> It is used to initialize the state of the parse. Elements number 1, 2, 6
> are ignored.
> -Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
> - If it is symbol `syntax-table', stop after the start of a comment or a
> +Sixth arg COMMENTSTOP non-nil means stop after the start of a comment.
> + If it is the symbol `syntax-table', stop after the start of a comment or a
> string, or after end of a comment or a string. */)
> (Lisp_Object from, Lisp_Object to, Lisp_Object targetdepth,
> Lisp_Object stopbefore, Lisp_Object oldstate, Lisp_Object commentstop)
> @@ -3527,15 +3576,17 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
> target = TYPE_MINIMUM (EMACS_INT); /* We won't reach this depth. */
> validate_region (&from, &to);
> + internalize_parse_state (oldstate, &state);
> scan_sexps_forward (&state, XINT (from), CHAR_TO_BYTE (XINT (from)),
> XINT (to),
> - target, !NILP (stopbefore), oldstate,
> + target, !NILP (stopbefore),
> (NILP (commentstop)
> ? 0 : (EQ (commentstop, Qsyntax_table) ? -1 : 1)));
> SET_PT_BOTH (state.location, state.location_byte);
> - return Fcons (make_number (state.depth),
> + return
> + Fcons (make_number (state.depth),
> Fcons (state.prevlevelstart < 0
> ? Qnil : make_number (state.prevlevelstart),
> Fcons (state.thislevelstart < 0
> @@ -3553,11 +3604,18 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
> ? Qsyntax_table
> : make_number (state.comstyle))
> : Qnil),
> - Fcons (((state.incomment
> - || (state.instring >= 0))
> - ? make_number (state.comstr_start)
> - : Qnil),
> - Fcons (state.levelstarts, Qnil))))))))));
> + Fcons (((state.incomment
> + || (state.instring >= 0))
> + ? make_number (state.comstr_start)
> + : Qnil),
> + Fcons (state.levelstarts,
> + Fcons (state.prev_syntax == Smax
> + ? Qnil
> + : make_number (state.prev_syntax),
> + Fcons (state.prev_comment_end == -1
> + ? Qnil
> + : make_number (state.prev_comment_end),
> + Qnil))))))))))));
> }
> \f
> void
>> Stefan
> --
> Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-18 4:49 ` Stefan Monnier
@ 2016-03-18 15:11 ` Alan Mackenzie
2016-03-18 15:22 ` Alan Mackenzie
2016-03-18 16:23 ` Stefan Monnier
0 siblings, 2 replies; 23+ messages in thread
From: Alan Mackenzie @ 2016-03-18 15:11 UTC (permalink / raw)
To: Stefan Monnier; +Cc: 23019
Hello, Stefan.
On Fri, Mar 18, 2016 at 12:49:07AM -0400, Stefan Monnier wrote:
> > Do this by adding two new fields to the parser state: the syntax of the last
> > character scanned, and the last end of comment scanned. This should make the
> > parser state complete.
> Thanks. I like the "syntax of the last character scanned", but I don't
> understand the reasoning behind "last end of comment scanned". Why is
> this relevant? Is it in case the "last character scanned" was a "slash
> ending a comment" so as to avoid treating "*/*" as both a comment closer and
> a subsequent opener?
That's exactly the reason.
> If so, I'm not sure I like it.
I don't really like it either.
> It sounds to me like there's a chance it's actually incomplete (e.g.
> it doesn't address the similar problem when the "last character
> scanned" is an end of a string which also happens to be a valid
> first-char of a comment-starter), and even if it isn't, it "feels
> ad-hoc" to me.
Now even I wouldn't have come up with that end-of-string scenario. ;-)
Such a scenario is presumably one reason why, in scan_sexps_forward, two
character comment delimiters are handled before strings.
> Would it be difficult to do the following instead:
> - get rid of element 11.
Done.
> - change element 10 so it's nil if the last char was an "end of
> something". Another way to look at it, is that the element 10 should
> only be non-nil if the "next lexeme" might start on that
> previous character.
I've tried this, and it's somewhat ugly. Setting the "previous_syntax"
to nil is also needed for the asterisk in "/*". The nil would appear to
mean "the syntactic value of the last character has already been used
up". So the "previous_syntax" is nil in the most interesting cases. It
also feels somewhat ad-hoc.
How about this idea: element 10 will record the syntax of the previous
character ONLY when it is potentially the first character of a two
character comment delimiter, otherwise it'll be nil. At least that's
being honest about what the thing's being used for.
> I also have a side question: IIUC your patch makes the 5th element
> redundant (can be replaced with a test whether "last char syntax" was
> "escape"), is that right?
It would appear to be, yes. We really can't get rid of element 5,
though, because there will surely be code out there that uses it. But
if I change element 10 as outlined above, element 5 will no longer be
redundant.
> Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-18 15:11 ` Alan Mackenzie
@ 2016-03-18 15:22 ` Alan Mackenzie
2016-03-18 16:23 ` Stefan Monnier
1 sibling, 0 replies; 23+ messages in thread
From: Alan Mackenzie @ 2016-03-18 15:22 UTC (permalink / raw)
To: Stefan Monnier; +Cc: 23019
Hello again, Stefan.
On Fri, Mar 18, 2016 at 03:11:55PM +0000, Alan Mackenzie wrote:
> Done.
> > - change element 10 so it's nil if the last char was an "end of
> > something". Another way to look at it, is that the element 10 should
> > only be non-nil if the "next lexeme" might start on that
> > previous character.
[ .... ]
> How about this idea: element 10 will record the syntax of the previous
> character ONLY when it is potentially the first character of a two
> character comment delimiter, otherwise it'll be nil. At least that's
> being honest about what the thing's being used for.
That's exactly what you suggested. Apologies for not reading your post
a bit more carefully. I think we're agreed, then. I'll implement it.
> > Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-18 15:11 ` Alan Mackenzie
2016-03-18 15:22 ` Alan Mackenzie
@ 2016-03-18 16:23 ` Stefan Monnier
2016-03-18 18:25 ` Alan Mackenzie
1 sibling, 1 reply; 23+ messages in thread
From: Stefan Monnier @ 2016-03-18 16:23 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: 23019
>> It sounds to me like there's a chance it's actually incomplete (e.g.
>> it doesn't address the similar problem when the "last character
>> scanned" is an end of a string which also happens to be a valid
>> first-char of a comment-starter), and even if it isn't, it "feels
>> ad-hoc" to me.
> Now even I wouldn't have come up with that end-of-string scenario. ;-)
I don't work in embedded systems, but Coq/Agda's total functions force
you to consider all possible cases.
> Such a scenario is presumably one reason why, in scan_sexps_forward, two
> character comment delimiters are handled before strings.
It doesn't handle the exact same situation, but it's closely related
indeed.
>> - change element 10 so it's nil if the last char was an "end of
>> something". Another way to look at it, is that the element 10 should
>> only be non-nil if the "next lexeme" might start on that
>> previous character.
> I've tried this, and it's somewhat ugly. Setting the "previous_syntax"
> to nil is also needed for the asterisk in "/*". The nil would appear to
> mean "the syntactic value of the last character has already been used
> up". So the "previous_syntax" is nil in the most interesting cases. It
> also feels somewhat ad-hoc.
> How about this idea: element 10 will record the syntax of the previous
> character ONLY when it is potentially the first character of a two
> character comment delimiter, otherwise it'll be nil. At least that's
> being honest about what the thing's being used for.
IIUC the only difference between what I (think I) suggested and what
you're proposing is that you want to return nil for the "prev is
backslash" whereas I was suggesting to return non-nil in that case.
[ AFAIK the only two-char elements we handle so far as the comment
delimiters and the backslash escapes. ]
Do I understand this right?
> It would appear to be, yes. We really can't get rid of element 5,
> though, because there will surely be code out there that uses it. But
> if I change element 10 as outlined above, element 5 will no longer be
> redundant.
I'd even be tempted to re-use element 5, although it might
conceivably break some code out there.
But even if we don't re-use element 5, I would actually much prefer to
render element 5 redundant.
Stefan
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-17 21:49 ` Alan Mackenzie
2016-03-18 4:49 ` Stefan Monnier
@ 2016-03-18 16:27 ` Stefan Monnier
2016-03-18 19:16 ` Alan Mackenzie
1 sibling, 1 reply; 23+ messages in thread
From: Stefan Monnier @ 2016-03-18 16:27 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: 23019
> (scan_sexps_forward): Remove a redundant state parameter. Access all `state'
> information via the address parameter `state'.
Have you taken a look at the performance impact of this part of the change?
I don't expect it will make much difference, but I'm actually wondering
whether it makes things slower or faster.
Stefan
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-18 16:23 ` Stefan Monnier
@ 2016-03-18 18:25 ` Alan Mackenzie
2016-03-18 19:36 ` Stefan Monnier
0 siblings, 1 reply; 23+ messages in thread
From: Alan Mackenzie @ 2016-03-18 18:25 UTC (permalink / raw)
To: Stefan Monnier; +Cc: 23019
Hello, Stefan.
On Fri, Mar 18, 2016 at 12:23:02PM -0400, Stefan Monnier wrote:
> >> - change element 10 so it's nil if the last char was an "end of
> >> something". Another way to look at it, is that the element 10 should
> >> only be non-nil if the "next lexeme" might start on that
> >> previous character.
> > I've tried this, and it's somewhat ugly. Setting the "previous_syntax"
> > to nil is also needed for the asterisk in "/*". The nil would appear to
> > mean "the syntactic value of the last character has already been used
> > up". So the "previous_syntax" is nil in the most interesting cases. It
> > also feels somewhat ad-hoc.
> > How about this idea: element 10 will record the syntax of the previous
> > character ONLY when it is potentially the first character of a two
> > character comment delimiter, otherwise it'll be nil. At least that's
> > being honest about what the thing's being used for.
> IIUC the only difference between what I (think I) suggested and what
> you're proposing is that you want to return nil for the "prev is
> backslash" whereas I was suggesting to return non-nil in that case.
> [ AFAIK the only two-char elements we handle so far as the comment
> delimiters and the backslash escapes. ]
We also have Scharquote, which scan_sexps_forward handles identically to
Sescape.
> Do I understand this right?
Yes, but I've no strong feelings on the matter.
> > It would appear to be, yes. We really can't get rid of element 5,
> > though, because there will surely be code out there that uses it. But
> > if I change element 10 as outlined above, element 5 will no longer be
> > redundant.
> I'd even be tempted to re-use element 5, although it might
> conceivably break some code out there.
I have bad feelings about that. Is it really worth the risk, just to
save one cons cell on a list that not that many instances of exist at
any time?
> But even if we don't re-use element 5, I would actually much prefer to
> render element 5 redundant.
OK. Here's an updated patch which does just that. Comments would be
welcome.
> Stefan
Amend parse-partial-sexp correctly to handle two character comment delimiters
Do this by adding a new field to the parser state: the syntax of the last
character scanned, should that be the first char of a (potential) two char
construct, nil otherwise.
This should make the parser state complete.
Also document element 9 of the parser state. Also refactor the code a bit.
* src/syntax.c (struct lisp_parse_state): Add a new field.
(SYNTAX_FLAGS_COMSTARTEND_FIRST): New function.
(internalize_parse_state): New function, extracted from scan_sexps_forward.
(back_comment): Call internalize_parse_state.
(forw_comment): Return the syntax of the last character scanned to the caller.
(Fforward_comment, scan_lists): New dummy variables, passed to forw_comment.
(scan_sexps_forward): Remove a redundant state parameter. Access all `state'
information via the address parameter `state'. Remove the code which converts
from external to internal form of `state'. Access buffer contents only from
`from' onwards. Reformulate code at the top of the main loop correctly to
recognize comment openers when starting in the middle of one. Call
forw_comment with extra argument (for return of final syntax value).
(Fparse_partial_sexp): Document elements 9, 10 of the parser state in the
doc string. Clarify the doc string in general. Call
internalize_parse_state. Take account of the new elements when consing up the
output parser state.
* doc/lispref/syntax.texi: (Parser State): Document element 9 and the new
element 10. Minor wording corrections (remove reference to "trivial cases").
(Low Level Parsing): Minor corrections.
diff --git a/doc/lispref/syntax.texi b/doc/lispref/syntax.texi
index d5a7eba..f81c164 100644
--- a/doc/lispref/syntax.texi
+++ b/doc/lispref/syntax.texi
@@ -791,10 +791,10 @@ Parser State
@subsection Parser State
@cindex parser state
- A @dfn{parser state} is a list of ten elements describing the state
-of the syntactic parser, after it parses the text between a specified
-starting point and a specified end point in the buffer. Parsing
-functions such as @code{syntax-ppss}
+ A @dfn{parser state} is a list of (currently) eleven elements
+describing the state of the syntactic parser, after it parses the text
+between a specified starting point and a specified end point in the
+buffer. Parsing functions such as @code{syntax-ppss}
@ifnottex
(@pxref{Position Parse})
@end ifnottex
@@ -851,15 +851,20 @@ Parser State
this element is @code{nil}.
@item
-Internal data for continuing the parsing. The meaning of this
-data is subject to change; it is used if you pass this list
-as the @var{state} argument to another call.
+The list of the positions of the currently open parentheses, starting
+with the outermost.
+
+@item
+When the last buffer position scanned was the (potential) first
+character of a two character construct (comment delimiter or
+escaped/char-quoted character pair), the @var{syntax-code}
+(@pxref{Syntax Table Internals}) of that position. Otherwise
+@code{nil}.
@end enumerate
Elements 1, 2, and 6 are ignored in a state which you pass as an
-argument to continue parsing, and elements 8 and 9 are used only in
-trivial cases. Those elements are mainly used internally by the
-parser code.
+argument to continue parsing. Elements 9 and 10 are mainly used
+internally by the parser code.
One additional piece of useful information is available from a
parser state using this function:
@@ -898,11 +903,11 @@ Low-Level Parsing
If the fourth argument @var{stop-before} is non-@code{nil}, parsing
stops when it comes to any character that starts a sexp. If
-@var{stop-comment} is non-@code{nil}, parsing stops when it comes to the
-start of an unnested comment. If @var{stop-comment} is the symbol
+@var{stop-comment} is non-@code{nil}, parsing stops after the start of
+an unnested comment. If @var{stop-comment} is the symbol
@code{syntax-table}, parsing stops after the start of an unnested
-comment or a string, or the end of an unnested comment or a string,
-whichever comes first.
+comment or a string, or after the end of an unnested comment or a
+string, whichever comes first.
If @var{state} is @code{nil}, @var{start} is assumed to be at the top
level of parenthesis structure, such as the beginning of a function
diff --git a/src/syntax.c b/src/syntax.c
index 249d0d5..e6a1942 100644
--- a/src/syntax.c
+++ b/src/syntax.c
@@ -81,6 +81,11 @@ SYNTAX_FLAGS_COMEND_SECOND (int flags)
return (flags >> 19) & 1;
}
static bool
+SYNTAX_FLAGS_COMSTARTEND_FIRST (int flags)
+{
+ return (flags & 0x50000) != 0;
+}
+static bool
SYNTAX_FLAGS_PREFIX (int flags)
{
return (flags >> 20) & 1;
@@ -153,6 +158,10 @@ struct lisp_parse_state
ptrdiff_t comstr_start; /* Position of last comment/string starter. */
Lisp_Object levelstarts; /* Char numbers of starts-of-expression
of levels (starting from outermost). */
+ int prev_syntax; /* Syntax of previous position scanned, when
+ that position (potentially) holds the first char
+ of a 2-char construct, i.e. comment delimiter
+ or Sescape, etc. Smax otherwise. */
};
\f
/* These variables are a cache for finding the start of a defun.
@@ -176,7 +185,8 @@ static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool);
static void scan_sexps_forward (struct lisp_parse_state *,
ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
- bool, Lisp_Object, int);
+ bool, int);
+static void internalize_parse_state (Lisp_Object, struct lisp_parse_state *);
static bool in_classes (int, Lisp_Object);
static void parse_sexp_propertize (ptrdiff_t charpos);
@@ -911,10 +921,11 @@ back_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
}
do
{
+ internalize_parse_state (Qnil, &state);
scan_sexps_forward (&state,
defun_start, defun_start_byte,
comment_end, TYPE_MINIMUM (EMACS_INT),
- 0, Qnil, 0);
+ 0, 0);
defun_start = comment_end;
if (!adjusted)
{
@@ -2314,7 +2325,9 @@ in_classes (int c, Lisp_Object iso_classes)
into *CHARPOS_PTR and the corresponding bytepos into *BYTEPOS_PTR.
Else, return false and store the charpos STOP into *CHARPOS_PTR, the
corresponding bytepos into *BYTEPOS_PTR and the current nesting
- (as defined for state.incomment) in *INCOMMENT_PTR.
+ (as defined for state->incomment) in *INCOMMENT_PTR. The
+ SYNTAX_WITH_FLAGS of the last character scanned in the comment is
+ stored into *last_syntax_ptr.
The comment end is the last character of the comment rather than the
character just after the comment.
@@ -2326,7 +2339,7 @@ static bool
forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
EMACS_INT nesting, int style, int prev_syntax,
ptrdiff_t *charpos_ptr, ptrdiff_t *bytepos_ptr,
- EMACS_INT *incomment_ptr)
+ EMACS_INT *incomment_ptr, int *last_syntax_ptr)
{
register int c, c1;
register enum syntaxcode code;
@@ -2346,6 +2359,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
*incomment_ptr = nesting;
*charpos_ptr = from;
*bytepos_ptr = from_byte;
+ *last_syntax_ptr = syntax;
return 0;
}
c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
@@ -2415,6 +2429,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
}
*charpos_ptr = from;
*bytepos_ptr = from_byte;
+ *last_syntax_ptr = syntax;
return 1;
}
@@ -2436,6 +2451,7 @@ between them, return t; otherwise return nil. */)
EMACS_INT count1;
ptrdiff_t out_charpos, out_bytepos;
EMACS_INT dummy;
+ int dummy2;
CHECK_NUMBER (count);
count1 = XINT (count);
@@ -2499,7 +2515,7 @@ between them, return t; otherwise return nil. */)
}
/* We're at the start of a comment. */
found = forw_comment (from, from_byte, stop, comnested, comstyle, 0,
- &out_charpos, &out_bytepos, &dummy);
+ &out_charpos, &out_bytepos, &dummy, &dummy2);
from = out_charpos; from_byte = out_bytepos;
if (!found)
{
@@ -2659,6 +2675,7 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
ptrdiff_t from_byte;
ptrdiff_t out_bytepos, out_charpos;
EMACS_INT dummy;
+ int dummy2;
bool multibyte_symbol_p = sexpflag && multibyte_syntax_as_symbol;
if (depth > 0) min_depth = 0;
@@ -2755,7 +2772,8 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
UPDATE_SYNTAX_TABLE_FORWARD (from);
found = forw_comment (from, from_byte, stop,
comnested, comstyle, 0,
- &out_charpos, &out_bytepos, &dummy);
+ &out_charpos, &out_bytepos, &dummy,
+ &dummy2);
from = out_charpos, from_byte = out_bytepos;
if (!found)
{
@@ -3119,7 +3137,7 @@ the prefix syntax flag (p). */)
}
\f
/* Parse forward from FROM / FROM_BYTE to END,
- assuming that FROM has state OLDSTATE (nil means FROM is start of function),
+ assuming that FROM has state STATE,
and return a description of the state of the parse at END.
If STOPBEFORE, stop at the start of an atom.
If COMMENTSTOP is 1, stop at the start of a comment.
@@ -3127,12 +3145,11 @@ the prefix syntax flag (p). */)
after the beginning of a string, or after the end of a string. */
static void
-scan_sexps_forward (struct lisp_parse_state *stateptr,
+scan_sexps_forward (struct lisp_parse_state *state,
ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t end,
EMACS_INT targetdepth, bool stopbefore,
- Lisp_Object oldstate, int commentstop)
+ int commentstop)
{
- struct lisp_parse_state state;
enum syntaxcode code;
int c1;
bool comnested;
@@ -3148,7 +3165,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
Lisp_Object tem;
ptrdiff_t prev_from; /* Keep one character before FROM. */
ptrdiff_t prev_from_byte;
- int prev_from_syntax;
+ int prev_from_syntax, prev_prev_from_syntax;
bool boundary_stop = commentstop == -1;
bool nofence;
bool found;
@@ -3165,6 +3182,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
do { prev_from = from; \
prev_from_byte = from_byte; \
temp = FETCH_CHAR_AS_MULTIBYTE (prev_from_byte); \
+ prev_prev_from_syntax = prev_from_syntax; \
prev_from_syntax = SYNTAX_WITH_FLAGS (temp); \
INC_BOTH (from, from_byte); \
if (from < end) \
@@ -3174,88 +3192,38 @@ do { prev_from = from; \
immediate_quit = 1;
QUIT;
- if (NILP (oldstate))
- {
- depth = 0;
- state.instring = -1;
- state.incomment = 0;
- state.comstyle = 0; /* comment style a by default. */
- state.comstr_start = -1; /* no comment/string seen. */
- }
- else
- {
- tem = Fcar (oldstate);
- if (!NILP (tem))
- depth = XINT (tem);
- else
- depth = 0;
-
- oldstate = Fcdr (oldstate);
- oldstate = Fcdr (oldstate);
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- /* Check whether we are inside string_fence-style string: */
- state.instring = (!NILP (tem)
- ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
- : -1);
+ depth = state->depth;
+ start_quoted = state->quoted;
+ prev_prev_from_syntax = Smax;
+ prev_from_syntax = state->prev_syntax;
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- state.incomment = (!NILP (tem)
- ? (INTEGERP (tem) ? XINT (tem) : -1)
- : 0);
-
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- start_quoted = !NILP (tem);
-
- /* if the eighth element of the list is nil, we are in comment
- style a. If it is non-nil, we are in comment style b */
- oldstate = Fcdr (oldstate);
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- state.comstyle = (NILP (tem)
- ? 0
- : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
- ? XINT (tem)
- : ST_COMMENT_STYLE));
-
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- state.comstr_start =
- RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
- oldstate = Fcdr (oldstate);
- tem = Fcar (oldstate);
- while (!NILP (tem)) /* >= second enclosing sexps. */
- {
- Lisp_Object temhd = Fcar (tem);
- if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
- curlevel->last = XINT (temhd);
- if (++curlevel == endlevel)
- curlevel--; /* error ("Nesting too deep for parser"); */
- curlevel->prev = -1;
- curlevel->last = -1;
- tem = Fcdr (tem);
- }
+ tem = state->levelstarts;
+ while (!NILP (tem)) /* >= second enclosing sexps. */
+ {
+ Lisp_Object temhd = Fcar (tem);
+ if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
+ curlevel->last = XINT (temhd);
+ if (++curlevel == endlevel)
+ curlevel--; /* error ("Nesting too deep for parser"); */
+ curlevel->prev = -1;
+ curlevel->last = -1;
+ tem = Fcdr (tem);
}
- state.quoted = 0;
- mindepth = depth;
-
curlevel->prev = -1;
curlevel->last = -1;
- SETUP_SYNTAX_TABLE (prev_from, 1);
- temp = FETCH_CHAR (prev_from_byte);
- prev_from_syntax = SYNTAX_WITH_FLAGS (temp);
- UPDATE_SYNTAX_TABLE_FORWARD (from);
+ state->quoted = 0;
+ mindepth = depth;
+
+ SETUP_SYNTAX_TABLE (from, 1);
/* Enter the loop at a place appropriate for initial state. */
- if (state.incomment)
+ if (state->incomment)
goto startincomment;
- if (state.instring >= 0)
+ if (state->instring >= 0)
{
- nofence = state.instring != ST_STRING_STYLE;
+ nofence = state->instring != ST_STRING_STYLE;
if (start_quoted)
goto startquotedinstring;
goto startinstring;
@@ -3266,11 +3234,8 @@ do { prev_from = from; \
while (from < end)
{
int syntax;
- INC_FROM;
- code = prev_from_syntax & 0xff;
- if (from < end
- && SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
+ if (SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
&& (c1 = FETCH_CHAR (from_byte),
syntax = SYNTAX_WITH_FLAGS (c1),
SYNTAX_FLAGS_COMSTART_SECOND (syntax)))
@@ -3280,32 +3245,39 @@ do { prev_from = from; \
/* Record the comment style we have entered so that only
the comment-end sequence of the same style actually
terminates the comment section. */
- state.comstyle
+ state->comstyle
= SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax);
comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax)
| SYNTAX_FLAGS_COMMENT_NESTED (syntax));
- state.incomment = comnested ? 1 : -1;
- state.comstr_start = prev_from;
+ state->incomment = comnested ? 1 : -1;
+ state->comstr_start = prev_from;
INC_FROM;
+ prev_from_syntax = Smax; /* the syntax has already been
+ "used up". */
code = Scomment;
}
- else if (code == Scomment_fence)
- {
- /* Record the comment style we have entered so that only
- the comment-end sequence of the same style actually
- terminates the comment section. */
- state.comstyle = ST_COMMENT_STYLE;
- state.incomment = -1;
- state.comstr_start = prev_from;
- code = Scomment;
- }
- else if (code == Scomment)
- {
- state.comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
- state.incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
- 1 : -1);
- state.comstr_start = prev_from;
- }
+ else
+ {
+ INC_FROM;
+ code = prev_from_syntax & 0xff;
+ if (code == Scomment_fence)
+ {
+ /* Record the comment style we have entered so that only
+ the comment-end sequence of the same style actually
+ terminates the comment section. */
+ state->comstyle = ST_COMMENT_STYLE;
+ state->incomment = -1;
+ state->comstr_start = prev_from;
+ code = Scomment;
+ }
+ else if (code == Scomment)
+ {
+ state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
+ state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
+ 1 : -1);
+ state->comstr_start = prev_from;
+ }
+ }
if (SYNTAX_FLAGS_PREFIX (prev_from_syntax))
continue;
@@ -3350,25 +3322,28 @@ do { prev_from = from; \
case Scomment_fence: /* Can't happen because it's handled above. */
case Scomment:
- if (commentstop || boundary_stop) goto done;
+ if (commentstop || boundary_stop) goto done;
startincomment:
/* The (from == BEGV) test was to enter the loop in the middle so
that we find a 2-char comment ender even if we start in the
middle of it. We don't want to do that if we're just at the
beginning of the comment (think of (*) ... (*)). */
found = forw_comment (from, from_byte, end,
- state.incomment, state.comstyle,
- (from == BEGV || from < state.comstr_start + 3)
- ? 0 : prev_from_syntax,
- &out_charpos, &out_bytepos, &state.incomment);
+ state->incomment, state->comstyle,
+ from == BEGV ? 0 : prev_from_syntax,
+ &out_charpos, &out_bytepos, &state->incomment,
+ &prev_from_syntax);
from = out_charpos; from_byte = out_bytepos;
- /* Beware! prev_from and friends are invalid now.
- Luckily, the `done' doesn't use them and the INC_FROM
- sets them to a sane value without looking at them. */
+ /* Beware! prev_from and friends (except prev_from_syntax)
+ are invalid now. Luckily, the `done' doesn't use them
+ and the INC_FROM sets them to a sane value without
+ looking at them. */
if (!found) goto done;
INC_FROM;
- state.incomment = 0;
- state.comstyle = 0; /* reset the comment style */
+ state->incomment = 0;
+ state->comstyle = 0; /* reset the comment style */
+ prev_from_syntax = Smax; /* Ensure "*|*" can't open a spurious new
+ comment. */
if (boundary_stop) goto done;
break;
@@ -3396,16 +3371,16 @@ do { prev_from = from; \
case Sstring:
case Sstring_fence:
- state.comstr_start = from - 1;
+ state->comstr_start = from - 1;
if (stopbefore) goto stop; /* this arg means stop at sexp start */
curlevel->last = prev_from;
- state.instring = (code == Sstring
+ state->instring = (code == Sstring
? (FETCH_CHAR_AS_MULTIBYTE (prev_from_byte))
: ST_STRING_STYLE);
if (boundary_stop) goto done;
startinstring:
{
- nofence = state.instring != ST_STRING_STYLE;
+ nofence = state->instring != ST_STRING_STYLE;
while (1)
{
@@ -3419,7 +3394,7 @@ do { prev_from = from; \
/* Check C_CODE here so that if the char has
a syntax-table property which says it is NOT
a string character, it does not end the string. */
- if (nofence && c == state.instring && c_code == Sstring)
+ if (nofence && c == state->instring && c_code == Sstring)
break;
switch (c_code)
@@ -3442,7 +3417,7 @@ do { prev_from = from; \
}
}
string_end:
- state.instring = -1;
+ state->instring = -1;
curlevel->prev = curlevel->last;
INC_FROM;
if (boundary_stop) goto done;
@@ -3461,25 +3436,96 @@ do { prev_from = from; \
stop: /* Here if stopping before start of sexp. */
from = prev_from; /* We have just fetched the char that starts it; */
from_byte = prev_from_byte;
+ prev_from_syntax = prev_prev_from_syntax;
goto done; /* but return the position before it. */
endquoted:
- state.quoted = 1;
+ state->quoted = 1;
done:
- state.depth = depth;
- state.mindepth = mindepth;
- state.thislevelstart = curlevel->prev;
- state.prevlevelstart
+ state->depth = depth;
+ state->mindepth = mindepth;
+ state->thislevelstart = curlevel->prev;
+ state->prevlevelstart
= (curlevel == levelstart) ? -1 : (curlevel - 1)->last;
- state.location = from;
- state.location_byte = from_byte;
- state.levelstarts = Qnil;
+ state->location = from;
+ state->location_byte = from_byte;
+ state->levelstarts = Qnil;
while (curlevel > levelstart)
- state.levelstarts = Fcons (make_number ((--curlevel)->last),
- state.levelstarts);
+ state->levelstarts = Fcons (make_number ((--curlevel)->last),
+ state->levelstarts);
+ state->prev_syntax = (SYNTAX_FLAGS_COMSTARTEND_FIRST (prev_from_syntax)
+ || state->quoted) ? prev_from_syntax : Smax;
immediate_quit = 0;
+}
+
+/* Convert a (lisp) parse state to the internal form used in
+ scan_sexps_forward. */
+static void
+internalize_parse_state (Lisp_Object external, struct lisp_parse_state *state)
+{
+ Lisp_Object tem;
+
+ if (NILP (external))
+ {
+ state->depth = 0;
+ state->instring = -1;
+ state->incomment = 0;
+ state->quoted = 0;
+ state->comstyle = 0; /* comment style a by default. */
+ state->comstr_start = -1; /* no comment/string seen. */
+ state->levelstarts = Qnil;
+ state->prev_syntax = Smax;
+ }
+ else
+ {
+ tem = Fcar (external);
+ if (!NILP (tem))
+ state->depth = XINT (tem);
+ else
+ state->depth = 0;
+
+ external = Fcdr (external);
+ external = Fcdr (external);
+ external = Fcdr (external);
+ tem = Fcar (external);
+ /* Check whether we are inside string_fence-style string: */
+ state->instring = (!NILP (tem)
+ ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
+ : -1);
+
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->incomment = (!NILP (tem)
+ ? (INTEGERP (tem) ? XINT (tem) : -1)
+ : 0);
+
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->quoted = !NILP (tem);
- *stateptr = state;
+ /* if the eighth element of the list is nil, we are in comment
+ style a. If it is non-nil, we are in comment style b */
+ external = Fcdr (external);
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->comstyle = (NILP (tem)
+ ? 0
+ : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
+ ? XINT (tem)
+ : ST_COMMENT_STYLE));
+
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->comstr_start =
+ RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->levelstarts = tem;
+
+ external = Fcdr (external);
+ tem = Fcar (external);
+ state->prev_syntax = NILP (tem) ? Smax : XINT (tem);
+ }
}
DEFUN ("parse-partial-sexp", Fparse_partial_sexp, Sparse_partial_sexp, 2, 6, 0,
@@ -3488,6 +3534,7 @@ Parsing stops at TO or when certain criteria are met;
point is set to where parsing stops.
If fifth arg OLDSTATE is omitted or nil,
parsing assumes that FROM is the beginning of a function.
+
Value is a list of elements describing final state of parsing:
0. depth in parens.
1. character address of start of innermost containing list; nil if none.
@@ -3501,16 +3548,22 @@ Value is a list of elements describing final state of parsing:
6. the minimum paren-depth encountered during this scan.
7. style of comment, if any.
8. character address of start of comment or string; nil if not in one.
- 9. Intermediate data for continuation of parsing (subject to change).
+ 9. List of positions of currently open parens, outermost first.
+10. When the last position scanned holds the first character of a
+ (potential) two character construct, the syntax of that position,
+ otherwise nil. That construct can be a two character comment
+ delimiter or an Escaped or Char-quoted character.
+11..... Possible further internal information used by `parse-partial-sexp'.
+
If third arg TARGETDEPTH is non-nil, parsing stops if the depth
in parentheses becomes equal to TARGETDEPTH.
-Fourth arg STOPBEFORE non-nil means stop when come to
+Fourth arg STOPBEFORE non-nil means stop when we come to
any character that starts a sexp.
Fifth arg OLDSTATE is a list like what this function returns.
It is used to initialize the state of the parse. Elements number 1, 2, 6
are ignored.
-Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
- If it is symbol `syntax-table', stop after the start of a comment or a
+Sixth arg COMMENTSTOP non-nil means stop after the start of a comment.
+ If it is the symbol `syntax-table', stop after the start of a comment or a
string, or after end of a comment or a string. */)
(Lisp_Object from, Lisp_Object to, Lisp_Object targetdepth,
Lisp_Object stopbefore, Lisp_Object oldstate, Lisp_Object commentstop)
@@ -3527,15 +3580,17 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
target = TYPE_MINIMUM (EMACS_INT); /* We won't reach this depth. */
validate_region (&from, &to);
+ internalize_parse_state (oldstate, &state);
scan_sexps_forward (&state, XINT (from), CHAR_TO_BYTE (XINT (from)),
XINT (to),
- target, !NILP (stopbefore), oldstate,
+ target, !NILP (stopbefore),
(NILP (commentstop)
? 0 : (EQ (commentstop, Qsyntax_table) ? -1 : 1)));
SET_PT_BOTH (state.location, state.location_byte);
- return Fcons (make_number (state.depth),
+ return
+ Fcons (make_number (state.depth),
Fcons (state.prevlevelstart < 0
? Qnil : make_number (state.prevlevelstart),
Fcons (state.thislevelstart < 0
@@ -3553,11 +3608,15 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
? Qsyntax_table
: make_number (state.comstyle))
: Qnil),
- Fcons (((state.incomment
- || (state.instring >= 0))
- ? make_number (state.comstr_start)
- : Qnil),
- Fcons (state.levelstarts, Qnil))))))))));
+ Fcons (((state.incomment
+ || (state.instring >= 0))
+ ? make_number (state.comstr_start)
+ : Qnil),
+ Fcons (state.levelstarts,
+ Fcons (state.prev_syntax == Smax
+ ? Qnil
+ : make_number (state.prev_syntax),
+ Qnil)))))))))));
}
\f
void
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply related [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-18 16:27 ` Stefan Monnier
@ 2016-03-18 19:16 ` Alan Mackenzie
2016-03-18 19:40 ` Stefan Monnier
0 siblings, 1 reply; 23+ messages in thread
From: Alan Mackenzie @ 2016-03-18 19:16 UTC (permalink / raw)
To: Stefan Monnier; +Cc: 23019
Hello, Stefan.
On Fri, Mar 18, 2016 at 12:27:36PM -0400, Stefan Monnier wrote:
> > (scan_sexps_forward): Remove a redundant state parameter. Access all `state'
> > information via the address parameter `state'.
> Have you taken a look at the performance impact of this part of the change?
> I don't expect it will make much difference, but I'm actually wondering
> whether it makes things slower or faster.
I didn't give all that much thought to it. With a "local" state,
state.field will be addressed as a constant offset from the stack frame
base register. With a "remote" state, state->field will be addressed as
a constant offset from some address register. Provided the processor
has enough registers available, it shouldn't make a difference. But on
an architecture with a restricted set of registers (?old 80x86), it might
make things slower if an address register needs to be repeatedly loaded,
or even repeatedly stacked around function calls.
I'm going to try timing it both ways: (parse-partial-sexp (point-min)
(point-max)) on xdisp.c (what else?):
Code with "->": 0.03793740272521973 seconds.
Code with "." : 0.03828787803649902 seconds.
So, at least on my machine, the "indirect" version is faster, by around
1%. Not a great difference, but I'm surprised by the way it went.
> Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-18 18:25 ` Alan Mackenzie
@ 2016-03-18 19:36 ` Stefan Monnier
2016-03-19 17:06 ` Alan Mackenzie
0 siblings, 1 reply; 23+ messages in thread
From: Stefan Monnier @ 2016-03-18 19:36 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: 23019
> We also have Scharquote, which scan_sexps_forward handles identically to
> Sescape.
Yes, it's two syntax codes which are 100% equivalent. An accident of
history I guess.
> I have bad feelings about that. Is it really worth the risk, just to
> save one cons cell on a list that not that many instances of exist at
> any time?
As you know, I like to take short term risks for long term benefits.
>> But even if we don't re-use element 5, I would actually much prefer to
>> render element 5 redundant.
> OK. Here's an updated patch which does just that. Comments would be
> welcome.
I'll take a closer look later, thanks.
Stefan
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-18 19:16 ` Alan Mackenzie
@ 2016-03-18 19:40 ` Stefan Monnier
0 siblings, 0 replies; 23+ messages in thread
From: Stefan Monnier @ 2016-03-18 19:40 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: 23019
> I didn't give all that much thought to it. With a "local" state,
> state.field will be addressed as a constant offset from the stack frame
> base register. With a "remote" state, state->field will be addressed as
> a constant offset from some address register. Provided the processor
> has enough registers available, it shouldn't make a difference. But on
> an architecture with a restricted set of registers (?old 80x86), it might
> make things slower if an address register needs to be repeatedly loaded,
> or even repeatedly stacked around function calls.
That was my first reaction as well. But my other self was telling me "I
can't say why, but my gut feeling says that this code is "cleaner"
and should hence be easier to optimize".
> So, at least on my machine, the "indirect" version is faster, by
> around 1%. Not a great difference, but I'm surprised by the way
> it went.
Thanks for the test. As expected, it's a wash, but it's good to confirm
that the cleaner version is at least no slower,
Stefan
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-18 19:36 ` Stefan Monnier
@ 2016-03-19 17:06 ` Alan Mackenzie
2016-03-20 1:30 ` Stefan Monnier
0 siblings, 1 reply; 23+ messages in thread
From: Alan Mackenzie @ 2016-03-19 17:06 UTC (permalink / raw)
To: Stefan Monnier; +Cc: 23019
Hello, Stefan.
On Fri, Mar 18, 2016 at 03:36:40PM -0400, Stefan Monnier wrote:
> > OK. Here's an updated patch which does just that. Comments would be
> > welcome.
> I'll take a closer look later, thanks.
I found some problems at ends of comments. The upshot is that
forw_comment must inform scan_sexps_forward, on a failed search, whether
the last character it scanned is still "syntactically live", or whether
that last character's syntax was "used up" in closing or opening a
comment. On a successful search, that character's syntax is always
"used up" in closing the comment.
Would you like to see the patch again, or should I just commit it?
> Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-19 17:06 ` Alan Mackenzie
@ 2016-03-20 1:30 ` Stefan Monnier
2016-03-20 13:41 ` Alan Mackenzie
2016-04-03 22:53 ` John Wiegley
0 siblings, 2 replies; 23+ messages in thread
From: Stefan Monnier @ 2016-03-20 1:30 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: 23019
> Would you like to see the patch again, or should I just commit it?
I'd like to hear what John thinks about the idea of re-using "nth 5"
instead of adding a new entry, but other than that, I think it's OK
to commit, thanks.
Stefan
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-20 1:30 ` Stefan Monnier
@ 2016-03-20 13:41 ` Alan Mackenzie
2016-04-03 22:53 ` John Wiegley
1 sibling, 0 replies; 23+ messages in thread
From: Alan Mackenzie @ 2016-03-20 13:41 UTC (permalink / raw)
To: 23019-done
Bug fixed.
On Sat, Mar 19, 2016 at 09:30:32PM -0400, Stefan Monnier wrote:
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-03-20 1:30 ` Stefan Monnier
2016-03-20 13:41 ` Alan Mackenzie
@ 2016-04-03 22:53 ` John Wiegley
2016-04-04 12:15 ` Stefan Monnier
2016-04-05 12:54 ` Alan Mackenzie
1 sibling, 2 replies; 23+ messages in thread
From: John Wiegley @ 2016-04-03 22:53 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Alan Mackenzie, 23019
>>>>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> I'd like to hear what John thinks about the idea of re-using "nth 5" instead
> of adding a new entry, but other than that, I think it's OK to commit,
> thanks.
How long has this stuff been out in the field? Do you think it's well known
enough that anyone is depending on the earlier behavior of the nth 5 value? I
have a feeling it's OK to re-use it.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-04-03 22:53 ` John Wiegley
@ 2016-04-04 12:15 ` Stefan Monnier
2016-04-05 12:54 ` Alan Mackenzie
1 sibling, 0 replies; 23+ messages in thread
From: Stefan Monnier @ 2016-04-04 12:15 UTC (permalink / raw)
To: John Wiegley; +Cc: Alan Mackenzie, 23019
>> I'd like to hear what John thinks about the idea of re-using "nth 5" instead
>> of adding a new entry, but other than that, I think it's OK to commit,
>> thanks.
> How long has this stuff been out in the field?
Many many years.
> Do you think it's well known enough that anyone is depending on the
> earlier behavior of the nth 5 value?
There are definitely packages which use the (nth 5 ..) value returned
from parse-partial-sexp. E.g. cperl-mode does:
state (parse-partial-sexp pre-B p))
(or (nth 3 state)
(nth 4 state)
(nth 5 state)
(error "`%s' inside `%s' BLOCK" A if-string))
as well as
(let ((pps (parse-partial-sexp (point) found)))
(or (nth 3 pps) (nth 4 pps) (nth 5 pps)))))
and verilog-mode does:
(setq state (parse-partial-sexp (point) end-mod-point 0 t nil))
(or (> (car state) 0) ; in parens
(nth 5 state) ; comment
))
sh-script also uses it, along with perl-mode.
> I have a feeling it's OK to re-use it.
That's also my feeling. All the uses I've found would be unaffected
(e.g. because they're in modes where there are no 2-char comment
markers, so there is really no change in behavior; or because it's only
used at positions which can't be in the middle of a 2-char comment
marker).
It's a "natural extension" of the previous meaning of "nth 5".
But admittedly, it's hard/impossible to find all uses, so I can't claim
with confidence that it won't break some code somewhere.
Stefan
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-04-03 22:53 ` John Wiegley
2016-04-04 12:15 ` Stefan Monnier
@ 2016-04-05 12:54 ` Alan Mackenzie
2016-04-05 13:50 ` Stefan Monnier
1 sibling, 1 reply; 23+ messages in thread
From: Alan Mackenzie @ 2016-04-05 12:54 UTC (permalink / raw)
To: John Wiegley; +Cc: 23019, Stefan Monnier
Hello, John.
On Sun, Apr 03, 2016 at 03:53:02PM -0700, John Wiegley wrote:
> >>>>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> > I'd like to hear what John thinks about the idea of re-using "nth 5" instead
> > of adding a new entry, but other than that, I think it's OK to commit,
> > thanks.
> How long has this stuff been out in the field? Do you think it's well known
> enough that anyone is depending on the earlier behavior of the nth 5 value? I
> have a feeling it's OK to re-use it.
My feeling is that it would be better not to change the definition of
the fifth element, but it's not a strong feeling.
One concern I have is that there is code out there which compensates for
the previous inadequate behaviour (I know there is in CC Mode), and it
may be more difficult to switch off this compensation if there isn't an
easy way to distinguish new from old, such as (> (length state) 10).
> --
> John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-04-05 12:54 ` Alan Mackenzie
@ 2016-04-05 13:50 ` Stefan Monnier
2016-04-05 14:44 ` Alan Mackenzie
0 siblings, 1 reply; 23+ messages in thread
From: Stefan Monnier @ 2016-04-05 13:50 UTC (permalink / raw)
To: Alan Mackenzie; +Cc: John Wiegley, 23019
> One concern I have is that there is code out there which compensates for
> the previous inadequate behaviour (I know there is in CC Mode), and it
> may be more difficult to switch off this compensation if there isn't an
> easy way to distinguish new from old, such as (> (length state) 10).
I'd be very surprised if other packages went to that trouble, but if
needed you can still distinguish the new from the old with something like:
(defconst pps-is-new
(let ((st (make-syntax-table)))
(modify-syntax-entry ?/ ". 14" st)
(with-temp-buffer
(with-syntax-table st
(insert "/")
(nth 5 (parse-partial-sexp (point-min) (point-max)))))))
-- Stefan
^ permalink raw reply [flat|nested] 23+ messages in thread
* bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.
2016-04-05 13:50 ` Stefan Monnier
@ 2016-04-05 14:44 ` Alan Mackenzie
0 siblings, 0 replies; 23+ messages in thread
From: Alan Mackenzie @ 2016-04-05 14:44 UTC (permalink / raw)
To: Stefan Monnier; +Cc: John Wiegley, 23019
Hello, Stefan.
On Tue, Apr 05, 2016 at 09:50:41AM -0400, Stefan Monnier wrote:
> > One concern I have is that there is code out there which compensates for
> > the previous inadequate behaviour (I know there is in CC Mode), and it
> > may be more difficult to switch off this compensation if there isn't an
> > easy way to distinguish new from old, such as (> (length state) 10).
> I'd be very surprised if other packages went to that trouble, but if
> needed you can still distinguish the new from the old with something like:
> (defconst pps-is-new
> (let ((st (make-syntax-table)))
> (modify-syntax-entry ?/ ". 14" st)
> (with-temp-buffer
> (with-syntax-table st
> (insert "/")
> (nth 5 (parse-partial-sexp (point-min) (point-max)))))))
It can certainly be done, yes, but that way it can only really be done
at set up time, wherease (> (length state) 10) could be done more or
less at any time.
It was just a small point, really.
> -- Stefan
--
Alan Mackenzie (Nuremberg, Germany).
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2016-04-05 14:44 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-03-15 9:13 bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance Alan Mackenzie
2016-03-15 9:35 ` Andreas Röhler
2016-03-15 10:15 ` Alan Mackenzie
2016-03-15 13:38 ` Andreas Röhler
2016-03-17 12:58 ` Stefan Monnier
2016-03-17 21:49 ` Alan Mackenzie
2016-03-18 4:49 ` Stefan Monnier
2016-03-18 15:11 ` Alan Mackenzie
2016-03-18 15:22 ` Alan Mackenzie
2016-03-18 16:23 ` Stefan Monnier
2016-03-18 18:25 ` Alan Mackenzie
2016-03-18 19:36 ` Stefan Monnier
2016-03-19 17:06 ` Alan Mackenzie
2016-03-20 1:30 ` Stefan Monnier
2016-03-20 13:41 ` Alan Mackenzie
2016-04-03 22:53 ` John Wiegley
2016-04-04 12:15 ` Stefan Monnier
2016-04-05 12:54 ` Alan Mackenzie
2016-04-05 13:50 ` Stefan Monnier
2016-04-05 14:44 ` Alan Mackenzie
2016-03-18 16:27 ` Stefan Monnier
2016-03-18 19:16 ` Alan Mackenzie
2016-03-18 19:40 ` Stefan Monnier
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).