Clarification needed: syntax-propertize vs font-lock-syntax-table

Clarification needed: syntax-propertize vs font-lock-syntax-table

[Wedler, Christoph]

The function `syntax-propertize' sets the syntax-table properties which
shadow the definition from the current syntax-table.  This function
calls the function in syntax-propertize-function whose docstring
explicitly allows to call `syntax-ppss'.

For this to work consistently, it is necessary that `syntax-propertize'
is called with the same (or very similar) syntax-table, at least for the
same buffer areas.  (Yes, I see the Todo at the beginning of syntax.el)
Or do I miss something?

The most prominent call of `syntax-propertize' is inside font-lock,
where the `font-lock-syntax-table' is active.  But there are also many
valid calls of this function (most via `syntax-ppss') where the standard
(syntax-table) is active.  (And there are missing calls - hello imenu.el)

What does this mean?  To be consistent, ...

 * Solution 1: the font-lock syntax table must not differ too much from
   the normal syntax table, i.e., the function in
   syntax-propertize-function should behave the same.  If so, it should
   be probably documented.

 * Solution 2: there is a local variable for an extra syntax table for
   `syntax-propertize'

Any other ideas?

With Solution 1, it would be excellent if the (nth 9 ppss),
parse-partial-sexp, would be made official.  Otherwise, the multi-mode
check in antlr-mode.el (are we in the action of a grammar?) could not
rely on (nth 0 ppss) anymore...

- Christoph

Re: Clarification needed: syntax-propertize vs font-lock-syntax-table

[Dmitry Gutov]

On 11/18/2015 08:06 PM, Wedler, Christoph wrote:

>   * Solution 1: the font-lock syntax table must not differ too much from
>     the normal syntax table, i.e., the function in

I think this is the current consensus. It shouldn't differ in paren, 
comment or string syntax. IOW, it should be syntax-ppss-compatible with 
the main one.

>     syntax-propertize-function should behave the same.  If so, it should
>     be probably documented.

It should.

> Any other ideas?

Support different syntax tables in different parts of the buffer. That 
must be a part of our eventual multi-major-mode solution.

RE: Clarification needed: syntax-propertize vs font-lock-syntax-table

[Wedler, Christoph]

> On 11/18/2015 08:06 PM, Wedler, Christoph wrote:

>>   * Solution 1: the font-lock syntax table must not differ too much from
>>     the normal syntax table, i.e., the function in

> I think this is the current consensus. It shouldn't differ in paren, 
> comment or string syntax. IOW, it should be syntax-ppss-compatible with 
> the main one.

OK, fine with me.

>     syntax-propertize-function should behave the same.  If so, it should
>     be probably documented.

> It should.

And syntax-propertize should be called at more places.  Currently, doing
it lazily does not really work, as it is not called when needed.  That
is why js-mode calls (syntax-propertize (point-max)) at the end (and I
will do the same in antlr-mode).


My other question was concerning (nth 9 ppss), i.e. the list of open
parentheses is very useful to have (and its also used in syntax.el
itself) -> I would like to have this to be official in the lisp
docstring as well - in src/syntax.c, we have

  struct lisp_parse_state
    {
      [...]
      Lisp_Object levelstarts; /* Char numbers of starts-of-expression
  				 of levels (starting from outermost).  */
    };


>> Any other ideas?

> Support different syntax tables in different parts of the buffer. That 
> must be a part of our eventual multi-major-mode solution.

In the case of antlr-mode (grammar with actions), it could be done as
long as it does not break the calulation of the "inner-mode-chunk-end",
e.g, with Python actions

   rule: "KEYWORD" { do_something {  } # do it };

the action ends with the final "}" (even though a naive Python-like
syntax-propertization would think of it as part of a comment)

Re: Clarification needed: syntax-propertize vs font-lock-syntax-table

[Dmitry Gutov]

On 11/20/2015 08:24 PM, Wedler, Christoph wrote:

> And syntax-propertize should be called at more places.  Currently, doing
> it lazily does not really work, as it is not called when needed.

You'll have to be more specific.

> That
> is why js-mode calls (syntax-propertize (point-max)) at the end (and I

Not anymore, it doesn't.

> will do the same in antlr-mode).

I'd prefer if you didn't.

> My other question was concerning (nth 9 ppss), i.e. the list of open
> parentheses is very useful to have (and its also used in syntax.el
> itself) -> I would like to have this to be official in the lisp
> docstring as well - in src/syntax.c, we have

I thought that's exactly what you meant by documenting the 9th element. 
Were there any other options?

I'm not the guy who can say if it's okay or not. Probably is. But it 
would be sad if after that we choose a way to adapt syntax-ppss for the 
multiple-modes case, and it will require saving a more complex structure 
in the 9th element.

> In the case of antlr-mode (grammar with actions), it could be done as
> long as it does not break the calulation of the "inner-mode-chunk-end",
> e.g, with Python actions
>
>     rule: "KEYWORD" { do_something {  } # do it };
>
> the action ends with the final "}" (even though a naive Python-like
> syntax-propertization would think of it as part of a comment)

Yes, of course we should try to make it break as few things as possible. 
Not sure if I could comment on this in any more detail.

RE: Clarification needed: syntax-propertize vs font-lock-syntax-table

[Wedler, Christoph]

>> And syntax-propertize should be called at more places.  Currently, doing
>> it lazily does not really work, as it is not called when needed.

> You'll have to be more specific.

Well, I realized that (with Emacs-24.4) the imenu list in antlr-mode
wasn't correctly before I put (syntax-propertize (point-max)) into
antlr-imenu-create-index-function.  I could imagine similar things in
other modes.

I have not seen other issues, but I do use font-lock (which also calls
syntax-propertize), as it is the default.  I could imagine that issues
could appear for people who have switched off font-lock.  (Needs to be
tested.)

>> That is why js-mode calls (syntax-propertize (point-max)) at the end
>>

> Y Not anymore, it doesn't.

You are right...

>> (and I will do the same in antlr-mode).

> I'd prefer if you didn't.

I probably define a variable for that which will be t for Emacs below
emacs-25.x and nil with the next release.

>> My other question was concerning (nth 9 ppss), i.e. the list of open
>> parentheses is very useful to have (and its also used in syntax.el
>> itself) -> I would like to have this to be official in the lisp
>> docstring as well - in src/syntax.c, we have

> I thought that's exactly what you meant by documenting the 9th element. 
> Were there any other options?

The other documentation request was for font-lock syntax-table as being
"syntax-ppss-compatible" (as you put it nicely).

> I'm not the guy who can say if it's okay or not. Probably is. But it 
> would be sad if after that we choose a way to adapt syntax-ppss for the 
> multiple-modes case, and it will require saving a more complex structure 
> in the 9th element.

Agreed.  How about a function like

(defun syntax-ppss-open-list-positions (ppss)
  "Get all syntactically open list position found in a syntactic scan.
PPSS is a scan state, as returned by `parse-partial-sexp' or `syntax-ppss'.
The start position of the outermost list comes first."
  (nth 9 ppss))

This function needs to be changed if the internal structure of (nth 9
ppss) is changed.

>> In the case of antlr-mode (grammar with actions), it could be done as
>> long as it does not break the calulation of the "inner-mode-chunk-end",
>> e.g, with Python actions
>>
>>     rule: "KEYWORD" { do_something {  } # do it };
>>
>> the action ends with the final "}" (even though a naive Python-like
>> syntax-propertization would think of it as part of a comment)

> Yes, of course we should try to make it break as few things as possible. 
> Not sure if I could comment on this in any more detail.

We might need something (an extra syntax flag?) which makes it easy to
"rewind" the ppss such that

  (equal (syntax-ppss inner-mode-end-position)
         (syntax-ppss inner-mode-start-position))

Re: Clarification needed: syntax-propertize vs font-lock-syntax-table

[Dmitry Gutov]

On 11/23/2015 07:05 PM, Wedler, Christoph wrote:

> Well, I realized that (with Emacs-24.4) the imenu list in antlr-mode
> wasn't correctly before I put (syntax-propertize (point-max)) into
> antlr-imenu-create-index-function.  I could imagine similar things in
> other modes.

In other functions, you mean. Yes, various facilities that depend on 
syntax properties should call syntax-propertize at appropriate times.

Note that calling (syntax-propertize (point-max)), as js-mode was doing, 
is not a guarantee of the file staying entirely up-to-date WRT syntax 
properties as the user edits it.

Neither is using font-lock: we fontify the file lazily, and thus also 
only propertize the buffer near its visible part.

> I probably define a variable for that which will be t for Emacs below
> emacs-25.x and nil with the next release.

But nothing has changed in Emacs 25 in that regard (I think?). We 
changed js-mode behavior, but mostly because it was a kludge anyway, and 
not because anything else is compensating for it now.

> The other documentation request was for font-lock syntax-table as being
> "syntax-ppss-compatible" (as you put it nicely).

I'd be happy to review the documentation patch. We can also ask Stefan 
to do that.

> Agreed.  How about a function like
>
> (defun syntax-ppss-open-list-positions (ppss)
>    "Get all syntactically open list position found in a syntactic scan.
> PPSS is a scan state, as returned by `parse-partial-sexp' or `syntax-ppss'.
> The start position of the outermost list comes first."
>    (nth 9 ppss))
>
> This function needs to be changed if the internal structure of (nth 9
> ppss) is changed.

Yes, that sounds good (maybe choose a shorter name?). But see above.

> We might need something (an extra syntax flag?) which makes it easy to
> "rewind" the ppss such that
>
>    (equal (syntax-ppss inner-mode-end-position)
>           (syntax-ppss inner-mode-start-position))

That sounds like it might require turning the current "list of openers" 
structure into a list-of-lists, where each element is the current "list 
of openers", plus some boundary information.

Or maybe handle that in some entirely different way, like keeping the 
current spss format, but allow the buffers to override the syntax-ppss 
logic via a newly introduced syntax-ppss-function variable.

Someone could try implementing the latter approach right now, using advice.

RE: Clarification needed: syntax-propertize vs font-lock-syntax-table

[Wedler, Christoph]

> Note that calling (syntax-propertize (point-max)), as js-mode was doing, 
> is not a guarantee of the file staying entirely up-to-date WRT syntax 
> properties as the user edits it.

I now checked the code: it is done via jit-lock-after-change in
after-change-functions.  As I have guessed, syntax-propertize currently
only works correctly if font-lock is switched on.

> Neither is using font-lock: we fontify the file lazily, and thus also 
> only propertize the buffer near its visible part.

Yes, but navigation command like M-x forward-list do not depend on
font-lock, but they do depend on syntax-propertize (consider parentheses
in generic strings).

> But nothing has changed in Emacs 25 in that regard (I think?). We 
> changed js-mode behavior, but mostly because it was a kludge anyway, and 
> not because anything else is compensating for it now.

Well, then I think that the change in js is wrong.  (Yes, a fully lazy
solution is better, but as long as it is not there...)

>> The other documentation request was for font-lock syntax-table as being
>> "syntax-ppss-compatible" (as you put it nicely).

> I'd be happy to review the documentation patch. We can also ask Stefan 
> to do that.

OK, I send a suggestion later this week, together with:

>> Agreed.  How about a function like
>>
>> (defun syntax-ppss-open-list-positions (ppss)
>>    "Get all syntactically open list position found in a syntactic scan.
>> PPSS is a scan state, as returned by `parse-partial-sexp' or `syntax-ppss'.
>> The start position of the outermost list comes first."
>>    (nth 9 ppss))
>>
>> This function needs to be changed if the internal structure of (nth 9
>> ppss) is changed.

> Yes, that sounds good (maybe choose a shorter name?). But see above.

Ok, I'll name it syntax-ppss-open-positions (like (nth 0 ppss) is called
syntax-ppss-depth, and not syntax-ppss-list-depth).

>> We might need something (an extra syntax flag?) which makes it easy to
>> "rewind" the ppss such that
>>
>>    (equal (syntax-ppss inner-mode-end-position)
>>           (syntax-ppss inner-mode-start-position))

> That sounds like it might require turning the current "list of openers" 
> structure into a list-of-lists, where each element is the current "list 
> of openers", plus some boundary information.

> Or maybe handle that in some entirely different way, like keeping the 
> current spss format, but allow the buffers to override the syntax-ppss 
> logic via a newly introduced syntax-ppss-function variable.

Well, that was the point of the above function: in the former case, the
function body is changed to (mapcar 'car (nth 9 ppss)), and will stay
the same in the latter.

Re: Clarification needed: syntax-propertize vs font-lock-syntax-table

[Dmitry Gutov]

On 11/24/2015 10:58 AM, Wedler, Christoph wrote:

> I now checked the code: it is done via jit-lock-after-change in
> after-change-functions.  As I have guessed, syntax-propertize currently
> only works correctly if font-lock is switched on.

font-lock depends on syntax-propertize. Not the other way around.

> Yes, but navigation command like M-x forward-list do not depend on
> font-lock, but they do depend on syntax-propertize (consider parentheses
> in generic strings).

Hm, actually, that has changed in 
ab21f61a552f038485d40218dfd94a16b843eb52. So yes, there's a difference 
between Emacs 24 and 25 here. Sorry for the confusion.

> Ok, I'll name it syntax-ppss-open-positions (like (nth 0 ppss) is called
> syntax-ppss-depth, and not syntax-ppss-list-depth).

Sounds good.

> Well, that was the point of the above function: in the former case, the
> function body is changed to (mapcar 'car (nth 9 ppss)), and will stay
> the same in the latter.

Sorry, I lost you here. Former case?