Re: Syntax tables for multiple modes [was: bug#22983: syntax-ppss returns wrong result.]

[Dmitry Gutov <dgutov@yandex.ru>]

On 03/20/2016 02:15 PM, Vitalie Spinu wrote:

> Well, font-lock-dont-widen is not respected even in c-mode. Look at
> c-before-context-fl-expand-region and semi-safe-place which are called directly
> or indirectly from c-font-lock-fontify-region.

Well, yes. c-mode is special, as usual. That should be workable if CC 
Mode starts using prog-widen instead of widen, though.

>> For indentation, we've introduced prog-indentation-context recently. And
>> indentation functions in programming modes are supposed to call prog-widen
>> instead of widen now.
>
> I was not aware of that. Not sure if it is a step in right direction though.

I'm not 100% happy with it either.

> `prog-indentation-context` looks fine to me but multi-modes already have their
> own wrappers for indentation which do just that according to their own semantics
> of modes/submodes/chunks/headers etc.

Too bad you were not around when this addition was discussed.

> The primary intent of `prog-indentation-context` is to be used in
> `prog-widen`. This part seems like a major complication. All mode authors now
> have to understand what is prog-widen, prog-first-column and
> prog-indentation-context. Why to burden prog-mode authors with notions that
> multi-mode engines can take care themselves?

IIRC, using first-column is fairly justified, the outer mode can't add 
extra indentation to the submode is a reliable, sane way (though I've 
also been hacking around that quite successfully). Here's the full 
discussion:

http://lists.gnu.org/archive/html/emacs-devel/2015-01/msg00431.html
http://lists.gnu.org/archive/html/emacs-devel/2015-02/msg00290.html

with my messages further down.

> It is also not clear to me why should prog-widen be used in indentation context
> only? It makes perfect sense for this function to be used in font-locking and
> syntax-propertize-function as well.

Indeed. In js-mode's case, the offending code is called from 
font-lock-keywords, for example.

> It's essentially a half-backed implementation of "hard widening" discussed
> earlier. Why not impose the widening restriction directly in `widen` then? Maybe
> bring widen to elisp and rename C widen into widen-internal. Then add generic
> `prog-hard-widen-limits` which would be checked along prog-indentation-context
> limits.

Right! At the very least, I we should extract the second element of 
prog-indentation-context into a separate variable, and make prog-widen 
more prominent.

But a proper implementation of hard-widen would be even better in my 
book. Although someone would need to comb through all low-level 
functions, at least, and decide which of them need to call 
widen-internal, and which will be fine with just widen.

Are you interested in working on a patch? Also Cc'ing Stefan.

Looking back on it, it seems prog-indentation-context was merged too 
early: it only has one usage so far, so it's not clear if the approach 
is generally viable.

Christoph sort of promised to add support in CC Mode, but then 
disappeared. Which is not so surprising, that stuff is difficult.

> Unless I miss something essential it's really not worth imposing such
> complexities on mode authors. Judging from the python.el, which is the only mode
> using prog-first-column so far, it's not a trivial task. Each mode author will
> basically have to implement indentation logic that mmm-mode or polymode already
> implement. And even then, multi-mode engines will probably need to overwrite
> that because the semantics of submode spans is either emacs-mode or
> multi-mode-engine specific.

This is not too different what I was saying, I think. That discussion is 
fairly long, though, and it veered off to the side.

AFAICT, though, the ultimate justification for having first-column is 
Python's indentation cycling behavior: 
http://lists.gnu.org/archive/html/emacs-devel/2015-02/msg01096.html

Which is not that convincing, but makes some things clearner anyway.

But the last element, previous-chunks, is still not used anywhere in 
Emacs. I think including it turned out to be a mistake, or at least 
premature.

>> syntax-propertize-function's aren't supposed to call widen at all, I think.
>
> This should probably be in the docs then.

Probably.

> Mode authors can decide to do loads of
> work in there. One instance is `markdown-mode` which caches all font-lock
> properties in syntax-propertize-function. While markdown-mode is clean and
> doesn't use widen anywhere, that might not be the case for other modes.

ruby-syntax-propertize also does some involved parsing, but as long as 
there's no `widen' there, we should be fine.