Re: Syntax tables for multiple modes [was: bug#22983: syntax-ppss returns wrong result.]

[Vitalie Spinu <spinuvit@gmail.com>]

>> On Sun, Mar 20 2016 17:58, Dmitry Gutov wrote:

> On 03/20/2016 02:15 PM, Vitalie Spinu wrote:

> IIRC, using first-column is fairly justified, the outer mode can't add extra
> indentation to the submode is a reliable, sane way

The inner mode cannot often make that decision either. Same inner mode can be
used in very different multi-mode contexts, each with their own semantics for
chunks/headers/indentation. Reducing all that to a simple (first-column
. previous-chunk) pair and letting inner mode do the job is surely not
enough. The only actor to make that decision should be multi-mode engine itself.

Instead of teaching modes about multi-modes, a much better idea is to introduce
`calculate-indent-function` which would accept POS and optional STRING-AFTER and
STRING-BEFORE. This function will return the indentation of STRING-AFTER at POS
assuming there is a virtual STRING-BEFORE just before POS.

This way, a multi-mode engine can call inner-mode's calculate-indent-function at
the end of previous chunk with STRING-AFTER being the line at point and
STRING-BEFORE being the content of current chunk. Most modes indent reliably
based on one previous line, so in 99% of the cases STRING-BEFORE can be nil and
multi-mode engine can call calculate-indent-function only on first line of the
current chunk (and that only for continuation chunks, which are a minority out
there). Then a lot of modes don't even care about what's in the current line, so
STRING-AFTER will be irrelevant as well. Thus most modes will not even need a an
implementation of calculate-indent-function.

This is both more general than prog-indentation-context and doesn't require
teaching major-modes about multi-modes. Moreover, a lot of major-modes already
have such a "calculator" in place.

>> It's essentially a half-backed implementation of "hard widening" discussed
>> earlier. Why not impose the widening restriction directly in `widen` then?
>> Maybe bring widen to elisp and rename C widen into widen-internal. Then add
>> generic `prog-hard-widen-limits` which would be checked along
>> prog-indentation-context limits.

> Right! At the very least, I we should extract the second element of
> prog-indentation-context into a separate variable, and make prog-widen more
> prominent.

Not sure about removing second element. Good thing about keeping all of them in
one place is for the indentation engine to be concerned with a single variable.

BTW, third argument should be renamed into PREVIOUS-CHUNK. The function returns
one chunk.

> But a proper implementation of hard-widen would be even better in my
> book. Although someone would need to comb through all low-level functions, at
> least, and decide which of them need to call widen-internal, and which will be
> fine with just widen.

No need to decide on widen-internal. All functions are free to call widen just
as they do before. It's 100% backward compatible. The only reason to use
`widen-internal` is to bring `widen` to elisp in order to allow for advise and
better debugging. Actually, with hard-widen-limits, there will be no need for
advice, so it can be kept in C.

Only consumers of `hard-widen-limits` should be concerned with its side
effects. But that's uniformly better than current situation when you cannot do
much about restricting widen.

In my experience hard-widen and parse-partial-sexp are the only hurdle in the
way of proper multi-modes. I don't remember a single problem that would occur
for a different reason.

BTW, I parse-partial-sexp must abide hard-widen-limits as well. This way the
request aired in bug#22983 of parse-partial-sexp == syntax-ppss will be
automatically satisfied. You won't need syntax-ppss-dont-widen either.

> Are you interested in working on a patch? Also Cc'ing Stefan.

My knowledge of emacs C internals is close to 0. Elisp side (and probably C
side) of this is trivial. I will look into it but I don't think I am the best
person for that.

> Looking back on it, it seems prog-indentation-context was merged too early: it
> only has one usage so far, so it's not clear if the approach is generally
> viable.

> Christoph sort of promised to add support in CC Mode, but then
> disappeared. Which is not so surprising, that stuff is difficult.

A patch that would require hunting every single mode out there and implementing
multi-modes locally should have been more carefully considered IMO. Emacs 25 is
not yet there, so it's not late to reconsider that decision.

> AFAICT, though, the ultimate justification for having first-column is Python's
> indentation cycling behavior:
> http://lists.gnu.org/archive/html/emacs-devel/2015-02/msg01096.html

> Which is not that convincing, but makes some things clearner anyway.

It's not convincing to me either. I use Christoph's indentation-0 trick in and
it indeed works reliably for all modes I have tried except python. But python's
issue can be fixed with a simple advice of python-indent-line-function, no need
to overhaul python indentation because of that. This is how it's now done in
polymode:

  https://github.com/vspinu/polymode/blob/master/polymode-compat.el#L189-L199


  Vitalie