Re: Syntax tables for multiple modes [was: bug#22983: syntax-ppss returns wrong result.]

[Stefan Monnier <monnier@iro.umontreal.ca>]

> parse-partial-sexp is called from code exclusively and it just happens
> that in multi-modes it is called outside of narrow region quite often.

How/why?  Can you give some concrete scenario?

> That's a major inconvenience. Why on earth one would need to
> take account in user narrowing for syntax parsing?

Because you need it for *everything* that looks at the buffer.
Why should parse-partial-sexp be different from (say) scan-sexps?

> If parse-partial-sexp could be made to always widen to hard limits it
> will automatically solve a bunch of problems. bug#22983 being one of them,

Bug#22983 should be fixed by widening, indeed, but it should be done in
syntax.el.  Widening in parse-partial-sexp would only address some cases
but not all (e.g. the syntax-begin-function cases or the
syntax-propertize-function cases).  Those other cases can only be fixed
in syntax.el.

> the ubiquitous out-of-range errors in font-lock in multi-modes being
> the most important one.

I'm not familiar with those, so if you could give some examples it
would help us judge if they would indeed benefit from a fix in
parse-partial-sexp rather than elsewhere.

>> Notice how syntax-ppss is different in this regard: since it doesn't
>> receive FROM, that same rule doesn't prevent syntax-ppss from widening
>> to (car hard-widen-limits).
> Well, not quite different. It has POS which might be outside of user narrowed
> range.

No: POS should be within point-min/max.

> Major mode author has to deal with the span explicitly as defined in
> previous-chunk in prog-indentation-context. Cognitively this is a more
> demanding task. Ask a new person to go and read the doc of
> prog-indentation-context and ask how much he or she understands of
> it. I read it and I think I understand most of it, but looking at all
> the usages of prog-widen and prog-first-column in python.el my brain
> gives up. Previous-chunk is not even used in python.el!

Replace all your widen calls with calls to `prog-widen' and you get
the same result (since (nth 1 prog-indentation-context) is basically
another name for your hard-widen-limit).  So I don't think prog-widen is
that very complicated.

As for prog-first-column the local major mode can just ignore it in
which case the multi-mode can do the same that you do.  It's only useful
if you need/want to provide a more complex behavior than what
polymode supports.  So, of course, it's more complex.

> The prog-calculate-indent-function is more general. You can call it on
> any buffer position (need not be last point in the previous span).

[ Note: In my mind, the "natural main case" for multi-mode indentation
  is when you call the indentation function on the *first position* of
  a span.  But you seem to look at it from the other end, where you call
  the indentation function on the *last position* of the previous span.
  I think I'm beginning to see why.  ]

Note that "is more general" here means that the major mode's function
has to handle more cases, so it would seem to fundamentally require more
work on the major mode's side.

> Current prog-indentation-context allows for possibility of a string to
> be inserted before begging in of current chunk.  STRING-BEFORE is more
> more general than that because of the arbitrary POS that it can be
> applied to. 

Good point.  I didn't think of that.  Do you make use of that
possibility, and/or can you give an example where it's useful?

> Here is a simple example when inner mode cannot decide by itself on
> the indentation.  Assume for concreteness a noweb header with some code
> immediately following the header:
>
>   <<foo, some_arg=4>>= some_call(blabla) 
>       some_other_call(blabla) ## indented by offset 2 with respect to header or prev_chunk
>
> How do you indent the some_call(blabal) after the header? The most
> meaningful way is to keep it untouched just as user defined it. If
> inner mode would indent it by itself it would give offset of 4. This
> is a simple example of header dependence.

Maybe it's because I'm not familiar with noweb, but I didn't understand
this example.  It looks like a very interesting example, so could you go
over it again in much more detail?


        Stefan