Re: /srv/bzr/emacs/trunk r101338: * lisp/emacs-lisp/syntax.el (syntax-ppss): More sanity check to catch

[Dmitry Gutov <dgutov@yandex.ru>]

On 12.02.2014 03:30, Stefan Monnier wrote:
> E.g. I had some font-lock code which would highlight an
> open-paren-in-column-0-in-string/comment with the `warning' face.
> So such an "incorrect" open paren would still cause incorrect
> highlighting, but the `warning' face on it would provide the clue as to
> what was the source of the problem.

I don't fully understand the explanation, but the logic "if 
syntax-beginning equals point, go to previous syntax-beginning" could've 
been handled in the specific syntax-beginning-function instead.

> Right, but that largely defeats the purpose of syntax-ppss (which is to
> use caching to speed up (parse-partial-sexp (point-min) (point))).

The optimization is still used if `syntax-ppss' is called several times 
during the syntax-propertization or fontification of one region.

Same with indentation, if we did that.

> To give you some background: I think syntax-begin-function is basically
> useless.  It's used in very few places (it used to be used in lisp-mode,
> but that was disabled recently, it's still used in js-mode, but it
> should probably be disabled there as well, and apparently mmm-mode also
> uses it, but these are the only cases I know) and is more trouble than
> it's worth.  It was meant and is designed as an optimization, but it is
> vanishingly often useful.

Okay, I can understand that.

> One option is to have a hook that takes a (POS . PPSS) pair, which
> syntax-ppss intends to use as a starting point for parsing, and return
> a new such pair to use instead, where the returned position should
> always be >= POS.

Sounds fine to me. As long as the hook is called at the same point 
`syntax-ppss' is called at, we can check whether POS is in the same 
region, look for nested submode regions between POS and point, and 
either discard the passed PPSS if the current subregion begins after 
POS, or manually `parse-partial-sexp' each piece of the current 
subregion (of the primary mode region, if we're there) between POS and 
some position closer to point.

We could parse the buffer till point itself, though. It wouldn't be 
harder coding-wise (we'll do `parse-partial-sexp's anyway), and that way 
the hook could be more flexible. Then the meaning of the hook would be 
"here's the last saved position and value, what will be the value at 
point?".

> This way, syntax-ppss could make full use of its cache, but mmm-mode
> could tell it about chunk boundaries (and decide what state to use at
> the beginning of a boundary).
>
> The main problem I see with this approach is that this hook would be
> called maybe too many times, so we'd want to improve the "fast path"
> (i.e. the first branch in syntax-ppss which tries to use
> syntax-ppss-last) so it can know when calling this new hook is unneeded.

Maybe we want that, but scanning the buffer for overlays should still be 
a) proportional to the distance between bounds, b) faster than 
`parse-partial-sexp', so at worst in mmm-mode the new scheme will just 
be slower than plain `syntax-ppss' by some constant ratio, on average.

> Maybe for that, the new hook should return not just a new (POS . PPSS)
> but also a "next-boundary" so we know we don't need to call this hook
> again as long as we're within POS...NEXT-BOUNDARY.

Not sure if it'll work. Suppose we're in some region, which spans 400 
chars after point, and then it's another region.

We call `syntax-ppss', happily report to it that the value at point (or 
some position near it) can be used until point + 400. Then move a few 
chars lower and delete the rest of the given region. NEXT-BOUNDARY 
becomes stale, and calling `syntax-ppss' from the region below can 
return a wrong value.

Using markers should work better, but maybe some problems are lurking 
there as well.