bug#22983: syntax-ppss returns wrong result.

[Dmitry Gutov <dgutov@yandex.ru>]

On 03/12/2016 12:15 AM, Alan Mackenzie wrote:

>> That part of specification can be rephrased.
>
> It's more than the specification which needs redoing.  The implementation
> (sometimes) returns the equivalent of (parse-partial-sexp (point-min)
> pos)), when point-min is not in a "safe place".

Sure. I just meant that we shouldn't get hung up on that element of the 
specification.

>>>> Changing syntax-ppss to be independent of narrowing will break it, and
>>>> we'll need to provide some alternative first.
>
>>> syntax-ppss is broken, and can't be fixed.
>
>> It's used ubiquitously, so it must be working.
>
> It might well be ubiquitous, but it's broken.

And yet, it can be fixed.

> Consider this: syntax-ppss
> will return the result of a parse based at point-min.  In general, the
> caller does not know whether point-min is in a string or not.  Therefore
> the result is of little value, UNLESS the caller takes special action,
> such as widening the buffer before every call to syntax-ppss.

You can say that.

>> Not entirely different, no. AFAIK, these are the semantics the vast
>> majority of its usages expect.
>
> But it's not the semantics these .el files get.  What's probably keeping
> them functional is the rarity with which buffers are narrowed to an
> "awkward" point-min.

Another thing that keeps it together, is that narrowing, as a user-level 
operator, is not that popular.

Personally, I consider it an anti-feature.

>> Except the multiple-major-mode case, which we'd ideally try to
>> accommodate, too.
>
> How does this code handle the changeover of syntax tables at a mode
> boundary?

The "inner" regions start with an "empty", top-level state. This is 
actually fine, because these are usually small enough not to benefit 
from the syntax-ppss cache too much (and syntax-ppss-last still helps).

The parts of the outer region following a subregion with different 
syntax table... rely on a few hacks, and a manual application of a 
`syntax-table' property when necessary. We need a better solution there, 
but it's probably out of scope for this discussion.

>> syntax-ppss-dont-widen would be nil by default, it would be an escape
>> hatch toward the current semantics, for when the caller knows how to
>> manage narrowings, etc.
>
> Ah, OK.  I think I see that now.  Maybe.  Surely the trouble is that
> either ALL calls or NONE must have s-p-dont-widen set.

Hmm, you're right. This variable still seems essential, but to be safe, 
mmm-mode and friends should probably also advise syntax-ppss, to always 
perform narrowing as appropriate.

> When that flag is
> toggled, all the caches have to be cleared.  Maybe there should be some
> initialisation flag in some initialisation function.  Or something like
> that.  (It's getting late!).

Is the syntax-ppss-dont-widen really relevant for your comment cache? It 
would be used only by certain major modes, and worst comes to worst, you 
could disable the cache in those buffers.

> It strikes me that the multiple major mode stuff could do with a
> substantially enhanced version of syntax-ppss which would smoothly handle
> going over a mode boundary.  But I don't know how you're implementing
> that.

So far, we're just wrapping the font-lock and indentation code, and 
otherwise hope for the best.