bug#22983: syntax-ppss returns wrong result.

[Alan Mackenzie <acm@muc.de>]

Hello, Dmitry.

On Fri, Mar 11, 2016 at 11:35:08PM +0200, Dmitry Gutov wrote:
> On 03/11/2016 11:24 PM, Alan Mackenzie wrote:

> > Er no, I meant what I wrote: the result of (syntax-ppss pos) must match
> > that of (parse-partial-sexp (point-min) pos).  I think ppss-0 and ppss-1
> > did actually match (but I can't quite remember).

> I imagine they didn't. I got the same value in all three cases, though, 
> so your scenario could use some revising.

Sorry about that.

> >> Considering narrowing can change point-min arbitrarily, specifying
> >> (syntax-ppss pos) as (parse-partial-sexp (point-min) pos) is a losing
> >> proposition if you want consistency.

> > Indeed.  But that is how syntax-ppss is specified, and (partially) how
> > it is implemented.

> That part of specification can be rephrased.

It's more than the specification which needs redoing.  The implementation
(sometimes) returns the equivalent of (parse-partial-sexp (point-min)
pos)), when point-min is not in a "safe place".

> >> Alas, we have some code out there that implements multiple-major-mode
> >> functionality using narrowing and some hacking of syntax-ppss-last
> >> syntax-ppss-cache values.

> >> Changing syntax-ppss to be independent of narrowing will break it, and
> >> we'll need to provide some alternative first.

> > syntax-ppss is broken, and can't be fixed.

> It's used ubiquitously, so it must be working.

It might well be ubiquitous, but it's broken.  Consider this: syntax-ppss
will return the result of a parse based at point-min.  In general, the
caller does not know whether point-min is in a string or not.  Therefore
the result is of little value, UNLESS the caller takes special action,
such as widening the buffer before every call to syntax-ppss.

> > The only sensible fix would be to specify that (syntax-ppss pos) is
> > the same as (parse-partial-sexp 1 pos).  But that is then a totally
> > different function, and there are around 200 uses in the Emacs
> > sources to check and fix, to say nothing of external code.

> Not entirely different, no. AFAIK, these are the semantics the vast 
> majority of its usages expect.

But it's not the semantics these .el files get.  What's probably keeping
them functional is the rarity with which buffers are narrowed to an
"awkward" point-min.

> Except the multiple-major-mode case, which we'd ideally try to
> accommodate, too.

How does this code handle the changeover of syntax tables at a mode
boundary?

> >> We could introduce a syntax-ppss-dont-widen variable, though. Similar to
> >> font-lock-dont-widen.

> > I'm trying to figure that out.  Wouldn't that still leave you with
> > problems when point-min is inside a string?

> syntax-ppss-dont-widen would be nil by default, it would be an escape 
> hatch toward the current semantics, for when the caller knows how to 
> manage narrowings, etc.

Ah, OK.  I think I see that now.  Maybe.  Surely the trouble is that
either ALL calls or NONE must have s-p-dont-widen set.  When that flag is
toggled, all the caches have to be cleared.  Maybe there should be some
initialisation flag in some initialisation function.  Or something like
that.  (It's getting late!).

It strikes me that the multiple major mode stuff could do with a
substantially enhanced version of syntax-ppss which would smoothly handle
going over a mode boundary.  But I don't know how you're implementing
that.

-- 
Alan Mackenzie (Nuremberg, Germany).