bug#30393: 24.4; cperl-mode: indentation failure

[Alan Mackenzie <acm@muc.de>]

Hello, Stefan

On Sun, Feb 11, 2018 at 17:53:38 -0500, Stefan Monnier wrote:
> > OK, but I suspect in practice, this would be impossible to debug for
> > lack of reproducibility.

> Those problems can be hard to debug, indeed.  But an inf-loop should at
> least be diagnosed as such fairly easily (even if its origin can be
> difficult to track down), so I don't think "in practice I haven't bumped
> into this problem yet" just because those problems stay undetected.

> > Another definite bug is that the syntax-ppss cache is not flushed when
> > the syntax-table is changed, whether with set-syntax-table or
> > modify-syntax-entry.

> That's right.  I haven't bumped into such issues yet, but here (contrary
> to the above problem) it might very well be because the error
> stays undetected.

.... and of course, the need to flush the cache when a syntax-table text
property is applied or removed.

> > This is critical, now that primitives depend on this cache.

> I can see two approaches to solve this problem:
> - hook into set-syntax-table and modify-syntax-entry or something
>   like that.  This will make it work right everywhere automatically, but
>   I'm afraid it could turn out to be difficult to make it efficient
>   (because of the cost of the tests needed to detect changes and more
>   importantly because of excessive flushing of the syntax-ppss cache).
> - provide new functions to let packages tell syntax-ppss about
>   such things.  E.g. a macro `with-new-syntax-context` (which would
>   be treated a bit like narrowing, maybe).  This would require changes
>   to packages that suffer from this problem but should give
>   better performance.

> I'd prefer the second option, but at the same time, I'm not completely sure
> what are the "typical" problem cases (which makes it hard to come up
> with good new functions/macros) other than the case where we use
> with-syntax-table (which is sometimes combined with a local narrowing)
> but some of those only tweak the "word-vs-symbol-vs-punctuation"
> settings so should ideally not flush the syntax-ppss cache.

> Also I don't actually know whether the "fully automatic" approach would
> actually turn out to be too expensive, it's just a gut feeling.

> > Would you please fix this, Stefan.

> It's fairly high up on my todo list, but I'm kinda swamped right now.

It has occurred to me over the last day or two that I have already
solved these problems (basically, with your first approach, hooking into
set-syntax-table and friends) in the comment-cache branch, and that the
approach taken could be used more or less unchanged in the current
master.

For set-syntax-table, it compares the old and the new syntax tables to
see if they are "literally the same" (i.e. process strings and comments
identically) or "literally different", and only in the latter case does
it flush the cache.  These comparisons, which are expensive, are cached
inside the syntax-tables (in "extra slots").  Similarly, on
modify-syntax-entry, the cache is flushed iff the change affects
literals.

Similarly, on setting or deleting a syntax-table text property, the
cache is flushed from that point if the change affects literals.  This
happens regardless of the setting of inhibit-modification-hooks, etc.

It is a fact that the vast bulk of libraries which use syntax-ppss use
only elements 3, 4, and 8, i.e. the ones relevant to literals, and
ignore everything else.  For these the scheme outlined above is
rigorous.  I have timed it in the comment-cache branch, scanning through
.../src/xdisp.c displaying each screen, and found no difference to the
approach without comment-cache.

For those few libraries which do use the full capabilities of the
parsing state, we would need to flush the cache on all
set-syntax-table's and so on.  Maybe.  Maybe this would be too expensive
in run time.

So the interface I propose would be two abnormal hooks, one for
"literally important" changes to the syntax, and the other for other
changes to the syntax.  The hook functions would take an optional
argument which would be nil for changes to the syntax table, or a buffer
position where a syntax-table property is being changed.

Mostly, only the first of these hooks need be used, the standard
function on them being syntax-ppss's flush function.  Major modes could
add syntax-ppss's flush function to the second hook (possibly through
some nice interface), should they use the non-literal parts of the parse
state.

One or two incidental changes would be needed, for example to fix the
infinite recursion in printing syntax-tables, caused by the mutual
presence of "literally the same/different" syntax tables in the extra
slots.  This would not be difficult.

Then, finally, if we can be bothered, we could put in a mechanism to
deal with changes in parse-sexp-lookup-properties and
parse-sexp-ignore-comments.

What do you think?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).