Re: forward-comment and syntax-ppss

[Dmitry Gutov <dgutov@yandex.ru>]

On 19.12.2016 16:17, Stefan Monnier wrote:

> Then I misunderstood.

Maybe I worded that too strongly. How about consider the major 
facilities case-by-case, and see if moving widening up the stack makes 
sense? Just doing that for indentation and font-lock (maybe there's even 
nothing to change there) would be a major win already.

> What is the problem that your proposal aims to fix?

To solve the "widen problem" that mixed-mode frameworks have with 
existing modes. Namely, when they delegate to some language-specific 
code, it could call `widen' before doing its work. Examples of 
language-specific code are the values of font-lock-keywords, 
indent-line-function and imenu-create-index-function (though the last 
one is less important).

And to solve them in a non-intrusive way that does not ask for new 
abstractions, and thus does not affect the rest of the discussion (about 
syntax-ppss, forward-comment, Info). That should simplify things.

>> This seems necessary to have a stable syntax cache, e.g. to have it
>> as a basis of comments navigation.
>
> But if we have something like syntax-ppss-parse-start, why not use that
> same info more generally when some function wants to widen (e.g. make
> it also replace font-lock-dont-widen)?

I'd imagine having a new variable syntax-dont-widen, that would serve 
both for syntax-ppss and font-lock. Still, that does not resolve the 
issue of indexing by point-min.

> But the problem is the linkage between the narrowing and the widening,
> not between the user-command and the widening.

There are different problems. I'm most focused on the issue of code 
undoing narrowing willy-nilly.

> In general, the command currently executed does not know who put the
> narrowing nor why, so it still doesn't know what's the intention behind
> it, and hence can't decide whether to widen or not, nor or much to widen.

Aside from the problem of Info (which we know can be implemented in a 
different way), an interactive command does know, with 98% probability, 
that it's the user who instituted the narrowing.

And even in Info-mode as it works now, we can provide an Info-specific 
widening command (bound to the usual key sequence) which would only 
widen to the current page instead of the whole buffer.

>> That's one option. But again, prog-widen doesn't solve everything, and you
>
> Not sure which problem you're referring to.  Do you mean that when
> syntax-ppss-parse-start is changed, syntax-ppss is not warned?

It can track the value of point-min, but it won't really know what to do 
with the caches corresponding to the previous point-min values. Keep 
them around? Discard some of them?

See the end of this message: 
http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00630.html

>> haven't commented on syntax-ppss's cache management ideas.
>
> Not sure on which part you want my input.

Some help choosing would be nice. :)

> I can't remember all that's
> been suggested, so here's just some random opinions of mine on the
> subject:
> - we could/should have a macro (with-new-syntax-ppss-context
>   ... &rest BODY) which runs BODY such that calls to syntax-ppss in
>   there can be cached within this scope but don't use the outer
>   syntax-ppss cache and don't affect it.  This could be used when
>   a command wants to temporarily use an ad-hoc syntax-table (typically
>   using with-syntax-table), for example.  It should probably be made to
>   obey the current narrowing limits, since such uses often additionally
>   want to restrict the parsing to a subpart of the buffer.

That seems sound. And, basically, all that's necessary for mixed-mode 
packages to work. Except for the "interleaving pieces of code" case, 
which I think Christopher mentioned is the case in Antlr. But that's an 
advanced problem.

> - if we introduce a syntax-ppss-parse-start, it should not have `ppss`
>   in its name, it should probably try to include some of the design of
>   indentation-context,

Which ones? Having both lower and upper bound? It's not indentation 
specific, so using the name is out of the question, I think.

> and it should probably be modified via functions
>   (rather than directly via setq), so that those functions can flush the
>   syntax-ppss cache when needed.

This sounds interesting. Care to add some details? Just a hook with 
functions that return an integer?

If we're going this way, an ability to substitute the cache, instead of 
just flushing it, sounds like a logical continuation of that idea. That 
might require a getter and setter for the cache, like we have for 
match-data.

We probably don't need to do that now, just keep the option open.

> This said, if needed it'd probably be
>   OK to have syntax-ppss auto-detect changes instead (after all,
>   syntax-ppss is still implemented in Elisp, so there is a lot of room
>   for speed up).

It might not we the fastest, but I'd like it to be algorithmically sound.

> I wouldn't do it in indent-for-tab-command, but I agree that it should
> be done in indent-according-to-mode rather than forcing every major
> mode's indentation function to widen.

Shouldn't indent-for-tab-command call indent-according-to-mode, then?

>> Same for font-lock.  And that takes care of the two known obstacles in
>> the mixed-major-mode space, with very little code or changes to
>> the API.
>
> Sure.  I think the current discussion is about "what should
> prog-widen/syntax-ppss-parse-start look like" more than "who/when should
> we call prog-widen".

I prefer to think of this as the next step.

>> Even if we add "something else", upon deciding that it's really needed,
>> later, the current proposal shouldn't hurt.
>
> If by "current proposal" you mean to prog-widen in
> indent-according-to-mode, then I agree it's a good idea.

Yes, and also agree about a similar rule for font-lock-keywords.

I really meant widen, though. Not sure what calling prog-widen in there 
would mean, semantically, because according to the previous discussion 
prog-narrowing was something a mixed-mode code would institute. But it's 
down the call stack from indent-according-to-mode (via a mixed-mode 
aware indent-line-function), not above it.