Re: Bug #25608 and the comment-cache branch

[Dmitry Gutov <dgutov@yandex.ru>]

On 22.02.2017 05:53, Stefan Monnier wrote:
>> I see, thanks. And I think that means that, ideally, it would work without
>> the caller having to adjust the syntax visibility bounds, or the like, as
>> long as the syntax table is correct and the beginning (or the end) of the
>> currently navigated comment is within view.
> 
> Right, but not reliably so: very often we need to parse backward not
> just until the matching starter but until the previous closer (to make
> sure the starter we saw was not itself within an earlier comment), and
> in other cases the mix of comment markers and string markers make it
> impossible to guess if we were really inside a comment, so we end up
> falling back on the forward-parse code.

Naturally, we'd need to save more information to be able to do that. 
E.g. propertize the end of a complex comment with the position of its 
beginning. Since the first time we go through a buffer is in the forward 
direction, getting that info would be inexpensive.

> Actually, it's very straightforward: the forward parse already gives us
> the beginning of the surrounding element, so we just re-do the forward
> parse from that spot.  It's just a matter of wrapping the code inside
> a loop.

You're likely a better judge of that. It does sound a bit convoluted to 
me (and having to deal with different kinds of comments adds its 
complexity), but not something that having a handful of tests wouldn't 
keep straight.

> No.  "Previously", we typically scan the line backward and stop as soon
> as we hit the previous \n (which tells us that no comment can start
> earlier than that if it finishes with a \n).
> 
> In a few cases, we do fallback on the forward parse code, in which case
> indeed we'll take longer, but those are normally rare (which is why this
> comment-cache and my syntax-ppss-patch haven't been installed yet: they
> improve the performance of a case that's somewhat infrequent).

I see, thanks.

>> Perhaps we could use the "generic comment bounds" syntax-table property to
>> delineate such difficult comments. If that idea sounds similar to
>> comment-cache, that is no accident.
> 
> Maybe.  Obviously, my syntax-ppss hammer makes me think that such
> alternate solutions aren't needed: syntax-ppss solves this case without
> having to try and come out with a clever way to detect which comments
> are tricky nor how to mark them.

The alternative tweak I had in mind would be applied somewhere around 
syntax-propertize. So it would be a matter of trading off one bit of 
complexity for another, still staying within the framework of syntax-ppss.

> Not completely sure either.  I've had vague ideas of adding some kind of
> hook to syntax-tables, i.e. add a new kind of syntax element which ends
> up calling an Elisp function of your choice so you can make it "do the
> right thing" for the particular construct.

I think just having paired syntactic elements would suffice. Or just 
propertizing the whole subregion with one text property span. Whichever 
would be easier to process.

Not sure about using the syntax-table property for this. In some weird 
cases there won't be a space of a newline to put these syntax-table 
values on. And a newline staying a newline might be syntactically 
important for the primary major mode somewhere.

Another thing to consider is that we would probably want to fontify the 
contents of all subregions normally, even when inside comments belonging 
to the outer mode. So the primitives used in 
font-lock-fontify-syntactically-region would need to be able to stop at 
those boundaries instead of automatically skipping over.

> So when scanning (forward or backward), if we bump into an element with
> that syntax (typically applied as a syntax-table text-property), we call
> the function which will know how to jump over the sub-region or will
> signal an "end of sub-region" error.

Just having those hooks won't be enough, we still don't have enough info 
how to syntax-propertize the subregion contents, for instance. So I'm 
not sure what the flexibility of using the functions here would buy us.