Re: Syntax-propertize and CC-mode [Was: Further CC-mode changes]

[Stefan Monnier <monnier@iro.umontreal.ca>]

> I've separated the discussion on syntax-propertize from the rest,
> because things are threatening to get out of hand.

Important note: I only provided a patch to use syntax-propertize for
awk-mode.  This case is much simpler to handle than the C/C++/Java cases
(at least, IIUC, the way you apply syntax-table properties in awk-mode
is easy to map to what syntax-propertize needs).

Using syntax-propertize for C/C++/Java is a whole other project, which
I hope to get to at some point, but I'll need to dig much deeper into
the belly of the beast before I can hope to do that.

> How is the lower limit determined (matching the upper limit POS)?

Syntax-propertize internally keeps track of "up-to-where have properties
been applied", so that it can provide the lower limit.

> At a guess, this is going to be anywhere down to BOB.

That can be, tho usually it's just "the beginning of the last buffer
modification".

> This isn't very lazy.  Why doesn't syntax-propertize take TWO
> parameters, BEG and END?

How would you know what is BEG without knowing what other packages
(indent.el, font-lock.el, ...) have already `syntax-propertize'd?

> Looking at the source, syntax-propertize is unoptimised.  It will always
> erase the syntax-table properties from the last buffer change onwards, in
> contrast to C++ Mode, which only erases them locally as needed.

syntax-propertize is lazy, but indeed it always assumes pessimistically
that any change to the buffer at POS might cause everything after POS to
be interpreted completely differently and hence require re-propertization.
So far, this has not proved to be problematic in any of the 20 or so
major modes that use syntax-propertize.

> There is surely an opportunity for better customisation here.

Yes, we could spice up syntax.el so that you can set some properties and
explain when modifications in previous text don't invalidate them.
We haven't seen a need for it so far.

> That is not true.  I need to know that syntax-propertize would nullify
> C++/Java Modes' careful optimisations.

syntax-propertize (as it stands) is probably not compatible with some of
the optimizations you use, indeed.  It probably gives you other
optimization opportunities, of course.  How well it would work is hard
to tell without a lot more investigation.

> I need to know whether or not syntax-propertize properly handles
> category properties.

By and large, `syntax-propertize' doesn't handle anything at all.

It does is keep track of the BEG argument for you, and it does erase the
`syntax-table' properties over BEG..END before calling
syntax-propertize-function.  But that's about all it does.

> I suspect it doesn't.  That involves reading the
> source code, for lack of documentation.

The source is all in a single function (called, you guessed,
syntax-propertize) which weighs in at 39 lines of Elisp code, a good
half of which has to do with calling
syntax-propertize-extend-region-functions.

> syntax-propertize needs a way of customizing its before-change function
> to avoid removing ST properties needlessly from large chunks of buffer.

Note that in 99% of the cases, there aren't any properties applied
further in the buffer anyway, so we don't actually "needlessly"
throwaway data on large chunks of buffer.

That's why so far, this optimization you're suggesting has not been
requested by anyone, even though modes such as (C)perl do have rather
complex/costly syntax-propertize-functions and are sometimes applied to
very large buffers as well.  IOW, I wouldn't worry about this problem
for now.  We'll cross this bridge when we get there.


        Stefan