AW: font-locking and open parens in column 0

AW: font-locking and open parens in column 0

[Mackenzie, Alan]

Hi, Martin! 

>-----Ursprüngliche Nachricht-----
>Von: martin rudalics [mailto:rudalics@gmx.at] 
>Gesendet: Donnerstag, 2. November 2006 19:32
>An: Mackenzie, Alan
>Cc: emacs-devel@gnu.org; Richard Stallman; Alan Mackenzie
>Betreff: Re: font-locking and open parens in column 0

>Good evening, Alan

> > The cause of this (as Martin (almost) discerned) is that the handling of
> > (eq open-paren-in-column-0-is-defun-start nil) in begining-of-defun-raw
> > hasn't been implemented.  The function just looks for a "(" in C0
> > regardless of that variable.
> >
> > "Clearly", when that variable is nil, a defun can begin at no place other
> > than a paren at the outermost level.  Therefore, the function must scan
> > the entire source file from BOB, as in the earliest days.

>No!  That would be a serious regression.  Font-locking should never be
>forced to scan from BOB.

         1         2         3         4         5         6         7  3
I agree with you.  :-)  However, my patch is to do with
beginning-of-defun-raw, not font-locking.  Are you saying that
open-paren-in-column-0-is-defun-start shouldn't exist at all?  When it is
nil, a paren in column 0 may not, of itself, be regarded as a defun
start.

Richard, please say what you think about this.

I would say, rather, that font-locking should not use b-o-defun-raw when
o-p-i-c-0-i-d-s is nil, except in exceptional circumstances.  The case
you spotted in syntax.c (and I've really no idea how you did ;-), is
such an exceptional case.  CC Mode caches parenthesis structures.

My patch did fix the bug (a whole screenful of misfontified string in
syntax.c), though, didn't it?

>Observe that this would only serve to handle
>the rare case where a user puts a paren in column zero of a C comment.
>I think warning about such parens as in emacs-lisp-mode is sufficient.
>Please try to make use of Richard's `font-lock-syntax-paren-check' from
>this thread as with

>     (put font-lock-beginning-of-syntax-function
>	 'font-lock-syntax-paren-check t)

>Moreover, any such code as yours should ....

You're talking about my patch to beginning-of-defun-raw, here, not the
stuff in CC Mode.  I don't have the source files to hand at the moment.

>(1) Consult `syntax-ppss' first.
>
>(2) Try to use the 9th field of the return value of 
>`parse-partial-sexp'
>to find the outermost paren instead of up-/forward-/backward-listing.
>
>(3) Crowd the cache of `syntax-ppss' in order to avoid further scans.

OK, thanks!  I'll need to learn about these optimisations.
begining-of-defun-raw, when used in font locking, could be distressingly
slow without them.

-- 
Alan Mackenzie (Ittersbach, Germany)



*******************************************
Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und loeschen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
 
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the contents in this e-mail is strictly forbidden.
*******************************************

Re: AW: font-locking and open parens in column 0

[martin rudalics]

Good afternoon, Alan

 >          1         2         3         4         5         6         7  3
 > I agree with you.  :-)  However, my patch is to do with
 > beginning-of-defun-raw, not font-locking.

My problem is with font-locking and I thought your patch wanted to
address that problem (as you also stated some lines below).

 > Are you saying that
 > open-paren-in-column-0-is-defun-start shouldn't exist at all?  When it is
 > nil, a paren in column 0 may not, of itself, be regarded as a defun
 > start.

I fail to understand the present state of things.  On the one hand,
`open-paren-in-column-0-is-defun-start' is customizable which means a
user should be able to set it and a major mode should respect that.  On
the other hand, c-mode deliberately sets this to nil.  I think users
should be free to express their choice here if they consider their
machine inapt for scanning from bob.

 > I would say, rather, that font-locking should not use b-o-defun-raw when
 > o-p-i-c-0-i-d-s is nil, except in exceptional circumstances.

Font-lock uses `syntax-ppss' which may call `syntax-begin-function'
which may be defined as `beginning-of-defun' which usually calls
`beginning-of-defun-raw'.  When I open a C file and jump to a position
before stealth fontification gets there, that's the way things behave.

 > The case
 > you spotted in syntax.c (and I've really no idea how you did ;-), is
 > such an exceptional case.

I spotted that incidentally when scrolling backwards through syntax.c.
Anyway, it *is* exceptional and thus should not warrant any major
change.  Richard's patch just comes in handy.

 > CC Mode caches parenthesis structures.

... which parallels the work of `syntax-ppss', hence we currently end up
with two caches for the same structures - I know c-mode has to work hard
to handle all sorts of older (X)Emacsen ...

 > My patch did fix the bug (a whole screenful of misfontified string in
 > syntax.c), though, didn't it?

It does fix it, and it's even pretty fast ;-).  But I still think the
"bug" is with the author who put the left paren in column zero of that
comment.  That author should be warned just as in emacs-lisp-mode.

 > You're talking about my patch to beginning-of-defun-raw, here, not the
 > stuff in CC Mode.  I don't have the source files to hand at the moment.
 >
 >
 >>(1) Consult `syntax-ppss' first.
 >>
 >>(2) Try to use the 9th field of the return value of
 >>`parse-partial-sexp'
 >>to find the outermost paren instead of up-/forward-/backward-listing.
 >>
 >>(3) Crowd the cache of `syntax-ppss' in order to avoid further scans.
 >
 >
 > OK, thanks!  I'll need to learn about these optimisations.
 > begining-of-defun-raw, when used in font locking, could be distressingly
 > slow without them.

Suggestion (1) obviously has nothing to do with font-locking (otherwise
we would end up with `syntax-ppss' calling `beginning-of-defun' calling
`syntax-ppss' ...) but with using `beginning-of-defun' interactively for
jumping to the beginning of the current function.  However, that's
something `c-beginning-of-defun' does perfectly for c-mode.

Re: font-locking and open parens in column 0

[Stefan Monnier]

>> > The cause of this (as Martin (almost) discerned) is that the handling of
>> > (eq open-paren-in-column-0-is-defun-start nil) in begining-of-defun-raw
>> > hasn't been implemented.  The function just looks for a "(" in C0
>> > regardless of that variable.

According to its docstring, the current behavior is correct:

   Normally a defun starts when there is a char with open-parenthesis
   syntax at the beginning of a line.

I think the bug is in cc-mode's use of beginning-of-defun.
BTW, the alternative used in elisp might be OK as well: flag with a big
`warning' face those spots in the file which match defun-prompt-regexp (or
where the line starts with an open paren) but which are inside a comment or
string.  This way you may get erroneous fontification, but the root cause is
clearly marked.



        Stefan

Re: AW: font-locking and open parens in column 0

[Richard Stallman]

    I agree with you.  :-)  However, my patch is to do with
    beginning-of-defun-raw, not font-locking.  Are you saying that
    open-paren-in-column-0-is-defun-start shouldn't exist at all?  When it is
    nil, a paren in column 0 may not, of itself, be regarded as a defun
    start.

    Richard, please say what you think about this.

I think it is correct to implement open-paren-in-column-0-is-defun-start
in the best way possible.  So please install your change.

However, it is also important to implement the font lock optimizations
as Stefan described them.  Can you do that now?  Also, warning about
misleading parens as Lisp mode does would be a good thing to implement
in C mode, whenever appropriate.

Stefan wrote:

    According to its docstring, the current behavior is correct:

       Normally a defun starts when there is a char with open-parenthesis
       syntax at the beginning of a line.

The purpose of open-paren-in-column-0-is-defun-start is to enable ot
disable this heuristic.  Setting it to nil disables the heuristic.
So I think his change is correct.

Re: AW: font-locking and open parens in column 0

[Stefan Monnier]

>        Normally a defun starts when there is a char with open-parenthesis
>        syntax at the beginning of a line.

> The purpose of open-paren-in-column-0-is-defun-start is to enable to
> disable this heuristic.  Setting it to nil disables the heuristic.
> So I think his change is correct.

Huh?  What do you mean by "heuristic".  The definition of
beginning-of-defun has never used syntax-tables or anything like that.
E.g. when defun-prompt-regexp is used it doesn't pay attention to
syntax-tables either.

open-paren-in-column-0-is-defun-start is only used as a heuristic in
syntax.c when skipping backward over comment.

There's another related heuristic, which is the use of beginning-of-defun
for syntax-begin-function.  This one is a heuristic specifically because
beginning-of-defun only works with regexps.  If we make beginning-of-defun
aware of strings and comments, then we'll have to warn every coder who uses
it in syntax-begin-function that it's not a good heuristic any more
(especially if beginning-of-defun uses syntax-ppss in which case we may get
into inf-loops).


        Stefan

Re: AW: font-locking and open parens in column 0

[Richard Stallman]

    > The purpose of open-paren-in-column-0-is-defun-start is to enable to
    > disable this heuristic.  Setting it to nil disables the heuristic.
    > So I think his change is correct.

    Huh?  What do you mean by "heuristic".

Emacs is looking for an open-paren at top level in list structure.
The heuristic is to take an open-paren at column zero and assume
it is at top level.

I am having trouble relating the rest of your message to this issue.
This is just a matter of implementing
open-paren-in-column-0-is-defun-start = nil to do what it says it will
do.

Re: AW: font-locking and open parens in column 0

[Stefan Monnier]

>> The purpose of open-paren-in-column-0-is-defun-start is to enable to
>> disable this heuristic.  Setting it to nil disables the heuristic.
>> So I think his change is correct.

>     Huh?  What do you mean by "heuristic".

> Emacs is looking for an open-paren at top level in list structure.

When?  We're talking about a patch to beginning-of-defun, right?
Beginning-of-defun has until now been defined by regexps.

> The heuristic is to take an open-paren at column zero and assume
> it is at top level.

The docstring of beginning-of-defun doesn't say anything about "toplevel".

AFAIK beginning-of-defun is a function in the same category as
forward-paragraph and things like that, which do not care much about the
global syntax state, but instead only pay attention to the immediately
surrounding text.

> I am having trouble relating the rest of your message to this issue.
> This is just a matter of implementing
> open-paren-in-column-0-is-defun-start = nil to do what it says it will do.

The discussion about beginning-of-defun is intricately tied to syntax-ppss
because:

1 - it started with a bug-report about wrong font-locking because
    beginning-of-defun is apparently used by cc-mode to get a safe starting
    point for syntactic analysis.
2 - beginning-of-defun has been used as a syntax-begin-function in several
    occasions, as a heuristic to find a safe starting point for
    syntactic analysis.
3 - changing beginning-of-defun to use parse-partial-sexp (or worse
    syntax-ppss) defeats the purpose of using it as a syntax-begin-function.
    It may even break such uses (e.g. in emacs-lisp-mode).
4 - the original motivation for the patch (i.e. point 1 above) is better
    addressed by not using beginning-of-defun and rely on syntax-ppss's
    cache instead.

I think the problem is that beginning-of-defun has many different possible
uses, not all of which are compatible:

1 - it can be used as a "move to toplevel" (i.e. outside of any syntactic
    element).  Currently it's a not reliable way to do that, but it's been
    used as a good heuristic.  Note that in some languages such a concept
    may not even be very meaningful: in languages whose files are commonly
    composed of only one toplevel element (typically a module or a class
    which then contains other elements inside themselves maybe classes or
    modules, ...).

2 - it can be used as a form of "backward-paragraph-for-prog-langs", to move
    to the beginning of a "block of text".  In case where defuns can be
    nested, this first only move to the beginning of the nested defun.

3 - a mix of the two: define some level of nesting (if any) as the main one
   (typically either the toplevel one, or if the toplevel is a single
   element, use the next level down) and move to the beginning of the defun
   at that level.

Interactive use mostly wants behavior 2 or 3.
A reliable way to get behavior 1 is to use syntax-ppss rather than
beginning-of-defun.

The proposed patch basically tries to make beginning-of-defun follow the
behavior number 1 and to make it do so reliably.  Given the availability of
syntax-ppss to get the same result, I don't think this patch is such
a good idea.

OTOH it might be a good idea indeed to change beginning-of-defun so that it
ignores regexp-matches if they're inside comments or strings.  But that'd be
a different patch, which would apply regardless of
open-paren-in-column-0-is-defun-start.


        Stefan

Re: AW: font-locking and open parens in column 0

[Richard Stallman]

    > Emacs is looking for an open-paren at top level in list structure.

    When?

Since around 1975.

	   We're talking about a patch to beginning-of-defun, right?
    Beginning-of-defun has until now been defined by regexps.

I don't think we are talking about the same question.
beginning-of-defun has a customization feature involving regexps, but
that isn't the point I'm talking about.  I'm talking about what it 
actually did, which (normally) was to find an open-paren in column 0.

You asked what I meant by heuristic and I explained it to you.

    > The heuristic is to take an open-paren at column zero and assume
    > it is at top level.

    The docstring of beginning-of-defun doesn't say anything about "toplevel".

So what?

    AFAIK beginning-of-defun is a function in the same category as
    forward-paragraph and things like that, which do not care much about the
    global syntax state, but instead only pay attention to the immediately
    surrounding text.

That is not how I think of it.

The original definition was to look for an open paren at top level.
But that was too slow, so I established the convention of assuming
that an open paren in column 0 would be taken as top level even if it
was not really at top level.

Re: AW: font-locking and open parens in column 0

[Richard Stallman]

    4 - the original motivation for the patch (i.e. point 1 above) is better
	addressed by not using beginning-of-defun and rely on syntax-ppss's
	cache instead.

The specific purpose of the change in beginning-of-defun-raw was simply to make
it correctly handle the case of open-paren-in-column-0-is-defun-start = nil
when there is no beginning-of-defun-function.

I know this discussion is part of a larger one, but I no longer
remember the rest of the issues in the larger discussion.

However, I did not realize that CC mode actually used that case.  I
thought it had a beginning-of-defun-function.  I was surprised just
now to see that it didn't have one.  Perhaps that should be changed.

However, taking out the use of syntax-ppss in beginning-of-defun-raw
isn't the right way to get good behavior in CC mode.  That would only
make it parse from the start of the file every time.

    3 - changing beginning-of-defun to use parse-partial-sexp (or worse
	syntax-ppss) defeats the purpose of using it as a syntax-begin-function.
	It may even break such uses (e.g. in emacs-lisp-mode).

Why would it break them?

(In any case, such a problem won't arise in Emacs Lisp mode, since
open-paren-in-column-0-is-defun-start = t.)

    I think the problem is that beginning-of-defun has many different possible
    uses, not all of which are compatible:

    1 - it can be used as a "move to toplevel" (i.e. outside of any syntactic
	element).  Currently it's a not reliable way to do that, but it's been
	used as a good heuristic.  Note that in some languages such a concept
	may not even be very meaningful: in languages whose files are commonly
	composed of only one toplevel element (typically a module or a class
	which then contains other elements inside themselves maybe classes or
	modules, ...).

    2 - it can be used as a form of "backward-paragraph-for-prog-langs", to move
	to the beginning of a "block of text".  In case where defuns can be
	nested, this first only move to the beginning of the nested defun.

Normally I'd expect it NOT to treat nested definitions as defuns.
Normally they would be entirely indented.

    3 - a mix of the two: define some level of nesting (if any) as the main one
       (typically either the toplevel one, or if the toplevel is a single
       element, use the next level down) and move to the beginning of the defun
       at that level.

You can get this behavior by adjusting the indentation
when open-paren-in-column-0-is-defun-start = t.

    A reliable way to get behavior 1 is to use syntax-ppss rather than
    beginning-of-defun.

That is a good point.  Maybe some uses of beginning-of-defun
(such as in Font Lock) ought to use syntax-ppss instead.

I think that is orthogonal to the question of this change
in beginning-of-defun-raw.

    The proposed patch basically tries to make beginning-of-defun follow the
    behavior number 1 and to make it do so reliably.

I don't see it that way.

    OTOH it might be a good idea indeed to change beginning-of-defun so that it
    ignores regexp-matches if they're inside comments or strings.  But that'd be
    a different patch, which would apply regardless of
    open-paren-in-column-0-is-defun-start.

I agree, that might be good.

Re: AW: font-locking and open parens in column 0

[martin rudalics]

 > However, I did not realize that CC mode actually used that case.  I
 > thought it had a beginning-of-defun-function.  I was surprised just
 > now to see that it didn't have one.  Perhaps that should be changed.

I've been asking before why c-mode cannot use `c-beginning-of-defun'
here.  Is it because syntax-table properties haven't been set up?  I
recall invoking `c-beginning-of-defun' in arbitrary not yet fontified
text and getting strange results.

 > However, taking out the use of syntax-ppss in beginning-of-defun-raw
 > isn't the right way to get good behavior in CC mode.  That would only
 > make it parse from the start of the file every time.

When scrolling through a buffer backwards, the current patch will cause
parsing from bob every time I scroll.

 > (In any case, such a problem won't arise in Emacs Lisp mode, since
 > open-paren-in-column-0-is-defun-start = t.)

`open-paren-in-column-0-is-defun-start' is a user option.

 >     A reliable way to get behavior 1 is to use syntax-ppss rather than
 >     beginning-of-defun.
 >
 > That is a good point.  Maybe some uses of beginning-of-defun
 > (such as in Font Lock) ought to use syntax-ppss instead.

This would introduce a circular dependency since `syntax-ppss' may call
`beginning-of-defun'.

 >     The proposed patch basically tries to make beginning-of-defun follow the
 >     behavior number 1 and to make it do so reliably.
 >
 > I don't see it that way.

I see it Stefan's way.

Re: AW: font-locking and open parens in column 0

[Stefan Monnier]

>     4 - the original motivation for the patch (i.e. point 1 above) is better
> 	addressed by not using beginning-of-defun and rely on syntax-ppss's
> 	cache instead.
[...]
> However, taking out the use of syntax-ppss in beginning-of-defun-raw
> isn't the right way to get good behavior in CC mode.

This is not what I'm suggesting: I'm saying that the places in cc-mode which
use beginning-of-defun-raw in order to find a safe parsing spot should be
changed to use syntax-ppss for that instead.
This is separate from whether or not beginning-of-defun uses syntax-ppss.

>     3 - changing beginning-of-defun to use parse-partial-sexp (or worse
> 	syntax-ppss) defeats the purpose of using it as
> 	a syntax-begin-function.  It may even break such uses (e.g. in
> 	emacs-lisp-mode).

> Why would it break them?

Because the whole point of using them is to avoid parsing from the beginning
of the buffer.  Maybe it won't burp (depending on how the circular
dependency "syntax-ppss => syntax-begin-function => beginning-of-defun =>
syntax-ppss" works out), but it will at least break the intent of the code.

>     I think the problem is that beginning-of-defun has many different possible
>     uses, not all of which are compatible:

>     1 - it can be used as a "move to toplevel" (i.e. outside of any syntactic
> 	element).  Currently it's a not reliable way to do that, but it's been
> 	used as a good heuristic.  Note that in some languages such a concept
> 	may not even be very meaningful: in languages whose files are commonly
> 	composed of only one toplevel element (typically a module or a class
> 	which then contains other elements inside themselves maybe classes or
> 	modules, ...).

>     2 - it can be used as a form of "backward-paragraph-for-prog-langs",
> 	to move to the beginning of a "block of text".  In case where defuns
> 	can be nested, this first only move to the beginning of the
> 	nested defun.

> Normally I'd expect it NOT to treat nested definitions as defuns.

That strongly depends on the language.  In languages such as Standard-ML,
it's pretty common to have big functions defined inside others.
It's probably true of many languages which use locally defined recursive
functions instead of loops.

In those languages it's not necessarily useful for beginning-of-defun to
only consider toplevel defuns (since there will often only be one or 2 in
teh whole file).

> Normally they would be entirely indented.

Yup.  The whole file is indented, basically.

>     3 - a mix of the two: define some level of nesting (if any) as the
>     main one (typically either the toplevel one, or if the toplevel is
>     a single element, use the next level down) and move to the beginning
>     of the defun at that level.

> You can get this behavior by adjusting the indentation
> when open-paren-in-column-0-is-defun-start = t.

But you may not want to fiddle with the indentation just for C-M-a to jump
the way you like it.  And the language may not allow you to either
(e.g. Haskell where indentation is significant).

>     A reliable way to get behavior 1 is to use syntax-ppss rather than
>     beginning-of-defun.

> That is a good point.  Maybe some uses of beginning-of-defun
> (such as in Font Lock) ought to use syntax-ppss instead.

Font-lock doesn't use beginning-of-defun.  Some clients of font-lock tell
syntaxx-ppss (via syntax-begin-function) to use beginning-of-defun as
a heuristic, for performance reasons.  E.g. emacs-lisp-mode.  Those are not
interested in beginning-of-defun becoming 100% reliable, since they
specifically trade the 100% reliability (which they'd get if they just left
syntax-begin-function alone) for speed.

> I think that is orthogonal to the question of this change
> in beginning-of-defun-raw.

That's odd.  The need for that change comes straight from such a "use
beginning-of-defun to find a safe parsing spot", and the impact of such
a change is mostly going to be on similar uses.


        Stefan

Re: font-locking and open parens in column 0

[Mackenzie, Alan]

Hi, Martin! 

>-----Ursprüngliche Nachricht-----
>Von: martin rudalics [mailto:rudalics@gmx.at] 
>Gesendet: Freitag, 3. November 2006 15:03
>An: Mackenzie, Alan
>Cc: Richard Stallman; emacs-devel@gnu.org; Alan Mackenzie
>Betreff: Re: AW: font-locking and open parens in column 0
>
>Good afternoon, Alan
>

[ .... ]

> > Are you saying that
> > open-paren-in-column-0-is-defun-start shouldn't exist at all?  When
> > it is nil, a paren in column 0 may not, of itself, be regarded as a
> > defun start.

>I fail to understand the present state of things.  On the one hand,
>`open-paren-in-column-0-is-defun-start' is customizable which means a
>user should be able to set it and a major mode should respect that.  On
>the other hand, c-mode deliberately sets this to nil.  I think users
>should be free to express their choice here if they consider their
>machine inapt for scanning from bob.

As far as I can see, the purpose of open-p-i-c-0-i-d-start wasn't
clearly recorded in the ChangeLog.  The value t is clear; I can't think
of any meaning for nil other than the one I've given it.  Certainly,
Martin Stjernholm assumed that meaning (goto a brace at top level) when
he coded up c-beginning-of-defun.  If this meaning is not given to it,
there doesn't seem to be any point in having the variable.

I think it was made customizable so that a user could set it for
"efficiency" (when his perl files' functions all have { at column 0) or
"rigour" (when he uses "k&r" placement of braces in his files.pl).  It
doesn't seem to have worked very well, so far.

CC Mode always sets o-p-i-c-0-i-d-start to nil, and caches the brace
structure to prevent excessive scanning from BOB.  After all, opic0id
being nil will always find BO-defun.  Setting it to t was an optimisation
for when computers were much less powerful than they are now - and this
causes quite a bit of inconvenience.

> > I would say, rather, that font-locking should not use b-o-defun-raw
> > when o-p-i-c-0-i-d-s is nil, except in exceptional circumstances.

>Font-lock uses `syntax-ppss' which may call `syntax-begin-function'
>which may be defined as `beginning-of-defun' which usually calls
>`beginning-of-defun-raw'.  When I open a C file and jump to a position
>before stealth fontification gets there, that's the way things behave.

OK

> > The case
> > you spotted in syntax.c (and I've really no idea how you did ;-), is
> > such an exceptional case.

>I spotted that incidentally when scrolling backwards through syntax.c.
>Anyway, it *is* exceptional and thus should not warrant any major
>change.  Richard's patch just comes in handy.

I confess I'm not familiar with this.  I've been without net access for
several weeks.  I'm currently at the mercy of Deutsche Telekom, who have
promised to get my new DSL connection working real soon now.  :-(

> > CC Mode caches parenthesis structures.

>... which parallels the work of `syntax-ppss', hence we currently end up
>with two caches for the same structures - I know c-mode has to work hard
>to handle all sorts of older (X)Emacsen ...

Not only that, CC Mode uses its cache for things other than font-locking.

> > My patch did fix the bug (a whole screenful of misfontified string in
> > syntax.c), though, didn't it?

>It does fix it, and it's even pretty fast ;-).  But I still think the
>"bug" is with the author who put the left paren in column zero of that
>comment.  That author should be warned just as in emacs-lisp-mode.

Should _HAVE_ been warned.  Martin Stjernholm has worked very hard to
remove that restriction from C Mode without sacrificing (much) speed.  It
would be a shame now to leave the restriction in Emacs 22.

[ .... ]

-- 
Alan.



*******************************************
Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und loeschen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
 
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the contents in this e-mail is strictly forbidden.
*******************************************

Re: font-locking and open parens in column 0

[Stefan Monnier]

> CC Mode always sets o-p-i-c-0-i-d-start to nil, and caches the brace
> structure to prevent excessive scanning from BOB.

Interestingly, it may not be sufficient.  There are basically only 2 uses of
open-paren-in-column-0-is-defun-start in Emacs:
- one in lisp.el to define beginning-of-defun (in order to know whether to
  add "^" to defun-prompt-regexp).
- one in syntax.c used in forward-comment when scanning comments backward.

beginning-of-defun has always been based on regexps and not on syntax, so
open-paren-in-column-0-is-defun-start in there has no significant impact
w.r.t. scanning from the beginning of the buffer or not: you can change
defun-prompt-regexp instead and get the same result.

OTOH in forward-comment it's used within C code with no way to hook
into it.  So the paren cache you use has no effect on this :-(
It's on my TODO list to change forward-comment so that it calls syntax-ppss
when needed rather than scan from the beginning of the defun/buffer.

> After all, opic0id being nil will always find BO-defun.  Setting it to
> t was an optimisation for when computers were much less powerful than they
> are now - and this causes quite a bit of inconvenience.

It's an algorithmic optimization, and since in computer science, problems
tend to get bigger as we get bigger machines, it's not clear that the
optimization is less useful now than it was then.  I do believe it is less
useful because we're lucky enough that our "problems" haven't gotten much
bigger: source code size is limited by human constraints more than by the
machine (except for computer-generated code, obviously), and we haven't
noticeably complexified our parsing technology (contrary to many other
programming environments).

>> ... which parallels the work of `syntax-ppss', hence we currently end up
>> with two caches for the same structures - I know c-mode has to work hard
>> to handle all sorts of older (X)Emacsen ...
> Not only that, CC Mode uses its cache for things other than font-locking.

syntax-ppss is used by font-lock but not only.  Whether you could use it for
the other places where you use your cache?  I don't know.  I hope so.
But if you can't I'd be interested to hear about it (mostly to figure out
in which direction syntax-ppss should be improved).

> Should _HAVE_ been warned.  Martin Stjernholm has worked very hard to
> remove that restriction from C Mode without sacrificing (much) speed.  It
> would be a shame now to leave the restriction in Emacs 22.

In that case, I believe a better patch than the one you proposed is to
change cc-fonts.el so as to set syntax-begin-function to nil.  After all,
the whole point of syntax-begin-function is to provide a *heuristic* that
can "quickly" find a safe starting point for parsing.  If you want your
parsing to be more reliable than a heuristic, then just don't use that.


        Stefan

Re: font-locking and open parens in column 0

[martin rudalics]

 > beginning-of-defun has always been based on regexps and not on syntax, so
 > open-paren-in-column-0-is-defun-start in there has no significant impact
 > w.r.t. scanning from the beginning of the buffer or not: you can change
 > defun-prompt-regexp instead and get the same result.

Does this mean we could use `syntax-ppss' and the car of the 9th field
instead and get rid of `open-paren-in-column-0-is-defun-start' once the
`forward-comment'/`find_defun_start' issue has been resolved?  Or,
better, give find_defun_start an additional argument, say "current", to
have it insist on finding the start of the current defun?

 > ... and we haven't
 > noticeably complexified our parsing technology (contrary to many other
 > programming environments).

Once `parse-partial-sexp' is able to handle alternative comment styles
or syntax-table properties (instead of the current text property ersatz)
parsing from bob will get more expensive.

 > syntax-ppss is used by font-lock but not only.  Whether you could use it for
 > the other places where you use your cache?  I don't know.  I hope so.
 > But if you can't I'd be interested to hear about it (mostly to figure out
 > in which direction syntax-ppss should be improved).

`syntax-ppss' *is* a major improvement.  It lacks a documentation of how
its cache gets filled and a way to customize that.

Re: font-locking and open parens in column 0

[martin rudalics]

 >> But I still think the
 >>"bug" is with the author who put the left paren in column zero of that
 >>comment.  That author should be warned just as in emacs-lisp-mode.
 >
 >
 > Should _HAVE_ been warned.  Martin Stjernholm has worked very hard to
 > remove that restriction from C Mode without sacrificing (much) speed.  It
 > would be a shame now to leave the restriction in Emacs 22.

Which means that Martin Stjernholm's change did not make it to
font-locking which, for c-mode, still uses the `beginning-of-defun' from
lisp-mode.  I see only two reasonable fixes to this:

(1) Set `syntax-begin-function' (or `beginning-of-defun-function') to
     `c-beginning-of-defun' if you trust that change.  This should,
     hopefully, populate your cache.

(2) Set `syntax-begin-function' to nil as Stefan suggested and let
     `syntax-ppss' do the work.  This should populate Stefan's cache.

Any other solution would be a shameful waste of resources since the
results of `parse-partial-sexp' from your original proposal don't get
stored anywhere (jumping to a position n in a c-mode buffer and
subsequently jumping to a position m < n would afford to rescan from
bob).  Also make sure that `font-lock-beginning-of-syntax-function'
nowhere interferes with these settings.

Ideally, there would be one and only one cache populated and consulted
by both - c-mode and `syntax-ppss'.

Re: font-locking and open parens in column 0

[Richard Stallman]

    CC Mode always sets o-p-i-c-0-i-d-start to nil, and caches the brace
    structure to prevent excessive scanning from BOB.  After all, opic0id
    being nil will always find BO-defun.  Setting it to t was an optimisation
    for when computers were much less powerful than they are now - and this
    causes quite a bit of inconvenience.

That optimization may still be important.  It is ok for CC mode to set
this to nil IF it provides another optimization which does the same
job.  For instance, using the cache.  If that makes the same
operations fast, then we don't need the paren-in-column-0
optimization.

There is some doubt about whether your cache is getting used
for all the operations that do this.

    >... which parallels the work of `syntax-ppss', hence we currently end up
    >with two caches for the same structures - I know c-mode has to work hard
    >to handle all sorts of older (X)Emacsen ...

    Not only that, CC Mode uses its cache for things other than font-locking.

Could CC mode use `syntax-ppss' for all those things?
If so, it could get rid of its own cache and rely on the
cache in `syntax-ppss'.

    >It does fix it, and it's even pretty fast ;-).  But I still think the
    >"bug" is with the author who put the left paren in column zero of that
    >comment.  That author should be warned just as in emacs-lisp-mode.

    Should _HAVE_ been warned.  Martin Stjernholm has worked very hard to
    remove that restriction from C Mode without sacrificing (much) speed.  It
    would be a shame now to leave the restriction in Emacs 22.

I agree that if CC mode is not confused by that paren in column 0
then there is no need to flag it for the user.

Re: font-locking and open parens in column 0

[Mackenzie, Alan]

Hi, Emacs!

[Apologies for not being able to reply with proper headers, since I'm
still without a network connection at home.  Please send any CCs to
acm@muc.de as well as my work address.]

Martin Rudalics wrote on Monday 11th September 2006, 15:05:55 +0200:

> (1) With emacs -Q open ~/src/syntax.c

> (2) Execute

>(defun foo ()
>  (interactive)
>  (re-search-forward "string-to-syntax")
>  (forward-line 6)
>  (recenter 0))

>This will fontify the entire body of `string_to_syntax' as a C string
>due to the left paren in column zero of the doc-string.

The cause of this (as Martin (almost) discerned) is that the handling of
(eq open-paren-in-column-0-is-defun-start nil) in begining-of-defun-raw
hasn't been implemented.  The function just looks for a "(" in C0
regardless of that variable.

"Clearly", when that variable is nil, a defun can begin at no place
other
than a paren at the outermost level.  Therefore, the function must scan
the entire source file from BOB, as in the earliest days.

The patch below implements this.  When applied (don't forget to rebuild
Emacs or M-x load-file lisp.elc, since lisp.elc is a preloaded file),
Emacs fontifies string-to-syntax properly after jumping there with M-x
foo.


2006-11-01  Alan Mackenzie  <acm@muc.de>

	* emacs-lisp/lisp.el (beginning-of-defun-raw): Code up the case
	(eq open-paren-in-column-0-is-defun-start nil) by searching for
	least nested open-paren.


*** lisp-1.74.el	2006-02-19 12:51:43.000000000 +0000
--- lisp.el	2006-11-01 22:03:56.313088952 +0000
***************
*** 208,229 ****
  
  If variable `beginning-of-defun-function' is non-nil, its value
  is called as a function to find the defun's beginning."
!   (interactive "p")
!   (if beginning-of-defun-function
!       (if (> (setq arg (or arg 1)) 0)
! 	  (dotimes (i arg)
! 	    (funcall beginning-of-defun-function))
! 	;; Better not call end-of-defun-function directly, in case
! 	;; it's not defined.
! 	(end-of-defun (- arg)))
!     (and arg (< arg 0) (not (eobp)) (forward-char 1))
      (and (re-search-backward (if defun-prompt-regexp
  				 (concat (if
open-paren-in-column-0-is-defun-start
  					     "^\\s(\\|" "")
  					 "\\(?:" defun-prompt-regexp
"\\)\\s(")
  			       "^\\s(")
! 			     nil 'move (or arg 1))
! 	 (progn (goto-char (1- (match-end 0)))) t)))
  
  (defvar end-of-defun-function nil
    "If non-nil, function for function `end-of-defun' to call.
--- 208,270 ----
  
  If variable `beginning-of-defun-function' is non-nil, its value
  is called as a function to find the defun's beginning."
!   (interactive "p") ; change this to "P", maybe, if we ever come to
pass ARG
! 		    ; to beginning-of-defun-function.
!   (unless arg (setq arg 1))		; The call might not be
interactive.
!   (cond
!    (beginning-of-defun-function
!     (if (> arg 0)
! 	(dotimes (i arg)
! 	  (funcall beginning-of-defun-function))
!       ;; Better not call end-of-defun-function directly, in case
!       ;; it's not defined.
!       (end-of-defun (- arg))))
! 
!    ((or defun-prompt-regexp open-paren-in-column-0-is-defun-start)
!     (and (< arg 0) (not (eobp)) (forward-char 1))
      (and (re-search-backward (if defun-prompt-regexp
  				 (concat (if
open-paren-in-column-0-is-defun-start
  					     "^\\s(\\|" "")
  					 "\\(?:" defun-prompt-regexp
"\\)\\s(")
  			       "^\\s(")
! 			     nil 'move arg)
! 	 (progn (goto-char (1- (match-end 0)))) t))
! 
!    (t
!     ;; Column 0 has no significance - so scan forward from BOB to see
how
!     ;; nested point is, then carry on from there.
!     (let ((floor (point-min))
! 	  (ceiling (point-max))
! 	  pps-state nesting-depth)
!       (save-restriction
! 	(widen)
! 	(setq pps-state (parse-partial-sexp (point-min) (point))
! 	      nesting-depth (nth 0 pps-state))
! 	;; Get outside of any string or comment.
! 	(if (nth 8 pps-state)
! 	    (goto-char (nth 8 pps-state)))
! 
! 	(cond
! 	 ((> arg 0)
! 	  (when (> nesting-depth 0)
! 	    (up-list (- nesting-depth))
! 	    (setq arg (1- arg)))
! 	  ;; We're now outside of any defun.
! 	  (backward-list arg)
! 	  (if (< (point) floor) (goto-char floor)))
! 
! 	 ((< arg 0)
! 	  (cond
! 	   ((> nesting-depth 0)
! 	    (up-list nesting-depth)
! 	    (setq arg (1+ arg)))
! 	   ((not (looking-at "\\s("))
! 	    ;; We're between defuns, and not at the start of one.
! 	    (setq arg (1+ arg))))
! 	  (forward-list (- arg))
! 	  (down-list)
! 	  (backward-char)
! 	  (if (> (point) ceiling) (goto-char ceiling)))))))))
  
  (defvar end-of-defun-function nil
    "If non-nil, function for function `end-of-defun' to call.


-- 
Alan Mackenzie (Ittersbach, Germany).



*******************************************
Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und loeschen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
 
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the contents in this e-mail is strictly forbidden.
*******************************************

Re: font-locking and open parens in column 0

[martin rudalics]

Good evening, Alan

 > The cause of this (as Martin (almost) discerned) is that the handling of
 > (eq open-paren-in-column-0-is-defun-start nil) in begining-of-defun-raw
 > hasn't been implemented.  The function just looks for a "(" in C0
 > regardless of that variable.
 >
 > "Clearly", when that variable is nil, a defun can begin at no place
 > other
 > than a paren at the outermost level.  Therefore, the function must scan
 > the entire source file from BOB, as in the earliest days.

No!  That would be a serious regression.  Font-locking should never be
forced to scan from BOB.  Observe that this would only serve to handle
the rare case where a user puts a paren in column zero of a C comment.
I think warning about such parens as in emacs-lisp-mode is sufficient.
Please try to make use of Richard's `font-lock-syntax-paren-check' from
this thread as with

     (put font-lock-beginning-of-syntax-function
	 'font-lock-syntax-paren-check t)

Moreover, any such code as yours should

(1) Consult `syntax-ppss' first.

(2) Try to use the 9th field of the return value of `parse-partial-sexp'
to find the outermost paren instead of up-/forward-/backward-listing.

(3) Crowd the cache of `syntax-ppss' in order to avoid further scans.