AW: font-locking and open parens in column 0

AW: font-locking and open parens in column 0

[Mackenzie, Alan]

Hi, Martin! 

>-----Ursprüngliche Nachricht-----
>Von: martin rudalics [mailto:rudalics@gmx.at] 
>Gesendet: Donnerstag, 2. November 2006 19:32
>An: Mackenzie, Alan
>Cc: emacs-devel@gnu.org; Richard Stallman; Alan Mackenzie
>Betreff: Re: font-locking and open parens in column 0

>Good evening, Alan

> > The cause of this (as Martin (almost) discerned) is that the handling of
> > (eq open-paren-in-column-0-is-defun-start nil) in begining-of-defun-raw
> > hasn't been implemented.  The function just looks for a "(" in C0
> > regardless of that variable.
> >
> > "Clearly", when that variable is nil, a defun can begin at no place other
> > than a paren at the outermost level.  Therefore, the function must scan
> > the entire source file from BOB, as in the earliest days.

>No!  That would be a serious regression.  Font-locking should never be
>forced to scan from BOB.

         1         2         3         4         5         6         7  3
I agree with you.  :-)  However, my patch is to do with
beginning-of-defun-raw, not font-locking.  Are you saying that
open-paren-in-column-0-is-defun-start shouldn't exist at all?  When it is
nil, a paren in column 0 may not, of itself, be regarded as a defun
start.

Richard, please say what you think about this.

I would say, rather, that font-locking should not use b-o-defun-raw when
o-p-i-c-0-i-d-s is nil, except in exceptional circumstances.  The case
you spotted in syntax.c (and I've really no idea how you did ;-), is
such an exceptional case.  CC Mode caches parenthesis structures.

My patch did fix the bug (a whole screenful of misfontified string in
syntax.c), though, didn't it?

>Observe that this would only serve to handle
>the rare case where a user puts a paren in column zero of a C comment.
>I think warning about such parens as in emacs-lisp-mode is sufficient.
>Please try to make use of Richard's `font-lock-syntax-paren-check' from
>this thread as with

>     (put font-lock-beginning-of-syntax-function
>	 'font-lock-syntax-paren-check t)

>Moreover, any such code as yours should ....

You're talking about my patch to beginning-of-defun-raw, here, not the
stuff in CC Mode.  I don't have the source files to hand at the moment.

>(1) Consult `syntax-ppss' first.
>
>(2) Try to use the 9th field of the return value of 
>`parse-partial-sexp'
>to find the outermost paren instead of up-/forward-/backward-listing.
>
>(3) Crowd the cache of `syntax-ppss' in order to avoid further scans.

OK, thanks!  I'll need to learn about these optimisations.
begining-of-defun-raw, when used in font locking, could be distressingly
slow without them.

-- 
Alan Mackenzie (Ittersbach, Germany)



*******************************************
Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und loeschen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
 
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized copying, disclosure or distribution of the contents in this e-mail is strictly forbidden.
*******************************************

Re: AW: font-locking and open parens in column 0

[martin rudalics]

Good afternoon, Alan

 >          1         2         3         4         5         6         7  3
 > I agree with you.  :-)  However, my patch is to do with
 > beginning-of-defun-raw, not font-locking.

My problem is with font-locking and I thought your patch wanted to
address that problem (as you also stated some lines below).

 > Are you saying that
 > open-paren-in-column-0-is-defun-start shouldn't exist at all?  When it is
 > nil, a paren in column 0 may not, of itself, be regarded as a defun
 > start.

I fail to understand the present state of things.  On the one hand,
`open-paren-in-column-0-is-defun-start' is customizable which means a
user should be able to set it and a major mode should respect that.  On
the other hand, c-mode deliberately sets this to nil.  I think users
should be free to express their choice here if they consider their
machine inapt for scanning from bob.

 > I would say, rather, that font-locking should not use b-o-defun-raw when
 > o-p-i-c-0-i-d-s is nil, except in exceptional circumstances.

Font-lock uses `syntax-ppss' which may call `syntax-begin-function'
which may be defined as `beginning-of-defun' which usually calls
`beginning-of-defun-raw'.  When I open a C file and jump to a position
before stealth fontification gets there, that's the way things behave.

 > The case
 > you spotted in syntax.c (and I've really no idea how you did ;-), is
 > such an exceptional case.

I spotted that incidentally when scrolling backwards through syntax.c.
Anyway, it *is* exceptional and thus should not warrant any major
change.  Richard's patch just comes in handy.

 > CC Mode caches parenthesis structures.

... which parallels the work of `syntax-ppss', hence we currently end up
with two caches for the same structures - I know c-mode has to work hard
to handle all sorts of older (X)Emacsen ...

 > My patch did fix the bug (a whole screenful of misfontified string in
 > syntax.c), though, didn't it?

It does fix it, and it's even pretty fast ;-).  But I still think the
"bug" is with the author who put the left paren in column zero of that
comment.  That author should be warned just as in emacs-lisp-mode.

 > You're talking about my patch to beginning-of-defun-raw, here, not the
 > stuff in CC Mode.  I don't have the source files to hand at the moment.
 >
 >
 >>(1) Consult `syntax-ppss' first.
 >>
 >>(2) Try to use the 9th field of the return value of
 >>`parse-partial-sexp'
 >>to find the outermost paren instead of up-/forward-/backward-listing.
 >>
 >>(3) Crowd the cache of `syntax-ppss' in order to avoid further scans.
 >
 >
 > OK, thanks!  I'll need to learn about these optimisations.
 > begining-of-defun-raw, when used in font locking, could be distressingly
 > slow without them.

Suggestion (1) obviously has nothing to do with font-locking (otherwise
we would end up with `syntax-ppss' calling `beginning-of-defun' calling
`syntax-ppss' ...) but with using `beginning-of-defun' interactively for
jumping to the beginning of the current function.  However, that's
something `c-beginning-of-defun' does perfectly for c-mode.

Re: font-locking and open parens in column 0

[Stefan Monnier]

>> > The cause of this (as Martin (almost) discerned) is that the handling of
>> > (eq open-paren-in-column-0-is-defun-start nil) in begining-of-defun-raw
>> > hasn't been implemented.  The function just looks for a "(" in C0
>> > regardless of that variable.

According to its docstring, the current behavior is correct:

   Normally a defun starts when there is a char with open-parenthesis
   syntax at the beginning of a line.

I think the bug is in cc-mode's use of beginning-of-defun.
BTW, the alternative used in elisp might be OK as well: flag with a big
`warning' face those spots in the file which match defun-prompt-regexp (or
where the line starts with an open paren) but which are inside a comment or
string.  This way you may get erroneous fontification, but the root cause is
clearly marked.



        Stefan

Re: AW: font-locking and open parens in column 0

[Richard Stallman]

    I agree with you.  :-)  However, my patch is to do with
    beginning-of-defun-raw, not font-locking.  Are you saying that
    open-paren-in-column-0-is-defun-start shouldn't exist at all?  When it is
    nil, a paren in column 0 may not, of itself, be regarded as a defun
    start.

    Richard, please say what you think about this.

I think it is correct to implement open-paren-in-column-0-is-defun-start
in the best way possible.  So please install your change.

However, it is also important to implement the font lock optimizations
as Stefan described them.  Can you do that now?  Also, warning about
misleading parens as Lisp mode does would be a good thing to implement
in C mode, whenever appropriate.

Stefan wrote:

    According to its docstring, the current behavior is correct:

       Normally a defun starts when there is a char with open-parenthesis
       syntax at the beginning of a line.

The purpose of open-paren-in-column-0-is-defun-start is to enable ot
disable this heuristic.  Setting it to nil disables the heuristic.
So I think his change is correct.

Re: AW: font-locking and open parens in column 0

[Stefan Monnier]

>        Normally a defun starts when there is a char with open-parenthesis
>        syntax at the beginning of a line.

> The purpose of open-paren-in-column-0-is-defun-start is to enable to
> disable this heuristic.  Setting it to nil disables the heuristic.
> So I think his change is correct.

Huh?  What do you mean by "heuristic".  The definition of
beginning-of-defun has never used syntax-tables or anything like that.
E.g. when defun-prompt-regexp is used it doesn't pay attention to
syntax-tables either.

open-paren-in-column-0-is-defun-start is only used as a heuristic in
syntax.c when skipping backward over comment.

There's another related heuristic, which is the use of beginning-of-defun
for syntax-begin-function.  This one is a heuristic specifically because
beginning-of-defun only works with regexps.  If we make beginning-of-defun
aware of strings and comments, then we'll have to warn every coder who uses
it in syntax-begin-function that it's not a good heuristic any more
(especially if beginning-of-defun uses syntax-ppss in which case we may get
into inf-loops).


        Stefan

Re: AW: font-locking and open parens in column 0

[Richard Stallman]

    > The purpose of open-paren-in-column-0-is-defun-start is to enable to
    > disable this heuristic.  Setting it to nil disables the heuristic.
    > So I think his change is correct.

    Huh?  What do you mean by "heuristic".

Emacs is looking for an open-paren at top level in list structure.
The heuristic is to take an open-paren at column zero and assume
it is at top level.

I am having trouble relating the rest of your message to this issue.
This is just a matter of implementing
open-paren-in-column-0-is-defun-start = nil to do what it says it will
do.

Re: AW: font-locking and open parens in column 0

[Stefan Monnier]

>> The purpose of open-paren-in-column-0-is-defun-start is to enable to
>> disable this heuristic.  Setting it to nil disables the heuristic.
>> So I think his change is correct.

>     Huh?  What do you mean by "heuristic".

> Emacs is looking for an open-paren at top level in list structure.

When?  We're talking about a patch to beginning-of-defun, right?
Beginning-of-defun has until now been defined by regexps.

> The heuristic is to take an open-paren at column zero and assume
> it is at top level.

The docstring of beginning-of-defun doesn't say anything about "toplevel".

AFAIK beginning-of-defun is a function in the same category as
forward-paragraph and things like that, which do not care much about the
global syntax state, but instead only pay attention to the immediately
surrounding text.

> I am having trouble relating the rest of your message to this issue.
> This is just a matter of implementing
> open-paren-in-column-0-is-defun-start = nil to do what it says it will do.

The discussion about beginning-of-defun is intricately tied to syntax-ppss
because:

1 - it started with a bug-report about wrong font-locking because
    beginning-of-defun is apparently used by cc-mode to get a safe starting
    point for syntactic analysis.
2 - beginning-of-defun has been used as a syntax-begin-function in several
    occasions, as a heuristic to find a safe starting point for
    syntactic analysis.
3 - changing beginning-of-defun to use parse-partial-sexp (or worse
    syntax-ppss) defeats the purpose of using it as a syntax-begin-function.
    It may even break such uses (e.g. in emacs-lisp-mode).
4 - the original motivation for the patch (i.e. point 1 above) is better
    addressed by not using beginning-of-defun and rely on syntax-ppss's
    cache instead.

I think the problem is that beginning-of-defun has many different possible
uses, not all of which are compatible:

1 - it can be used as a "move to toplevel" (i.e. outside of any syntactic
    element).  Currently it's a not reliable way to do that, but it's been
    used as a good heuristic.  Note that in some languages such a concept
    may not even be very meaningful: in languages whose files are commonly
    composed of only one toplevel element (typically a module or a class
    which then contains other elements inside themselves maybe classes or
    modules, ...).

2 - it can be used as a form of "backward-paragraph-for-prog-langs", to move
    to the beginning of a "block of text".  In case where defuns can be
    nested, this first only move to the beginning of the nested defun.

3 - a mix of the two: define some level of nesting (if any) as the main one
   (typically either the toplevel one, or if the toplevel is a single
   element, use the next level down) and move to the beginning of the defun
   at that level.

Interactive use mostly wants behavior 2 or 3.
A reliable way to get behavior 1 is to use syntax-ppss rather than
beginning-of-defun.

The proposed patch basically tries to make beginning-of-defun follow the
behavior number 1 and to make it do so reliably.  Given the availability of
syntax-ppss to get the same result, I don't think this patch is such
a good idea.

OTOH it might be a good idea indeed to change beginning-of-defun so that it
ignores regexp-matches if they're inside comments or strings.  But that'd be
a different patch, which would apply regardless of
open-paren-in-column-0-is-defun-start.


        Stefan

Re: AW: font-locking and open parens in column 0

[Richard Stallman]

    > Emacs is looking for an open-paren at top level in list structure.

    When?

Since around 1975.

	   We're talking about a patch to beginning-of-defun, right?
    Beginning-of-defun has until now been defined by regexps.

I don't think we are talking about the same question.
beginning-of-defun has a customization feature involving regexps, but
that isn't the point I'm talking about.  I'm talking about what it 
actually did, which (normally) was to find an open-paren in column 0.

You asked what I meant by heuristic and I explained it to you.

    > The heuristic is to take an open-paren at column zero and assume
    > it is at top level.

    The docstring of beginning-of-defun doesn't say anything about "toplevel".

So what?

    AFAIK beginning-of-defun is a function in the same category as
    forward-paragraph and things like that, which do not care much about the
    global syntax state, but instead only pay attention to the immediately
    surrounding text.

That is not how I think of it.

The original definition was to look for an open paren at top level.
But that was too slow, so I established the convention of assuming
that an open paren in column 0 would be taken as top level even if it
was not really at top level.

Re: AW: font-locking and open parens in column 0

[Richard Stallman]

    4 - the original motivation for the patch (i.e. point 1 above) is better
	addressed by not using beginning-of-defun and rely on syntax-ppss's
	cache instead.

The specific purpose of the change in beginning-of-defun-raw was simply to make
it correctly handle the case of open-paren-in-column-0-is-defun-start = nil
when there is no beginning-of-defun-function.

I know this discussion is part of a larger one, but I no longer
remember the rest of the issues in the larger discussion.

However, I did not realize that CC mode actually used that case.  I
thought it had a beginning-of-defun-function.  I was surprised just
now to see that it didn't have one.  Perhaps that should be changed.

However, taking out the use of syntax-ppss in beginning-of-defun-raw
isn't the right way to get good behavior in CC mode.  That would only
make it parse from the start of the file every time.

    3 - changing beginning-of-defun to use parse-partial-sexp (or worse
	syntax-ppss) defeats the purpose of using it as a syntax-begin-function.
	It may even break such uses (e.g. in emacs-lisp-mode).

Why would it break them?

(In any case, such a problem won't arise in Emacs Lisp mode, since
open-paren-in-column-0-is-defun-start = t.)

    I think the problem is that beginning-of-defun has many different possible
    uses, not all of which are compatible:

    1 - it can be used as a "move to toplevel" (i.e. outside of any syntactic
	element).  Currently it's a not reliable way to do that, but it's been
	used as a good heuristic.  Note that in some languages such a concept
	may not even be very meaningful: in languages whose files are commonly
	composed of only one toplevel element (typically a module or a class
	which then contains other elements inside themselves maybe classes or
	modules, ...).

    2 - it can be used as a form of "backward-paragraph-for-prog-langs", to move
	to the beginning of a "block of text".  In case where defuns can be
	nested, this first only move to the beginning of the nested defun.

Normally I'd expect it NOT to treat nested definitions as defuns.
Normally they would be entirely indented.

    3 - a mix of the two: define some level of nesting (if any) as the main one
       (typically either the toplevel one, or if the toplevel is a single
       element, use the next level down) and move to the beginning of the defun
       at that level.

You can get this behavior by adjusting the indentation
when open-paren-in-column-0-is-defun-start = t.

    A reliable way to get behavior 1 is to use syntax-ppss rather than
    beginning-of-defun.

That is a good point.  Maybe some uses of beginning-of-defun
(such as in Font Lock) ought to use syntax-ppss instead.

I think that is orthogonal to the question of this change
in beginning-of-defun-raw.

    The proposed patch basically tries to make beginning-of-defun follow the
    behavior number 1 and to make it do so reliably.

I don't see it that way.

    OTOH it might be a good idea indeed to change beginning-of-defun so that it
    ignores regexp-matches if they're inside comments or strings.  But that'd be
    a different patch, which would apply regardless of
    open-paren-in-column-0-is-defun-start.

I agree, that might be good.

Re: AW: font-locking and open parens in column 0

[martin rudalics]

 > However, I did not realize that CC mode actually used that case.  I
 > thought it had a beginning-of-defun-function.  I was surprised just
 > now to see that it didn't have one.  Perhaps that should be changed.

I've been asking before why c-mode cannot use `c-beginning-of-defun'
here.  Is it because syntax-table properties haven't been set up?  I
recall invoking `c-beginning-of-defun' in arbitrary not yet fontified
text and getting strange results.

 > However, taking out the use of syntax-ppss in beginning-of-defun-raw
 > isn't the right way to get good behavior in CC mode.  That would only
 > make it parse from the start of the file every time.

When scrolling through a buffer backwards, the current patch will cause
parsing from bob every time I scroll.

 > (In any case, such a problem won't arise in Emacs Lisp mode, since
 > open-paren-in-column-0-is-defun-start = t.)

`open-paren-in-column-0-is-defun-start' is a user option.

 >     A reliable way to get behavior 1 is to use syntax-ppss rather than
 >     beginning-of-defun.
 >
 > That is a good point.  Maybe some uses of beginning-of-defun
 > (such as in Font Lock) ought to use syntax-ppss instead.

This would introduce a circular dependency since `syntax-ppss' may call
`beginning-of-defun'.

 >     The proposed patch basically tries to make beginning-of-defun follow the
 >     behavior number 1 and to make it do so reliably.
 >
 > I don't see it that way.

I see it Stefan's way.

Re: AW: font-locking and open parens in column 0

[Stefan Monnier]

>     4 - the original motivation for the patch (i.e. point 1 above) is better
> 	addressed by not using beginning-of-defun and rely on syntax-ppss's
> 	cache instead.
[...]
> However, taking out the use of syntax-ppss in beginning-of-defun-raw
> isn't the right way to get good behavior in CC mode.

This is not what I'm suggesting: I'm saying that the places in cc-mode which
use beginning-of-defun-raw in order to find a safe parsing spot should be
changed to use syntax-ppss for that instead.
This is separate from whether or not beginning-of-defun uses syntax-ppss.

>     3 - changing beginning-of-defun to use parse-partial-sexp (or worse
> 	syntax-ppss) defeats the purpose of using it as
> 	a syntax-begin-function.  It may even break such uses (e.g. in
> 	emacs-lisp-mode).

> Why would it break them?

Because the whole point of using them is to avoid parsing from the beginning
of the buffer.  Maybe it won't burp (depending on how the circular
dependency "syntax-ppss => syntax-begin-function => beginning-of-defun =>
syntax-ppss" works out), but it will at least break the intent of the code.

>     I think the problem is that beginning-of-defun has many different possible
>     uses, not all of which are compatible:

>     1 - it can be used as a "move to toplevel" (i.e. outside of any syntactic
> 	element).  Currently it's a not reliable way to do that, but it's been
> 	used as a good heuristic.  Note that in some languages such a concept
> 	may not even be very meaningful: in languages whose files are commonly
> 	composed of only one toplevel element (typically a module or a class
> 	which then contains other elements inside themselves maybe classes or
> 	modules, ...).

>     2 - it can be used as a form of "backward-paragraph-for-prog-langs",
> 	to move to the beginning of a "block of text".  In case where defuns
> 	can be nested, this first only move to the beginning of the
> 	nested defun.

> Normally I'd expect it NOT to treat nested definitions as defuns.

That strongly depends on the language.  In languages such as Standard-ML,
it's pretty common to have big functions defined inside others.
It's probably true of many languages which use locally defined recursive
functions instead of loops.

In those languages it's not necessarily useful for beginning-of-defun to
only consider toplevel defuns (since there will often only be one or 2 in
teh whole file).

> Normally they would be entirely indented.

Yup.  The whole file is indented, basically.

>     3 - a mix of the two: define some level of nesting (if any) as the
>     main one (typically either the toplevel one, or if the toplevel is
>     a single element, use the next level down) and move to the beginning
>     of the defun at that level.

> You can get this behavior by adjusting the indentation
> when open-paren-in-column-0-is-defun-start = t.

But you may not want to fiddle with the indentation just for C-M-a to jump
the way you like it.  And the language may not allow you to either
(e.g. Haskell where indentation is significant).

>     A reliable way to get behavior 1 is to use syntax-ppss rather than
>     beginning-of-defun.

> That is a good point.  Maybe some uses of beginning-of-defun
> (such as in Font Lock) ought to use syntax-ppss instead.

Font-lock doesn't use beginning-of-defun.  Some clients of font-lock tell
syntaxx-ppss (via syntax-begin-function) to use beginning-of-defun as
a heuristic, for performance reasons.  E.g. emacs-lisp-mode.  Those are not
interested in beginning-of-defun becoming 100% reliable, since they
specifically trade the 100% reliability (which they'd get if they just left
syntax-begin-function alone) for speed.

> I think that is orthogonal to the question of this change
> in beginning-of-defun-raw.

That's odd.  The need for that change comes straight from such a "use
beginning-of-defun to find a safe parsing spot", and the impact of such
a change is mostly going to be on similar uses.


        Stefan