Last steps for pretesting

Last steps for pretesting

[Richard Stallman]

As far as I can see, all we need to do before we start pretesting is

1. fix the recently reported bugs
2. update the copyright notices (all files need 2006)
3. check a few manual files:

     anti.texi   glossary.texi   macos.texi
     maintaining.texi   misc.texi   msdog.texi  trouble.texi

and some of the tasks from the start of FOR-RELEASE.

** update facemenu-unlisted-faces.

** Regenerate the postscript files of the reference cards in etc.

** Ask maintainers of refcard translations to update them.

** Check what should be deleted or updated in MORE.STUFF.

Re: Last steps for pretesting

[Ramprasad B]

[-- Attachment #1.1: Type: text/plain, Size: 509 bytes --]

Richard Stallman <rms@gnu.org> wrote:
 
 >  2. update the copyright notices (all files need 2006)
 
 1. Do we have to add `2006' also for a file which hasn't been edited in 2006 ?     
 2. If a file has 2001 copyright year, how is 2006 copyright year added ?
     Is it like 2001, 2006 or 2001-2006 or 2001, 2002, 2003, 2004, 2005, 2006 ?
 
 Thanks.
 --
 Ramprasad B
 
 
		
---------------------------------
Yahoo! Messenger with Voice. Make PC-to-Phone Calls to the US (and 30+ countries) for 2¢/min or less.

[-- Attachment #1.2: Type: text/html, Size: 684 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Last steps for pretesting

[Eli Zaretskii]

> Date: Mon, 17 Apr 2006 01:01:58 -0700 (PDT)
> From: Ramprasad B <ramprasad_i82@yahoo.com>
> Cc: emacs-devel@gnu.org
> 
> Richard Stallman <rms@gnu.org> wrote:
>  
>  >  2. update the copyright notices (all files need 2006)
>  
>  1. Do we have to add `2006' also for a file which hasn't been edited in 2006 ?     

Yes.

>  2. If a file has 2001 copyright year, how is 2006 copyright year added ?
>      Is it like 2001, 2006 or 2001-2006 or 2001, 2002, 2003, 2004, 2005, 2006 ?

I think just add 2006, but please wait for Richard to give the
definitive answer.

Re: Last steps for pretesting

[Richard Stallman]

     1. Do we have to add `2006' also for a file which hasn't been edited in 2006 ?     

Yes.  All files should list 2006.

     2. If a file has 2001 copyright year, how is 2006 copyright year added ?
	 Is it like 2001, 2006 or 2001-2006 or 2001, 2002, 2003, 2004, 2005, 2006 ?

All the files have 2005, now, don't they?
Do you see any file which lacks 2005 in its copyright notice?

So you just have to add 2006 at the end.

There should be a function copyright-update which does this job.

Re: Last steps for pretesting

[Bill Wohler]

Richard Stallman <rms@gnu.org> writes:

> As far as I can see, all we need to do before we start pretesting is
>
...
> and some of the tasks from the start of FOR-RELEASE.

    ** Ensure MH-E 8.0 has been released.
    Assigned to Bill Wohler <wohler@newt.com>.

Inches (centimeters) away. I'm down to two more tasks.

> ** Check what should be deleted or updated in MORE.STUFF.

Wasn't aware of this file. Added URL for MH-E.

I'm glad to hear we're getting close!

-- 
Bill Wohler <wohler@newt.com>  http://www.newt.com/wohler/  GnuPG ID:610BD9AD
Maintainer of comp.mail.mh FAQ and MH-E. Vote Libertarian!
If you're passed on the right, you're in the wrong lane.

Re: Last steps for pretesting

[Ramprasad B]

--- Richard Stallman <rms@gnu.org> wrote:

> Yes.  All files should list 2006.

Ok.
 
> All the files have 2005, now, don't they?
> Do you see any file which lacks 2005 in its copyright notice?

Yes. Many, like in emacs/admin -> admin.el,  alloc-colors.c, etc.
 
> There should be a function copyright-update which does this job.

Will do that.

Thanks.

--
Ramprasad B

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: Last steps for pretesting

[Richard Stallman]

    > All the files have 2005, now, don't they?
    > Do you see any file which lacks 2005 in its copyright notice?

    Yes. Many, like in emacs/admin -> admin.el,  alloc-colors.c, etc.

The admin directory is not terribly important, since it isn't part of
the release.

It also is not very big, so it would not be hard to fix the files in it.
So how about if you fix them too?

Re: Last steps for pretesting

[Alan Mackenzie]

Hi, Richard, Hi, all!

On Sun, 16 Apr 2006, Richard Stallman wrote:

>As far as I can see, all we need to do before we start pretesting is

>1. fix the recently reported bugs
>2. update the copyright notices (all files need 2006)
>3. check a few manual files:

I'm struggling somewhat to get CC Mode in a presentable state.  I'm
getting a steady trickle of syntax analysis bugs at bug-cc-mode (e.g.,
the "DEFUN" problem) which I'm fixing at about the same rate they're
coming in at.  Thanks to Michaël Cadilhac[*] for sending a patch in with
"his" bug!  I currently have two or three such bugs pending at the moment
(I've lost count).

[*] He tells me his copyright assignment papers are currently being
processed.

Besides these, there are some issues with connecting CC Mode up with
Emacs 22:  Foremost amongst them are:

(i) Fixing the submode indicators (the "/la" of "C/la" on the mode line)
so that their doc-string is displayed when the mouse hovers over them.

(ii) Writing font-lock-expand-region functions, one for CC Mode as a
whole (to replace (setq font-lock-lines-before 2)) and one for AWK Mode
(to replace the advising of the Font Lock after-change functions).

(iii) Tracking down the "CC Mode loads CL" problem.  (I've made good
progress on this).

(iv) A problem in Emacs (See: Subject: Problem mit symlinks,
locate-library and load-history.  Date: Tue, 19 Mar 2006).  This is
caused by (a) load-history storing a promiscuous mix of true file names
and file names containing symbolic links; (b) eval-after-load only
recognising files as being already loaded if they're the ones currently
showing on load-path.  I've half-solved this problem by writing a
function to search load-history properly, which I posted to emacs-devel.
Richard tentatively approved it.  Would it be possible to define
load-history such that all file names in it are true names?

Of these, I could use help for (i) and (iv); (ii) is fairly trivial,
(iii) is something it is best for me to do.

There are also several relatively minor issues, some of which I've
reported to emacs-devel, some of which other people have reported to
bug-cc-mode, which haven't been addressed yet.

I feel a bit like the wet blanket at a party at the moment.  I'm not
going to get all of these things finished this week, or even next week.
Maybe the week after.  Sorry.

Richard, when do you envisage going into pretest?

-- 
Alan.

Re: Last steps for pretesting

[Stefan Monnier]

> (ii) Writing font-lock-expand-region functions, one for CC Mode as a
> whole (to replace (setq font-lock-lines-before 2)) and one for AWK Mode
> (to replace the advising of the Font Lock after-change functions).

BTW, I still intend to move font-lock-extend-region-function from
after-change-functions to font-lock-fontify-region (or maybe even
jit-lock-fontify).

So if you really manage to use the current font-lock-extend-region-function
(called from after-change-functions) in a way that's robust, efficient, and
can't be done as efficiently/elegantly/robustly with a hook in
font-lock-fontify-region (or with the font-lock-multiline property),
please scream.

> (iv) A problem in Emacs (See: Subject: Problem mit symlinks,
> locate-library and load-history.  Date: Tue, 19 Mar 2006).  This is
> caused by (a) load-history storing a promiscuous mix of true file names
> and file names containing symbolic links; (b) eval-after-load only
> recognising files as being already loaded if they're the ones currently
> showing on load-path.  I've half-solved this problem by writing a
> function to search load-history properly, which I posted to emacs-devel.
> Richard tentatively approved it.  Would it be possible to define
> load-history such that all file names in it are true names?

As mentioned in that thread, I think it's better to "truify" names
on-the-fly when you need it rather than directly inside load-history.


        Stefan

Re: Last steps for pretesting

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> As mentioned in that thread, I think it's better to "truify" names
> on-the-fly when you need it rather than directly inside
> load-history.

Uh, but the truification has to happen when loading, anyway.  One
should not replicate the operation unnecessarily at load time, true,
but since the info has to be generated anyway, one could think about
how to make it accessible without additional overhead.

If I am completely bonkers, do have a pointer to that thread?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Evening, Stefan!

On Wed, 19 Apr 2006, Stefan Monnier wrote:

>> (ii) Writing font-lock-expand-region functions, one for CC Mode as a
>> whole (to replace (setq font-lock-lines-before 2)) and one for AWK Mode
>> (to replace the advising of the Font Lock after-change functions).

>BTW, I still intend to move font-lock-extend-region-function from
>after-change-functions to font-lock-fontify-region (or maybe even
>jit-lock-fontify).

That would radically change the meaning of the function.

Please keep Font Lock working equivalently with and without JIT switched
on.  Edebugging through font-lock-fontify-region (for debugging a mode's
font lock settings) is a nightmare when JIT is enabled.

How about the following compromise?  font-lock-extend-region-function
should get called BOTH in the two after-change functions AND in
jit-lock-fontify-now and font-lock-default-fontify-region.  The third
parameter (OLD-LEN) gets bound to nil in the new case.  In either case
the extend-region function is entitled to return nil.

My AWK extend region function would then look like this:

(defun c-awk-font-lock-extend-region (beg end old-len)
  (cons (c-awk-beginning-of-logical-line beg)
        (if old-len
            ;; From an after-change-function
            (c-awk-end-of-change-region beg end old-len)
          ;; From a fontifying routine
          (c-awk-end-of-logical-line end))))

>So if you really manage to use the current
>font-lock-extend-region-function (called from after-change-functions) in
>a way that's robust, efficient, .....

So far, yes (except, perhaps, for the need of an extend-region function
in j-l-f-n and f-l-d-f-r).

>.... and can't be done as efficiently/elegantly/robustly with a hook in
>font-lock-fontify-region (or with the font-lock-multiline property),
>please scream.

AAAAAAAAARRRRRRRRRRRRRRGGGGGGGGGGGGGGGGGGHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!

I don't think we're going to agree on the relative elegance of
font-lock-extend-region-function vs. font-lock-multiline.  I think that
the former is more elegant than the latter, and you think the latter is
the more elegant.

I'm convinced that either approach would be robust (hey, we're competent
hackers ;-).

However, I think that f-l-extend-region-function can be done more
efficiently by the major mode maintainer, in the sense that it will
require less time and effort reading manuals, reading fine source, and
coding.

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (true file names in load-history?)

[Alan Mackenzie]

Evening again, Stefan!

On Wed, 19 Apr 2006, Stefan Monnier wrote:

>> (iv) A problem in Emacs (See: Subject: Problem mit symlinks,
>> locate-library and load-history.  Date: Tue, 19 Mar 2006).  This is
>> caused by (a) load-history storing a promiscuous mix of true file names
>> and file names containing symbolic links; (b) eval-after-load only
>> recognising files as being already loaded if they're the ones currently
>> showing on load-path.  I've half-solved this problem by writing a
>> function to search load-history properly, which I posted to emacs-devel.
>> Richard tentatively approved it.  Would it be possible to define
>> load-history such that all file names in it are true names?

>As mentioned in that thread, I think it's better to "truify" names
>on-the-fly when you need it rather than directly inside load-history.

I think that's wrong.

Every time you evaluate an eval-after-load form, you've got to scan the
list of loaded files in load-history.  The way you suggest, the names in
load-history would need to be "truified" (Hey, that's a lovely word!)
many times, once for each eval-after-load.

If you "truify" the names as you enter them into load-history, you only
need to do it once.  

However, I think it is urgently necessary to define rigorously what file
name goes into load-history.  The lack of this rigour is what triggered a
bug in CC Mode "File mode specification error: (void-variable
c-font-lock-keywords-3)".

I think the absolute true file name would be the best.

>        Stefan

-- 
Alan.

Re: Last steps for pretesting

[Richard Stallman]

    (iii) Tracking down the "CC Mode loads CL" problem.  (I've made good
    progress on this).

I sent you some info about it yesterday.

Re: Last steps for pretesting

[Ramprasad B]

[-- Attachment #1.1: Type: text/plain, Size: 389 bytes --]

Richard Stallman <rms@gnu.org> wrote:
 
 > The admin directory is not terribly important, since it isn't part of
 > the release.
 
 Ok.
 
 > So how about if you fix them too?
 
 Yes fixed emacs/admin completely and fixing some more too.
 
 Thanks.
 --
 Ramprasad B
 
 
 
 
		
---------------------------------
New Yahoo! Messenger with Voice. Call regular phones from your PC and save big.

[-- Attachment #1.2: Type: text/html, Size: 564 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, again, Stefan!

On Wed, 19 Apr 2006, Alan Mackenzie wrote:

>On Wed, 19 Apr 2006, Stefan Monnier wrote:

>>So if you really manage to use the current
>>font-lock-extend-region-function (called from after-change-functions)
>>in a way that's robust, efficient, .....

>So far, yes (except, perhaps, for the need of an extend-region function
>in j-l-f-n and f-l-d-f-r).

>>.... and can't be done as efficiently/elegantly/robustly with a hook in
>>font-lock-fontify-region (or with the font-lock-multiline property),
>>please scream.

>AAAAAAAAARRRRRRRRRRRRRRGGGGGGGGGGGGGGGGGGHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!

>I don't think we're going to agree on the relative elegance of
>font-lock-extend-region-function vs. font-lock-multiline.  I think that
>the former is more elegant than the latter, and you think the latter is
>the more elegant.

>I'm convinced that either approach would be robust (hey, we're competent
>hackers ;-).

>However, I think that f-l-extend-region-function can be done more
>efficiently by the major mode maintainer, in the sense that it will
>require less time and effort reading manuals, reading fine source, and
>coding.

Also critically important:  font-lock-extend-region-function is
documented in the Elisp manual.  The font-lock-multiline property isn't.
Would you please write this documentation now.  Then it would be much
easier for Richard, me and everybody else to compare the two approaches.
Thanks in anticipation!

>>        Stefan

>Alan.

-- 
Alan.

Re: Last steps for pretesting

[Reiner Steib]

On Mon, Apr 17 2006, Richard Stallman wrote:

> 3. check a few manual files:
>
>      anti.texi   glossary.texi   macos.texi
>      maintaining.texi   misc.texi   msdog.texi  trouble.texi

xresources.texi is missing here.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> BTW, I still intend to move font-lock-extend-region-function from
>> after-change-functions to font-lock-fontify-region (or maybe even
>> jit-lock-fontify).

> That would radically change the meaning of the function.

Of course.

> Please keep Font Lock working equivalently with and without JIT switched
> on.  Edebugging through font-lock-fontify-region (for debugging a mode's
> font lock settings) is a nightmare when JIT is enabled.

Yes, OK, don't worry about the jit side-comment.

> How about the following compromise?  font-lock-extend-region-function
> should get called BOTH in the two after-change functions AND in
> jit-lock-fontify-now and font-lock-default-fontify-region.

No, my problem is specifically with calling it from after-change-functions.

>> So if you really manage to use the current
>> font-lock-extend-region-function (called from after-change-functions) in
>> a way that's robust, efficient, .....

> So far, yes (except, perhaps, for the need of an extend-region function
> in j-l-f-n and f-l-d-f-r).

So your code needs extend-region both in a-c-f and in f-l-d-f-r ?

>> .... and can't be done as efficiently/elegantly/robustly with a hook in
>> font-lock-fontify-region (or with the font-lock-multiline property),
>> please scream.

> AAAAAAAAARRRRRRRRRRRRRRGGGGGGGGGGGGGGGGGGHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!

> I don't think we're going to agree on the relative elegance of
> font-lock-extend-region-function vs. font-lock-multiline.  I think that
> the former is more elegant than the latter, and you think the latter is
> the more elegant.

Of course, we don't have to agree on every detail.

> I'm convinced that either approach would be robust (hey, we're competent
> hackers ;-).

I'm convinced that only using a-c-f would *not* be robust, no matter how
competent the hacker.

> However, I think that f-l-extend-region-function can be done more
> efficiently by the major mode maintainer, in the sense that it will
> require less time and effort reading manuals, reading fine source, and
> coding.

more efficiently than what?


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Thu, 20 Apr 2006, Stefan Monnier wrote:

>>> BTW, I still intend to move font-lock-extend-region-function from
>>> after-change-functions to font-lock-fontify-region (or maybe even
>>> jit-lock-fontify).

>> That would radically change the meaning of the function.

>Of course.

>> How about the following compromise?  font-lock-extend-region-function
>> should get called BOTH in the two after-change functions AND in
>> jit-lock-fontify-now and font-lock-default-fontify-region.

>No, my problem is specifically with calling it from
>after-change-functions.

You mean in calling it ONLY from a-c-f?  Or from calling it in
after-change-functions at all?

>>> So if you really manage to use the current
>>> font-lock-extend-region-function (called from after-change-functions)
>>> in a way that's robust, efficient, .....

>> So far, yes (except, perhaps, for the need of an extend-region
>> function in j-l-f-n and f-l-d-f-r).

>So your code needs extend-region both in a-c-f and in f-l-d-f-r ?

Yes.  The region needs extending in two distinct circumstances:
(i) In after-change, to determine the entire region which needs
  refontifying;
(ii) In jit-lock-fontify-now, to ensure that each chunk starts and stops
  at a "safe place".
These are logically disjoint.  But it seems that the major mode code to
do each of them is so similar, that we might as well just have a single
function, font-lock-extend-region-function, called in two different ways.

>>> .... and can't be done as efficiently/elegantly/robustly with a hook
>>> in font-lock-fontify-region (or with the font-lock-multiline
>>> property), please scream.

>> AAAAAAAAARRRRRRRRRRRRRRGGGGGGGGGGGGGGGGGGHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!

>> I don't think we're going to agree on the relative elegance of
>> font-lock-extend-region-function vs. font-lock-multiline.  I think
>> that the former is more elegant than the latter, and you think the
>> latter is the more elegant.

>Of course, we don't have to agree on every detail.

:-)

>> I'm convinced that either approach would be robust (hey, we're competent
>> hackers ;-).

>I'm convinced that only using a-c-f would *not* be robust, no matter how
>competent the hacker.

I agree with you here.

>> However, I think that f-l-extend-region-function can be done more
>> efficiently by the major mode maintainer, in the sense that it will
>> require less time and effort reading manuals, reading fine source, and
>> coding.

>more efficiently than what?

More efficiently than setting font-lock-multiline text properties.

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> So your code needs extend-region both in a-c-f and in f-l-d-f-r ?

> Yes.  The region needs extending in two distinct circumstances:
> (i) In after-change, to determine the entire region which needs
>   refontifying;
> (ii) In jit-lock-fontify-now, to ensure that each chunk starts and stops
>   at a "safe place".
> These are logically disjoint.  But it seems that the major mode code to
> do each of them is so similar, that we might as well just have a single
> function, font-lock-extend-region-function, called in two different ways.

IIRC part (i) needs more than an after-change-function: it also needs
a before-change-function.  And (again IIRC) it's the part that can already
be solved by placing (from one of the font-lock-keywords rules)
a font-lock-multiline property.  Basically it's the part that makes sure
that if a multiline element loses its multiline status, the whole
(previously atomic) text gets properly refreshed.

I find it very hard to believe that b-f-c + a-f-c ends up being as
efficient/elegant/robust as placing a font-lock-multiline property.  In all
likely hood, adding the font-lock-multiline property is a matter of adding
a single line of code at the right place (the place where you fontify
the corresponding multiline entity).

Here is the problems I see with your proposal:
- an `extend-region' hook in font-lock-fontify-region is needed
- the font-lock-multiline property should be enough in all cases to make
  it unnecessary to use an after-change-function hook.
- adding an after-change-function hook additionally to
  a font-lock-fontify-region hook means making font-lock that much more
  complex to use and maintain, especially since that after-change-function
  basically can't be used without also adding a before-change-function.
- an after-change-function is expensive, and even more when it comes bundled
  with a before-change-function, and even more so when compared to
  code that's only run during font-locking.


        Stefan

Re: Last steps for pretesting

[Stefan Monnier]

> Uh, but the truification has to happen when loading, anyway.

No, AFAIK "truification" doesn't currently happen at load time.


        Stefan

Re: Last steps for pretesting

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Uh, but the truification has to happen when loading, anyway.
>
> No, AFAIK "truification" doesn't currently happen at load time.

I would consider it surprising if Emacs was able to load files without
finding out where they are.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Last steps for pretesting

[Stefan Monnier]

>>> Uh, but the truification has to happen when loading, anyway.
>> No, AFAIK "truification" doesn't currently happen at load time.
> I would consider it surprising if Emacs was able to load files without
> finding out where they are.

I'd be surprised if you didn't know that if "/foo" is a symlink to "/bar",
Emacs doesn't need to know it in order to load "/foo/toto/titi.elc", i.e. it
doesn't need to first convert it to "/bar/toto/titi.elc".
There must some kind of of misunderstanding.


        Stefan

Re: Last steps for pretesting

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Uh, but the truification has to happen when loading, anyway.
>>> No, AFAIK "truification" doesn't currently happen at load time.
>> I would consider it surprising if Emacs was able to load files without
>> finding out where they are.
>
> I'd be surprised if you didn't know that if "/foo" is a symlink to
> "/bar", Emacs doesn't need to know it in order to load
> "/foo/toto/titi.elc", i.e. it doesn't need to first convert it to
> "/bar/toto/titi.elc".  There must some kind of of misunderstanding.

I probably confused the issue with making file names absolute or
resolving them in load-path.

Sorry,

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Thu, 20 Apr 2006, Stefan Monnier wrote:

>>> So your code needs extend-region both in a-c-f and in f-l-d-f-r ?

>> Yes.  The region needs extending in two distinct circumstances:
>> (i) In after-change, to determine the entire region which needs
>>   refontifying;
>> (ii) In jit-lock-fontify-now, to ensure that each chunk starts and stops
>>   at a "safe place".
>> These are logically disjoint.  But it seems that the major mode code to
>> do each of them is so similar, that we might as well just have a single
>> function, font-lock-extend-region-function, called in two different ways.

>IIRC part (i) needs more than an after-change-function: it also needs
>a before-change-function.

That would depend on the major mode, I think.  AWK mode needs a
before-change function.  The CC Mode replacement for (setq
font-lock-lines-before 2) will not need a b-c function.

However, using the f-l-multiline property approach instead will also need
a before-change function for exactly the same reasons - or else it will
require the major mode to maintain the f-l-multiline property
permanently.  That would be a major overhead.

>And (again IIRC) it's the part that can already be solved by placing
>(from one of the font-lock-keywords rules) a font-lock-multiline
>property.

I'm not sure.  Can a font-lock-keyword rule get access to after-change's
OLD-LEN in an reasonable fashion?

If a change consists of yanking 20k of text into a buffer, the Major Mode
must not splat font-lock-multiline onto 20k + 20 bytes, since this would
defeat the purpose of JIT lock (fontifying a bit at a time).  The Major
Mode would have to apply f-l-m separately to the head and tail of the
(large) region separately, but in the normal case (small change region)
would have to combine them as a single region.  It would need logic do
distinguish these two cases.  This is inelegant.

>Basically it's the part that makes sure that if a multiline
>element loses its multiline status, the whole (previously atomic) text
>gets properly refreshed.

How is this to be done with the multiline property?

>I find it very hard to believe that b-f-c + a-f-c ends up being as
>efficient/elegant/robust as placing a font-lock-multiline property.  In
>all likely hood, adding the font-lock-multiline property is a matter of
>adding a single line of code at the right place (the place where you
>fontify the corresponding multiline entity).

Stefan, the font-lock-multiline property is still undocumented.  That
makes it very frustrating, very difficult to evaluate the paragraph
you've just written, very difficult to criticise it constructively (I
think I've perhaps already done enough of the other sort of criticism
;-).  

Exactly where the "right place" is to add that single line of code is
completely opaque to me at the moment.  I could probably work it out with
several hours study of the Font Lock source and documentation, but I
don't want to do that at the moment - I've got enough on my plate.

Could you please document the font-lock-multiline mechanism immediately,
including details on how best to access the after-change-functions's
OLD-LEN, and details of how and where to hook in the "(put-text-property a
b 'font-lock-multiline t)" call.  Either that, or tell me where to find
an example of this text property being used, preferably one which uses
OLD-LEN.

Or, perhaps you could recode the following AWK Mode fragment into an
equivalent which uses f-l-multiline instead of f-l-expand-region
function.  PLEASE!

#########################################################################
(defvar c-awk-old-EOLL 0)
(make-variable-buffer-local 'c-awk-old-EOLL)
;; End of logical line following the region which is about to be changed.  Set
;; in c-awk-before-change and used in c-awk-font-lock-extend-region.

(defun c-awk-before-change (beg end)
;; This function is called exclusively from the before-change-functions hook.
;; It does two things: Finds the end of the (logical) line on which END lies,
;; and clears c-awk-NL-prop text properties from this point onwards.
  (save-restriction
    (save-excursion
      (setq c-awk-old-EOLL (c-awk-end-of-logical-line end))
      (c-save-buffer-state nil
       (c-awk-clear-NL-props end (point-max))))))
(add-hook 'before-change-functions c-awk-before-change nil t)

(defun c-awk-end-of-change-region (beg end old-len)
  ;; Find the end of the region which needs to be font-locked after a change.
  ;; This is the end of the logical line on which the change happened, either
  ;; as it was before the change, or as it is now, whichever is later.
  ;; N.B. point is left undefined.
  (max (+ (- c-awk-old-EOLL old-len) (- end beg))
       (c-awk-end-of-logical-line end)))

(defun c-awk-font-lock-extend-region (beg end old-len)
  (cons (c-awk-beginning-of-logical-line beg)
        (if old-len
            ;; From an after-change-function
            (c-awk-end-of-change-region beg end old-len)
          ;; From a fontifying routine
          (c-awk-end-of-logical-line end))))
(setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)
#########################################################################

>Here is the problems I see with your proposal:
>- an `extend-region' hook in font-lock-fontify-region is needed

It is indeed.  It is a simple patch, and I am ready and willing to write
it, both for font-core.el and modes.texi.  I do first ask that it's
accepted in principle, though.

>- the font-lock-multiline property should be enough in all cases to make
>  it unnecessary to use an after-change-function hook.

f-l-extend-region-function also makes a (separate) after-change-f hook
unnecessary.

>- adding an after-change-function hook additionally to
>  a font-lock-fontify-region hook means making font-lock that much more
>  complex to use and maintain, especially since that after-change-function
>  basically can't be used without also adding a before-change-function.

I disagree with everything in that paragraph.  :-)

>- an after-change-function is expensive, and even more when it comes bundled
>  with a before-change-function, and even more so when compared to
>  code that's only run during font-locking.

Huh?  Font Locking _is_ expensive.  A before-change function need not be
expensive (e.g.  c-awk-before-change, see above), and calling it is not
expensive.  I suspect that using the f-l-multiline property will
absolutely mandate an extra after-change function - how else can it
access OLD-LEN?

Stefan, please show me the code!  (Or the documentation!).  How does a
hacker use the font-lock-multiline property?  In detail.

-- 
Alan.

Re: Last steps for pretesting

[Alan Mackenzie]

On Thu, 20 Apr 2006, Stefan Monnier wrote:

>>>> Uh, but the truification has to happen when loading, anyway.
>>> No, AFAIK "truification" doesn't currently happen at load time.
>> I would consider it surprising if Emacs was able to load files without
>> finding out where they are.

>I'd be surprised if you didn't know that if "/foo" is a symlink to "/bar",
>Emacs doesn't need to know it in order to load "/foo/toto/titi.elc", i.e. it
>doesn't need to first convert it to "/bar/toto/titi.elc".
>There must some kind of of misunderstanding.

Emacs DOES need to know, every time it evaluates an eval-after-load.

For example, if this form is evaluated:

    (eval-after-load "/mnt/hda7/emacs/lisp/cc-fonts.elc" '(...))

and load-history has as one of its elements

    ("/home/acm/emacs/emacs/lisp/cc-fonts.elc" ....), ; a symlink

then each stored file name must be truified at _every_ eval-after-load.

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> IIRC part (i) needs more than an after-change-function: it also needs
>> a before-change-function.

> That would depend on the major mode, I think.  AWK mode needs a
> before-change function.  The CC Mode replacement for (setq
> font-lock-lines-before 2) will not need a b-c function.

Simulating font-lock-lines-before only needs an `extend-region' hook in
font-lock-default-fontify-region, a subject about which we already agree.

> However, using the f-l-multiline property approach instead will also need
> a before-change function for exactly the same reasons

That's news to me.  Could you explain the scenario you have in mind?

> - or else it will require the major mode to maintain the f-l-multiline
> property permanently.

The font-lock-multiline property is removed by font-lock-unfontify-region
and it is expected that it gets re-added (if needed) every time you fontify
a piece of text.  It's never seemed to be a big deal.
No before-change-function in sight.

>> And (again IIRC) it's the part that can already be solved by placing
>> (from one of the font-lock-keywords rules) a font-lock-multiline
>> property.

> I'm not sure.  Can a font-lock-keyword rule get access to after-change's
> OLD-LEN in an reasonable fashion?

What for?  The font-lock-multiline it needs to add is for use by the next
font-lock round.  It says: "next time you fontify this area, make sure you
refontify this piece atomically".

> If a change consists of yanking 20k of text into a buffer, the Major Mode
> must not splat font-lock-multiline onto 20k + 20 bytes, since this would
> defeat the purpose of JIT lock (fontifying a bit at a time).

The font-lock-multiline property is added not to the whole fontified text,
but just to a particular region of text that matched a pattern (a
"keyword").  I think you're absolutely right: I have to document it because
you seem to completely misunderstand it.

> Could you please document the font-lock-multiline mechanism immediately,
> including details on how best to access the after-change-functions's
> OLD-LEN, and details of how and where to hook in the "(put-text-property a
> b 'font-lock-multiline t)" call.  Either that, or tell me where to find
> an example of this text property being used, preferably one which uses
> OLD-LEN.

AFAIK the OLD-LEN is never needed.  See font-lock.el for an example of code
that sets font-lock-multiline property: if a regexp matches several lines,
it puts the font-lock-multiline property on the wholse matched text (this
is only done if the font-lock-multiline variable is set).

> Or, perhaps you could recode the following AWK Mode fragment into an
> equivalent which uses f-l-multiline instead of f-l-expand-region
> function.  PLEASE!

I'll try to, although I'd need to see the font-lock-keywords part (where
I'll have to add the code).

> #########################################################################
> (defvar c-awk-old-EOLL 0)
> (make-variable-buffer-local 'c-awk-old-EOLL)
> ;; End of logical line following the region which is about to be changed.  Set
> ;; in c-awk-before-change and used in c-awk-font-lock-extend-region.

> (defun c-awk-before-change (beg end)
> ;; This function is called exclusively from the before-change-functions hook.
> ;; It does two things: Finds the end of the (logical) line on which END lies,
> ;; and clears c-awk-NL-prop text properties from this point onwards.
>   (save-restriction
>     (save-excursion
>       (setq c-awk-old-EOLL (c-awk-end-of-logical-line end))
>       (c-save-buffer-state nil
>        (c-awk-clear-NL-props end (point-max))))))
> (add-hook 'before-change-functions c-awk-before-change nil t)

> (defun c-awk-end-of-change-region (beg end old-len)
>   ;; Find the end of the region which needs to be font-locked after a change.
>   ;; This is the end of the logical line on which the change happened, either
>   ;; as it was before the change, or as it is now, whichever is later.
>   ;; N.B. point is left undefined.
>   (max (+ (- c-awk-old-EOLL old-len) (- end beg))
>        (c-awk-end-of-logical-line end)))

> (defun c-awk-font-lock-extend-region (beg end old-len)
>   (cons (c-awk-beginning-of-logical-line beg)
>         (if old-len
>             ;; From an after-change-function
>             (c-awk-end-of-change-region beg end old-len)
>           ;; From a fontifying routine
>           (c-awk-end-of-logical-line end))))
> (setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)
> #########################################################################

IIUC, something like the following should do:

(defun c-awk-font-lock-extend-region (beg end)
  (cons (c-awk-beginning-of-logical-line beg)
        (c-awk-end-of-logical-line end)))
(setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)

(defconst c-awk-font-lock-keywords
   '(...
     (".*\\\\\n.*"
      (0 (progn (put-text-property (match-beginning 0) (match-end 0)
                                   'font-lock-multiline t)
                nil)))
     ...))

although I suspect that you already have a font-lock-keyword that matches
something akin to ".*\\\\\n.*" where you can add the put-text-property.

>> Here is the problems I see with your proposal:
>> - an `extend-region' hook in font-lock-fontify-region is needed

> It is indeed.  It is a simple patch, and I am ready and willing to write
> it, both for font-core.el and modes.texi.  I do first ask that it's
> accepted in principle, though.

It's completely accepted.  I just hope we can call it
"font-lock-extend-region-function".

>> - the font-lock-multiline property should be enough in all cases to make
>> it unnecessary to use an after-change-function hook.

> f-l-extend-region-function also makes a (separate) after-change-f hook
> unnecessary.

Which f-l-extend-region-function?  The one called from after-change-f?
Then it's what I meant by "an after-change-function hook".  The fact that
it's not called directly from a-c-f but via f-t-a-c-f is not very
important for this discussion.

>> - an after-change-function is expensive, and even more when it comes bundled
>> with a before-change-function, and even more so when compared to
>> code that's only run during font-locking.
> Huh?  Font Locking _is_ expensive.

Yup, but (thanks to jit-lock) it's not executed after each and every
buffer modification.

> A before-change function need not be expensive (e.g.  c-awk-before-change,
> see above), and calling it is not expensive.

The code will have probably the same cost whether it's called from a-c-f or
from font-lock, but in one case it'll be called (much) more often.

> I suspect that using the f-l-multiline property will absolutely mandate an
> extra after-change function - how else can it access OLD-LEN?

Why would it need OLD-LEN?


        Stefan

Re: Last steps for pretesting

[Richard Stallman]

    xresources.texi is missing here.

I think we will move most or all of that out of the Emacs manual.
It is not necessary for editing with Emacs.

Re: Last steps for pretesting

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     xresources.texi is missing here.
>
> I think we will move most or all of that out of the Emacs manual.
> It is not necessary for editing with Emacs.

I strongly disagree.  The X resources (and equivalents) are the only
way to specify startup fonts and geometry in a way that makes them
work on Emacs startup such that the window manager has a chance to
accommodate them.

X resources and startup options are so important that they are one of
the few things mentioned in man pages of programs even when separate
and more elaborate info pages exist.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Good morning, Stefan!

On Thu, 20 Apr 2006, Stefan Monnier wrote:

>>> IIRC part (i) needs more than an after-change-function: it also needs
>>> a before-change-function.

>> That would depend on the major mode, I think.  AWK mode needs a
>> before-change function.  The CC Mode replacement for (setq
>> font-lock-lines-before 2) will not need a b-c function.

>Simulating font-lock-lines-before only needs an `extend-region' hook in
>font-lock-default-fontify-region, a subject about which we already agree.

We do.  :-)

>> - or else it will require the major mode to maintain the f-l-multiline
>> property permanently.

>The font-lock-multiline property is removed by font-lock-unfontify-region
>and it is expected that it gets re-added (if needed) every time you fontify
>a piece of text.  It's never seemed to be a big deal.
>No before-change-function in sight.

That's why I think it won't work in AWK mode.  For Font locking triggered
by a change, you need the details of the _change_, not merely how the
buffer looks after it.

>>> And (again IIRC) it's the part that can already be solved by placing
>>> (from one of the font-lock-keywords rules) a font-lock-multiline
>>> property.

>> I'm not sure.  Can a font-lock-keyword rule get access to after-change's
>> OLD-LEN in an reasonable fashion?

>What for?  The font-lock-multiline it needs to add is for use by the next
>font-lock round.  It says: "next time you fontify this area, make sure you
>refontify this piece atomically".

The font locking is triggered by the Change.  Therefore it needs details
of that change  to determine what region of text needs refontifying.  It
is up to the major mode maintainer, whether she needs those details or
not.  AWK Mode, for example, does need them.

>> If a change consists of yanking 20k of text into a buffer, the Major Mode
>> must not splat font-lock-multiline onto 20k + 20 bytes, since this would
>> defeat the purpose of JIT lock (fontifying a bit at a time).

>The font-lock-multiline property is added not to the whole fontified text,
>but just to a particular region of text that matched a pattern (a
>"keyword").  I think you're absolutely right: I have to document it because
>you seem to completely misunderstand it.

:-)  Thanks!

>> Could you please document the font-lock-multiline mechanism
>> immediately, including details on how best to access the
>> after-change-functions's OLD-LEN, and details of how and where to hook
>> in the "(put-text-property a b 'font-lock-multiline t)" call.  Either
>> that, or tell me where to find an example of this text property being
>> used, preferably one which uses OLD-LEN.

>AFAIK the OLD-LEN is never needed.

It is most definitely needed.  I don't think you have understood what I
have been trying to say.  :-(

>See font-lock.el for an example of code that sets font-lock-multiline
>property: if a regexp matches several lines, it puts the
>font-lock-multiline property on the wholse matched text (this is only
>done if the font-lock-multiline variable is set).

Perhaps I have truly misunderstood you.  I thought you were telling me
that some code contained within the major mode would do the
(put-text-property ... 'font-lock-multiline ...).  Are you really saying
that the major mode merely has to set up its font-lock-keywords so that
font-lock.el finds the right places to apply the text property?

>> Or, perhaps you could recode the following AWK Mode fragment into an
>> equivalent which uses f-l-multiline instead of f-l-expand-region
>> function.  PLEASE!

>I'll try to, although I'd need to see the font-lock-keywords part (where
>I'll have to add the code).

It's right at the end of cc-fonts.el.

>> #########################################################################
>> (defvar c-awk-old-EOLL 0)
>> (make-variable-buffer-local 'c-awk-old-EOLL)
>> ;; End of logical line following the region which is about to be changed.  Set
>> ;; in c-awk-before-change and used in c-awk-font-lock-extend-region.

>> (defun c-awk-before-change (beg end)
>> ;; This function is called exclusively from the before-change-functions hook.
>> ;; It does two things: Finds the end of the (logical) line on which END lies,
>> ;; and clears c-awk-NL-prop text properties from this point onwards.
>>   (save-restriction
>>     (save-excursion
>>       (setq c-awk-old-EOLL (c-awk-end-of-logical-line end))
>>       (c-save-buffer-state nil
>>        (c-awk-clear-NL-props end (point-max))))))
>> (add-hook 'before-change-functions c-awk-before-change nil t)

>> (defun c-awk-end-of-change-region (beg end old-len)
>>   ;; Find the end of the region which needs to be font-locked after a change.
>>   ;; This is the end of the logical line on which the change happened, either
>>   ;; as it was before the change, or as it is now, whichever is later.
>>   ;; N.B. point is left undefined.
>>   (max (+ (- c-awk-old-EOLL old-len) (- end beg))
>>        (c-awk-end-of-logical-line end)))

>> (defun c-awk-font-lock-extend-region (beg end old-len)
>>   (cons (c-awk-beginning-of-logical-line beg)
>>         (if old-len
>>             ;; From an after-change-function
>>             (c-awk-end-of-change-region beg end old-len)
>>           ;; From a fontifying routine
>>           (c-awk-end-of-logical-line end))))
>> (setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)
>> #########################################################################

>IIUC, something like the following should do:

>(defun c-awk-font-lock-extend-region (beg end)
>  (cons (c-awk-beginning-of-logical-line beg)
>        (c-awk-end-of-logical-line end))) <==================================
>(setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)

That, quite demonstrably, WON'T do, since that will fail to extend the
region to what was the end of the logical line before the change.

>(defconst c-awk-font-lock-keywords
>   '(...
>     (".*\\\\\n.*"
>      (0 (progn (put-text-property (match-beginning 0) (match-end 0)
>                                   'font-lock-multiline t)
>                nil)))
>     ...))

OK, I think I'm beginning to get the idea.



>>> Here is the problems I see with your proposal:
>>> - an `extend-region' hook in font-lock-fontify-region is needed

>> It is indeed.  It is a simple patch, and I am ready and willing to write
>> it, both for font-core.el and modes.texi.  I do first ask that it's
>> accepted in principle, though.

>It's completely accepted.  I just hope we can call it
>"font-lock-extend-region-function".

It has been accepted (by Richard, no less), as part of the
font-lock-after-change-function/jit-lock-after-change.  I'm proposing to
enhance it to do the necessary region extension from within
f-l-default-fontify-region/j-l-fontify-now too.

>>> - the font-lock-multiline property should be enough in all cases to make
>>> it unnecessary to use an after-change-function hook.

>> f-l-extend-region-function also makes a (separate) after-change-f hook
>> unnecessary.

>Which f-l-extend-region-function?  The one called from after-change-f?

Yes.

>Then it's what I meant by "an after-change-function hook".  The fact that
>it's not called directly from a-c-f but via f-t-a-c-f is not very
>important for this discussion.

I think it's important.  It saves the major mode maintainer from having
to install his code as a separate a-c-function, and juggling them around
to make sure they get called in the correct order.

>>> - an after-change-function is expensive, and even more when it comes
>>> bundled with a before-change-function, and even more so when compared
>>> to code that's only run during font-locking.
>> Huh?  Font Locking _is_ expensive.

>Yup, but (thanks to jit-lock) it's not executed after each and every
>buffer modification.

OK.  I think your point is that the f-l-extend-region-function called
from f-l-a-f-f/j-l-a-c is expensive because it is called after _every_
change.  That is potentially true, and it is up to the major mode writer
to ensure that this isn't a problem.  In C Mode, for example,
c-extend-region-function (as it will be in CC Mode 5.32/Emacs 23.1) will
be an extensive thing, involving sophisticated caching, sneaky
heuristics, and so on, so as it will run fast enough.  In AWK Mode, the
function is trivial enough that it's already not a problem.

>> A before-change function need not be expensive (e.g.  c-awk-before-change,
>> see above), and calling it is not expensive.

>The code will have probably the same cost whether it's called from a-c-f or
>from font-lock, but in one case it'll be called (much) more often.

Yes.  But that calling from a-c-f is essential to correct font locking.

>> I suspect that using the f-l-multiline property will absolutely
>> mandate an extra after-change function - how else can it access
>> OLD-LEN?

>Why would it need OLD-LEN?

Stefan, I've tried to answer this question several times already, but
I've clearly not made a good job of it.  I'm getting very frustrated with
not being able to get my point across to you.

PLEASE go through this canonical example in all its gory detail and tell
me you've understood it.  Please tell me that you understand what I'm
doing in the c-awk-before-change, and what I'm doing with OLD-LEN.
Please tell me you see why the way you suggested (89 lines higher up)
wouldn't work.

If you can then tell me a way of achieving the same effect WITHOUT the
use of before-change and/or OLD-LEN, I'd be happy to hear about it.

OK, here goes!

#########################################################################
In AWK Mode:

Point is at EOL 3, and we delete the backslash there.

1. "string \
2. over \
3. several \       <========= point is at EOL 3, about to delete the \
4. #lines."

1. In c-awk-before-change (see above for the code), we calculate
c-awk-old-EOLL ("end of old logical line").  This give 36, (EOL 4).

2. The \ gets deleted, and the text looks like this:

1. "string \
2. over \
3. several 
4. #lines."

3. jit-lock-after-change gets called as (jit-lock-after-change 26 26 1),
which in its turn calls (c-awk-font-lock-extend-region 26 26 1).  This
"1" is the OLD-LEN, of course.

4. c-awk-f-l-e-r calls (c-awk-end-of-change-region 26 26 1).  This does
the following:
(i) It determines the current position of the PREVIOUS end of logical
line: (+ (- c-awk-old-EOLL old-len) (- end beg)),
      => (+ (- 36 1) (- 26 26))
      => 35
NOTE THE USE OF old-len.

(ii) It determines the current end of logical line:
    (c-awk-end-of-logical 26)
     => 26.  This is EOL 3

(iii) It selects the greater of these two positions:
    (max (+ (- c-awk-old-EOLL old-len) (- end beg))
         (c-awk-end-of-logical-line end))
    => (max 35 26)
    => 35.
This 35 is the current EOL 4.

(iv) 35 is returned to jit-lock-after-change in the cons (1 . 35), the
extended fontification region.
#########################################################################

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> Good morning, Stefan!

Good morning Alan,

> The font locking is triggered by the Change.  Therefore it needs details
> of that change  to determine what region of text needs refontifying.  It

No, it only needs to know what the current text looks like (obviously) and
how the text was font-locked last time (to properly remove highlighting
where it doesn't apply any more).  This last part canbe obtained via the
font-lock-multiline property.

> Perhaps I have truly misunderstood you.  I thought you were telling me
> that some code contained within the major mode would do the
> (put-text-property ... 'font-lock-multiline ...).  Are you really saying
> that the major mode merely has to set up its font-lock-keywords so that
> font-lock.el finds the right places to apply the text property?

I've generally been assuming that your code would explicitly do
a put-text-property, but indeed font-lock.el includes some code which tries
to do that for you.

>> IIUC, something like the following should do:

>> (defun c-awk-font-lock-extend-region (beg end)
>> (cons (c-awk-beginning-of-logical-line beg)
>> (c-awk-end-of-logical-line end))) <==================================
>> (setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)

> That, quite demonstrably, WON'T do, since that will fail to extend the
> region to what was the end of the logical line before the change.

Of course it's not enough: it only takes care of making sure current atomic
elements are properly fontified, but not that previously atomic elements are
properly refontified.  That's what the font-lock-multiline property is for.

>>>> Here is the problems I see with your proposal:
>>>> - an `extend-region' hook in font-lock-fontify-region is needed

>>> It is indeed.  It is a simple patch, and I am ready and willing to write
>>> it, both for font-core.el and modes.texi.  I do first ask that it's
>>> accepted in principle, though.

>> It's completely accepted.  I just hope we can call it
>> "font-lock-extend-region-function".

> It has been accepted (by Richard, no less), as part of the
> font-lock-after-change-function/jit-lock-after-change.  I'm proposing to
> enhance it to do the necessary region extension from within
> f-l-default-fontify-region/j-l-fontify-now too.

I consider the two as separate: the first is already installed (but I want
it removed) and the second is not installed yet, but we agree that it should
be added.  I.e. I want it moved from a-c-f to f-l-d-f-r (it should not be
added to j-l-fontify-now).

>>>> - the font-lock-multiline property should be enough in all cases to make
>>>> it unnecessary to use an after-change-function hook.

>>> f-l-extend-region-function also makes a (separate) after-change-f hook
>>> unnecessary.

>> Which f-l-extend-region-function?  The one called from after-change-f?

> Yes.

>> Then it's what I meant by "an after-change-function hook".  The fact that
>> it's not called directly from a-c-f but via f-t-a-c-f is not very
>> important for this discussion.

> I think it's important.  It saves the major mode maintainer from having
> to install his code as a separate a-c-function, and juggling them around
> to make sure they get called in the correct order.

Yes it is important in general.  But this discussion is about the need to
have *any* a-c-f hook.  If we need one, its place is in f-l-a-c-f, indeed,
but I argue that we don't need one, so the precise place is not really
relevant to the discussion.

>> The code will have probably the same cost whether it's called from a-c-f or
>> from font-lock, but in one case it'll be called (much) more often.
> Yes.  But that calling from a-c-f is essential to correct font locking.

That's what I claim is not true.

> #########################################################################
> In AWK Mode:

> Point is at EOL 3, and we delete the backslash there.

> 1. "string \
> 2. over \
> 3. several \       <========= point is at EOL 3, about to delete the \
> 4. #lines."

OK: if the text had never been font-locked before, an a hook in
f-l-fontify-region is all we need, right?  So the interesting case is when
the text had already been fontified.
In this case, the whole multiline-line has had a font-lock-multiline added,
so at this point in your example, the multiline-line is 100% covered by
a font-lock-multiline property.

> 1. In c-awk-before-change (see above for the code), we calculate
> c-awk-old-EOLL ("end of old logical line").  This give 36, (EOL 4).

Here let's say instead we don't do anything.

> 2. The \ gets deleted, and the text looks like this:

> 1. "string \
> 2. over \
> 3. several 
> 4. #lines."

OK, so the whole text from line 1 to line 4 still has a font-lock-multiline
property.

> 3. jit-lock-after-change gets called as (jit-lock-after-change 26 26 1),
> which in its turn calls (c-awk-font-lock-extend-region 26 26 1).  This
> "1" is the OLD-LEN, of course.

Here let's also say that instead we don't do anything special.

> 4. c-awk-f-l-e-r calls (c-awk-end-of-change-region 26 26 1).  This does
> the following:
> (i) It determines the current position of the PREVIOUS end of logical
> line: (+ (- c-awk-old-EOLL old-len) (- end beg)),
>       => (+ (- 36 1) (- 26 26))
>       => 35
> NOTE THE USE OF old-len.

> (ii) It determines the current end of logical line:
>     (c-awk-end-of-logical 26)
>      => 26.  This is EOL 3

> (iii) It selects the greater of these two positions:
>     (max (+ (- c-awk-old-EOLL old-len) (- end beg))
>          (c-awk-end-of-logical-line end))
>     => (max 35 26)
>     => 35.
> This 35 is the current EOL 4.

> (iv) 35 is returned to jit-lock-after-change in the cons (1 . 35), the
> extended fontification region.
> #########################################################################

Here let's also say that instead we don't do anything special.

But when we later get to f-l-fontify-region, the code looks at the
font-lock-multiline property at the boundary (i.e. at the beginning and end
of line 3) and noticies that it's set, so it extends the region to be
fontified to span all 4 lines.  Just what you wanted all along.

Look ma!  No old-len!  Tadaaa!

If you insert text (instead of removing it), it gets a bit more interesting
(because the inserted text doesn't have a font-lock-multiline property)
but it basically works along the same lines.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Good evening, Stefan!

On Fri, 21 Apr 2006, Stefan Monnier wrote:

>> Good morning, Stefan!

>Good morning Alan,

>> The font locking is triggered by the Change.  Therefore it needs details
>> of that change  to determine what region of text needs refontifying.  It

>No, it only needs to know what the current text looks like (obviously) and
>how the text was font-locked last time (to properly remove highlighting
>where it doesn't apply any more).  This last part can be obtained via the
>font-lock-multiline property.

OK.  I think you're right here, but I'm not 100% sure.

>> Perhaps I have truly misunderstood you.  I thought you were telling me
>> that some code contained within the major mode would do the
>> (put-text-property ... 'font-lock-multiline ...).  Are you really
>> saying that the major mode merely has to set up its font-lock-keywords
>> so that font-lock.el finds the right places to apply the text
>> property?

>I've generally been assuming that your code would explicitly do
>a put-text-property, but indeed font-lock.el includes some code which tries
>to do that for you.

It would do this, presumably, in an after-change (or before-change)
function, either directly on one of these hooks or called from
font-lock-after-change.

>>> IIUC, something like the following should do:

>>> (defun c-awk-font-lock-extend-region (beg end)
>>> (cons (c-awk-beginning-of-logical-line beg)
>>> (c-awk-end-of-logical-line end))) <==================================
>>> (setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)

>> That, quite demonstrably, WON'T do, since that will fail to extend the
>> region to what was the end of the logical line before the change.

>Of course it's not enough: it only takes care of making sure current
>atomic elements are properly fontified, but not that previously atomic
>elements are properly refontified.  That's what the font-lock-multiline
>property is for.

OK.  So the font-lock-multiline property is put on exactly a region of
text which needs fontifying atomically, yet straddles a line break.  An
after-change function (and/or a before-change function) is what will do
this putting and erasing of f-l-multiline.

>>>>> Here is the problems I see with your proposal:
>>>>> - an `extend-region' hook in font-lock-fontify-region is needed

>>>> It is indeed.  It is a simple patch, and I am ready and willing to write
>>>> it, both for font-core.el and modes.texi.  I do first ask that it's
>>>> accepted in principle, though.

>>> It's completely accepted.  I just hope we can call it
>>> "font-lock-extend-region-function".

>> It has been accepted (by Richard, no less), as part of the
>> font-lock-after-change-function/jit-lock-after-change.  I'm proposing to
>> enhance it to do the necessary region extension from within
>> f-l-default-fontify-region/j-l-fontify-now too.

>I consider the two as separate: the first is already installed (but I want
>it removed) and the second is not installed yet, but we agree that it should
>be added.  I.e. I want it moved from a-c-f to f-l-d-f-r (it should not be
>added to j-l-fontify-now).

I think it's more accurate just to say you want to remove the
extend-region stuff from f-l-after-change.  I want it to stay.  :-)
Why don't we call the thing we need in f-l-d-f-r
"font-lock-extend-CHUNK-function", since it's more likely to be a
jit-lock chunk than a region supplied by the user or major mode?

What is your objection to f-l-extend-region-f in a-c-f?  Is it because it
might gobble too much processor time?

It needs to be in j-l-fontify-now, so that that function knows what bytes
to apply the 'fontified property to.  Some refactoring of the code might
be helpful - say, extracting `font-lock-core-fontify-region' from
font-lock-default-fontify-region, this new function fontifying exactly
the region it is given, and doing it immediately.

>>>>> - the font-lock-multiline property should be enough in all cases to
>>>>> make it unnecessary to use an after-change-function hook.

>>>> f-l-extend-region-function also makes a (separate) after-change-f hook
>>>> unnecessary.

>>> Which f-l-extend-region-function?  The one called from after-change-f?

>> Yes.

>>> Then it's what I meant by "an after-change-function hook".  The fact that
>>> it's not called directly from a-c-f but via f-t-a-c-f is not very
>>> important for this discussion.

>> I think it's important.  It saves the major mode maintainer from having
>> to install his code as a separate a-c-function, and juggling them around
>> to make sure they get called in the correct order.

>Yes it is important in general.  But this discussion is about the need to
>have *any* a-c-f hook.  If we need one, its place is in f-l-a-c-f, indeed,
>but I argue that we don't need one, so the precise place is not really
>relevant to the discussion.

I can't see how a major mode can manipulate f-l-multiline properties
anywhere BUT an a-c-f (or a before-c-f).

>>> The code will have probably the same cost whether it's called from
>>> a-c-f or from font-lock, but in one case it'll be called (much) more
>>> often.
>> Yes.  But that calling from a-c-f is essential to correct font
>> locking.

>That's what I claim is not true.

I can't see that yet, but I'm trying to.  I think you're saying that
instead of AWK Mode recording information in c-awk-old-EOLL ("c-awk old
end of logical line") in the before-change function, it would make some
setting of font-lock-multiline over the pertinent part of the buffer.

>> #########################################################################
>> In AWK Mode:

>> Point is at EOL 3, and we delete the backslash there.

>> 1. "string \
>> 2. over \
>> 3. several \       <========= point is at EOL 3, about to delete the \
>> 4. #lines."

>OK: if the text had never been font-locked before, a hook in
>f-l-fontify-region is all we need, right?  So the interesting case is
>when the text had already been fontified.  In this case, the whole
>multiline-line has had a font-lock-multiline added, so at this point in
>your example, the multiline-line is 100% covered by a
>font-lock-multiline property.

OK.

>> 1. In c-awk-before-change (see above for the code), we calculate
>> c-awk-old-EOLL ("end of old logical line").  This give 36, (EOL 4).

>Here let's say instead we don't do anything.

OK.

>> 2. The \ gets deleted, and the text looks like this:

>> 1. "string \
>> 2. over \
>> 3. several 
>> 4. #lines."

>OK, so the whole text from line 1 to line 4 still has a font-lock-multiline
>property.

OK.

>> 3. jit-lock-after-change gets called as (jit-lock-after-change 26 26 1),
>> which in its turn calls (c-awk-font-lock-extend-region 26 26 1).  This
>> "1" is the OLD-LEN, of course.

>Here let's also say that instead we don't do anything special.

OK.

>> 4. c-awk-f-l-e-r calls (c-awk-end-of-change-region 26 26 1).  This does
>> the following:
>> (i) It determines the current position of the PREVIOUS end of logical
>> line: (+ (- c-awk-old-EOLL old-len) (- end beg)),
>>       => (+ (- 36 1) (- 26 26))
>>       => 35
>> NOTE THE USE OF old-len.

>> (ii) It determines the current end of logical line:
>>     (c-awk-end-of-logical 26)
>>      => 26.  This is EOL 3

>> (iii) It selects the greater of these two positions:
>>     (max (+ (- c-awk-old-EOLL old-len) (- end beg))
>>          (c-awk-end-of-logical-line end))
>>     => (max 35 26)
>>     => 35.
>> This 35 is the current EOL 4.

>> (iv) 35 is returned to jit-lock-after-change in the cons (1 . 35), the
>> extended fontification region.
>> #########################################################################

>Here let's also say that instead we don't do anything special.

OK.

>But when we later get to f-l-fontify-region, the code looks at the
>font-lock-multiline property at the boundary (i.e. at the beginning and end
>of line 3) and noticies that it's set, so it extends the region to be
>fontified to span all 4 lines.  Just what you wanted all along.

OK, but .....

>Look ma!  No old-len!  Tadaaa!

.... but some code, somewhere, sometime needs to adjust the f-l-m
property, removing it from L4.  In general, it will also need to _split_
the text property into L1,2,3 and L4,5,...  Otherwise, a f-l-m region
could gradually engulf an entire buffer, making a nonsense of jit-lock.
(OK, this would be an extreme situation.  :-)

>If you insert text (instead of removing it), it gets a bit more
>interesting (because the inserted text doesn't have a
>font-lock-multiline property) but it basically works along the same
>lines.

Something, somewhere, sometime has to analyse the newly inserted code,
putting f-l-m properties throughout it, I think, and adjusting those at
the boundaries of the new region.  Is this to be done at the time of the
change or at the time it gets font locked?  I think it has to be done at
the change, otherwise the f-l-m settings will be wrong when it comes time
to fontify a chunk.

#########################################################################

SUMMARY:  

1. I think font-lock-extend-region-function, called as it is from
j-l-after-change, is a good thing.  It is a correct way of giving a major
mode an opportunity to massage a fontification region.  I think it should
stay.  You think it should be removed.

2. f-l-d-f-r / j-l-f-now absolutely need a
"font-lock-extend-chunk-function" functionality (regardless of its
precise name).  We agree on this. 

3. You have convinced me that it is possible to use the f-l-m properties.

4. I think that f-l-extend-region-f, called from f-l-after-change, is
likely to be more convenient and less error prone than the direct
manipulation of the f-l-multiline property by major mode code.

5. I suspect you think that f-l-extend-region-f is likely to consume more
processor time than direct manipulation of the f-l-m property. 

Is there any reason why we can't leave both methods side by side in the
code?

>        Stefan

BTW:  I'll be away from the net over the next couple of days.  ;-)

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> I've generally been assuming that your code would explicitly do
>> a put-text-property, but indeed font-lock.el includes some code which tries
>> to do that for you.

> It would do this, presumably, in an after-change (or before-change)
> function, either directly on one of these hooks or called from
> font-lock-after-change.

No, it's done in font-lock-default-fontify-region.

> OK.  So the font-lock-multiline property is put on exactly a region of
> text which needs fontifying atomically, yet straddles a line break.  An
> after-change function (and/or a before-change function) is what will do
> this putting and erasing of f-l-multiline.

No, it's done in font-lock-default-fontify-region.

> I think it's more accurate just to say you want to remove the
> extend-region stuff from f-l-after-change.  I want it to stay.  :-)
> Why don't we call the thing we need in f-l-d-f-r
> "font-lock-extend-CHUNK-function", since it's more likely to be a
> jit-lock chunk than a region supplied by the user or major mode?

Since it's a hook run from font-lock-default-fontify-region, it makes sense
to call it "extend-region", don't you think?  Naming (and documenting)
something based on how it is used rather than what it does is usually
a bad idea.

> What is your objection to f-l-extend-region-f in a-c-f?  Is it because it
> might gobble too much processor time?

That and the fact that it's not necessary, and that to use it (as seen in
your example) you need a good bit more work/code than doing it "the other
way" (with code solely run from font-lock-default-fontify-region).

> It needs to be in j-l-fontify-now, so that that function knows what bytes
> to apply the 'fontified property to.

This is not needed for correctness.  It's only a matter of avoiding
redundant work.  This need wouldn't be new (it has existed since Emacs-21
becaue of font-lock-multiline) and doesn't need to be addressed right away.

>> Yes it is important in general.  But this discussion is about the need to
>> have *any* a-c-f hook.  If we need one, its place is in f-l-a-c-f, indeed,
>> but I argue that we don't need one, so the precise place is not really
>> relevant to the discussion.

> I can't see how a major mode can manipulate f-l-multiline properties
> anywhere BUT an a-c-f (or a before-c-f).

What about the sample solution I provided: it's all manipulated from
font-lock-keywords (i.e. from font-lock-default-fontify-region)?

>> Look ma!  No old-len!  Tadaaa!
> .... but some code, somewhere, sometime needs to adjust the f-l-m
> property, removing it from L4.  In general, it will also need to _split_
> the text property into L1,2,3 and L4,5,...  Otherwise, a f-l-m region
> could gradually engulf an entire buffer, making a nonsense of jit-lock.
> (OK, this would be an extreme situation.  :-)

Yes, of course.  It's done this way: font-lock-unfontify-region (called from
font-lock-default-fontify-region) removes the font-lock-mutliline property
on the whole fontified region jsut before proceeding with the actual
fontification, so the font-lock-keywords need to re-add it at those places
where it's still needed.

>> If you insert text (instead of removing it), it gets a bit more
>> interesting (because the inserted text doesn't have a
>> font-lock-multiline property) but it basically works along the same
>> lines.

> Something, somewhere, sometime has to analyse the newly inserted code,
> putting f-l-m properties throughout it, I think, and adjusting those at
> the boundaries of the new region.  Is this to be done at the time of the
> change or at the time it gets font locked?  I think it has to be done at
> the change, otherwise the f-l-m settings will be wrong when it comes time
> to fontify a chunk.

Let me extend my example.  In the last episode, we stopped just before
refontification was to take place and had seen that the f-l-m property was
going to make the refontification apply to all 4 lines, as needed.
The fontification will work as follows:

5. font-lock-default-fontify-region gets asked to refontify line 3.

6. it extends the refontified region at beg and at end because of the
   presence of f-l-m property.  So the refontified region now spans all
   4 lines.

7. Prior to applying the new fontification, font-lock-unfontify-region is
   called which removes all `face' properties and `font-lock-multiline'
   properties on all 4 lines.

8. the actual refontification takes place, using font-lock-keywords.
   During this step, the rule

     (".*\\\\\n.*"
      (0 (progn (put-text-property (match-beginning 0) (match-end 0)
                                   'font-lock-multiline t)
                nil)))

   is used to re-add a font-lock-multiline property to the first 3 lines
   (the 4th is not part of the multiline-line any more).

   If we had remoed the \ on line 2 instead, the refontification would have
   added the font-lock-multiline to lines 1&2 and then 2&3 (the two
   separated by a \n which is does not have this property).

See, it does analyse the new text to adjust the font-lock-multiline property.

> 1. I think font-lock-extend-region-function, called as it is from
> j-l-after-change, is a good thing.  It is a correct way of giving a major
> mode an opportunity to massage a fontification region.  I think it should
> stay.  You think it should be removed.

Yup.

> 2. f-l-d-f-r / j-l-f-now absolutely need a
> "font-lock-extend-chunk-function" functionality (regardless of its
> precise name).  We agree on this. 

Yup.

> 3. You have convinced me that it is possible to use the f-l-m properties.

Good.

> 4. I think that f-l-extend-region-f, called from f-l-after-change, is
> likely to be more convenient and less error prone than the direct
> manipulation of the f-l-multiline property by major mode code.

You mean adding a single put-text-property call in your font-lock-keywords
rule is more error prone than cooking up a b-c-f plus an a-c-f plus some way
to store info between the two so that the a-c-f can figure out what
happened, ...?

> 5. I suspect you think that f-l-extend-region-f is likely to consume more
> processor time than direct manipulation of the f-l-m property.

The problemis not what is done but when.  after-change-fucntions get run
more often than font-lock-fontify-region.

> Is there any reason why we can't leave both methods side by side in
> the code?

Unnecessary complexity.  And it will mislead people into thinking that
using a a-c-f is a good way to solve their problem.


        Stefan

RE: Last steps for pretesting (font-lock-extend-region-function)

[Drew Adams]

Do we get a summary, once you have reached fixed point? Thx.    

Re: Last steps for pretesting

[Richard Stallman]

    Richard, when do you envisage going into pretest?

It's a matter of fixing the pending bugs.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan.

On Fri, 21 Apr 2006, Stefan Monnier wrote:

[ .... ]

>> What is your objection to f-l-extend-region-f in a-c-f?  Is it because it
>> might gobble too much processor time?

>That and the fact that it's not necessary, and that to use it (as seen
>in your example) you need a good bit more work/code than doing it "the
>other way" (with code solely run from font-lock-default-fontify-region).

It will need an order of magnitude less work for f-l-extend-region-f than
for f-l-multiline properties.  At least, it will for those hackers who
aren't totally familiar with the internals of font-lock-keywords, etc.

f-l-e-r-f only needs to be applied at the extremes of the region.  f-l-m
needs to be applied throughout the entire region.  I doubt very much
whether f-l-m would use less processing power than f-l-e-r-f.  Also,
f-l-extend-region-function (called from f-l-a-c-f/j-l-a-c) will work
EVERY time, because of its directness and simplicity.  Using only f-l-m
will fail in some circumstances.

In the most general situation, a regexp in f-l-keywords will be
inadequate - a defun will be needed.  This will be at least as
complicated as the one for f-l-extend-region-f.

Writing a f-l-extend-region-f does not require any detailed knowledge of
font-lock-keywords - thus the two problems of determining the region to
fontify and of specifying how to fontify it are kept separate, hence
simplified.  Abusing f-l-keywords to set exotic text properties (as
against its primary purpose, to set faces) requires contorted thinking.
To write a f-l-e-r-f, one need only read a single page of the elisp
manual.  

It's me that's going to have to implement this mechanism for CC Mode (all
modes other than AWK Mode).  It's going to be difficult enough as it is,
without the added complications of the difficult to understand structure
font-lock-keywords.

>> It needs to be in j-l-fontify-now, so that that function knows what
>> bytes to apply the 'fontified property to.

>This is not needed for correctness.  It's only a matter of avoiding
>redundant work.  This need wouldn't be new (it has existed since
>Emacs-21 becaue of font-lock-multiline) and doesn't need to be addressed
>right away.

It might cause errors, and it will certainly increase run time.  It isn't
much effort to be correct, here.  Why not do it right?  (Yes, I'm
prepared to do this patch, too.)

[ .... ]

>> I can't see how a major mode can manipulate f-l-multiline properties
>> anywhere BUT an a-c-f (or a before-c-f).

>What about the sample solution I provided: it's all manipulated from
>font-lock-keywords (i.e. from font-lock-default-fontify-region)?

It won't work in every case.  An after-change function (at the very
least) will need to be added to your solution to make it robust.

>>> Look ma!  No old-len!  Tadaaa!
>> .... but some code, somewhere, sometime needs to adjust the f-l-m
>> property, removing it from L4.  In general, it will also need to
>> _split_ the text property into L1,2,3 and L4,5,...  Otherwise, a f-l-m
>> region could gradually engulf an entire buffer, making a nonsense of
>> jit-lock.  (OK, this would be an extreme situation.  :-)

>Yes, of course.  It's done this way: font-lock-unfontify-region (called
>from font-lock-default-fontify-region) removes the font-lock-mutliline
>property on the whole fontified region just before proceeding with the
>actual fontification, so the font-lock-keywords need to re-add it at
>those places where it's still needed.

OK.

[ .... ]

>> Something, somewhere, sometime has to analyse the newly inserted code,
>> putting f-l-m properties throughout it, I think, and adjusting those
>> at the boundaries of the new region.  Is this to be done at the time
>> of the change or at the time it gets font locked?  I think it has to
>> be done at the change, otherwise the f-l-m settings will be wrong when
>> it comes time to fontify a chunk.

>Let me extend my example.  In the last episode, we stopped just before
>refontification was to take place and had seen that the f-l-m property was
>going to make the refontification apply to all 4 lines, as needed.
>The fontification will work as follows:

>5. font-lock-default-fontify-region gets asked to refontify line 3.

>6. it extends the refontified region at beg and at end because of the
>   presence of f-l-m property.  So the refontified region now spans all
>   4 lines.

>7. Prior to applying the new fontification, font-lock-unfontify-region is
>   called which removes all `face' properties and `font-lock-multiline'
>   properties on all 4 lines.

>8. the actual refontification takes place, using font-lock-keywords.
>   During this step, the rule

>     (".*\\\\\n.*"
>      (0 (progn (put-text-property (match-beginning 0) (match-end 0)
>                                   'font-lock-multiline t)
>                nil)))

>   is used to re-add a font-lock-multiline property to the first 3 lines
>   (the 4th is not part of the multiline-line any more).

>   If we had remoed the \ on line 2 instead, the refontification would have
>   added the font-lock-multiline to lines 1&2 and then 2&3 (the two
>   separated by a \n which is does not have this property).

>See, it does analyse the new text to adjust the font-lock-multiline property.

It sounds good in theory, but hasn't yet been tried in practice, AFAIK.
Or has it?  For which major mode?  I really don't want to have to be the
first person to try your approach.  I don't like it and don't think it
will work very well.

>> 1. I think font-lock-extend-region-function, called as it is from
>> j-l-after-change, is a good thing.  It is a correct way of giving a
>> major mode an opportunity to massage a fontification region.  I think
>> it should stay.  You think it should be removed.

>Yup.

>> 2. f-l-d-f-r / j-l-f-now absolutely need a
>> "font-lock-extend-chunk-function" functionality (regardless of its
>> precise name).  We agree on this. 

>Yup.

>> 3. You have convinced me that it is possible to use the f-l-m
>> properties.

>Good.

>> 4. I think that f-l-extend-region-f, called from f-l-after-change, is
>> likely to be more convenient and less error prone than the direct
>> manipulation of the f-l-multiline property by major mode code.

>You mean adding a single put-text-property call in your
>font-lock-keywords rule is more error prone than cooking up a b-c-f plus
>an a-c-f plus some way to store info between the two so that the a-c-f
>can figure out what happened, ...?

Yes, absolutely.  There are so many special cases, font-lock-keywords has
a complicated structure, it would be very easy to put the property on
".*\\\\$" rather than ".*\\\\\n", for example. 

>> 5. I suspect you think that f-l-extend-region-f is likely to consume
>> more processor time than direct manipulation of the f-l-m property.

>The problem is not what is done but when.  after-change-functions get run
>more often than font-lock-fontify-region.

No it doesn't.  font-lock-fontify-region gets run at every change (unless
jit-lock-defer-time is non-nil - it's nil by default).  f-l-f-r gets run
additionally for background fontification.

>> Is there any reason why we can't leave both methods side by side in
>> the code?

>Unnecessary complexity.  And it will mislead people into thinking that
>using a a-c-f is a good way to solve their problem.

after-change-functions is the canonical way of solving problems related
to buffer changes.  This includes Font Locking.

I say to you again - your solution is not robust.  I don't think it's
been tried at all (correct me if I'm wrong).  A variant form of
f-l-extend-region-f (the advice on the f-l a-c-functions, as implemented
for AWK Mode) has been in use since June 2003, and so far no bugs related
to it have been reported by users.  We are both aware of a refinement
which is needed, namely calling some sort of f-l-extend-region-f from
f-l-d-f-r and j-l-f-n.

So - please leave the better tested mechanism in place.

>        Stefan

-- 
Alan.

P.S.  We're boring everybody else as well as ourselves.  :-(

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>>> What is your objection to f-l-extend-region-f in a-c-f?  Is it because it
>>> might gobble too much processor time?

>> That and the fact that it's not necessary, and that to use it (as seen
>> in your example) you need a good bit more work/code than doing it "the
>> other way" (with code solely run from font-lock-default-fontify-region).

> It will need an order of magnitude less work for f-l-extend-region-f than
> for f-l-multiline properties.

Are you talking about CPU-work or programmer work?
I was talking about programmer work, and I've shown you the code for
font-lock-multiline: it's quite a bit shorter and more straightforward than
your hack using b-c-f and a-c-f: just match the multiline element, place
a font-lock-multiline property on it, and you're set.

> At least, it will for those hackers who aren't totally familiar with the
> internals of font-lock-keywords, etc.

You don't have to be familiar with the internals.  At least not more than
necessary to write the proper foo-before-change-function plus
foo-font-lock-lock-extend-region-after-change.

> f-l-e-r-f only needs to be applied at the extremes of the region.  f-l-m
> needs to be applied throughout the entire region.  I doubt very much
> whether f-l-m would use less processing power than f-l-e-r-f.

As I said, the important aspect is not how much time it takes each time it's
executed, but how many times it's executed.  In an after-change-function, it
risks being executed enough times to beomce visible to the user.

> Also, f-l-extend-region-function (called from f-l-a-c-f/j-l-a-c) will work
> EVERY time, because of its directness and simplicity.

Reality check?  Here's your code:

   (defvar c-awk-old-EOLL 0)
   (make-variable-buffer-local 'c-awk-old-EOLL)
   ;; End of logical line following the region which is about to be changed.
   ;; Set in c-awk-before-change and used in c-awk-font-lock-extend-region.
   
   (defun c-awk-before-change (beg end)
   ;; This function is called exclusively from the before-change-functions hook.
   ;; It does two things: Finds the end of the (logical) line on which END lies,
   ;; and clears c-awk-NL-prop text properties from this point onwards.
     (save-restriction
       (save-excursion
         (setq c-awk-old-EOLL (c-awk-end-of-logical-line end))
         (c-save-buffer-state nil
          (c-awk-clear-NL-props end (point-max))))))
   (add-hook 'before-change-functions c-awk-before-change nil t)
   
   (defun c-awk-end-of-change-region (beg end old-len)
     ;; Find the end of the region which needs to be font-locked after a change.
     ;; This is the end of the logical line on which the change happened, either
     ;; as it was before the change, or as it is now, whichever is later.
     ;; N.B. point is left undefined.
     (max (+ (- c-awk-old-EOLL old-len) (- end beg))
          (c-awk-end-of-logical-line end)))

Here's my alternative implementation:

   (defconst c-awk-font-lock-keywords
      '(...
        (".*\\\\\n.*"
         (0 (progn (put-text-property (match-beginning 0) (match-end 0)
                                      'font-lock-multiline t)
                   nil)))
        ...))

Now which one's more direct and simple, really?

> Using only f-l-m will fail in some circumstances.

Please backup such claims.

> In the most general situation, a regexp in f-l-keywords will be
> inadequate - a defun will be needed.  This will be at least as
> complicated as the one for f-l-extend-region-f.

So you mean in the most general case, it'll be no worse than your code.
Here, we agree ;-)

> Writing a f-l-extend-region-f does not require any detailed knowledge of
> font-lock-keywords - thus the two problems of determining the region to
> fontify and of specifying how to fontify it are kept separate, hence
> simplified.  Abusing f-l-keywords to set exotic text properties (as
> against its primary purpose, to set faces) requires contorted thinking.
> To write a f-l-e-r-f, one need only read a single page of the elisp
> manual.

And then one needs to think hard about how to actually write the code, what
are the relevant odd cases one has to worry about, where to store the info
you need from b-c-f to a-c-f, ... so much fun, indeed.

Before/after-change-functions are great hammers, but you know what they
say about hammers.

> It's me that's going to have to implement this mechanism for CC Mode (all
> modes other than AWK Mode).  It's going to be difficult enough as it is,
> without the added complications of the difficult to understand the structure
> font-lock-keywords.

Just take a deep breath, a couple of steps back, and try to look at the
example use of font-lock-multiline above as if it were very simple.

>>> It needs to be in j-l-fontify-now, so that that function knows what
>>> bytes to apply the 'fontified property to.

>> This is not needed for correctness.  It's only a matter of avoiding
>> redundant work.  This need wouldn't be new (it has existed since
>> Emacs-21 becaue of font-lock-multiline) and doesn't need to be addressed
>> right away.
> It might cause errors,

Again, please backup such claims.

> and it will certainly increase run time.

In some cases it will, yes.  Those cases tend to suffer from other
performance problems anyway, tho (i.e. they typically involve large regions
that need to be rehighlighted atomically, so even if you fix this
jit-lock-fontify-now performance misfeature the underlying performance
issue will still bite you and you'll end up having to use a different
solution for which the jit-lock-fontify-now problem won't apply either).
At least this was my experience with smerge-mode and it didn't seem
particularly specific to my situation.

> It isn't much effort to be correct, here.  Why not do it right?  (Yes, I'm
> prepared to do this patch, too.)

It's enough changes that I don't think it should go into Emacs-22.  I prefer
keeping the same performance misfeature that we already had in Emacs-21 and
release Emacs-22 a few days earlier.

But if you want to fix it, here's how a good fix should work IMNSHO:
- every function that get added to jit-lock-functions (such as
  font-lock-fontify-region) should be changed to return the region
  it actually jit'd.
- jit-lock-fontify-now should then take the intersection of the returned
  regions to determine where to set the `fontified' property.

>> What about the sample solution I provided: it's all manipulated from
>> font-lock-keywords (i.e. from font-lock-default-fontify-region)?

> It won't work in every case.  An after-change function (at the very
> least) will need to be added to your solution to make it robust.

Again, I see no evidence of this.  Back up your claim.

> Yes, absolutely.  There are so many special cases, font-lock-keywords has
> a complicated structure, it would be very easy to put the property on
> ".*\\\\$" rather than ".*\\\\\n", for example.

The only really important part where the font-lock-multiline property should
be added is the text that covers all the "backslash newlines" of
a multiline line.  Whether it includes the last ".*$" or the leading "^.*"
doesn't really matter.
It's not nearly as brittle as you think.

>>> 5. I suspect you think that f-l-extend-region-f is likely to consume
>>> more processor time than direct manipulation of the f-l-m property.

>> The problem is not what is done but when.  after-change-functions get run
>> more often than font-lock-fontify-region.

> No it doesn't.  font-lock-fontify-region gets run at every change (unless
> jit-lock-defer-time is non-nil - it's nil by default).

No, it's run at every redisplay.  For self-insert-command, it's usually the
same as "every change", but for many other commands it can be quite
different (or even for self-insert-command, if you trigger an abbrev
expansion or if you trigger auto-fill, ...).

"at every redisplay" basically means "at human pace", whereas "after every
change" means "at CPU pace".

> I say to you again - your solution is not robust.  I don't think it's
> been tried at all (correct me if I'm wrong).  A variant form of
> f-l-extend-region-f (the advice on the f-l a-c-functions, as implemented
> for AWK Mode) has been in use since June 2003, and so far no bugs related
> to it have been reported by users.

Without using a hook in font-lock-default-fontify-region you can't claim
it's robust: I can easily cook up a case where it fails.

> We are both aware of a refinement which is needed, namely calling some
> sort of f-l-extend-region-f from f-l-d-f-r.

It's not a refinement.  It's a requirement for correctness.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> I say to you again - your solution is not robust.  I don't think it's
> been tried at all (correct me if I'm wrong).

Of course it has, in smerge-mode (since Emacs-21.1) which I use daily, in
cvs-status (since Emacs-21.1) in perl-mode (since some time in late 2004
IIRC), in sh-script.el (a bit before perl-mode IIRC), in message.el (where
it's been buggy at times because I didn't understand that I needed
a font-lock-default-fontify-region hook).

They do all suffer from the missing font-lock-default-fontify-region hook,
just like your code, of course.

> So - please leave the better tested mechanism in place.

I don't claim it's buggy or even more buggy than font-lock-multiline.
It's just an ugly kludge, much more heavyweight codewise (this is
objectively the case in terms of lines of code, and I hope my comparison
above convinces a few more people that it's also subjectively the case).


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> I say to you again - your solution is not robust.  I don't think it's
> been tried at all (correct me if I'm wrong).  A variant form of
> f-l-extend-region-f (the advice on the f-l a-c-functions, as implemented
> for AWK Mode) has been in use since June 2003, and so far no bugs related
> to it have been reported by users.  We are both aware of a refinement
> which is needed, namely calling some sort of f-l-extend-region-f from
> f-l-d-f-r and j-l-f-n.

> So - please leave the better tested mechanism in place.

I feel like Alice in wonderland: you talk as if I wanted to remove
a well-established hook in favor of some new fangled feature, whereas the
font-lock-multiline property is the one that's been in Emacs since 21.1,
whereas your hook is the one that's new and that you want to add to
Emacs-22.

I think in order to justify this new hook you need to show that the
pre-existing font-lock-multiline solution is not good enough in some cases.
I know you have a distaste for it, but I find it to be a far cry from
showing that it's not good enough.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Tomas Zerolo]

[-- Attachment #1.1: Type: text/plain, Size: 328 bytes --]

On Mon, Apr 24, 2006 at 07:28:44PM +0000, Alan Mackenzie wrote:

[on font lock]

> P.S.  We're boring everybody else as well as ourselves.  :-(

No way. I don't know about yourselves, but I'm not bored. I'm following
and learning a lot from the discusion, even if I'm not understanding
everything.

Thanks
-- tomás

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Last steps for pretesting (font-lock-extend-region-function)

[Ralf Angeli]

* Stefan Monnier (2006-04-24) writes:

> I was talking about programmer work, and I've shown you the code for
> font-lock-multiline: it's quite a bit shorter and more straightforward than
> your hack using b-c-f and a-c-f: just match the multiline element, place
> a font-lock-multiline property on it, and you're set.

Do you even have to bother with placing font-lock-multiline properties
manually?  In AUCTeX we simply set the font-lock-multiline variable to
t and font-lock added the property automatically.

-- 
Ralf

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Mon, 24 Apr 2006, Stefan Monnier wrote:

>> I say to you again - your solution is not robust.  I don't think it's
>> been tried at all (correct me if I'm wrong).  A variant form of
>> f-l-extend-region-f (the advice on the f-l a-c-functions, as implemented
>> for AWK Mode) has been in use since June 2003, and so far no bugs related
>> to it have been reported by users.  We are both aware of a refinement
>> which is needed, namely calling some sort of f-l-extend-region-f from
>> f-l-d-f-r and j-l-f-n.

>> So - please leave the better tested mechanism in place.

>I feel like Alice in wonderland:

So do I.  :-)  I'm still not 100% sure that we're arguing about exactly
the same thing.

>you talk as if I wanted to remove a well-established hook in favor of
>some new fangled feature, whereas the font-lock-multiline property is
>the one that's been in Emacs since 21.1, whereas your hook is the one
>that's new and that you want to add to Emacs-22.

It has already been added.  I'm desperately trying to stop you from
taking it away again, because that would make my (Emacs) life much more
difficult.

>I think in order to justify this new hook you need to show that the
>pre-existing font-lock-multiline solution is not good enough in some
>cases.

I don't.  I merely need to point out that Richard specifically asked for
this hook to be installed.  When he did so, he was aware that it was
called from an after-change function.

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Mon, 24 Apr 2006, Stefan Monnier wrote:

>> I say to you again - your solution is not robust.  I don't think it's
>> been tried at all (correct me if I'm wrong).

>Of course it has, in smerge-mode (since Emacs-21.1) which I use daily, in
>cvs-status (since Emacs-21.1) in perl-mode (since some time in late 2004
>IIRC), in sh-script.el (a bit before perl-mode IIRC), in message.el (where
>it's been buggy at times because I didn't understand that I needed
>a font-lock-default-fontify-region hook).

>They do all suffer from the missing font-lock-default-fontify-region
>hook, just like your code, of course.

>> So - please leave the better tested mechanism in place.

>I don't claim it's buggy or even more buggy than font-lock-multiline.
>It's just an ugly kludge, .....

I can't understand at all why you feel that.  I really can't.  To me,
f-l-e-r-f seems natural, simple and obvious.  Optionally setting a hook
variable to a function is done countless times in Emacs.  I can't see why
font-lock-extend-region-function is any more ugly than, say,
adaptive-fill-function.

>....much more heavyweight codewise (this is objectively the case in
>terms of lines of code, and I hope my comparison above convinces a few
>more people that it's also subjectively the case).

This is not the case.  We're agreed that, for the f-l-multiline mechanism
to work, it needs supplementing by a f-l-extend-region-function hook.  If
we need this hook anyway, why complicate it with f-l-multiline when the
hook works just fine on its own?

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Ralf!

On Tue, 25 Apr 2006, Ralf Angeli wrote:

>* Stefan Monnier (2006-04-24) writes:

>> I was talking about programmer work, and I've shown you the code for
>> font-lock-multiline: it's quite a bit shorter and more straightforward than
>> your hack using b-c-f and a-c-f: just match the multiline element, place
>> a font-lock-multiline property on it, and you're set.

>Do you even have to bother with placing font-lock-multiline properties
>manually?  In AUCTeX we simply set the font-lock-multiline variable to
>t and font-lock added the property automatically.

Yes, absolutely!  There are two separate things here which Stefan is
failing to distinguish:

(i) The matching of keywords which span line breaks;

(ii) Extending a font-lock region to include all text needing
refontification.

If I've understood font-lock-multiline properly, only manually setting
f-l-m can achieve purpose (ii).

>Ralf

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Mon, 24 Apr 2006, Stefan Monnier wrote:

>>>> What is your objection to f-l-extend-region-f in a-c-f?  Is it because it
>>>> might gobble too much processor time?

>>> That and the fact that it's not necessary, and that to use it (as seen
>>> in your example) you need a good bit more work/code than doing it "the
>>> other way" (with code solely run from font-lock-default-fontify-region).

>> It will need an order of magnitude less work for f-l-extend-region-f than
>> for f-l-multiline properties.

>Are you talking about CPU-work or programmer work?

Programmer work.  I can state that with 100% certainty when that
programmer is me.  ;-(

>I was talking about programmer work, and I've shown you the code for
>font-lock-multiline: it's quite a bit shorter and more straightforward
>than your hack using b-c-f and a-c-f: just match the multiline element,
>place a font-lock-multiline property on it, and you're set.

"just", you write.  ;-)  The complexity of font-lock-keywords is an order
of magnitude higher than that of a defun, or a short sequence of related
defuns.  There's the complexity of where to quote, where to omit quotes,
wondering whether to put the "region" regexps before or after the face
setting regexps, whether to make f-l-m start-sticky or stop-sticky.... 

I find the elisp manual page which describes font-lock-keywords
("Search-based Fontification") all but inpenetrable.  It's reduced me to
tears on several occasions.  I've just spent five minutes looking through
it, trying to find the description of the form you suggested to me:

     (".*\\\\\n.*"
      (0 (progn (put-text-property (match-beginning 0) (match-end 0)
                                   'font-lock-multiline t)
                nil)))
 
.  I didn't find it.

Consider the difficulty of testing.  With the approach I've used, it's a
simple matter of C-u C-M-x, and edebugging through the function with M-:.
With your f-l-m approach, you've got to delve into the bowels of
font-lock, find the appropriate function (far from trivial: it's
f-l-default-f-r, I think), instrument it, f and blind as recursive levels
of jit lock build up on your screen, turn off jit-lock (a far from
trivial operation when you've not done it before - it's not documented),
start the whole process again.

>> At least, it will for those hackers who aren't totally familiar with
>> the internals of font-lock-keywords, etc.

>You don't have to be familiar with the internals.  At least not more
>than necessary to write the proper foo-before-change-function plus
>foo-font-lock-lock-extend-region-after-change.

With respect, Stefan, that's rubbish.  You need to know where and how to
write the above Lisp form (the one starting (".*\\\\n.*" ...)) into
f-l-keywords.  For me, that's perhaps half an hour to an hour of
frustration trying to find the right place in "Search-based
Fontification".

>> f-l-e-r-f only needs to be applied at the extremes of the region.
>> f-l-m needs to be applied throughout the entire region.  I doubt very
>> much whether f-l-m would use less processing power than f-l-e-r-f.

>As I said, the important aspect is not how much time it takes each time
>it's executed, but how many times it's executed.  In an
>after-change-function, it risks being executed enough times to become
>visible to the user.

[Hey, "beomce" was a difficult typo to correct ;-]

What about the run-time overhead of having an extra text property over
the entire buffer?  The f-l-e-r-f, as you envisage it (being called only
from f-l-default-fontify-r) will also be called at virtually every buffer
change, since almost every change causes a redisplay.

>> Also, f-l-extend-region-function (called from f-l-a-c-f/j-l-a-c) will
>> work EVERY time, because of its directness and simplicity.

>Reality check?  Here's your code:

>   (defvar c-awk-old-EOLL 0)
>   (make-variable-buffer-local 'c-awk-old-EOLL)
>   ;; End of logical line following the region which is about to be changed.
>   ;; Set in c-awk-before-change and used in c-awk-font-lock-extend-region.
   
>   (defun c-awk-before-change (beg end)
>   ;; This function is called exclusively from the before-change-functions hook.
>   ;; It does two things: Finds the end of the (logical) line on which END lies,
>   ;; and clears c-awk-NL-prop text properties from this point onwards.
>     (save-restriction
>       (save-excursion
>         (setq c-awk-old-EOLL (c-awk-end-of-logical-line end))
>         (c-save-buffer-state nil
>          (c-awk-clear-NL-props end (point-max))))))
>   (add-hook 'before-change-functions c-awk-before-change nil t)
   
>   (defun c-awk-end-of-change-region (beg end old-len)
>     ;; Find the end of the region which needs to be font-locked after a change.
>     ;; This is the end of the logical line on which the change happened, either
>     ;; as it was before the change, or as it is now, whichever is later.
>     ;; N.B. point is left undefined.
>     (max (+ (- c-awk-old-EOLL old-len) (- end beg))
>          (c-awk-end-of-logical-line end)))

>Here's my alternative implementation:

>   (defconst c-awk-font-lock-keywords
>      '(...
>        (".*\\\\\n.*"
>         (0 (progn (put-text-property (match-beginning 0) (match-end 0)
>                                      'font-lock-multiline t)
>                   nil)))
>        ...))

>Now which one's more direct and simple, really?

The one which is adequately commented.  The one which doesn't use an
obscure list structure that you need to look up in a manual.  The one
with thin, obvious interfaces with the rest of Emacs.  The one which is
trivial to debug with edebug.  The one which doesn't conflate determining
the font-lock region with actually fontifying it.  The one which runs
faster on a large region.

[ .... ]

>> It's me that's going to have to implement this mechanism for CC Mode
>> (all modes other than AWK Mode).  It's going to be difficult enough as
>> it is, without the added complications of the difficult to understand
>> the structure font-lock-keywords.

>Just take a deep breath, a couple of steps back, and try to look at the
>example use of font-lock-multiline above as if it were very simple.

I've tried, but I just don't have the intellectual capability to cope
with such complexity without the greatest of difficulty.  You obviously
do.  Complicated nested structures make my eyes glaze over.  There's a
good chance I'm not the only Emacs hacker who has this sort of trouble.

There's an assymetry between our positions: by threatening to remove the
f-l-e-r-f hook (after-change version), you're trying to force me to use
the f-l-m mechanism.  You're trying to impose your taste and judgement on
me, and I resent this quite a lot.  I think the f-l-m text property is a
revolting kludge, but I'm not trying to do away with it.  I'm happy
enough for other hackers to use it, though I'd rather not have to
maintain their code afterwards.

Is there any chance we could bring this discussion to an end, now?  The
things we've been talking about are important, but this discussion has
taken up an enormous amount of my time, and probably yours too.  We've
both got other urgent things to do.

[ .... ]

>> It isn't much effort to be correct, here.  Why not do it right?  (Yes,
>> I'm prepared to do this patch, too.)

>It's enough changes that I don't think it should go into Emacs-22.  I
>prefer keeping the same performance misfeature that we already had in
>Emacs-21 and release Emacs-22 a few days earlier.

>But if you want to fix it, here's how a good fix should work IMNSHO:
>- every function that get added to jit-lock-functions (such as
>  font-lock-fontify-region) should be changed to return the region
>  it actually jit'd.
>- jit-lock-fontify-now should then take the intersection of the returned
>  regions to determine where to set the `fontified' property.

My fix is simpler: to recognise that f-l-default-fontify-region performs
two disparate tasks: (i) Extending the region to be fontified; (ii)
Fontifying this region.  By extracting (ii) into a separate function,
that function could be called directly from jit-lock-fontify-now,
bypassing the redundant second massaging of the region.

[ .... ]

>> We are both aware of a refinement which is needed, namely calling some
>> sort of f-l-extend-region-f from f-l-d-f-r.

>It's not a refinement.  It's a requirement for correctness.

It is, and we need it.  It needs a name - here are the possibilities:

(i) It uses the same hook, font-lock-extend-region-function, as
f-l-after-change-functions, setting the third parameter, OLD-LEN, to nil;

(ii) It uses a distinct hook with a new name, say,
font-lock-extend-chunk-function;

(iii) It usurps the name f-l-e-r-f from its current use, abolishing the
region extension mechanism in the f-l-after-change-f/j-l-after-change.

I'd prefer (i), but would have no trouble with (ii).  I absolutely do NOT
want (iii).

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> On Mon, 24 Apr 2006, Stefan Monnier wrote:

[...]

> I find the elisp manual page which describes font-lock-keywords
> ("Search-based Fontification") all but inpenetrable.  It's reduced me to
> tears on several occasions.  I've just spent five minutes looking through
> it, trying to find the description of the form you suggested to me:
>
>      (".*\\\\\n.*"
>       (0 (progn (put-text-property (match-beginning 0) (match-end 0)
>                                    'font-lock-multiline t)
>                 nil)))
>  
> .  I didn't find it.
>

[...]

>>You don't have to be familiar with the internals.  At least not more
>>than necessary to write the proper foo-before-change-function plus
>>foo-font-lock-lock-extend-region-after-change.
>
> With respect, Stefan, that's rubbish.  You need to know where and how to
> write the above Lisp form (the one starting (".*\\\\n.*" ...)) into
> f-l-keywords.  For me, that's perhaps half an hour to an hour of
> frustration trying to find the right place in "Search-based
> Fontification".

[...]

>>Here's my alternative implementation:
>
>>   (defconst c-awk-font-lock-keywords
>>      '(...
>>        (".*\\\\\n.*"
>>         (0 (progn (put-text-property (match-beginning 0) (match-end 0)
>>                                      'font-lock-multiline t)
>>                   nil)))
>>        ...))
>
>>Now which one's more direct and simple, really?
>
> The one which is adequately commented.

Not relevant.

> The one which doesn't use an obscure list structure that you need to
> look up in a manual.  The one with thin, obvious interfaces with the
> rest of Emacs.

It appears to me like you are uncomfortable around font lock
expressions.  And I doubt that anybody having worked with font lock
entries and syntax table entries has not teared out his hairs in
frustration several times round.

But I don't think that the solution is trying to move everything
somewhere else.  It would rather be sensible to create appropriate
accessor functions that save the programmer from having to deal with
the internal list structures.

We do similar things with the "event" data structure by having posn-*
accessor functions.  font-lock should be made more programmer-friendly
instead of delegating stuff to various ad-hoc hooks.

> I've tried, but I just don't have the intellectual capability to
> cope with such complexity without the greatest of difficulty.  You
> obviously do.  Complicated nested structures make my eyes glaze
> over.  There's a good chance I'm not the only Emacs hacker who has
> this sort of trouble.

Then we need to create functions or macros that build and access the
data structures.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> you talk as if I wanted to remove a well-established hook in favor of
>> some new fangled feature, whereas the font-lock-multiline property is
>> the one that's been in Emacs since 21.1, whereas your hook is the one
>> that's new and that you want to add to Emacs-22.

> It has already been added.

It hasn't been added in released code yet.

> I'm desperately trying to stop you from taking it away again, because that
> would make my (Emacs) life much more difficult.

Actually no: as I've shown with my sample alternative solution, it would
only force you to make you life simpler.

>> I think in order to justify this new hook you need to show that the
>> pre-existing font-lock-multiline solution is not good enough in some
>> cases.

> I don't.  I merely need to point out that Richard specifically asked for
> this hook to be installed.

No: he agreed to install it after you claimed it was needed.  As a matter of
fact, it was needed in font-lock-default-fontify-region (where you didn't
add it) but not in font-lock-after-change-function.

> When he did so, he was aware that it was called from an
> after-change function.

It seemed clear to him that an "extend-region" hook was a good idea and we
all agree that it is.  But I highly doubt it that he took the time to
understand enough to be able to judge whether placing it in
font-lock-after-change-function was a good choice or not.  


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> I don't claim it's buggy or even more buggy than font-lock-multiline.
>> It's just an ugly kludge, .....

> I can't understand at all why you feel that.  I really can't.  To me,
> f-l-e-r-f seems natural, simple and obvious.  Optionally setting a hook
> variable to a function is done countless times in Emacs.  I can't see why
> font-lock-extend-region-function is any more ugly than, say,
> adaptive-fill-function.

I'm not saying the hook is a kludge: I'm saying the code that uses it is
a kludge, compared to the code that uses font-lock-multiline.

>> ....much more heavyweight codewise (this is objectively the case in
>> terms of lines of code, and I hope my comparison above convinces a few
>> more people that it's also subjectively the case).

> This is not the case.  We're agreed that, for the f-l-multiline mechanism
> to work, it needs supplementing by a f-l-extend-region-function hook.

Yes, it needs such a hook called from font-lock-default-fontify-region.
Just like your current hook that's called from
font-lock-after-change-function.  This is orthogonal.

> If we need this hook anyway, why complicate it with f-l-multiline when the
> hook works just fine on its own?

There are two alternatives when writing your code in cc-mode:
- use a hook in f-l-d-f-r + font-lock-multiline.
  The font-lock-multiline part of the code is a couple lines added to
  font-lock-keywords.
- use a hook in f-l-d-f-r + a hook in f-l-a-c-f + a hook in b-c-f.
  The *-change-function part of the code is a good bit more than
  a couple lines.

In which way does font-lock-multiline complicate "it"?


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> I find the elisp manual page which describes font-lock-keywords
> ("Search-based Fontification") all but inpenetrable.  It's reduced me to
> tears on several occasions.  I've just spent five minutes looking through
> it, trying to find the description of the form you suggested to me:

>      (".*\\\\\n.*"
>       (0 (progn (put-text-property (match-beginning 0) (match-end 0)
>                                    'font-lock-multiline t)
>                 nil)))
 
> .  I didn't find it.

Then the problem isn't font-lock-multiline.  Please don't mix up matters.
This form is the standard form (except you haven't noticed yet that the
FACENAME part of the font-lock-keywords is not a face symbol but an Elisp
expression whose value after exavluation should be a face).  It's:

 (MATCHER HIGHLIGHT ...)

where MATCHER is ".*\\\\\n.*" and HIGHLIGHT is of the form MATCH-HIGHLIGHT
which is of the form (SUBEXP FACENAME [OVERRIDE [LAXMATCH]]) where SUBEXP is
0 and FACENAME is (progn ...).

> With respect, Stefan, that's rubbish.  You need to know where and how to
> write the above Lisp form (the one starting (".*\\\\n.*" ...)) into
> f-l-keywords.  For me, that's perhaps half an hour to an hour of
> frustration trying to find the right place in "Search-based
> Fontification".

If you don't know how font-lock-keywords work, then learn that first before
trying to work on the font-lock support for cc-mode.  That'll save you a lot
more time.

> What about the run-time overhead of having an extra text property over
> the entire buffer?

It's not over the entire buffer.  And I haven't seen any evidence that it's
noticeable (especially compared the number of `face' properties or of the
`fontified' property added to the whole buffer).

> The f-l-e-r-f, as you envisage it (being called only from
> f-l-default-fontify-r) will also be called at virtually every buffer
> change, since almost every change causes a redisplay.

AFAIK, the case where a single change takes place between redisplay is the
case where there is no performance issue: it takes place at the user's
pace anyway.  The important case is when Emacs has more work to do in
response to a user's action, i.e. there's more work to be done until the
next redisplay.

>> Just take a deep breath, a couple of steps back, and try to look at the
>> example use of font-lock-multiline above as if it were very simple.

> I've tried, but I just don't have the intellectual capability to cope
> with such complexity without the greatest of difficulty.

The whole point is that you shouldn't have to.  It's like recursion (tho not
nearly as elegant, I must admit): have faith, don't try to simulate it all
in your head, and then it becomes simple.

> You obviously do.  Complicated nested structures make my eyes glaze over.
> There's a good chance I'm not the only Emacs hacker who has this sort
> of trouble.

Any Emacs hacker that needs to write complex syntax highlighting rules will
have to learn it anyway unless he prefers to circumvent font-lock and roll
his own instead.  Sounds like NIH to me.

> There's an assymetry between our positions: by threatening to remove the
> f-l-e-r-f hook (after-change version), you're trying to force me to use
> the f-l-m mechanism.

Yes.

> You're trying to impose your taste and judgement on me, and I resent this
> quite a lot.

I'm as close as it gets to a font-lock maintainer.  So from where I stand,
you're trying to impose your taste and judgment on me.

> I think the f-l-m text property is a revolting kludge, but
> I'm not trying to do away with it.  I'm happy enough for other hackers to
> use it, though I'd rather not have to maintain their code afterwards.

> Is there any chance we could bring this discussion to an end, now?  The
> things we've been talking about are important, but this discussion has
> taken up an enormous amount of my time, and probably yours too.  We've
> both got other urgent things to do.

3 ways this can end:
- Richard says that you're right.
- You convince me that the a-c-f code is better for some realistic case
  (e.g. yours).
- I remove the hook in a-c-f.

> My fix is simpler: to recognise that f-l-default-fontify-region performs
> two disparate tasks: (i) Extending the region to be fontified; (ii)
> Fontifying this region.  By extracting (ii) into a separate function,
> that function could be called directly from jit-lock-fontify-now,
> bypassing the redundant second massaging of the region.

It has to be in font-lock-default-fontify-buffer since ther'es no guarantee
this is only called from jit-lock-fontify-now.  So you're suggesting to add
a redundant call to the extend-region hook.  Maybe it's simpler but it's
ugly (because of the redundance, and because it adds a dependency between
jit-lock and font-lock).

> (i) It uses the same hook, font-lock-extend-region-function, as
> f-l-after-change-functions, setting the third parameter, OLD-LEN, to nil;

> (ii) It uses a distinct hook with a new name, say,
> font-lock-extend-chunk-function;

> (iii) It usurps the name f-l-e-r-f from its current use, abolishing the
> region extension mechanism in the f-l-after-change-f/j-l-after-change.

(iv) it reuses the name font-lock-extend-region-function and renames the
     a-c-f hook to font-lock-after-change-extend-region-function.

This will happen hopefully today.  At the same time
font-lock-after-change-extend-region-function will be moved from font-core
to font-lock (where it belongs) and its make-variable-buffer-local call will
be removed as well (use make-local-variable manually if you need it).


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Good afternoon, Stefan!

On Tue, 25 Apr 2006, Stefan Monnier wrote:

>>> you talk as if I wanted to remove a well-established hook in favor of
>>> some new fangled feature, whereas the font-lock-multiline property is
>>> the one that's been in Emacs since 21.1, whereas your hook is the one
>>> that's new and that you want to add to Emacs-22.

>> It has already been added.

>It hasn't been added in released code yet.

>> I'm desperately trying to stop you from taking it away again, because
>> that would make my (Emacs) life much more difficult.

>Actually no: as I've shown with my sample alternative solution, it would
>only force you to make you life simpler.

You have shown nothing of the kind.  You're in no better position to
decide what is objectively simpler than I am, since neither of is is
unbiassed in this debate.  I'm in a better position than you to know what
makes MY life simpler.  Please respect that.

>>> I think in order to justify this new hook you need to show that the
>>> pre-existing font-lock-multiline solution is not good enough in some
>>> cases.

>> I don't.  I merely need to point out that Richard specifically asked for
>> this hook to be installed.

>No: he agreed to install it after you claimed it was needed.  As a
>matter of fact, it was needed in font-lock-default-fontify-region (where
>you didn't add it) but not in font-lock-after-change-function.

He requested it.  Subject: Re: [sigra@home.se: C++-mode: Syntax
highlighting: wrong color for function identifier depending on the kind
of whitespace that follows].

RMS, 2005-02-12:
>Here's the change I was talking about.  Would someone please adapt this
>_now_ to the current sources, and install it?  Then please rename
>before-font-lock-after-change-function to
>font-lock-extend-region-function, and rename
>font-lock-run-before-after-change-hook to font-lock-extend-region?

That was a positive request, not an acquiescence.  In fact, a year
earlier in the same thread:

RMS, 2005-03-11:
>I think we should add that hook now.
>It is simple and easy to understand.
>Even if in some cases it is not a good solution,
>it is a good solution part of the time.
>That is enough reason.

It seems clear that Richard totally understood my patch.  However, he is
the best person to say what actually meant.  At the time, you saw its
purpose as being subtlely different from my intention, and we spent some
time at loggerheads because of this.  As a result of that discussion, it
became clear to me that we need a similar hook to adjust the 500-byte
chunks produced by jit.

>> When he did so, he was aware that it was called from an after-change
>> function.

>It seemed clear to him that an "extend-region" hook was a good idea and
>we all agree that it is.  But I highly doubt it that he took the time to
>understand enough to be able to judge whether placing it in
>font-lock-after-change-function was a good choice or not.  

RMS: "It is simple and easy to understand."  I think he can speak for
himself on this matter, and we should let him.

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Tue, 25 Apr 2006, Stefan Monnier wrote:

>> I find the elisp manual page which describes font-lock-keywords
>> ("Search-based Fontification") all but inpenetrable.  It's reduced me to
>> tears on several occasions.  I've just spent five minutes looking through
>> it, trying to find the description of the form you suggested to me:

>>      (".*\\\\\n.*"
>>       (0 (progn (put-text-property (match-beginning 0) (match-end 0)
>>                                    'font-lock-multiline t)
>>                 nil)))

>> .  I didn't find it.

>This form is the standard form (except you haven't noticed yet that the
>FACENAME part of the font-lock-keywords is not a face symbol but an Elisp
>expression whose value after exavluation should be a face).  It's:

> (MATCHER HIGHLIGHT ...)

>where MATCHER is ".*\\\\\n.*" and HIGHLIGHT is of the form MATCH-HIGHLIGHT
>which is of the form (SUBEXP FACENAME [OVERRIDE [LAXMATCH]]) where SUBEXP is
>0 and FACENAME is (progn ...).

OK.  So FACENAME comes back as nil (which presumably means "don't apply a
facename property") and the side effect is applying f-l-multiline if the
buffer line matches the regexp.

>> With respect, Stefan, that's rubbish.  You need to know where and how
>> to write the above Lisp form (the one starting (".*\\\\n.*" ...)) into
>> f-l-keywords.  For me, that's perhaps half an hour to an hour of
>> frustration trying to find the right place in "Search-based
>> Fontification".

>If you don't know how font-lock-keywords work, then learn that first
>before trying to work on the font-lock support for cc-mode.  That'll
>save you a lot more time.

I cannot learn it, any more than I could learn logarithm tables when such
were still in use.  I need to go through the elisp page every single
time.  The same applies to things like c-offsets-alist too, by the way.

>> What about the run-time overhead of having an extra text property over
>> the entire buffer?

>It's not over the entire buffer.

Load a buffer and wait for it to be jit-locked.  The entire buffer will
have been scanned and this text property, in effect, set to t or nil on
every single byte.

>And I haven't seen any evidence that it's noticeable (especially
>compared the number of `face' properties or of the `fontified' property
>added to the whole buffer).

I'm sure it's not.

>> The f-l-e-r-f, as you envisage it (being called only from
>> f-l-default-fontify-r) will also be called at virtually every buffer
>> change, since almost every change causes a redisplay.

>AFAIK, the case where a single change takes place between redisplay is the
>case where there is no performance issue: it takes place at the user's
>pace anyway.  The important case is when Emacs has more work to do in
>response to a user's action, i.e. there's more work to be done until the
>next redisplay.

Is it really that important to responsiveness whether the delay happens
at the redisplay or immediately after the change?  With the after-change
f-l-e-r-f, only the two boundaries of the region get checked, plus any
chunk boundaries when this is bigger than 500 bytes.  With the extra
pattern in f-l-keywords, then entire region gets checked.

An after-change f-l-e-r-f needs to be fast enough, and if it's not, that
might be a good reason for using f-l-m instead.  However, f-l-m is more
expensive in aggregate - it marks _every_ sequence which must be
atomically fontified, just in case that sequence might straddle a chunk
boundary in the future.  Not only that, it erases and recalculates the
property every time it fontifies a region, even when there has been no
buffer change in that region.  After-change f-l-e-r-f only checks for
those region boundaries it needs to know immediately.

[ .... ]

>I'm as close as it gets to a font-lock maintainer.  So from where I
>stand, you're trying to impose your taste and judgment on me.

I'm as close as it gets to a CC Mode maintainer at the moment.  What's
constraining my freedom in that department now?  ;-)

[ .... ]

>> My fix is simpler: to recognise that f-l-default-fontify-region
>> performs two disparate tasks: (i) Extending the region to be
>> fontified; (ii) Fontifying this region.  By extracting (ii) into a
>> separate function, that function could be called directly from
>> jit-lock-fontify-now, bypassing the redundant second massaging of the
>> region.

>It has to be in font-lock-default-fontify-buffer since there's no
>guarantee this is only called from jit-lock-fontify-now.

No.  It has to be _called_ from font-lock-default-fontify-buffer.

>So you're suggesting to add a redundant call to the extend-region hook.
>Maybe it's simpler but it's ugly (because of the redundance, and because
>it adds a dependency between jit-lock and font-lock).

No.  I'm suggesting refactoring the font lock code a little so that each
path through it calls the extend-region hook exactly once.  Once in the
jit-lock "branch" and once in the plain font lock "branch".

>> (i) It uses the same hook, font-lock-extend-region-function, as
>> f-l-after-change-functions, setting the third parameter, OLD-LEN, to nil;

>> (ii) It uses a distinct hook with a new name, say,
>> font-lock-extend-chunk-function;

>> (iii) It usurps the name f-l-e-r-f from its current use, abolishing the
>> region extension mechanism in the f-l-after-change-f/j-l-after-change.

>(iv) it reuses the name font-lock-extend-region-function and renames the
>     a-c-f hook to font-lock-after-change-extend-region-function.

Or that.  It's a good job our terminals are wider than 80 columns
nowadays.  :-)

>This will happen hopefully today.

Hopefully yes!

>At the same time font-lock-after-change-extend-region-function will be
>moved from font-core to font-lock (where it belongs) and its
>make-variable-buffer-local call will be removed as well (use
>make-local-variable manually if you need it).

Surely not - that variable is essentially buffer local, and it makes no
sense for it not to be so.  To remove the make-variable-buffer-local call
could lead to the same problems which have bedevilled paragraph-start and
paragraph-separate.

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> OK.  So FACENAME comes back as nil (which presumably means "don't apply a
> facename property") and the side effect is applying f-l-multiline if the
> buffer line matches the regexp.

Right.

>>> What about the run-time overhead of having an extra text property over
>>> the entire buffer?
>> It's not over the entire buffer.
> Load a buffer and wait for it to be jit-locked.  The entire buffer will
> have been scanned and this text property, in effect, set to t or nil on
> every single byte.

Actually the `nil' value is not added, so the property is only added where
it's non-nil.  Hence it's not added to the entire buffer.

>> And I haven't seen any evidence that it's noticeable (especially
>> compared the number of `face' properties or of the `fontified' property
>> added to the whole buffer).
> I'm sure it's not.

Good.

> Is it really that important to responsiveness whether the delay happens
> at the redisplay or immediately after the change?  With the after-change
> f-l-e-r-f, only the two boundaries of the region get checked, plus any
> chunk boundaries when this is bigger than 500 bytes.  With the extra
> pattern in f-l-keywords, then entire region gets checked.

But a single command may do thousands of buffer-modifications.

Look at it this way.  Let's define things as follows:
- a single buffer-modification, without your hook, takes time T1
- font-locking, without my font-lock-multiline thingy, takes time T2
- your hook takes times N1*T1
- my font-lock-multiline takes time N2*T2

So your approach can slow down user-visible execution by a fator N1, whereas
mine can slow it down by a factor N2.  So the important question (to me
anyway) is whether N1 is expected to be bigger or smaller than N2.

Note that in all likelyhood T2 is much larger than T1, so it should be
expected that N1 is significantly larger than N2.

> expensive in aggregate - it marks _every_ sequence which must be
> atomically fontified, just in case that sequence might straddle a chunk
> boundary in the future.

No: it's only applied to those atomic elements which do straddle
a line boundary.

>> I'm as close as it gets to a font-lock maintainer.  So from where I
>> stand, you're trying to impose your taste and judgment on me.

> I'm as close as it gets to a CC Mode maintainer at the moment.  What's
> constraining my freedom in that department now?  ;-)

We're talking about font-lock code here.  You're free to do whatever you like
in cc-mode, including adding an advice to font-lock-after-change-function.

>> It has to be in font-lock-default-fontify-buffer since there's no
>> guarantee this is only called from jit-lock-fontify-now.

> No.  It has to be _called_ from font-lock-default-fontify-buffer.

RIght, I meant to say that there's no guarantee
font-lock-default-fontify-buffer will be called from jit-lock-fontify-now,
which is why a call to extend-region-function needs to be in
font-lock-default-fontify-buffer.

>> So you're suggesting to add a redundant call to the extend-region hook.
>> Maybe it's simpler but it's ugly (because of the redundance, and because
>> it adds a dependency between jit-lock and font-lock).

> No.  I'm suggesting refactoring the font lock code a little so that each
> path through it calls the extend-region hook exactly once.  Once in the
> jit-lock "branch" and once in the plain font lock "branch".

That's still redundancy (admittedly less so because it's only static
redundancy).  In my book, if you can get away with putting it it at only
1 spot, it's better from a software engineering point of view.

>> At the same time font-lock-after-change-extend-region-function will be
>> moved from font-core to font-lock (where it belongs) and its
>> make-variable-buffer-local call will be removed as well (use
>> make-local-variable manually if you need it).

> Surely not - that variable is essentially buffer local, and it makes no
> sense for it not to be so.  To remove the make-variable-buffer-local call
> could lead to the same problems which have bedevilled paragraph-start and
> paragraph-separate.

It's a variable which should be modified via `add-hook' (which has
a parameter to specify whether it should be applied buffer-locally or not),
not via setq.  So there is need for make-variable-buffer-local.

Especially since it does make sense to add stuff to it globally as well:
Think about someone who wants to re-add the font-lock-extra-lines feature.
Similarly the font-lock-multiline property could be handled by moving the
code from font-lock-default-fontify-region to this hook.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> Surely not - that variable is essentially buffer local, and it makes no
>> sense for it not to be so.  To remove the make-variable-buffer-local call
>> could lead to the same problems which have bedevilled paragraph-start and
>> paragraph-separate.

> It's a variable which should be modified via `add-hook' (which has
> a parameter to specify whether it should be applied buffer-locally or not),
> not via setq.  So there is need for make-variable-buffer-local.

> Especially since it does make sense to add stuff to it globally as well:
> Think about someone who wants to re-add the font-lock-extra-lines feature.
> Similarly the font-lock-multiline property could be handled by moving the
> code from font-lock-default-fontify-region to this hook.

Hmmm...  sorry 'bout that: I was assuming changing it a list of functions,
but indeed it's not necessarily a good idea.

Still: even if it's a simple variable, you shouldn't set it via `setq' but
via font-lock-defaults (which will use make-local-variable for you), so
there's still no justification for make-variable-buffer-local.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Ralf Angeli]

* Alan Mackenzie (2006-04-25) writes:

> On Tue, 25 Apr 2006, Ralf Angeli wrote:
>
>>Do you even have to bother with placing font-lock-multiline properties
>>manually?  In AUCTeX we simply set the font-lock-multiline variable to
>>t and font-lock added the property automatically.
>
> Yes, absolutely!  There are two separate things here which Stefan is
> failing to distinguish:
>
> (i) The matching of keywords which span line breaks;
>
> (ii) Extending a font-lock region to include all text needing
> refontification.
>
> If I've understood font-lock-multiline properly, only manually setting
> f-l-m can achieve purpose (ii).

As far as I can see font-lock cares about that itself in case of
keyword fontification if the value of the variable
`font-lock-multiline' is t.  There are `(put-text-property
... 'font-lock-multiline t)' calls in
`font-lock-fontify-achored-keywords' and
`font-lock-fontify-keywords-region'.  As I wrote before, AUCTeX uses
this feature without manually adding `font-lock-multiline' properties.

-- 
Ralf

Re: Last steps for pretesting (font-lock-extend-region-function)

[Ralf Angeli]

* Alan Mackenzie (2006-04-24) writes:

> after-change-functions is the canonical way of solving problems related
> to buffer changes.  This includes Font Locking.

But extending the region in the after-change case does not help with
"normal" fontification by region as e.g. done by jit-lock.

> I say to you again - your solution is not robust.  I don't think it's
> been tried at all (correct me if I'm wrong).

Hopefully I understand "your solution" correctly here if I am assuming
this relates to putting the hook into
`font-lock-default-fontify-region'.  I've implemented a variant of
this approach for AUCTeX by using a custom-made function for
`font-lock-fontify-region-function' which calls other functions for
extending the region and then calls `font-lock-default-fontify-region'
with the extended region as arguments.  There is no special code in
`after-change-functions' and the approach actually works quite well.

> We are both aware of a refinement
> which is needed, namely calling some sort of f-l-extend-region-f from
> f-l-d-f-r and j-l-f-n.

If putting the hook into `font-lock-default-fontify-region' works
reliably, why do you need it in `after-change-functions' and
`jit-lock-fontify-now' as well?

-- 
Ralf

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> I was talking about programmer work, and I've shown you the code for
>> font-lock-multiline: it's quite a bit shorter and more straightforward
>> than your hack using b-c-f and a-c-f: just match the multiline element,
>> place a font-lock-multiline property on it, and you're set.

> Do you even have to bother with placing font-lock-multiline properties
> manually?  In AUCTeX we simply set the font-lock-multiline variable to
> t and font-lock added the property automatically.

Of course, you can indeed just set the variable, but this has more impact on
performance (e.g. every time it matches a keyword, font-lock will then check
whether it spans several lines, even if the keyword pattern is something
silly like "foo" which clearly can't span multiple lines), and will not
always work correctly if the MATCHER was a function rather than a regexp.

I suggested to do it manually because it is more explicit, it gives you
more control, and most importantly I haven't seen the part of
c-awk-font-lock-keywords which cares about multiple lines, so I haven't
convinced myself that it would work for his example.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>>> I was talking about programmer work, and I've shown you the code for
>>> font-lock-multiline: it's quite a bit shorter and more straightforward than
>>> your hack using b-c-f and a-c-f: just match the multiline element, place
>>> a font-lock-multiline property on it, and you're set.

>> Do you even have to bother with placing font-lock-multiline properties
>> manually?  In AUCTeX we simply set the font-lock-multiline variable to
>> t and font-lock added the property automatically.

> Yes, absolutely!  There are two separate things here which Stefan is
> failing to distinguish:

> (i) The matching of keywords which span line breaks;

This can only reliably be handled by a hook in
font-lock-default-fontify-region.

> (ii) Extending a font-lock region to include all text needing
> refontification.

> If I've understood font-lock-multiline properly, only manually setting
> f-l-m can achieve purpose (ii).

No, setting the font-lock-multiline variable will (if all goes well) take
care of (ii), but won't reliably take care of (i).

Setting the font-lock-multiline property by hand will take care of (ii) but
not of (i).


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> Hopefully I understand "your solution" correctly here if I am assuming
> this relates to putting the hook into
> `font-lock-default-fontify-region'.  I've implemented a variant of
> this approach for AUCTeX by using a custom-made function for
> `font-lock-fontify-region-function' which calls other functions for
> extending the region and then calls `font-lock-default-fontify-region'
> with the extended region as arguments.  There is no special code in
> `after-change-functions' and the approach actually works quite well.

Hey, that's a very good point.  There isn't any real need for a new
"extend-region" hook in font-lock-default-fontify-region since we can simply
use the font-lock-fontify-region-function hook for that purpose.

Thanks for pointing it out,


        Stefan

font-lock-multiline for cc-awk.

[Stefan Monnier]

AFAICT the patch below converts cc-awk to use font-lock-multiline instead of
an after-change-function.


        Stefan


--- cc-awk.el	06 Mar 2006 14:26:48 -0500	1.9
+++ cc-awk.el	25 Apr 2006 16:08:37 -0400	
@@ -677,7 +677,13 @@
     (goto-char (1+ beg))
     (or (eobp)
         (while (search-forward "\"" end t)
-          (c-put-char-property (1- (point)) 'syntax-table '(1))))))
+          (c-put-char-property (1- (point)) 'syntax-table '(1)))))
+  ;; Make sure that this element will be properly refontified next time
+  ;; around even if it spans several lines.
+  (save-excursion
+    (goto-char beg)
+    (if (search-forward "\n" end t)
+        (put-text-property beg end 'font-lock-multiline t))))
 
 (defun c-awk-syntax-tablify-string ()
   ;; Point is at the opening " or _" of a string.  Set the syntax-table
@@ -759,10 +765,12 @@
 ;; (iv) Inside a comment, all syntax-table properties are cleared.
 ;;
 ;; This function does hidden buffer changes.
-  (let (anchor
-	(anchor-state-/div nil)) ; t means a following / would be a div sign.
     (c-awk-beginning-of-logical-line) ; ACM 2002/7/21.  This is probably redundant.
+  ;; Redundant if called from font-lock.  --Stef
     (c-clear-char-properties (point) lim 'syntax-table)
+  (let (anchor
+	(anchor-state-/div nil)
+        (linebeg (point))) ; t means a following / would be a div sign.
     ;; Once round the next loop for each string, regexp, or div sign
     (while (progn
              ;; Skip any "harmless" lines before the next tricky one.
@@ -776,83 +784,19 @@
       (setq anchor-state-/div
             (if (looking-at "_?\"")
                 (c-awk-syntax-tablify-string)
+              (if (save-excursion (re-search-backward "\n" linebeg t))
+                  ;; Whether / is interpreted as a div or not depends on all
+                  ;; the text between linebeg and point, so if something is
+                  ;; changed there, even if it's on another line, we need to
+                  ;; refontify this piece of text.
+                  ;; Note that jit-lock-contextually also takes care of
+                  ;; that, so this is only necessary if we want it to work
+                  ;; without jit-lock-contextually or if we want to avoid
+                  ;; the jit-lock-context-time delay.
+                  (put-text-property linebeg (point) 'font-lock-multiline t))
               (c-awk-syntax-tablify-/ anchor anchor-state-/div))))
     nil))
 
-
-;; ACM, 2002/07/21: Thoughts: We need an AWK Mode after-change function to set
-;; the syntax-table properties even when font-lock isn't enabled, for the
-;; subsequent use of movement functions, etc.  However, it seems that if font
-;; lock _is_ enabled, we can always leave it to do the job.
-(defvar c-awk-old-EOLL 0)
-(make-variable-buffer-local 'c-awk-old-EOLL)
-;; End of logical line following the region which is about to be changed.  Set
-;; in c-awk-before-change and used in c-awk-after-change.
-
-(defun c-awk-before-change (beg end)
-;; This function is called exclusively from the before-change-functions hook.
-;; It does two things: Finds the end of the (logical) line on which END lies,
-;; and clears c-awk-NL-prop text properties from this point onwards.
-;;
-;; This function might do hidden buffer changes.
-  (save-restriction
-    (save-excursion
-      (setq c-awk-old-EOLL (c-awk-end-of-logical-line end))
-      (c-save-buffer-state nil
-       (c-awk-clear-NL-props end (point-max))))))
-
-(defun c-awk-end-of-change-region (beg end old-len)
-  ;; Find the end of the region which needs to be font-locked after a change.
-  ;; This is the end of the logical line on which the change happened, either
-  ;; as it was before the change, or as it is now, whichever is later.
-  ;; N.B. point is left undefined.
-  ;;
-  ;; This function might do hidden buffer changes.
-  (max (+ (- c-awk-old-EOLL old-len) (- end beg))
-       (c-awk-end-of-logical-line end)))
-
-(defun c-awk-after-change (beg end old-len)
-;; This function is called exclusively as an after-change function in
-;; AWK Mode.  It ensures that the syntax-table properties get set in the
-;; changed region.  However, if font-lock is enabled, this function does
-;; nothing, since an enabled font-lock after-change function will always do
-;; this.
-;;
-;; This function might do hidden buffer changes.
-  (unless (and (boundp 'font-lock-mode) font-lock-mode)
-    (save-restriction
-      (save-excursion
-	(save-match-data
-	  (setq end (c-awk-end-of-change-region beg end old-len))
-	  (c-awk-beginning-of-logical-line beg)
-	  (c-save-buffer-state nil  ; So that read-only status isn't affected.
-                                        ; (e.g. when first loading the buffer)
-	    (c-awk-set-syntax-table-properties end)))))))
-
-;; ACM 2002/5/25.  When font-locking is invoked by a buffer change, the region
-;; specified by the font-lock after-change function must be expanded to
-;; include ALL of any string or regexp within the region.  The simplest way to
-;; do this in practice is to use the beginning/end-of-logical-line functions.
-;; Don't overlook the possibility of the buffer change being the "recapturing"
-;; of a previously escaped newline.
-(defmacro c-awk-advise-fl-for-awk-region (function)
-  `(defadvice ,function (before get-awk-region activate)
-;; When font-locking an AWK Mode buffer, make sure that any string/regexp is
-;; completely font-locked.
-  (when (eq major-mode 'awk-mode)
-    (save-excursion
-      (ad-set-arg 1 (c-awk-end-of-change-region
-                     (ad-get-arg 0)     ; beg
-                     (ad-get-arg 1)     ; end
-                     (ad-get-arg 2)))   ; old-len
-      (ad-set-arg 0 (c-awk-beginning-of-logical-line (ad-get-arg 0)))))))
-
-(c-awk-advise-fl-for-awk-region font-lock-after-change-function)
-(c-awk-advise-fl-for-awk-region jit-lock-after-change)
-(c-awk-advise-fl-for-awk-region lazy-lock-defer-rest-after-change)
-(c-awk-advise-fl-for-awk-region lazy-lock-defer-line-after-change)
-
-\f
 ;; ACM 2002/9/29.  Movement functions, e.g. for C-M-a and C-M-e
 
 ;; The following three regexps differ from those earlier on in cc-awk.el in
@@ -1011,5 +955,5 @@
 \f
 (cc-provide 'cc-awk)			; Changed from 'awk-mode, ACM 2002/5/21
 
-;;; arch-tag: c4836289-3aa4-4a59-9934-9ccc2bacccf3
+;; arch-tag: c4836289-3aa4-4a59-9934-9ccc2bacccf3
 ;;; awk-mode.el ends here

Re: Last steps for pretesting (font-lock-extend-region-function)

[Ralf Angeli]

* Stefan Monnier (2006-04-25) writes:

>> Hopefully I understand "your solution" correctly here if I am assuming
>> this relates to putting the hook into
>> `font-lock-default-fontify-region'.  I've implemented a variant of
>> this approach for AUCTeX by using a custom-made function for
>> `font-lock-fontify-region-function' which calls other functions for
>> extending the region and then calls `font-lock-default-fontify-region'
>> with the extended region as arguments.  There is no special code in
>> `after-change-functions' and the approach actually works quite well.
>
> Hey, that's a very good point.  There isn't any real need for a new
> "extend-region" hook in font-lock-default-fontify-region since we can simply
> use the font-lock-fontify-region-function hook for that purpose.
>
> Thanks for pointing it out,

Now I feel a bit guilty that I didn't mention this earlier.  Anyway, I
thought that a hook in f-l-d-f-r might be useful despite the
possibility mentioned above as a "documentable" and common way of
extending the region.  But you are right, it's not really necessary.
But then, IMO, a hook in `after-change-functions' is not strictly
necessary either.  Its only benefit might be that refontification of
the extended region happens faster.

-- 
Ralf

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> I was talking about programmer work, and I've shown you the code for
>> font-lock-multiline: it's quite a bit shorter and more straightforward than
>> your hack using b-c-f and a-c-f: just match the multiline element, place
>> a font-lock-multiline property on it, and you're set.

> Do you even have to bother with placing font-lock-multiline properties
> manually?  In AUCTeX we simply set the font-lock-multiline variable to
> t and font-lock added the property automatically.

[ Now that I've actually looked at his code: ]
Actually, yes he does because the font-lock-multiline variable is only
applied to font-lock-keywords, whereas his multiline needs come from
font-lock-syntactic-keywords, and also because he doesn't actually use
font-lock-syntactic-keywords, he just puts there a single MATCHER which does
the work by hand.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> Now I feel a bit guilty that I didn't mention this earlier.  Anyway, I
> thought that a hook in f-l-d-f-r might be useful despite the
> possibility mentioned above as a "documentable" and common way of
> extending the region.  But you are right, it's not really necessary.

I was thinking of documenting this use of font-lock-fontify-region-function
in the new "multi line font lock elements" node of the elisp manual.
Do you think a separate hook is still necessary?

The way I see it, a separate hook would only really make sense if it were
a "normal" hook where you can place several functions, so that minor modes
can also use it reliably and easily.

> But then, IMO, a hook in `after-change-functions' is not strictly
> necessary either.  Its only benefit might be that refontification of
> the extended region happens faster.

I don't understand what you're saying here.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Ralf Angeli]

* Stefan Monnier (2006-04-25) writes:

>> Now I feel a bit guilty that I didn't mention this earlier.  Anyway, I
>> thought that a hook in f-l-d-f-r might be useful despite the
>> possibility mentioned above as a "documentable" and common way of
>> extending the region.  But you are right, it's not really necessary.
>
> I was thinking of documenting this use of font-lock-fontify-region-function
> in the new "multi line font lock elements" node of the elisp manual.
> Do you think a separate hook is still necessary?

If the above mentioned use of f-l-f-r-f will be documented, the need
for a hook diminishes even further.  It would only serve as a more
convenient way of achieving the same thing.  So, no, I don't think the
hook is still necessary.

> The way I see it, a separate hook would only really make sense if it were
> a "normal" hook where you can place several functions, so that minor modes
> can also use it reliably and easily.

Yes.  In AUCTeX I used a regular list, not a hook, for this purpose.

>> But then, IMO, a hook in `after-change-functions' is not strictly
>> necessary either.  Its only benefit might be that refontification of
>> the extended region happens faster.
>
> I don't understand what you're saying here.

When experimenting with extension of the region to be fontified in
AUCTeX I advised `jit-lock-after-change' to do this extension and with
that piece of advice refontification of a multiline construct after a
buffer change happened immediately while without the advice it took a
few moments.  Possibly as long as `jit-lock-context-time' IIUC.

-- 
Ralf

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> When experimenting with extension of the region to be fontified in
> AUCTeX I advised `jit-lock-after-change' to do this extension and with
> that piece of advice refontification of a multiline construct after a
> buffer change happened immediately while without the advice it took a
> few moments.  Possibly as long as `jit-lock-context-time' IIUC.

That seems to be unrelated to any kind of hook in font-lock-fontify-region.
It seems what happens is that without the advice the text doesn't get
a font-lock-multiline property but happens to still be correctly refreshed
thanks to jit-lock-contextually.  Or something like that.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Good evening, Stefan!

On Tue, 25 Apr 2006, Stefan Monnier wrote:

>> I can't understand at all why you feel that.  I really can't.  To me,
>> f-l-e-r-f seems natural, simple and obvious.  Optionally setting a
>> hook variable to a function is done countless times in Emacs.  I can't
>> see why font-lock-extend-region-function is any more ugly than, say,
>> adaptive-fill-function.

>I'm not saying the hook is a kludge: I'm saying the code that uses it is
>a kludge, compared to the code that uses font-lock-multiline.

OK.  I feel that existing code in AWK Mode is natural.  (OK, I wrote it,
so I'm not in a position to judge it dispassionately, but ...).  It notes
the logical EOL before a change.  It notes the logical EOL after a
change.  The later one is the end position for fontification.

By contrast, the f-l-multiline method (ab)uses the structure for setting
face properties to set region boundaries for possible future changes.

>>> ....much more heavyweight codewise (this is objectively the case in
>>> terms of lines of code, and I hope my comparison above convinces a
>>> few more people that it's also subjectively the case).

>> This is not the case.  We're agreed that, for the f-l-multiline
>> mechanism to work, it needs supplementing by a
>> f-l-extend-region-function hook.

>Yes, it needs such a hook called from font-lock-default-fontify-region.
>Just like your current hook that's called from
>font-lock-after-change-function.  This is orthogonal.

Agreed.

[ .... ]

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Evening, Stefan!

On Tue, 25 Apr 2006, Stefan Monnier wrote:

>> OK.  So FACENAME comes back as nil (which presumably means "don't apply a
>> facename property") and the side effect is applying f-l-multiline if the
>> buffer line matches the regexp.

>Right.

>>>> What about the run-time overhead of having an extra text property over
>>>> the entire buffer?
>>> It's not over the entire buffer.
>> Load a buffer and wait for it to be jit-locked.  The entire buffer will
>> have been scanned and this text property, in effect, set to t or nil on
>> every single byte.

>Actually the `nil' value is not added, so the property is only added where
>it's non-nil.  Hence it's not added to the entire buffer.

Maybe the 'nil' isn't explicitly added, but the entire font-lock region
(or buffer) has to be scanned.  Every time a region gets fontified, the
f-l-m properties are erased and every multiline sequence in that region
is recalculated.  In the approach I favour, only the bits at the region
boundaries need checking, and then only at the time they are needed.

This continual erasure and recalculation of f-l-m properties could have a
serious impact on the speed of C Mode, when that finally acquires a
suitable region calcualtion function - because this function will
essentially have to be c-beginning-of-statement-1, with various
strategies, such as cacheing, to speed it up.  Calculating this function
for _every_ statement within the region would be prohibitively expensive.

>>> And I haven't seen any evidence that it's noticeable (especially
>>> compared the number of `face' properties or of the `fontified'
>>> property added to the whole buffer).
>> I'm sure it's not.

>Good.

>> Is it really that important to responsiveness whether the delay
>> happens at the redisplay or immediately after the change?  With the
>> after-change f-l-e-r-f, only the two boundaries of the region get
>> checked, plus any chunk boundaries when this is bigger than 500 bytes.
>> With the extra pattern in f-l-keywords, then entire region gets
>> checked.

>But a single command may do thousands of buffer-modifications.

What is a typical number for an "advanced" command?

>Look at it this way.  Let's define things as follows:
>- a single buffer-modification, without your hook, takes time T1
>- font-locking, without my font-lock-multiline thingy, takes time T2
>- your hook takes times N1*T1
>- my font-lock-multiline takes time N2*T2

>So your approach can slow down user-visible execution by a fator N1,
>whereas mine can slow it down by a factor N2.  So the important question
>(to me anyway) is whether N1 is expected to be bigger or smaller than
>N2.

Equally important is T1 and T2.

>Note that in all likelyhood T2 is much larger than T1, so it should be
>expected that N1 is significantly larger than N2.

You've given a good argument for keeping both approaches available.

>> expensive in aggregate - it marks _every_ sequence which must be
>> atomically fontified, just in case that sequence might straddle a chunk
>> boundary in the future.

>No: it's only applied to those atomic elements which do straddle
>a line boundary.

My understanding of the form you were suggesting for AWK Mode's
f-l-keywords, ("^.*\\\\\n" . (progn ......)), was that this regexp would
be matched against ALL lines in the region, marking all multi-lines,
in case they might in the future come to straddle chunk boundaries.

>>> I'm as close as it gets to a font-lock maintainer.  So from where I
>>> stand, you're trying to impose your taste and judgment on me.

>> I'm as close as it gets to a CC Mode maintainer at the moment.  What's
>> constraining my freedom in that department now?  ;-)

>We're talking about font-lock code here.  You're free to do whatever you
>like in cc-mode, including adding an advice to
>font-lock-after-change-function.

Richard has specifically required this advice to be eradicated, on the
grounds that if such advice is necessary there's some deficiency in the
underlying functionality, and that deficiency should be fixed.

[ .... ]

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Tue, 25 Apr 2006, Stefan Monnier wrote:

>>> Surely not - that variable is essentially buffer local, and it makes no
>>> sense for it not to be so.  To remove the make-variable-buffer-local call
>>> could lead to the same problems which have bedevilled paragraph-start and
>>> paragraph-separate.

>> It's a variable which should be modified via `add-hook' (which has
>> a parameter to specify whether it should be applied buffer-locally or not),
>> not via setq.  So there is need for make-variable-buffer-local.

>> Especially since it does make sense to add stuff to it globally as well:
>> Think about someone who wants to re-add the font-lock-extra-lines feature.
>> Similarly the font-lock-multiline property could be handled by moving the
>> code from font-lock-default-fontify-region to this hook.

>Hmmm...  sorry 'bout that: I was assuming changing it a list of functions,
>but indeed it's not necessarily a good idea.

>Still: even if it's a simple variable, you shouldn't set it via `setq'
>but via font-lock-defaults (which will use make-local-variable for you),
>so there's still no justification for make-variable-buffer-local.

I don't think XEmacs has got this mechanism yet, so portable code will be
carrying on using `setq' for some time.

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Ralf!

On Tue, 25 Apr 2006, Ralf Angeli wrote:

>* Alan Mackenzie (2006-04-25) writes:

>> On Tue, 25 Apr 2006, Ralf Angeli wrote:

>>>Do you even have to bother with placing font-lock-multiline properties
>>>manually?  In AUCTeX we simply set the font-lock-multiline variable to
>>>t and font-lock added the property automatically.

>> Yes, absolutely!  There are two separate things here which Stefan is
>> failing to distinguish:

>> (i) The matching of keywords which span line breaks;

>> (ii) Extending a font-lock region to include all text needing
>> refontification.

>> If I've understood font-lock-multiline properly, only manually setting
>> f-l-m can achieve purpose (ii).

>As far as I can see font-lock cares about that itself in case of
>keyword fontification if the value of the variable
>`font-lock-multiline' is t.  There are `(put-text-property
>... 'font-lock-multiline t)' calls in
>`font-lock-fontify-achored-keywords' and
>`font-lock-fontify-keywords-region'.

Yes, but as far as I can see, these function are capable of putting the
text property only on bytes between BEG and END.  It cannot modify BEG by
applying the property to bytes before BEG.

>As I wrote before, AUCTeX uses this feature without manually adding
>`font-lock-multiline' properties.

>Ralf

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Ralf!

On Tue, 25 Apr 2006, Ralf Angeli wrote:

>* Alan Mackenzie (2006-04-24) writes:

>> after-change-functions is the canonical way of solving problems related
>> to buffer changes.  This includes Font Locking.

>But extending the region in the after-change case does not help with
>"normal" fontification by region as e.g. done by jit-lock.

>> I say to you again - your solution is not robust.  I don't think it's
>> been tried at all (correct me if I'm wrong).

>Hopefully I understand "your solution" correctly here if I am assuming
>this relates to putting the hook into
>`font-lock-default-fontify-region'.

Yes - more precisely, putting the hook ONLY into f-l-d-f-r.

>I've implemented a variant of this approach for AUCTeX by using a
>custom-made function for `font-lock-fontify-region-function' which calls
>other functions for extending the region and then calls
>`font-lock-default-fontify-region' with the extended region as
>arguments.  There is no special code in `after-change-functions' and the
>approach actually works quite well.

>> We are both aware of a refinement which is needed, namely calling some
>> sort of f-l-extend-region-f from f-l-d-f-r and j-l-f-n.

>If putting the hook into `font-lock-default-fontify-region' works
>reliably, why do you need it in `after-change-functions' and
>`jit-lock-fontify-now' as well?

It doesn't work reliably.  The region which needs fontification after a
buffer change depends on that _change_, not merely on what the buffer
looks like after the change.  If the f-l-extend-region-f isn't called
from after-change-functions, necessary information gets lost.  Take, once
more, the standard canonical example:

1: "string \
2: over \
3: several  <======= point is here, having just deleted a character.
4: lines."

Point is at EOL 3, having just deleted a character.  A hook in
f-l-after-change-functions cannot determine whether L4 belongs to the
region, without knowing whether the deleted character was a \.  A hook is
thus necessary in font-lock-after-change-functions too, to record this
information in some fashion.

>Ralf

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Tue, 25 Apr 2006, Stefan Monnier wrote:

>> Hopefully I understand "your solution" correctly here if I am assuming
>> this relates to putting the hook into
>> `font-lock-default-fontify-region'.  I've implemented a variant of
>> this approach for AUCTeX by using a custom-made function for
>> `font-lock-fontify-region-function' which calls other functions for
>> extending the region and then calls `font-lock-default-fontify-region'
>> with the extended region as arguments.  There is no special code in
>> `after-change-functions' and the approach actually works quite well.

>Hey, that's a very good point.  There isn't any real need for a new
>"extend-region" hook in font-lock-default-fontify-region since we can
>simply use the font-lock-fontify-region-function hook for that purpose.

Would this not be very inconvenient?  Somebody needing the extend-region
functionality in f-l-d-f-r would then have to copy the entire source of
the default function, amend the source, make the
f-l-fontify-region-funtion buffer local, then set it.

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Miles Bader]

Alan Mackenzie <acm@muc.de> writes:
>>Actually no: as I've shown with my sample alternative solution, it would
>>only force you to make you life simpler.
>
> You have shown nothing of the kind.  You're in no better position to
> decide what is objectively simpler than I am, since neither of is is
> unbiassed in this debate.  I'm in a better position than you to know what
> makes MY life simpler.  Please respect that.

I gotta say -- I know nothing about the subject, but as far as the code
posted, Stefan's looks a _lot_ simpler.  More specifically, I get the
sense I could figure out what Stefan's code is doing by reading the docs
a bit, whereas the corresponding code from Alan seems impenetrable.

Maybe that doesn't matter to you Alan, because you're immersed in your
current solution, but I think it would be good to think about future
maintainers too.

-Miles
-- 
We are all lying in the gutter, but some of us are looking at the stars.
-Oscar Wilde

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> Would this not be very inconvenient?  Somebody needing the extend-region
> functionality in f-l-d-f-r would then have to copy the entire source of
> the default function, amend the source, make the
> f-l-fontify-region-funtion buffer local, then set it.

Of course not, just use a function like

   (defun c-awk-font-lock-fontify-region (beg end &rest args)
     (apply 'font-lock-default-fontify-region
            (c-awk-beginning-of-logical-line beg)
            (c-awk-end-of-logical-line end)
            args))


-- Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> I'm not saying the hook is a kludge: I'm saying the code that uses it is
>> a kludge, compared to the code that uses font-lock-multiline.

> OK.  I feel that existing code in AWK Mode is natural.  (OK, I wrote it,
> so I'm not in a position to judge it dispassionately, but ...).  It notes
> the logical EOL before a change.  It notes the logical EOL after a
> change.  The later one is the end position for fontification.

Why doesn't the BOL before change matter?
Why does the EOL before change matter?

> By contrast, the f-l-multiline method (ab)uses the structure for setting
> face properties to set region boundaries for possible future changes.

Abuses the structure of what?

There two issues about multiline elements:
- properly notice them when they appear
- properly *re*fontify them when they change

The first is taken care of by font-lock-fontify-region-function.
The second talks about *re*fontification, so it's natural that it would
involves saving some info during fontification for the next time around.

Doing it in a-c-f is fundamentally not the right place since it doesn't know
whether and how the changed text was fontified: it works, of course, but
it's not particularly more natural.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>> If putting the hook into `font-lock-default-fontify-region' works
>> reliably, why do you need it in `after-change-functions' and
>> `jit-lock-fontify-now' as well?

> It doesn't work reliably.

On its own, it doesn't.  Just like an after-change-function on its own is
not reliable, and a font-lock-multiline on its own is not reliable.  But he
uses font-lock-multiline as well, and that coupled with
font-lock-default-fontify-region *is* reliable.

> 1: "string \
> 2: over \
> 3: several  <======= point is here, having just deleted a character.
> 4: lines."

Come on' Alan, we've been through this: font-lock-multiline handles it
just fine.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

> This continual erasure and recalculation of f-l-m properties could have a
> serious impact on the speed of C Mode, when that finally acquires a
> suitable region calcualtion function - because this function will
> essentially have to be c-beginning-of-statement-1, with various
> strategies, such as cacheing, to speed it up.  Calculating this function
> for _every_ statement within the region would be prohibitively expensive.

Look at my suggested patch for cc-awk and you'll see that you never need to
compute anything extra: you just have to place the font-lock-multiline
property at the moment where you highlight the multiline element (at which
point you already have all the necessary info).

My experience has been that this has always been the case.  And it's not
a coincidence.

>> Look at it this way.  Let's define things as follows:
>> - a single buffer-modification, without your hook, takes time T1
>> - font-locking, without my font-lock-multiline thingy, takes time T2
>> - your hook takes times N1*T1
>> - my font-lock-multiline takes time N2*T2

>> So your approach can slow down user-visible execution by a fator N1,
>> whereas mine can slow it down by a factor N2.  So the important question
>> (to me anyway) is whether N1 is expected to be bigger or smaller than
>> N2.

> Equally important is T1 and T2.

Both T1 and T2 are outside the scope of this discussion, so I'm not sure why
you find them important.

>> Note that in all likelyhood T2 is much larger than T1, so it should be
>> expected that N1 is significantly larger than N2.
> You've given a good argument for keeping both approaches available.

Why?  Because you think that in some potential hypothetical case, T2 might
be smaller than T1?  In my experience not only is T2 always much larger than
T1, but N2 is also tiny (because it's really nothing more than adding a few
put-text-property calls).  So the reverse case sounds hopelessly unlikely
to me.  Get back to me when you've found one.

>>> expensive in aggregate - it marks _every_ sequence which must be
>>> atomically fontified, just in case that sequence might straddle a chunk
>>> boundary in the future.

>> No: it's only applied to those atomic elements which do straddle
>> a line boundary.

> My understanding of the form you were suggesting for AWK Mode's
> f-l-keywords, ("^.*\\\\\n" . (progn ......)), was that this regexp would
> be matched against ALL lines in the region, marking all multi-lines,
> in case they might in the future come to straddle chunk boundaries.

Oh yes, that's the case, sorry I misunderstood your use of "chunk" (even
though it was indeed quite clear, my fault).  So the only correction I'd
make to your claim is that it's not "_every_ sequence which must be
atomically fontified", but "_every_ multiline sequence which must be
atomically fontified" (that was probably meant implicitly).

>>>> I'm as close as it gets to a font-lock maintainer.  So from where I
>>>> stand, you're trying to impose your taste and judgment on me.

>>> I'm as close as it gets to a CC Mode maintainer at the moment.  What's
>>> constraining my freedom in that department now?  ;-)

>> We're talking about font-lock code here.  You're free to do whatever you
>> like in cc-mode, including adding an advice to
>> font-lock-after-change-function.

> Richard has specifically required this advice to be eradicated, on the
> grounds that if such advice is necessary there's some deficiency in the
> underlying functionality, and that deficiency should be fixed.

Yes, and there is no such deficiency, as shown with my sample patch, using
font-lock-multiline.


        Stefan

Re: Last steps for pretesting (font-lock-extend-region-function)

[Ralf Angeli]

* Alan Mackenzie (2006-04-26) writes:

> On Tue, 25 Apr 2006, Ralf Angeli wrote:
>
>>As far as I can see font-lock cares about that itself in case of
>>keyword fontification if the value of the variable
>>`font-lock-multiline' is t.  There are `(put-text-property
>>... 'font-lock-multiline t)' calls in
>>`font-lock-fontify-achored-keywords' and
>>`font-lock-fontify-keywords-region'.
>
> Yes, but as far as I can see, these function are capable of putting the
> text property only on bytes between BEG and END.  It cannot modify BEG by
> applying the property to bytes before BEG.

Sure, but that was not was I was after anyway.  I just wanted to
mention a way for the f-l-m property to be applied automatically.  As
this only works for keyword fontification and not for syntactic
keyword fontification (which your code is using at some point as
Stefan pointed out) this point is moot now, however.

-- 
Ralf

Re: font-lock-multiline for cc-awk.

[Alan Mackenzie]

Hi, Stefan!

On Tue, 25 Apr 2006, Stefan Monnier wrote:

>AFAICT the patch below converts cc-awk to use font-lock-multiline
>instead of an after-change-function.

You've rubbed out my after-change function!  Heeeellllllllllpppppppp!!

The after-change function is needed to set syntax-table text properties,
since this must be done regardless of whether font lock is enabled.  In
fact, it's a massive pain in the posterior that font lock erases these
text properties without getting permission from the major mode.  This is
the only reason that c-awk-set-syntax-table-properties is hooked into
font-lock-syntactic keywords - essentially, the syntax-table properties
are an after-change thing.

I don't want to mix up the syntax-table and f-l-multiline properties,
because they're very different things.  Doing so would weaken the
existing functions, which are strong single-purpose functions.  To do
things this sort of way, I'd put in a new function
c-awk-set-fl-multiline-properties (a sister function to
c-awk-set-syntax-table-properties, with the same calling conventions).
The existing function loops round string/regexp/division sign constructs,
whereas the new one would be looping round lines.  They don't mix very
happily.

But mainly, I don't want to be applying f-l-m properties throughout the
region[*].  They should be applied only at the region boundaries when
these need extending, and only when font lock is enabled, since they serve
no purpose otherwise.

[*] As you're aware, I don't want to use f-l-m properties at all.  But if
I must use them, I only want them where absolutely needed.

And you've rubbed out my before-change function!  Ruuuuuunnnnnnnn
awwwwwaaaayyyyyyy!!!!!!!  This is needed for other things besides
calculating font lock boundaries.

Why not leave things alone until Richard has settled the whole matter?

>        Stefan

[ Patch tentatively applied and examined. ]

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan~

On Wed, 26 Apr 2006, Stefan Monnier wrote:

>> Would this not be very inconvenient?  Somebody needing the extend-region
>> functionality in f-l-d-f-r would then have to copy the entire source of
>> the default function, amend the source, make the
>> f-l-fontify-region-funtion buffer local, then set it.

>Of course not, just use a function like

>   (defun c-awk-font-lock-fontify-region (beg end &rest args)
>     (apply 'font-lock-default-fontify-region
>            (c-awk-beginning-of-logical-line beg)
>            (c-awk-end-of-logical-line end)
>            args))

Heh!  Thanks, I should have spotted that myself!

That wouldn't work in the general case, though, because the current
implementation of f-l-d-f-r would foul up the BEG and END parameters
unless they happen to be at line boundaries.  This is another reason for
the refactoring I've been suggesting, to extract a core function from
f-l-d-f-r which fontifies exactly the region it's given.

>-- Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Wed, 26 Apr 2006, Stefan Monnier wrote:

>>> I'm not saying the hook is a kludge: I'm saying the code that uses it is
>>> a kludge, compared to the code that uses font-lock-multiline.

>> OK.  I feel that existing code in AWK Mode is natural.  (OK, I wrote it,
>> so I'm not in a position to judge it dispassionately, but ...).  It notes
>> the logical EOL before a change.  It notes the logical EOL after a
>> change.  The later one is the end position for fontification.

>Why doesn't the BOL before change matter?

Maybe it does/would do in some major modes.  It happens not to have any effect
in AWK.

>Why does the EOL before change matter?

Because a buffer change can alter this EOL's syntactic context.

>> By contrast, the f-l-multiline method (ab)uses the structure for setting
>> face properties to set region boundaries for possible future changes.

>Abuses the structure of what?

font-lock-keywords.  The purpose of this structure is to fontify strings
matching specific regular expressions.  Using it to expand the region to
be font-locked at a future fontification seems utterly disconnected with
this purpose.

>There two issues about multiline elements:
>- properly notice them when they appear
>- properly *re*fontify them when they change

I cannot bring myself to consider the extension of a font lock region as
"the manipulation of a multiline element".  It just feels too contorted.
I have always understood the term "multiline element" to mean an element
within the font-lock region which crosses a line boundary.  [Such things
occur a lot in Texinfo Mode, for example, and turning on the
f-l-multiline variable does indeed make these elements fontify properly.]

>The first is taken care of by font-lock-fontify-region-function.

The AWK Mode regular expressions, even those that can straddle line
breaks, match just fine without having to turn on the f-l-multiline
variable. 

>The second talks about *re*fontification, so it's natural that it would
>involves saving some info during fontification for the next time around.

I think I would agree, except that font-lock brutally erases these
properties at every fontification, only to have to recalculate them.
This wastes processor time, probably quite a lot of it.  What is the
point of doing this, unless there has been a buffer change?  The same
applies to the syntax-table property.

>Doing it in a-c-f is fundamentally not the right place since it doesn't
>know whether and how the changed text was fontified: it works, of
>course, but it's not particularly more natural.

To know how text is to be refontified, you need to know how it has been
changed.  before/after-change-functions is a perfectly natural place to
find this out.

>        Stefan

-- 
Alan.

Re: Last steps for pretesting (font-lock-extend-region-function)

[Stefan Monnier]

>>>> I'm not saying the hook is a kludge: I'm saying the code that uses it is
>>>> a kludge, compared to the code that uses font-lock-multiline.

>>> OK.  I feel that existing code in AWK Mode is natural.  (OK, I wrote it,
>>> so I'm not in a position to judge it dispassionately, but ...).  It notes
>>> the logical EOL before a change.  It notes the logical EOL after a
>>> change.  The later one is the end position for fontification.
>> Why doesn't the BOL before change matter?
> Maybe it does/would do in some major modes.  It happens not to have any
> effect in AWK.
>> Why does the EOL before change matter?
> Because a buffer change can alter this EOL's syntactic context.

But why does it matter?
Honestly, neither explanation seems very clear to me.  I don't know what
others think, but this is an example of the complexity in your code.
You end up having to discuss various scenarios to justify your code, which
leaves the reader with the lingering question "but are all the relevant
scenarios covered?".

Clearly, we 're biased, but I find that the code using font-lock-multiline
doesn't suffer from such problems.  It does suffer from the fact that you
have to trust font-lock's use of the font-lock-multiline property, but if
you assume it works, then all you have to do is "place the property on the
chunk of text that needs to be refontified if any part gets changed".

> font-lock-keywords.  The purpose of this structure is to fontify strings
> matching specific regular expressions.  Using it to expand the region to
> be font-locked at a future fontification seems utterly disconnected with
> this purpose.

My sample code using the ".*\\(\\\\\n.*\\)*" regexp was just a quick way to
show you what it could look like without having to delve into your code.
Check my cc-awk patch: it shows what typically happens in practice, which is
that you place the put-text-property right where you add the face (or
syntax-table property), so it's a lot more natural.

>> There two issues about multiline elements:
>> - properly notice them when they appear
>> - properly *re*fontify them when they change

> I cannot bring myself to consider the extension of a font lock region as
> "the manipulation of a multiline element".

I don't understand what you're saying here.

> It just feels too contorted.  I have always understood the term "multiline
> element" to mean an element within the font-lock region which crosses
> a line boundary.

IIUC, that's how I understand it as well, except I wouldn't restrict it to
the "font-lock region".

>> The first is taken care of by font-lock-fontify-region-function.

> The AWK Mode regular expressions, even those that can straddle line
> breaks, match just fine without having to turn on the f-l-multiline
> variable.

I don't see how my text and yours are related here.

>> The second talks about *re*fontification, so it's natural that it would
>> involves saving some info during fontification for the next time around.

> I think I would agree, except that font-lock brutally erases these
> properties at every fontification, only to have to recalculate them.
> This wastes processor time, probably quite a lot of it.

No it doesn't *particularly* "waste processor time quite a lot":  it's just
what font-lock does for a living: erase text-properties and the re-add them
(usually it only does it for `face', sometimes it does it for `syntax-table'
and/or `font-lock-multiline', and it can do that for other properties (see
font-lock-extra-managed-props)).

> What is the point of doing this, unless there has been a buffer change?

If font-lock really happens to remove a font-lock-multiline property, it
means that somebody placed it there before.  Since this property is normally
placed during font-locking, it implies that we're actually *re*fontifying.
*Re*fontification normally takes place in response to a buffer change.

So, the "removal + re-addition" of the font-lock-multiline property (just
like any other property managed by font-lock) indeed only happens if there
has been a buffer change.

>> Doing it in a-c-f is fundamentally not the right place since it doesn't
>> know whether and how the changed text was fontified: it works, of
>> course, but it's not particularly more natural.

> To know how text is to be refontified, you need to know how it has been
> changed.

The font-lock-multiline property is a "living" proof that this is not true.


        Stefan

Re: font-lock-multiline for cc-awk.

[Stefan Monnier]

>> AFAICT the patch below converts cc-awk to use font-lock-multiline
>> instead of an after-change-function.

> You've rubbed out my after-change function!  Heeeellllllllllpppppppp!!

Well, it was just a sample patch.

> In fact, it's a massive pain in the posterior that font lock erases these
> text properties without getting permission from the major mode.

Huh?  font-lock only clears the syntax-table properties if you specifically
ask it to manage that property (by setting font-lock-syntactic-keywords).

> I don't want to mix up the syntax-table and f-l-multiline properties,
> because they're very different things.

On the contrary, they belong together: the font-lock-multiline accompagnies
the syntax-table property to say "this syntax-table property depends on this
chunk of text", so that if a part of this chunk of text is changed, the
syntax-table property can be recomputed.

> Doing so would weaken the
> existing functions, which are strong single-purpose functions.  To do
> things this sort of way, I'd put in a new function
> c-awk-set-fl-multiline-properties (a sister function to
> c-awk-set-syntax-table-properties, with the same calling conventions).
> The existing function loops round string/regexp/division sign constructs,
> whereas the new one would be looping round lines.

A multiline-line without strings or regexps doesn't need to be refontified
atomically because none of the other patterns you use can span or depend on
multiple lines of text.

> They don't mix very happily.

On the contrary: they belong together.

> But mainly, I don't want to be applying f-l-m properties throughout the
> region[*].  They should be applied only at the region boundaries when
> these need extending, and only when font lock is enabled, since they serve
> no purpose otherwise.

> [*] As you're aware, I don't want to use f-l-m properties at all.  But if
> I must use them, I only want them where absolutely needed.

Look, I'm just showing this code to try and get you to understand how
font-lock-multiline properties work.  It seems like you've been thinking
much too hard about your a-c-f code and can't think outside of
that setting.

[ Reminds me of "a calculus of capabilities" and "alias types": they present
  a new way to guarantee the safety of "manual" memory
  management primitives.  Contrary to previous appoaches (which tended to be
  very complex and incomplete or much too restrictive), they allow
  dangling pointers.  Many people found it terribly difficult to believe to
  understand that a system that allows dangling pointers can be safe w.r.t
  manual memory management: all previous approaches cared about checking
  that "free" is safe, whereas those new approaches only care about checking
  that the memory access is safe: i.e. they simply disallow access after
  a "free" rather than disallow "free" if the object is still "live" (which
  they typically define as "reachable"). ]

> Why not leave things alone until Richard has settled the whole matter?

Of course, I didn't intend to apply the patch.


        Stefan