Re: [patch] use font-lock

Re: [patch] use font-lock

[Lennart Borgman (gmail)]

Daniel Colascione wrote:
> 
> 
> I've converted nxml to font-lock. I used the existing fontification 
> machinery
> and put it inside a cc-mode-style matcher. Efficiency and output are the
> same, but:
> 
> 1) hi-lock-mode works now
> 2) all conventional font-locking functions work as expected. For 
> example, you
> can turn fontification on and off with M-x font-lock-mode.
> 3) font-lock-add-keywords DTRT
> 4) multiple-major-modes modes should be able to use nXML fontification now,
> though this remains untested

Hi Daniel,

This sounds very interesting. As you might know in nXhtml I am using 
nxml-mode with mumamo (which is a framework for multiple major modes in 
a buffer). It works good even though mumamo requires that major modes 
uses font-lock.

You might wonder how that can be the case. To make it work I implemented 
a workaround where I use the parsing capabilities from nxml-mode to 
check that the files follows the DTD specified syntax, but syntax 
highlighting from another mode (xml-mode/html-mode) that supports font-lock.

There is one very disturbing thing with my solution: I can't stop 
nxml-mode from parsing the whole buffer. It parses also those parts 
where mumamo has assigned another major mode. (I hoped that someone some 
day might have the time and skill to look into this, but I did not have 
them.)

Does you solution handle this problem? If it does, then how does it 
handle it? Does font-lock-fontify-region-function handle also the 
parsing of the xml code? That would be great, but it seems difficult.

Another thing that would be great would be integration with CEDET. As 
you have probably seen nxml-mode is a part of CVS Emacs and CEDET will 
hopefully soon be. Eric Ludlam has done very much work on CEDET recently.

If the completion offered by nxml-mode could be used together with CEDET 
that would be very good. (nXhtml currently offer this in a visible way 
separately, but I believe the long term solution is to go with CEDET - 
at least as an option.)

BTW, there is a problem with hi-lock. It uses text properties which may 
be hidden by overlays. IMO it should use overlays with high priorities. 
(That seems to be the easiest solution.)

> I've also added a new function:
> nxml-debug-region: Interactive function. Activate the region and call
> nxml-debug-region. The new region is what nxml thinks should be 
> re-fontified
> if the original region is changed.
> 
> The new code probably works only on Emacs 22. Lightly tested, but it 
> seems to
> handle corner highlighting cases fine. I removed a bunch of code that
> was "for redisplay", since I'm assuming font-lock handles those funky bits.
> 
> In addition to the patch, I've attached a set of files that demonstrates
> extending nXML mode to work with the Genshi template engine. The examples
> ought to work with some slight modification of the embedded paths.

Genshi was new to me. I will add it to mumamo.el.

How did you do the integration with xhtml?


> (Does the Relax NG compact syntax offer a way to say "include the next file
> for this document type"?)


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/emacs-nxml-mode/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/emacs-nxml-mode/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:emacs-nxml-mode-digest@yahoogroups.com 
    mailto:emacs-nxml-mode-fullfeatured@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    emacs-nxml-mode-unsubscribe@yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: [emacs-nxml-mode] [patch] use font-lock

[Daniel Colascione]

On Friday 23 May 2008, Lennart Borgman (gmail) wrote:
> You might wonder how that can be the case. To make it work I implemented
> a workaround where I use the parsing capabilities from nxml-mode to
> check that the files follows the DTD specified syntax, but syntax
> highlighting from another mode (xml-mode/html-mode) that supports
> font-lock.

I saw that you used that approach, and decided to just alter nXML-mode 
instead.

I don't really understand why nXML was written to not use font-lock in the 
first place. cc-mode had used the complex-matcher approach for a long time, 
portably and with few problems. But from having read the font-lock 
documentation, one wouldn't suppose this kind of power was available. The 
trick of making a matcher that does all the fontification itself and just 
returns 'nil' is not documented. Perhaps it should be.

> There is one very disturbing thing with my solution: I can't stop
> nxml-mode from parsing the whole buffer. It parses also those parts
> where mumamo has assigned another major mode. (I hoped that someone some
> day might have the time and skill to look into this, but I did not have
> them.)

> Does you solution handle this problem? If it does, then how does it
> handle it? Does font-lock-fontify-region-function handle also the
> parsing of the xml code? That would be great, but it seems difficult.

I haven't looked at mumamo mode in detail. How does mumamo isolate major modes 
to particular areas of the buffer? If you're just narrowing the buffer, I 
think going through nXML and removing all the calls to WIDEN should be 
sufficient. (Or replacing them with some kind of mumamo-widen.)

cc-mode also widens the buffer. What's the difference?

> Another thing that would be great would be integration with CEDET. As
> you have probably seen nxml-mode is a part of CVS Emacs and CEDET will
> hopefully soon be. Eric Ludlam has done very much work on CEDET recently.

That's good news. I've followed the "IDE features" thread on emacs-devel with 
excitement.

> If the completion offered by nxml-mode could be used together with CEDET
> that would be very good. (nXhtml currently offer this in a visible way
> separately, but I believe the long term solution is to go with CEDET -
> at least as an option.)

I don't use CEDET, but if it's getting into Emacs, I might as well give it 
another whirl. IIRC, CEDET requires a buffer parser to just generate a set of 
tags. What else is required?

> BTW, there is a problem with hi-lock. It uses text properties which may
> be hidden by overlays. IMO it should use overlays with high priorities.
> (That seems to be the easiest solution.)

IMHO, font-lock itself should use overlays. But ignoring that distinction, 
hi-lock dynamically adding font-lock keywords is the right way to go. 

HIGHLIGHT elements already support a kind of priority through the OVERRIDE 
flag, which can be t, prepend, or append, and through ordering on 
font-lock-keywords.

> Genshi was new to me. I will add it to mumamo.el.
>
> How did you do the integration with xhtml?

I extended the XHTML schema to support Genshi and XInclude elements and 
attributes; I include the original XHTML schema and augment it. 

It'd be nice if nXML extended Relax NG to support some kind of schema plugin 
mechanism, basically automating what's in qtmstr-xhtml.rnc: e.g. some way of 
saying "elements ns:a, ns:b, and ns:c from mylanguage.rnc are at block level 
and can contain block-level elements, and ns:d is inline, but can't contain 
any children. Oh, and attributes ns:foo and ns:bar can attach to all 
elements".

As it is, the best approach for Genshi, IMHO, would be some kind of minor 
mode. This mode would add the appropriate font-lock keywords and generate a 
temporary schema like the one in qtmstr-xhtml.rnc, only pointing to the 
correct xhtml.rnc.

Re: [patch] use font-lock

[Lennart Borgman (gmail)]

Daniel Colascione wrote:
> 
> 
> On Friday 23 May 2008, Lennart Borgman (gmail) wrote:
>  > You might wonder how that can be the case. To make it work I implemented
>  > a workaround where I use the parsing capabilities from nxml-mode to
>  > check that the files follows the DTD specified syntax, but syntax
>  > highlighting from another mode (xml-mode/html-mode) that supports
>  > font-lock.
> 
> I saw that you used that approach, and decided to just alter nXML-mode
> instead.
> 
> I don't really understand why nXML was written to not use font-lock in the
> first place. cc-mode had used the complex-matcher approach for a long time,
> portably and with few problems. But from having read the font-lock
> documentation, one wouldn't suppose this kind of power was available. The
> trick of making a matcher that does all the fontification itself and just
> returns 'nil' is not documented. Perhaps it should be.

Do you mean font-lock-fontify-region-function?

>  > There is one very disturbing thing with my solution: I can't stop
>  > nxml-mode from parsing the whole buffer. It parses also those parts
>  > where mumamo has assigned another major mode. (I hoped that someone some
>  > day might have the time and skill to look into this, but I did not have
>  > them.)
> 
>  > Does you solution handle this problem? If it does, then how does it
>  > handle it? Does font-lock-fontify-region-function handle also the
>  > parsing of the xml code? That would be great, but it seems difficult.
> 
> I haven't looked at mumamo mode in detail. How does mumamo isolate major 
> modes
> to particular areas of the buffer? If you're just narrowing the buffer, I
> think going through nXML and removing all the calls to WIDEN should be
> sufficient. (Or replacing them with some kind of mumamo-widen.)

Mumamo now defines what I call "multi major modes". (Previously there 
was a minor mode called mumamo-mode, but that is obsolete.)

Mumamo defines its own font-lock-fontify-region-function that does the 
job. In this function it narrows the buffer to chunks where each chunk 
has its own major mode.

So the main question is: does the parsing happens in the 
font-lock-fontify-region-function that you have defined in nxml-mode? (I 
guess you have defined such a function?)

One thing I think need to be taken care of is nxml-mode starting state 
in chunks handled by mumamo. Is there a way to do that?


> cc-mode also widens the buffer. What's the difference?

Sorry, I do not understand what you mean here.


> I don't use CEDET, but if it's getting into Emacs, I might as well give it
> another whirl. IIRC, CEDET requires a buffer parser to just generate a 
> set of
> tags. What else is required?

I do not know. Maybe this discussion should be carried over to Emacs 
devel which I think Eric Ludlam reads?


>  > BTW, there is a problem with hi-lock. It uses text properties which may
>  > be hidden by overlays. IMO it should use overlays with high priorities.
>  > (That seems to be the easiest solution.)
> 
> IMHO, font-lock itself should use overlays.

That would probably be a performance problem.

> But ignoring that distinction,
> hi-lock dynamically adding font-lock keywords is the right way to go.

Yes, it is very nice.

>  > Genshi was new to me. I will add it to mumamo.el.
>  >
>  > How did you do the integration with xhtml?
> 
> I extended the XHTML schema to support Genshi and XInclude elements and
> attributes; I include the original XHTML schema and augment it.

Thanks, very nice. That mean you can handle <?pyton ... ?> and for 
example py:content="var".

mumamo can help in two ways here:

1) inside <?pyton ... ?> it can use a genshi-mode major mode if there is 
one.

2) inside {% pyton ... %} it can do the same.


> It'd be nice if nXML extended Relax NG to support some kind of schema 
> plugin
> mechanism, basically automating what's in qtmstr-xhtml.rnc: e.g. some 
> way of
> saying "elements ns:a, ns:b, and ns:c from mylanguage.rnc are at block 
> level
> and can contain block-level elements, and ns:d is inline, but can't contain
> any children. Oh, and attributes ns:foo and ns:bar can attach to all
> elements".

Good idea IMO.


> As it is, the best approach for Genshi, IMHO, would be some kind of minor
> mode. This mode would add the appropriate font-lock keywords and generate a
> temporary schema like the one in qtmstr-xhtml.rnc, only pointing to the
> correct xhtml.rnc.

Maybe. But with mumamo it could be a major mode, written with those 
tools that there already are in Emacs.

This could be combined with the schema extending mechanism you suggest.

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/emacs-nxml-mode/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/emacs-nxml-mode/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:emacs-nxml-mode-digest@yahoogroups.com 
    mailto:emacs-nxml-mode-fullfeatured@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    emacs-nxml-mode-unsubscribe@yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: [patch] use font-lock

[Daniel Colascione]

On Friday 23 May 2008 06:50:19 pm Lennart Borgman (gmail) wrote:
> Do you mean font-lock-fontify-region-function?

Sort of. I'm talking about using a function as a matcher keyword in 
font-lock-keywords. This function always returns nil, but while searching, 
sets the face property itself between (point) and its one parameter.

The advantage of using the function-matcher approach is that it interacts 
nicely with other users of font-lock-keywords. cc-mode has used it forever, 
and it seems to work well.

AFAICS, this trick is undocumented.

> Mumamo defines its own font-lock-fontify-region-function that does the
> job. In this function it narrows the buffer to chunks where each chunk
> has its own major mode.
>
> So the main question is: does the parsing happens in the
> font-lock-fontify-region-function that you have defined in nxml-mode? (I
> guess you have defined such a function?)

Yes, in that the after-change function that updates bookkeeping information is 
called from the font-lock machinery (via 
font-lock-extend-after-change-function). But it doesn't matter here, because 
at least nxml still keeps some buffer-global information, like the end of the 
XML prolog (which is assume to extend from the start of the buffer to 
nxml-prolog-end).

> One thing I think need to be taken care of is nxml-mode starting state
> in chunks handled by mumamo. Is there a way to do that?

I don't see an easy way to do that. The problem is that nxml needs to widen 
the buffer to see whether the prolog has changed, whether it's scanned as far 
as a given change, and so on. And then it assumes the buffer is contiguous. 
But I didn't write it; there may be a way to deal with the problem.

I'm just speculating here, but what if there were some indirect overlay 
property? It would have the behavior of using, as the overlay content, the 
contents of another buffer. Moving the point in the parent would move the 
point in the child. The child buffer would have its own font-lock setup, and 
the resulting highlighting would show up in the parent.

Using a system like that, you could stitch together multiple child buffers to 
form a parent that looked like it had multiple major modes. And each major 
mode of a child buffer wouldn't know or care about the parent.

> > IMHO, font-lock itself should use overlays.
>
> That would probably be a performance problem.

And a compatibility nightmare, you're right. But I think it would have been a 
better conceptual model.


------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/emacs-nxml-mode/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/emacs-nxml-mode/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:emacs-nxml-mode-digest@yahoogroups.com 
    mailto:emacs-nxml-mode-fullfeatured@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    emacs-nxml-mode-unsubscribe@yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: [patch] use font-lock

[Lennart Borgman (gmail)]

Daniel Colascione wrote:
> On Friday 23 May 2008 06:50:19 pm Lennart Borgman (gmail) wrote:
>  > Do you mean font-lock-fontify-region-function?
> 
> Sort of. I'm talking about using a function as a matcher keyword in
> font-lock-keywords. This function always returns nil, but while searching,
> sets the face property itself between (point) and its one parameter.
> 
> The advantage of using the function-matcher approach is that it interacts
> nicely with other users of font-lock-keywords. cc-mode has used it forever,
> and it seems to work well.
> 
> AFAICS, this trick is undocumented.

Is not that documented in `font-lock-keywords'? But maybe you mean it 
could be more explicit?


>  > Mumamo defines its own font-lock-fontify-region-function that does the
>  > job. In this function it narrows the buffer to chunks where each chunk
>  > has its own major mode.
>  >
>  > So the main question is: does the parsing happens in the
>  > font-lock-fontify-region-function that you have defined in nxml-mode? (I
>  > guess you have defined such a function?)
> 
> Yes, in that the after-change function that updates bookkeeping 
> information is
> called from the font-lock machinery (via
> font-lock-extend-after-change-function).

I am not sure what you mean here. Mumamo calls font-lock-fontify-region 
(after trying to set the variables needed for the chunks major mode). 
Does that work together with your approach?


>  > One thing I think need to be taken care of is nxml-mode starting state
>  > in chunks handled by mumamo. Is there a way to do that?
> 
> I don't see an easy way to do that. The problem is that nxml needs to widen
> the buffer to see whether the prolog has changed, whether it's scanned 
> as far
> as a given change, and so on. And then it assumes the buffer is contiguous.
> But I didn't write it; there may be a way to deal with the problem.

It is a general problem. Js2 has a similar problem. (I am cc:ing Steve 
Yegge too ;-)


> I'm just speculating here, but what if there were some indirect overlay
> property? It would have the behavior of using, as the overlay content, the
> contents of another buffer. Moving the point in the parent would move the
> point in the child. The child buffer would have its own font-lock setup, 
> and
> the resulting highlighting would show up in the parent.
> 
> Using a system like that, you could stitch together multiple child 
> buffers to
> form a parent that looked like it had multiple major modes. And each major
> mode of a child buffer wouldn't know or care about the parent.

I have been thinking about different such possibilities. AFAICS it would 
be much simpler if the parsing modes (like nxml-mode) are aware of what 
should be parsed in the buffers. Other possibilities tends to be very 
complex.

So I would suggest that something like mumamo divides the buffer into 
chunks with different major modes. Then the parsing mode and mumamo (or 
something similar) should communicate to tell what chunks should be 
parsed and what chunks should not be parsed. (The major mode itself is 
not enought to tell if a chunk should be parsed or not. For example a 
<style> ... </style> chunk in xhtml can be parsed by nxml-mode. Ie 
nxml-mode does not care much about the content of it.)

Doing it that way I see two problems:

1) The parser must be able to somehow support a "parse region" 
functionality.

2) Parsing states for the different chunks must be initialized somehow. 
I would suggest carrying over the last state from the previous chunk if 
needed. (Sometimes assuming that the parsing state at the beginning of 
the chunk is the "zero state" is enough, sometimes not.)

As you noticed the parser must somehow start from the beginning of the file.

Font lock could cheat a bit and start in the middle of a file so parsing 
and font-lock style syntax highlighting do not have the same needs. 
Maybe that can be handled by doing the syntax highlighting in two 
passes, the first pass similar to font-lock currents way and the second 
pass doing parsing?

This would allow editing with the benefit of syntax highlighting 
(font-lock style) while the parser works in the background, doing the 
second pass. When the support of the parser is needed (for example for 
completion) the user of course would have to wait for the parser to 
catch up.



How would such a scheme fit js2 and the changed nxml-mode? (I guess 
there is more work to support it in both modes and a framework similar 
to jit-lock would have to be used for the parsing.)

------------------------------------

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/emacs-nxml-mode/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/emacs-nxml-mode/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:emacs-nxml-mode-digest@yahoogroups.com 
    mailto:emacs-nxml-mode-fullfeatured@yahoogroups.com

<*> To unsubscribe from this group, send an email to:
    emacs-nxml-mode-unsubscribe@yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Re: [emacs-nxml-mode] [patch] use font-lock

[Stefan Monnier]

>> I've converted nxml to font-lock.  I used the existing fontification
>> machinery and put it inside a cc-mode-style matcher.  Efficiency and
>> output are the same, but:
>> 
>> 1) hi-lock-mode works now
>> 2) all conventional font-locking functions work as expected.
>>    For example, you can turn fontification on and off with
>>    M-x font-lock-mode.
>> 3) font-lock-add-keywords DTRT
>> 4) multiple-major-modes modes should be able to use nXML fontification now,
>>    though this remains untested

Sounds great.  Would you submit your patch to emacs-devel@gnu.org?


        Stefan


PS: BTW, is the Emacs CVS now the "official" nxml repository?