Why js2-mode in Emacs 23.2?

Why js2-mode in Emacs 23.2?

[Deniz Dogan]

Hi

I just found out that js2-mode will be shipped with Emacs 23.2. I'd
just like to say that while js2-mode has some very nice features, I
and several other JavaScript programmers enjoy "espresso-mode" by
Daniel Colascione far more. It is not only faster (which should come
as no surprise to anyone) but also *maintained*.

Has there been any discussion here about which JavaScript mode people
want? If so, could someone provide me with a link to that discussion?
If not, let's have that discussion now!

-- 
Deniz Dogan

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 1:34 AM, Deniz Dogan<deniz.a.m.dogan@gmail.com> wrote:
> Hi
>
> I just found out that js2-mode will be shipped with Emacs 23.2. I'd
> just like to say that while js2-mode has some very nice features, I
> and several other JavaScript programmers enjoy "espresso-mode" by
> Daniel Colascione far more. It is not only faster (which should come
> as no surprise to anyone) but also *maintained*.
>
> Has there been any discussion here about which JavaScript mode people
> want? If so, could someone provide me with a link to that discussion?
> If not, let's have that discussion now!


I don't know what others say, but I really think there are room for
two. One more basic (in the sense that it does not parse the file) and
a parser (that will hopefully fit the CEDET framework).

In the case of HTML for example we have the basic html-mode and nxml-mode.

There has long been talk of getting Daniel's/Karl's mode into Emacs.
Maybe you can help getting it done?

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 9, 2009, at 7:38 PM, Lennar Borgman wrote:
> There has long been talk of getting Daniel's/Karl's mode into Emacs.
> Maybe you can help getting it done?

The copyright on Karl's mode appears to be held by the FSF, and I  
already have papers filed. Just say the word and it's in.

Re: Why js2-mode in Emacs 23.2?

[Deniz Dogan]

2009/8/10 Daniel Colascione <danc@merrillpress.com>:
> On Aug 9, 2009, at 7:38 PM, Lennar Borgman wrote:
>>
>> There has long been talk of getting Daniel's/Karl's mode into Emacs.
>> Maybe you can help getting it done?
>
> The copyright on Karl's mode appears to be held by the FSF, and I already
> have papers filed. Just say the word and it's in.

Isn't espresso-mode basically a superset of Karl's mode? I don't
really see a reason to include the latter, as it would probably only
draw unnecessary attention from the better JavaScript modes.

-- 
Deniz Dogan

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 1:46 AM, Daniel Colascione<danc@merrillpress.com> wrote:
> On Aug 9, 2009, at 7:38 PM, Lennar Borgman wrote:
>>
>> There has long been talk of getting Daniel's/Karl's mode into Emacs.
>> Maybe you can help getting it done?
>
> The copyright on Karl's mode appears to be held by the FSF, and I already
> have papers filed. Just say the word and it's in.

It is up to Chong and Stefan of course.

Could you please make sure espresso-mode follows Emacs conventions? I
am not sure but I think there were some thing that could be fixed.

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 1:50 AM, Deniz Dogan<deniz.a.m.dogan@gmail.com> wrote:

> Isn't espresso-mode basically a superset of Karl's mode? I don't
> really see a reason to include the latter, as it would probably only
> draw unnecessary attention from the better JavaScript modes.

Yes, of course. I just wanted to draw attention to that fact.

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 9, 2009, at 7:50 PM, Deniz Dogan wrote:
> Isn't espresso-mode basically a superset of Karl's mode? I don't
> really see a reason to include the latter, as it would probably only
> draw unnecessary attention from the better JavaScript modes.

Yep. My point is that even though espresso-mode is based on Karl's  
mode, Karl doesn't need to get involved because his code already has  
the correct magic words attached.

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 9, 2009, at 7:55 PM, Lennart Borgman wrote:
> It is up to Chong and Stefan of course.

Of course.

> Could you please make sure espresso-mode follows Emacs conventions? I
> am not sure but I think there were some thing that could be fixed.

The only issue I'm aware of is that espresso-mode syntactically  
fontifies the whole buffer
when starting up to ensure that regular expressions are marked  
correctly. The comment
that explains it is here:

   ;; Important to fontify the whole buffer syntactically! If we don't,
   ;; then we might have regular expression literals that aren't marked
   ;; as strings, which will screw up parse-partial-sexp, scan-lists,
   ;; etc. and and produce maddening "unbalanced parenthesis" errors.
   ;; When we attempt to find the error and scroll to the portion of
   ;; the buffer containing the problem, JIT-lock will apply the
   ;; correct syntax to the regular expresion literal and the problem
   ;; will mysteriously disappear.

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 1:58 AM, Daniel Colascione<danc@merrillpress.com> wrote:
> On Aug 9, 2009, at 7:55 PM, Lennart Borgman wrote:
>>
>> It is up to Chong and Stefan of course.
>
> Of course.
>
>> Could you please make sure espresso-mode follows Emacs conventions? I
>> am not sure but I think there were some thing that could be fixed.
>
> The only issue I'm aware of is that espresso-mode syntactically fontifies
> the whole buffer
> when starting up to ensure that regular expressions are marked correctly.
> The comment
> that explains it is here:
>
>  ;; Important to fontify the whole buffer syntactically! If we don't,
>  ;; then we might have regular expression literals that aren't marked
>  ;; as strings, which will screw up parse-partial-sexp, scan-lists,
>  ;; etc. and and produce maddening "unbalanced parenthesis" errors.
>  ;; When we attempt to find the error and scroll to the portion of
>  ;; the buffer containing the problem, JIT-lock will apply the
>  ;; correct syntax to the regular expresion literal and the problem
>  ;; will mysteriously disappear.


Ah, yes, that was what broke it in mumamo. Could you please find a way
to fix this? There a different levels of fontification in JIT-lock
that probably can help.

Other things to look for are of course key bindings and using
define-derived-mode. (And probably more that I have forgotten.)

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 9, 2009, at 8:00 PM, Lennart Borgman wrote:
> On Mon, Aug 10, 2009 at 1:58 AM, Daniel Colascione<danc@merrillpress.com 
> > wrote:
>>
>>  ;; Important to fontify the whole buffer syntactically!
>
> Ah, yes, that was what broke it in mumamo. Could you please find a way
> to fix this? There a different levels of fontification in JIT-lock
> that probably can help.

It's not really JIT-lock's fault. It's that regular expressions are  
not stringified until they're seen by JIT-lock. Instead of using  
syntactic-keywords to mark regular expressions, we could scan the  
buffer manually. But that would cause exactly the same problems that  
the fontificaton call would, if I understand correctly.

Is there another way to solve the problem? Can mumamo tell espresso to  
ignore certain parts of the buffer?

> Other things to look for are of course key bindings and using
> define-derived-mode. (And probably more that I have forgotten.)

Keybindings are standard, IIRC. espresso-mode does override the normal  
tag binding, but it implements the same functionality in a different  
way. That sort of keybinding is explicitly allowed.

As for define-derived-mode: nope, espresso-mode doesn't use that.

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 2:06 AM, Daniel Colascione<danc@merrillpress.com> wrote:
> On Aug 9, 2009, at 8:00 PM, Lennart Borgman wrote:
>>
>> On Mon, Aug 10, 2009 at 1:58 AM, Daniel Colascione<danc@merrillpress.com>
>> wrote:
>>>
>>>  ;; Important to fontify the whole buffer syntactically!
>>
>> Ah, yes, that was what broke it in mumamo. Could you please find a way
>> to fix this? There a different levels of fontification in JIT-lock
>> that probably can help.
>
> It's not really JIT-lock's fault. It's that regular expressions are not
> stringified until they're seen by JIT-lock. Instead of using
> syntactic-keywords to mark regular expressions, we could scan the buffer
> manually. But that would cause exactly the same problems that the
> fontificaton call would, if I understand correctly.

Could not JIT-lock do a second pass?


> Is there another way to solve the problem? Can mumamo tell espresso to
> ignore certain parts of the buffer?

Well, it does, but espresso kind of ignores that ... ;-)

Just telling it where to scan will however not help since
espresso-mode will be called every time a user enters a chunk with
espresso-mode.


>> Other things to look for are of course key bindings and using
>> define-derived-mode. (And probably more that I have forgotten.)
>
> Keybindings are standard, IIRC. espresso-mode does override the normal tag
> binding, but it implements the same functionality in a different way. That
> sort of keybinding is explicitly allowed.

Nice.


> As for define-derived-mode: nope, espresso-mode doesn't use that.

It is a minor change but will help keeping it upto date later.

Re: Why js2-mode in Emacs 23.2?

[Leo]

On 2009-08-10 00:58 +0100, Daniel Colascione wrote:
>> Could you please make sure espresso-mode follows Emacs conventions? I
>> am not sure but I think there were some thing that could be fixed.
>
> The only issue I'm aware of is that espresso-mode syntactically
> fontifies the whole buffer

eval-after-load should not be used in emacs packages.

-- 
Leo's Emacs uptime: 6 days, 0 hours, 15 minutes, 17 seconds

Re: Why js2-mode in Emacs 23.2?

[Leo]

On 2009-08-10 00:38 +0100, Lennart Borgman wrote:
>> I just found out that js2-mode will be shipped with Emacs 23.2. I'd
>
>> just like to say that while js2-mode has some very nice features, I
>> and several other JavaScript programmers enjoy "espresso-mode" by
>> Daniel Colascione far more. It is not only faster (which should come
>> as no surprise to anyone) but also *maintained*.
>>
>> Has there been any discussion here about which JavaScript mode people
>> want? If so, could someone provide me with a link to that discussion?
>> If not, let's have that discussion now!
>
>
> I don't know what others say, but I really think there are room for
> two. One more basic (in the sense that it does not parse the file) and
> a parser (that will hopefully fit the CEDET framework).

Why not fix bugs in js2-mode? Having two packages providing similar
features is a burden to maintainers as well as users.

-- 
Leo's Emacs uptime: 6 days, 0 hours, 24 minutes, 52 seconds

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 9, 2009, at 8:17 PM, Lennart Borgman wrote:
> Could not JIT-lock do a second pass?

>> Is there another way to solve the problem? Can mumamo tell espresso  
>> to
>> ignore certain parts of the buffer?
>
> Well, it does, but espresso kind of ignores that ... ;-)
>
> Just telling it where to scan will however not help since
> espresso-mode will be called every time a user enters a chunk with
> espresso-mode.

A second pass when? The problem is that functions that have nothing to  
do with fontification operate improperly unless correct text  
properties are attached to certain bits of the buffer.

Consider the regular expression /}/. syntactic-keywords are used to  
mark the whole expression as a string. If forward-sexp encounters that  
expression before the expression is marked as a string, then it'll  
interpret the '}' incorrectly. When the user actually enters a '}', we  
fontify it correctly. However, when we first start up, we need to  
fonyify the whole buffer so that we don't trip over
one of these regular expressions if we look at it before jit-lock gets  
around to it.

Can espresso just perform the regexp-marking scan 1) on only the  
region given to it by mumamo, and 2) only once, with some kind of  
indication given by mumamo of whether we're seen this particular chunk  
before?


>> As for define-derived-mode: nope, espresso-mode doesn't use that.
>
> It is a minor change but will help keeping it upto date later.

So just use define-derived-mode to inherit from fundamental-mode?

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 9, 2009, at 8:32 PM, Leo wrote:
> Why not fix bugs in js2-mode? Having two packages providing similar
> features is a burden to maintainers as well as users.

Because js2-mode is an abomination? Oh, it's not so bad that it's slow  
and unmaintained: that description applies to lots of code in Emacs.  
The problem is that js2-mode operates fundamentally differently from  
other modes, eschews font-lock, parses the buffer *asynchronously*,  
and generally goes against every possible convention for implementing  
a major mode.

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 9, 2009, at 8:18 PM, Leo wrote:

> On 2009-08-10 00:58 +0100, Daniel Colascione wrote:
>>> Could you please make sure espresso-mode follows Emacs  
>>> conventions? I
>>> am not sure but I think there were some thing that could be fixed.
>>
>> The only issue I'm aware of is that espresso-mode syntactically
>> fontifies the whole buffer
>
> eval-after-load should not be used in emacs packages.

Why not?

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 2:46 AM, Daniel Colascione<danc@merrillpress.com> wrote:
> On Aug 9, 2009, at 8:17 PM, Lennart Borgman wrote:
>>
>> Could not JIT-lock do a second pass?
>
>>> Is there another way to solve the problem? Can mumamo tell espresso to
>>> ignore certain parts of the buffer?
>>
>> Well, it does, but espresso kind of ignores that ... ;-)
>>
>> Just telling it where to scan will however not help since
>> espresso-mode will be called every time a user enters a chunk with
>> espresso-mode.
>
> A second pass when?

I see. It will not help here.

> The problem is that functions that have nothing to do
> with fontification operate improperly unless correct text properties are
> attached to certain bits of the buffer.

You can handle this by checking for fontification and do it when
necessary and tell JIT lock you have done it.

See mlinks.el in nXhtml for an example of how to do it. (Look in the
repository, I have just implemented this.)

Maybe you can take that functionality and makes it into a separate
library. I have not had time to do that but I think it would be good.


> Consider the regular expression /}/. syntactic-keywords are used to mark the
> whole expression as a string. If forward-sexp encounters that expression
> before the expression is marked as a string, then it'll interpret the '}'
> incorrectly. When the user actually enters a '}', we fontify it correctly.
> However, when we first start up, we need to fonyify the whole buffer so that
> we don't trip over
> one of these regular expressions if we look at it before jit-lock gets
> around to it.
>
> Can espresso just perform the regexp-marking scan 1) on only the region
> given to it by mumamo, and 2) only once, with some kind of indication given
> by mumamo of whether we're seen this particular chunk before?

I think the recipe above will work for mumamo too.


>>> As for define-derived-mode: nope, espresso-mode doesn't use that.
>>
>> It is a minor change but will help keeping it upto date later.
>
> So just use define-derived-mode to inherit from fundamental-mode?

Yes. (Or from nil.)

Re: Why js2-mode in Emacs 23.2?

[Stefan Monnier]

>   ;; Important to fontify the whole buffer syntactically! If we don't,
>   ;; then we might have regular expression literals that aren't marked
>   ;; as strings, which will screw up parse-partial-sexp, scan-lists,
>   ;; etc. and and produce maddening "unbalanced parenthesis" errors.
>   ;; When we attempt to find the error and scroll to the portion of
>   ;; the buffer containing the problem, JIT-lock will apply the
>   ;; correct syntax to the regular expresion literal and the problem
>   ;; will mysteriously disappear.

Yes, this is a general problem.  Basically font-lock-syntactic-keywords
should be moved out of font-lock into syntax.el (and provide a new
feature like (syntax-ensure-properties BEG END) to request application
of those rules) and things like syntax-ppss, indentation, forward-sexp,
... should make sure it's applied to the portion of the buffer they
look at.


        Stefan

Re: Why js2-mode in Emacs 23.2?

[Stefan Monnier]

>> Why not fix bugs in js2-mode? Having two packages providing similar
>> features is a burden to maintainers as well as users.

> Because js2-mode is an abomination? Oh, it's not so bad that it's slow
> and unmaintained: that description applies to lots of code in Emacs.
> The problem is that js2-mode operates fundamentally differently from
> other modes, eschews font-lock, parses the buffer *asynchronously*,
> and generally goes against every possible convention for implementing
> a major mode.

Here's the story as I remember it: a year or two ago, someone asked for
a Javascript major mode to be included.  It sounded like a good idea,
and some people pointed out the then-brand-new js2-mode which was pretty
impressive featurewise.
Then time passed and js2-mode didn't get installed early enough.  So it
finally got installed recently.  I think it's good.  AFAIK its author is
still around and maintaining it (he's the one who installed it into the
CVS and he made some changes at this occasion, such as bringing all the
code into a single file).
So I do believe js2-mode is maintained.  Also, while it it unusual,
I think it's a very interesting experiment in how Emacs could evolve
(similarly to CEDET).

This also means that an alternative javascript-mode might be acceptable.
But you guys need to agree on which one will be the default mode (I
suggest to make auto-mode-alist bind *.js files to javascript-mode and
then make that into an alias that can easily be changed).  Also the
alternative mode needs to be as lean/simple/clean as possible (e.g. when
bumping into limitations of the generic Emacs support code, it'd be
better to live with it until it gets fixed (and/or get the generic code
fixed) than trying to work around it).


        Stefan

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 4:47 AM, Stefan Monnier<monnier@iro.umontreal.ca> wrote:
>>   ;; Important to fontify the whole buffer syntactically! If we don't,
>>   ;; then we might have regular expression literals that aren't marked
>>   ;; as strings, which will screw up parse-partial-sexp, scan-lists,
>>   ;; etc. and and produce maddening "unbalanced parenthesis" errors.
>>   ;; When we attempt to find the error and scroll to the portion of
>>   ;; the buffer containing the problem, JIT-lock will apply the
>>   ;; correct syntax to the regular expresion literal and the problem
>>   ;; will mysteriously disappear.
>
> Yes, this is a general problem.  Basically font-lock-syntactic-keywords
> should be moved out of font-lock into syntax.el (and provide a new
> feature like (syntax-ensure-properties BEG END) to request application
> of those rules) and things like syntax-ppss, indentation, forward-sexp,
> ... should make sure it's applied to the portion of the buffer they
> look at.

I am not sure I understand this, but thinking of it again, could not
Daniel use a font-lock-fontify-region-function that first does a
fontification without keywords (if it is there it is needed).

Or otherwise use what I suggested before.

Re: Why js2-mode in Emacs 23.2?

[Miles Bader]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> So I do believe js2-mode is maintained.  Also, while it it unusual,
> I think it's a very interesting experiment in how Emacs could evolve
> (similarly to CEDET).

"Evolve"?  Is js2-mode's roll-everything-from-scratch code better than
the existing infrastructure that it ignores (e.g. font-lock)?

-Miles

-- 
The car has become... an article of dress without which we feel uncertain,
unclad, and incomplete.  [Marshall McLuhan, Understanding Media, 1964]

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 5:24 AM, Miles Bader<miles@gnu.org> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> So I do believe js2-mode is maintained.  Also, while it it unusual,
>> I think it's a very interesting experiment in how Emacs could evolve
>> (similarly to CEDET).
>
> "Evolve"?  Is js2-mode's roll-everything-from-scratch code better than
> the existing infrastructure that it ignores (e.g. font-lock)?


Someone made nxml-mode use (a bit of) font-lock. Can't the same be made for js2?

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 9, 2009, at 11:27 PM, Lennart Borgman wrote:
> Someone made nxml-mode use (a bit of) font-lock. Can't the same be  
> made for js2?

That was me. And no, js2 can't use a similar technique without a lot  
of work. And even if it could, it'd still be working slowly and  
asynchronously. Why use that instead of a mode that actually uses  
Emacs infrastructure?

Re: Why js2-mode in Emacs 23.2?

[Jason Rumney]

Lennart Borgman wrote:
> Someone made nxml-mode use (a bit of) font-lock. Can't the same be made for js2?
>   

Maybe someone has done that since, but the changes I made shortly after 
it was merged were merely to use the same faces as font-lock instead of 
reinventing a new set.

Re: Why js2-mode in Emacs 23.2?

[Xah Lee]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

On Sun, Aug 9, 2009 at 8:24 PM, Miles Bader <miles@gnu.org> wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> > So I do believe js2-mode is maintained.  Also, while it it unusual,
> > I think it's a very interesting experiment in how Emacs could evolve
> > (similarly to CEDET).
>
> "Evolve"?  Is js2-mode's roll-everything-from-scratch code better than
> the existing infrastructure that it ignores (e.g. font-lock)?


I'll have to agree with Stefan.

js2, nxml, they are fundamentally different than traditional emacs modes, in
that each have a parser that fully understands the languages at syntax
level.

emacs's syntax coloring is based on text pattern matching. It might work
faster or more smooth right now, but from technology point of view, its one
generation behind.

of the 2 approach, the full parser is also a order more difficult to write.

it'd be nice if emacs provides syntax parsing infrastructure.... nmxl, js2,
cedet, can be leads in this direction and see how things work out.

Including expesso-mode is also very good i think, as it provide tried and
true solution that just works.

my 2 ¢. :)

  Xah

[-- Attachment #2: Type: text/html, Size: 1674 bytes --]

Re: Why js2-mode in Emacs 23.2?

[Xah Lee]

[-- Attachment #1: Type: text/plain, Size: 236 bytes --]

On Sun, Aug 9, 2009 at 10:51 PM, Xah Lee <xahlee@gmail.com> wrote:

> I'll have to agree with Stefan...
>

egads, elide that sentence for me please. :) Didn't mean look at this as
some disagreement. Sorry for the no content mail.

 Xah

[-- Attachment #2: Type: text/html, Size: 835 bytes --]

Re: Why js2-mode in Emacs 23.2?

[Deniz Dogan]

2009/8/10 Stefan Monnier <monnier@iro.umontreal.ca>:
>>> Why not fix bugs in js2-mode? Having two packages providing similar
>>> features is a burden to maintainers as well as users.
>
>> Because js2-mode is an abomination? Oh, it's not so bad that it's slow
>> and unmaintained: that description applies to lots of code in Emacs.
>> The problem is that js2-mode operates fundamentally differently from
>> other modes, eschews font-lock, parses the buffer *asynchronously*,
>> and generally goes against every possible convention for implementing
>> a major mode.
>
> Here's the story as I remember it: a year or two ago, someone asked for
> a Javascript major mode to be included.  It sounded like a good idea,
> and some people pointed out the then-brand-new js2-mode which was pretty
> impressive featurewise.
> Then time passed and js2-mode didn't get installed early enough.  So it
> finally got installed recently.  I think it's good.  AFAIK its author is
> still around and maintaining it (he's the one who installed it into the
> CVS and he made some changes at this occasion, such as bringing all the
> code into a single file).
> So I do believe js2-mode is maintained.  Also, while it it unusual,
> I think it's a very interesting experiment in how Emacs could evolve
> (similarly to CEDET).

Yes, it seems to be maintained now. Last time I checked a new version
had not been released for a year or so.

> This also means that an alternative javascript-mode might be acceptable.
> But you guys need to agree on which one will be the default mode (I
> suggest to make auto-mode-alist bind *.js files to javascript-mode and
> then make that into an alias that can easily be changed).  Also the
> alternative mode needs to be as lean/simple/clean as possible (e.g. when
> bumping into limitations of the generic Emacs support code, it'd be
> better to live with it until it gets fixed (and/or get the generic code
> fixed) than trying to work around it).

I see no reason to make javascript-mode the default for JavaScript
files. I definitely think espresso-mode would be a better choice, even
though users would have to see "Espresso" in the mode line, which
could possibly confuse some people, but that's the only drawback as
far as I'm concerned. What I would really *not* like to see though is
js2-mode as the default.

-- 
Deniz Dogan

Re: Why js2-mode in Emacs 23.2?

[Miles Bader]

Xah Lee <xahlee@gmail.com> writes:
> egads, elide that sentence for me please. :) Didn't mean look at this as
> some disagreement. Sorry for the no content mail.

The rest of your mail was very useful though.

-Miles

-- 
You can hack anything you want, with TECO and DDT.

Re: Why js2-mode in Emacs 23.2?

[Carsten Dominik]

On Aug 10, 2009, at 2:18 AM, Leo wrote:

> On 2009-08-10 00:58 +0100, Daniel Colascione wrote:
>>> Could you please make sure espresso-mode follows Emacs  
>>> conventions? I
>>> am not sure but I think there were some thing that could be fixed.
>>
>> The only issue I'm aware of is that espresso-mode syntactically
>> fontifies the whole buffer
>
> eval-after-load should not be used in emacs packages.

Why not?

- Carsten

Re: Why js2-mode in Emacs 23.2?

[Leo]

On 2009-08-10 01:49 +0100, Daniel Colascione wrote:
[...]
>> eval-after-load should not be used in emacs packages.
>
> Why not?

On 2009-08-10 08:06 +0100, Carsten Dominik wrote:
[...]
>> eval-after-load should not be used in emacs packages.
>
> Why not?
>
> - Carsten

I seem to recall RMS said that sometime ago. It is also documented  in
the elisp manual:

,----[ (info "(elisp)Hooks for Loading") ]
|    In general, well-designed Lisp programs should not use this feature.
| The clean and modular ways to interact with a Lisp library are (1)
| examine and set the library's variables (those which are meant for
| outside use), and (2) call the library's functions.  If you wish to do
| (1), you can do it immediately--there is no need to wait for when the
| library is loaded.  To do (2), you must load the library (preferably
| with `require').
| 
|    But it is OK to use `eval-after-load' in your personal
| customizations if you don't feel they must meet the design standards
| for programs meant for wider use.
`----

Best,
-- 
Leo's Emacs uptime: 6 days, 8 hours, 38 minutes, 34 seconds

Re: Why js2-mode in Emacs 23.2?

[CHENG Gao]

*On Mon, 10 Aug 2009 09:44:58 +0100
* Also sprach Leo <sdl.web@gmail.com>:

> On 2009-08-10 01:49 +0100, Daniel Colascione wrote:
> [...]
>>> eval-after-load should not be used in emacs packages.
>>
>> Why not?
>
> On 2009-08-10 08:06 +0100, Carsten Dominik wrote:
> [...]
>>> eval-after-load should not be used in emacs packages.
>>
>> Why not?
>>
>> - Carsten
>
> I seem to recall RMS said that sometime ago. It is also documented  in
> the elisp manual:
>
> ,----[ (info "(elisp)Hooks for Loading") ]
> |    In general, well-designed Lisp programs should not use this feature.
> | The clean and modular ways to interact with a Lisp library are (1)
> | examine and set the library's variables (those which are meant for
> | outside use), and (2) call the library's functions.  If you wish to do
> | (1), you can do it immediately--there is no need to wait for when the
> | library is loaded.  To do (2), you must load the library (preferably
> | with `require').
> | 
> |    But it is OK to use `eval-after-load' in your personal
> | customizations if you don't feel they must meet the design standards
> | for programs meant for wider use.
> `----
>
> Best,

It's in ELISP info manual, under "Emacs Lisp Coding Conventions".

Re: Why js2-mode in Emacs 23.2?

[Leo]

On 2009-08-10 09:54 +0100, CHENG Gao wrote:
>> I seem to recall RMS said that sometime ago. It is also documented  in
>> the elisp manual:
>>
>> ,----[ (info "(elisp)Hooks for Loading") ]
>> |    In general, well-designed Lisp programs should not use this feature.
>> | The clean and modular ways to interact with a Lisp library are (1)
>> | examine and set the library's variables (those which are meant for
>> | outside use), and (2) call the library's functions.  If you wish to do
>> | (1), you can do it immediately--there is no need to wait for when the
>> | library is loaded.  To do (2), you must load the library (preferably
>> | with `require').
>> | 
>> |    But it is OK to use `eval-after-load' in your personal
>> | customizations if you don't feel they must meet the design standards
>> | for programs meant for wider use.
>> `----
>>
>> Best,
>
> It's in ELISP info manual, under "Emacs Lisp Coding Conventions".

Yes. for convenience:

,----[ (info "(elisp)Coding Conventions") ]
| * Avoid using `eval-after-load' in libraries and packages (*note
|   Hooks for Loading::).  This feature is meant for personal
|   customizations; using it in a Lisp program is unclean, because it
|   modifies the behavior of another Lisp file in a way that's not
|   visible in that file.  This is an obstacle for debugging, much
|   like advising a function in the other package.
`----

-- 
Leo's Emacs uptime: 6 days, 9 hours, 23 minutes, 20 seconds

Re: Why js2-mode in Emacs 23.2?

[Richard Riley]

Leo <sdl.web@gmail.com> writes:

> On 2009-08-10 09:54 +0100, CHENG Gao wrote:
>>> I seem to recall RMS said that sometime ago. It is also documented  in
>>> the elisp manual:
>>>
>>> ,----[ (info "(elisp)Hooks for Loading") ]
>>> |    In general, well-designed Lisp programs should not use this feature.
>>> | The clean and modular ways to interact with a Lisp library are (1)
>>> | examine and set the library's variables (those which are meant for
>>> | outside use), and (2) call the library's functions.  If you wish to do
>>> | (1), you can do it immediately--there is no need to wait for when the
>>> | library is loaded.  To do (2), you must load the library (preferably
>>> | with `require').
>>> | 
>>> |    But it is OK to use `eval-after-load' in your personal
>>> | customizations if you don't feel they must meet the design standards
>>> | for programs meant for wider use.
>>> `----
>>>
>>> Best,
>>
>> It's in ELISP info manual, under "Emacs Lisp Coding Conventions".
>
> Yes. for convenience:
>
> ,----[ (info "(elisp)Coding Conventions") ]
> | * Avoid using `eval-after-load' in libraries and packages (*note
> |   Hooks for Loading::).  This feature is meant for personal
> |   customizations; using it in a Lisp program is unclean, because it
> |   modifies the behavior of another Lisp file in a way that's not
> |   visible in that file.  This is an obstacle for debugging, much
> |   like advising a function in the other package.
> `----

It strikes me as no more unclean than loading a package and then calling
any of its "public" interfaces. Since for sure the package maintainer
does not know what the end user has called or modified after it is
loaded regardless.

I fail to see the weakness in (pseudo code)

"After the package has loaded perform the following possibly time
consuming tasks"

compared to

"load the package and perform the following possibly time consuming
tasks".

other than the first one is likely much more efficient in terms of it being
autoloaded when needed. 

Re: Why js2-mode in Emacs 23.2?

[Carsten Dominik]

On Aug 10, 2009, at 10:44 AM, Leo wrote:

> On 2009-08-10 01:49 +0100, Daniel Colascione wrote:
> [...]
>>> eval-after-load should not be used in emacs packages.
>>
>> Why not?
>
> On 2009-08-10 08:06 +0100, Carsten Dominik wrote:
> [...]
>>> eval-after-load should not be used in emacs packages.
>>
>> Why not?
>>
>> - Carsten
>
> I seem to recall RMS said that sometime ago. It is also documented  in
> the elisp manual:
>
> ,----[ (info "(elisp)Hooks for Loading") ]
> |    In general, well-designed Lisp programs should not use this  
> feature.
> | The clean and modular ways to interact with a Lisp library are (1)
> | examine and set the library's variables (those which are meant for
> | outside use), and (2) call the library's functions.  If you wish  
> to do
> | (1), you can do it immediately--there is no need to wait for when  
> the
> | library is loaded.  To do (2), you must load the library (preferably
> | with `require').
> |
> |    But it is OK to use `eval-after-load' in your personal
> | customizations if you don't feel they must meet the design standards
> | for programs meant for wider use.
> `----


I see.  Still, I can imagine good use cases for it.  For example,
if your package needs to do some tweaking to correctly interact with
some library, but if that library is only used by very few people.
In this case you'd want to avoid loading the library just to apply  
your tweaks.
I'd say this is a good case for using eval-after-load in a normal  
Emacs library.

Here is an example from Org-mode:

(eval-after-load "imenu"
   '(progn
      (add-hook 'imenu-after-jump-hook
	       (lambda ()
		 (if (eq major-mode 'org-mode)
		     (org-show-context 'org-goto))))))

When imenu jumps into a location in a file that is hidden by outline  
folding,
then the context of point needs to be revealed.  However, loading imenu
just to do this setup is useless for user who do use imenu at all.


- Carsten

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 7:51 AM, Xah Lee<xahlee@gmail.com> wrote:

> it'd be nice if emacs provides syntax parsing infrastructure.... nmxl, js2,
> cedet, can be leads in this direction and see how things work out.


I think CEDET includes a framework for building parsers.

Beside that it can also use other parsers. As I understand it it is
for example possible to (in priinciple use the nxml-mode parser in
CEDET, but you could also write a new xml parser using Semantic (which
is part of CEDET).

Re: Why js2-mode in Emacs 23.2?

[Leo]

On 2009-08-10 11:41 +0100, Carsten Dominik wrote:
> I see. Still, I can imagine good use cases for it. For example, if
> your package needs to do some tweaking to correctly interact with some
> library, but if that library is only used by very few people. In this
> case you'd want to avoid loading the library just to apply your
> tweaks. I'd say this is a good case for using eval-after-load in a
> normal Emacs library.
> 
> Here is an example from Org-mode:
>
> (eval-after-load "imenu"
>   '(progn
>      (add-hook 'imenu-after-jump-hook
> 	       (lambda ()
> 		 (if (eq major-mode 'org-mode)
> 		     (org-show-context 'org-goto))))))
>
> When imenu jumps into a location in a file that is hidden by outline
> folding, then the context of point needs to be revealed. However,
> loading imenu just to do this setup is useless for user who do use
> imenu at all.

I heavily use eval-after-load in my .emacs.

It seems to me that when there is a need to use eval-after-load in a
package included in Emacs some bad-design is happening. For example,
imenu probably needs to provide a better interface for other packages to
interact with it. What org is doing, in this case, is using
eval-after-load to workaround it. Anyway I am not sure, maybe RMS or the
maintainer could clarify on the use of eval-after-load.

Best,

Leo

-- 
Leo's Emacs uptime: 6 days, 12 hours, 54 minutes, 23 seconds

Re: Why js2-mode in Emacs 23.2?

[Stefan Monnier]

> I am not sure I understand this, but thinking of it again, could not
> Daniel use a font-lock-fontify-region-function that first does a
> fontification without keywords (if it is there it is needed).

You mean that there can be workarounds; well sure there might be.
But font-lock was not written with this use in mind, so I'd
expect snags.  And there's no reason why it should be bound to
font-lock.  And we still need to make sure that this gets called
properly when doing forward-sexp, etc...


        Stefan

Re: Why js2-mode in Emacs 23.2?

[Stephen Eilert]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

On Sun, Aug 9, 2009 at 9:48 PM, Daniel Colascione <danc@merrillpress.com>wrote:

> On Aug 9, 2009, at 8:32 PM, Leo wrote:
>
>> Why not fix bugs in js2-mode? Having two packages providing similar
>> features is a burden to maintainers as well as users.
>>
>
> Because js2-mode is an abomination? Oh, it's not so bad that it's slow and
> unmaintained: that description applies to lots of code in Emacs. The problem
> is that js2-mode operates fundamentally differently from other modes,
> eschews font-lock, parses the buffer *asynchronously*, and generally goes
> against every possible convention for implementing a major mode.
>
>
>
I do not think that was done without a very good reason (and there's a
lengthy post explaining it), unless the author is a complete masochist.

This is most likely pointing out serious deficiencies in Emacs
infrastructure that have been neglected.


--Stephen

programmer, n:
       A red eyed, mumbling mammal capable of conversing with inanimate
monsters.

[-- Attachment #2: Type: text/html, Size: 1493 bytes --]

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 4:05 PM, Stephen Eilert<spedrosa@gmail.com> wrote:

> I do not think that was done without a very good reason (and there's a
> lengthy post explaining it), unless the author is a complete masochist.
>
> This is most likely pointing out serious deficiencies in Emacs
> infrastructure that have been neglected.

Where is that post?

Re: Why js2-mode in Emacs 23.2?

[Deniz Dogan]

2009/8/10 Stephen Eilert <spedrosa@gmail.com>:
>
>
> On Sun, Aug 9, 2009 at 9:48 PM, Daniel Colascione <danc@merrillpress.com>
> wrote:
>>
>> On Aug 9, 2009, at 8:32 PM, Leo wrote:
>>>
>>> Why not fix bugs in js2-mode? Having two packages providing similar
>>> features is a burden to maintainers as well as users.
>>
>> Because js2-mode is an abomination? Oh, it's not so bad that it's slow and
>> unmaintained: that description applies to lots of code in Emacs. The problem
>> is that js2-mode operates fundamentally differently from other modes,
>> eschews font-lock, parses the buffer *asynchronously*, and generally goes
>> against every possible convention for implementing a major mode.
>>
>>
>
> I do not think that was done without a very good reason (and there's a
> lengthy post explaining it), unless the author is a complete masochist.
>
> This is most likely pointing out serious deficiencies in Emacs
> infrastructure that have been neglected.
>
>
> --Stephen

Has someone invited Steve to join this discussion yet?

-- 
Deniz Dogan

Re: Why js2-mode in Emacs 23.2?

[Deniz Dogan]

3

2009/8/10 Lennart Borgman <lennart.borgman@gmail.com>:
> On Mon, Aug 10, 2009 at 4:05 PM, Stephen Eilert<spedrosa@gmail.com> wrote:
>
>> I do not think that was done without a very good reason (and there's a
>> lengthy post explaining it), unless the author is a complete masochist.
>>
>> This is most likely pointing out serious deficiencies in Emacs
>> infrastructure that have been neglected.
>
> Where is that post?
>
>
>

I'm assuming this is what Stephen was referring to:
http://steve-yegge.blogspot.com/2008/03/js2-mode-new-javascript-mode-for-emacs.html

-- 
Deniz Dogan

Re: Why js2-mode in Emacs 23.2?

[Stefan Monnier]

>> So I do believe js2-mode is maintained.  Also, while it it unusual,
>> I think it's a very interesting experiment in how Emacs could evolve
>> (similarly to CEDET).

> "Evolve"?  Is js2-mode's roll-everything-from-scratch code better than
> the existing infrastructure that it ignores (e.g. font-lock)?

;-)

I'm talking about the underlying algorithms.  E.g. I'd like to see
Emacs's infrastructure move towards providing general parsing technology
for all modes.  So the idea is that the functionality offered by
js2-mode should at some point be doable without having to
roll-everything-from-scratch.


        Stefan

Re: Why js2-mode in Emacs 23.2?

[Stefan Monnier]

> I see no reason to make javascript-mode the default for JavaScript
> files.

Beep! Thanks for playing!
Bug you misunderstood: javascript-mode should be an alias either for
espresso-mode or for js2-mode.

> What I would really *not* like to see though is js2-mode as
> the default.

I don't care which default is chosen.  Just please agree on one.


        Stefan

Re: Why js2-mode in Emacs 23.2?

[Stefan Monnier]

> It seems to me that when there is a need to use eval-after-load in a
> package included in Emacs some bad-design is happening. For example,
> imenu probably needs to provide a better interface for other packages to
> interact with it.

Exactly.  It's just like defadvice: it's a great tool for users (and
external package maintainers), and it can be handy when the right fix
is hard to get (e.g. requires a serious redesign or heavy
argumentation), but it's still in the "to be avoided" category.


        Stefan

Re: Why js2-mode in Emacs 23.2?

[Lennart Borgman]

On Mon, Aug 10, 2009 at 4:41 PM, Deniz Dogan<deniz.a.m.dogan@gmail.com> wrote:

> Has someone invited Steve to join this discussion yet?


I think he is reading Emacs devel, at least he wrote some messages a
few days ago.

I would be surprised if he had missed this thread, but you might
perhaps mail him privately and ask.

eval-after-load not harmful after all (Was: Re: Why js-2mode?)

[Daniel Colascione]

On Aug 10, 2009, at 5:26 AM, Leo wrote:
> On 2009-08-10 09:54 +0100, CHENG Gao wrote:
> ,----[ (info "(elisp)Coding Conventions") ]
> | * Avoid using `eval-after-load' in libraries and packages (*note
> |   Hooks for Loading::).  This feature is meant for personal
> |   customizations; using it in a Lisp program is unclean, because it
> |   modifies the behavior of another Lisp file in a way that's not
> |   visible in that file.  This is an obstacle for debugging, much
> |   like advising a function in the other package.

Actually, I think the use of eval-after-load can be a *good* thing.

As other people have mentioned, sometimes the author of a mode would  
like to add support to some other mode that provides a certain  
feature, e.g., hideshow. Requiring that rarely-used feature mode is  
inefficient, so what seems to happen in practice is that these feature  
mdoes grow lists of major modes with which they word. Consider hs- 
minor-mode, which by default has c-mode, c++-mode, bibtex-mode, and  
java-mode, or speeedbar's speedbar-obj-alist. The problem is that  
these lists are seldom updated, and more importantly, cannot really be  
exhaustive. In fact, these lists are a blatant layering violation,  
because lower-level functions like hideshow should not be required to  
know anything in particular about C.

It's not really reasonable to expect the user to fill in the gaps.  
There are too many of them (the number of modes times the number of  
additional feature modes), and the user might not know enough to be  
able to customize the feature mode appropriately.

eval-after-load provides a perfect solution. If foo-mode can detect  
that hideshow has been loaded, it can customize hideshow to handle foo  
primitives appropriately --- after all, it's foo-mode --- and still  
not incur the overhead of loading hideshow every time. (Also, hideshow  
doesn't need to be patched to add support for foo.)

Granted, eval-after-load does have the potential to enable spooky  
action at a distance and make code hard to understand. But I don't  
think that fear is relevant when eval-after-load forms call functions   
specifically designed to be used by third parties to add support for a  
new environment of some sort.

I think the coding convention should be revised.

RE: eval-after-load not harmful after all (Was: Re: Why js-2mode?)

[Drew Adams]

In Icicles, I sometimes `require' other libraries explicitly. But in a few cases
I use `eval-after-load' instead - for exactly the kind of thing that Carsten
described.

I agree with Daniel, Carsten, and others that `eval-after-load' should not be
demonized.

The manual's text should be changed to suggest using it only when appropriate -
and then describe when it might be appropriate. IOW, we should _explain_ this,
so users understand it.


---

The manual sometimes seems to have difficulty dealing with the tradeoff of
providing (a) simple guidelines and (b) information that can suggest behavior
that runs counter to the simple guidelines in particular contexts. (No, I don't
have specific passages in mind, off the top of my head.)

The Elisp manual is above all a reference manual. Users expect it to be fairly
complete, even if that sometimes means presenting complex material that is not
black-and-white.

The idea is to _both_ (a) give the 80-95% general guideline (and its rationale)
and (b) give additional info that can address the other 20-5% of use cases.
What's important is to explain what's happening and why: rationale, use cases,
maybe examples.

The two, opposite dangers are:

(1) Oversimplifying, presenting only a "rule" that gets memorized, promulgated,
and promoted like a catechism. Not sufficiently teaching the why: what's
happening, rationale, exceptions, use cases.

(2) Not sufficiently drawing attention to the main practice that we recommend
for general use most of the time. Drowning the reader in a complex mass of
countervailing considerations.

It's not necessarily easy to get this right, to handle both the general case and
the specifics. Providing rationale and reasoning goes a long way to presenting
these two sides. The goal, after all, is not just respect of "the rule" but user
understanding.

Re: eval-after-load not harmful after all

[Stefan Monnier]

> I agree with Daniel, Carsten, and others that `eval-after-load' should
> not be demonized.

I don't know what makes you think it's "demonized".  We explicitly
support and document the feature, so we clearly think it has value.
But we also want to make it clear that within Emacs's own code, its use
is discouraged (but not disallowed).


        Stefan

"Font-lock is limited to text matching" is a myth

[Daniel Colascione]

On Monday 10 August 2009, Xah Lee wrote:
> emacs's syntax coloring is based on text pattern matching. It might work
> faster or more smooth right now, but from technology point of view, its one
> generation behind.

This is a common misconception I'd like to dispel. font-lock is *NOT* limited to using regular expressions. Yes, regular expressions are its default mode of operation, and for most languages, they are sufficient. But font-lock has the ability to mix and match several different kinds of fontification. Let's not reinvent the wheel.

There are two approaches to go beyond regular expression matching in font-lock: first, one can use a function as a font-lock matcher instead of a regular expression. That function can be arbitrarily complex, and can look at as much of the buffer as is necessary to fontify correctly. When this function locates a pattern to match, it returns control to font-lock, which fontifies it like any other interesting tidbit it found.

If that's not good enough, you can also have your font-lock "matcher" actually perform the fontification itself. cc-mode, espresso-mode, and nxml-mode use this technique. The basic idea is to find interesting patterns in the region of interest, then call font-lock-apply-highlight on them to perform the actual fontification. Because the "matcher" function is still using the font-lock infrastructure, font-lock will respect the fontification and behave correctly in other respects.

Using one of these two techniques allows a mode to use arbitrarily-complex fontification while still integrating with user-supplied keywords (e.g., from font-lock-add-keywords), various font-lock enhancement modes like jit-lock, and all other normal Emacs infrastructure.

It's *really* not necessary to bypass font-lock to get fontification to be as complex as you want. Just look at cc-mode.

> it'd be nice if emacs provides syntax parsing infrastructure.... nmxl, js2,
> cedet, can be leads in this direction and see how things work out.

It already does. It's called font-lock. If you want some kind of infrastructure that uses a grammar to perform fontification, I believe CEDET, which will be included in Emacs, can be wired up that way [I think? I'm not quite sure whether Semantic's tags are generic enough for fontification].

But using font-lock is not "a generation behind". Really, most editors' hard-coded highlighting features are a generation behind font-lock!

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

2009/8/10 Daniel Colascione <danc@merrillprint.com>:

> If that's not good enough, you can also have your font-lock "matcher" actually perform the fontification itself. cc-mode, espresso-mode, and nxml-mode use this technique. The basic idea is to find interesting patterns in the region of interest, then call font-lock-apply-highlight on them to perform the actual fontification. Because the "matcher" function is still using the font-lock infrastructure, font-lock will respect the fontification and behave correctly in other respects.


mumamo uses this font-lock for dividing the buffer into chunks with
different major modes. It ties into the font-lock framework using
font-lock-fontify-region-function which I think is an entry point
parsers could and should use. (But that is more beliefe ..., someone
who knows better please tell me.)

You said that js2 could not use font-lock or did I misunderstand you?
I do not understand why it can't use that entry point. (Cc:ing Steve.)

If it can't then maybe there is something that could be added to font-lock?


>> it'd be nice if emacs provides syntax parsing infrastructure.... nmxl, js2,
>> cedet, can be leads in this direction and see how things work out.
>
> It already does. It's called font-lock. If you want some kind of infrastructure that uses a grammar to perform fontification, I believe CEDET, which will be included in Emacs, can be wired up that way [I think? I'm not quite sure whether Semantic's tags are generic enough for fontification].


I think Xah meant the grammars. Maybe CEDET is not yet using
font-lock, I do not know how old JIT-lock, which is an essential part
for this, is.

Thanks for a good overview, Daniel. Perhaps you want to explain once
again the difficulties that espresso-mode have with this? What is
missing? I have had a lot of trouble in mumamo, but I thought it was
because I am making things not really expected by the frame work (and
some things are missing in Emacs to really make the corner cases
possible AFAICS).

Re: "Font-lock is limited to text matching" is a myth

[Stefan Monnier]

>> it'd be nice if emacs provides syntax parsing infrastructure.... nmxl, js2,
>> cedet, can be leads in this direction and see how things work out.
> It already does. It's called font-lock.

No, font-lock doesn't do any parsing.  parse-partial-sexp does a very
limited kind of parsing, and font-lock makes use of it, but font-lock
itself doesn't do any parsing.
Your font-lock-keywords may of course do an arbitrarily complex form of
parsing, but it's still not font-lock that does it.


        Stefan

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Mon, Aug 10, 2009 at 8:47 PM, Stefan Monnier<monnier@iro.umontreal.ca> wrote:
>>> it'd be nice if emacs provides syntax parsing infrastructure.... nmxl, js2,
>>> cedet, can be leads in this direction and see how things work out.
>> It already does. It's called font-lock.
>
> No, font-lock doesn't do any parsing.  parse-partial-sexp does a very
> limited kind of parsing, and font-lock makes use of it, but font-lock
> itself doesn't do any parsing.


I think Daniel suggested that it is a frame-work that can be used for
book-keeping of how far parsing has gone (the state must be kept
elsewhere) and parsing while Emacs is idle. Don't you think it can be
used for that?


> Your font-lock-keywords may of course do an arbitrarily complex form of
> parsing, but it's still not font-lock that does it.
>
>
>        Stefan
>

Re: Why js2-mode in Emacs 23.2?

[Stephen Eilert]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]

On Mon, Aug 10, 2009 at 11:37 AM, Lennart Borgman <lennart.borgman@gmail.com
> wrote:

> On Mon, Aug 10, 2009 at 4:05 PM, Stephen Eilert<spedrosa@gmail.com> wrote:
>
> > I do not think that was done without a very good reason (and there's a
> > lengthy post explaining it), unless the author is a complete masochist.
> >
> > This is most likely pointing out serious deficiencies in Emacs
> > infrastructure that have been neglected.
>
> Where is that post?
>

Not here, but in his blog.

http://steve-yegge.blogspot.com/2008/03/js2-mode-new-javascript-mode-for-emacs.html


--Stephen

programmer, n:
       A red eyed, mumbling mammal capable of conversing with inanimate
monsters.

[-- Attachment #2: Type: text/html, Size: 1211 bytes --]

Re: "Font-lock is limited to text matching" is a myth

[David Engster]

[Added Eric Ludlam to Cc]

Lennart Borgman <lennart.borgman@gmail.com> writes:
> 2009/8/10 Daniel Colascione <danc@merrillprint.com>:
>> It already does. It's called font-lock. If you want some kind of
>> infrastructure that uses a grammar to perform fontification, I
>> believe CEDET, which will be included in Emacs, can be wired up that
>> way [I think? I'm not quite sure whether Semantic's tags are generic
>> enough for fontification].
>
>
> I think Xah meant the grammars. Maybe CEDET is not yet using
> font-lock, I do not know how old JIT-lock, which is an essential part
> for this, is.

CEDET can create formatted strings from the tags, and it supports using
font-lock for that (see file semantic-format.el). There's also the
minor-mode 'semantic-decoration' (semantic-decorate.el) which applies
styles to certain tags in the buffer while editing, and this mode does
not use font-lock. Since Semantic usually does not parse every
expression in the buffer, this decoration is currently limited to stuff
like include statements, function declarations or protected class
members. (Eric, please correct me if I'm wrong here).

-David

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Mon, Aug 10, 2009 at 10:42 PM, David Engster<deng@randomsample.de> wrote:
>> I think Xah meant the grammars. Maybe CEDET is not yet using
>> font-lock, I do not know how old JIT-lock, which is an essential part
>> for this, is.
>
> CEDET can create formatted strings from the tags, and it supports using
> font-lock for that (see file semantic-format.el). There's also the
> minor-mode 'semantic-decoration' (semantic-decorate.el) which applies
> styles to certain tags in the buffer while editing, and this mode does
> not use font-lock.


Thanks David. What is the reason that font-lock is not used there?

I do not know much about it, but am a bit curious. Does CEDET use the
font-lock/JIT-lock frame work for handling parsing requirements?


> Since Semantic usually does not parse every
> expression in the buffer, this decoration is currently limited to stuff
> like include statements, function declarations or protected class
> members. (Eric, please correct me if I'm wrong here).
>
> -David
>

Re: "Font-lock is limited to text matching" is a myth

[Eric M. Ludlam]

Hi,

  CEDET, or more specifically, the Semantic parser/generator framework
uses font-lock for the following:

  * Pre-defined faces for colorizing text strings.
  * A simple test to see if point is in a comment or string.  (ie - not
code.)

  The concept of using the Semantic parser/generator framework for
improving font-locking accuracy has come up many times.  No-one to my
knowledge has attempted to mix the two.

  You can create a parser with the parser generator, and in the grammar
you create, put any kind of Lisp code you want.  You can also define
your own grammar-only macros.  (ie. there is a TAG macro for creating a
new tag at some point in a grammar.)  Thus, you could create a 'FONT'
macro, and use it to colorize a buffer while parsing the buffer.

  The PROS are that you could use a formal grammar to choose coloring
patterns.  That grammar could be shared by the tagging system.

  The CONS are that everything in Semantic is set up to parse the entire
buffer in one pass, and to parse logical sub-sections only after a full
parse has been done.

  Font lock/jit-lock is much better in the way it colorizes buffer
sections, and also the way it does so just before display (or so it
seems to me anyway) which you don't want to do w/ Semantic due to the
overhead of the parsing system.

  It probably DOES make sense to mix the two.  For example, font-lock
could refer to the tagging state to determine the logical bounds of an
edited entity, cutting down on partial expression matching problems.
Font lock could even be rigged up to just use the lexical analyzer for
some analysis if that makes sense.

  I would imagine that the parsing engine in Semantic, if it is deemed
critical by the maintainers, will get faster if key components are
integrated into the C code.  I would imagine a font-lock replacement
could be implemented fairly easily, but getting it to work as quickly as
font lock would take a fair effort.

  Lastly, as David Engster stated, CEDET has decoration tools that
decorate entire tags in some way, such as putting a line on top of
functions.  This is a separate decoration activity not related to font
lock, and something font lock would not be able to do reliably.

Enjoy
Eric


On Mon, 2009-08-10 at 22:51 +0200, Lennart Borgman wrote:
> On Mon, Aug 10, 2009 at 10:42 PM, David Engster<deng@randomsample.de> wrote:
> >> I think Xah meant the grammars. Maybe CEDET is not yet using
> >> font-lock, I do not know how old JIT-lock, which is an essential part
> >> for this, is.
> >
> > CEDET can create formatted strings from the tags, and it supports using
> > font-lock for that (see file semantic-format.el). There's also the
> > minor-mode 'semantic-decoration' (semantic-decorate.el) which applies
> > styles to certain tags in the buffer while editing, and this mode does
> > not use font-lock.
> 
> 
> Thanks David. What is the reason that font-lock is not used there?
> 
> I do not know much about it, but am a bit curious. Does CEDET use the
> font-lock/JIT-lock frame work for handling parsing requirements?
> 
> 
> > Since Semantic usually does not parse every
> > expression in the buffer, this decoration is currently limited to stuff
> > like include statements, function declarations or protected class
> > members. (Eric, please correct me if I'm wrong here).
> >
> > -David
> >

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Tue, Aug 11, 2009 at 12:06 AM, Eric M. Ludlam<eric@siege-engine.com> wrote:

Hi Eric,

>  The concept of using the Semantic parser/generator framework for
> improving font-locking accuracy has come up many times.  No-one to my
> knowledge has attempted to mix the two.


Maybe that can easier be done if Semantic parser use
font-lock/JIT-lock timers and marking to keep track of what need to be
reparsed? (It is just a wild idea perhaps.)


>  The CONS are that everything in Semantic is set up to parse the entire
> buffer in one pass, and to parse logical sub-sections only after a full
> parse has been done.


So you do a first pass with coarse parsing and then you look in the
sub-sections for details? Is this strictly necessary? I guess you are
looking for top level definitions in the first pass?

Could that pass have its own state and continue upon demand (when an
item is not recognized) or is such a logic impossible?

(I guess font-lock/JIT-lock could be improved to help with keeping
track of what parts of the buffer that have been parsed/maybe
fontified.)


>  I would imagine that the parsing engine in Semantic, if it is deemed
> critical by the maintainers, will get faster if key components are
> integrated into the C code.

Is that part stable?


> I would imagine a font-lock replacement
> could be implemented fairly easily, but getting it to work as quickly as
> font lock would take a fair effort.

Maybe a cooperation would be better.


>  Lastly, as David Engster stated, CEDET has decoration tools that
> decorate entire tags in some way, such as putting a line on top of
> functions.  This is a separate decoration activity not related to font
> lock, and something font lock would not be able to do reliably.

Why not if it asks the parser?


> Enjoy
> Eric

RE: eval-after-load not harmful after all (Was: Re: Why js-2mode?)

[Stephen J. Turnbull]

Drew Adams writes:
 > In Icicles, I sometimes `require' other libraries explicitly. But in a few cases
 > I use `eval-after-load' instead - for exactly the kind of thing that Carsten
 > described.
 > 
 > I agree with Daniel, Carsten, and others that `eval-after-load' should not be
 > demonized.

I have no objection to use of `eval-after-load' in user or site files
that are disabled by emacs -q.  But `eval-after-load' in core files is
like smoking in an elevator next to a pregnant woman.  It's very
convenient, maybe even "necessary", from the point of view of the
user of the feature, but he doesn't bear the pain of debugging a
Heisenbug that appears for the user of random-package-using-e-a-l.

I note that all the advocates of e-a-l are package maintainers and UI
types; all the deprecators are core.  This is an inherent tension, and
I think it should be resolved in favor of protecting the core.

RE: eval-after-load not harmful after all (Was: Re: Why js-2mode?)

[Drew Adams]

> I have no objection to use of `eval-after-load' in user or site files
> that are disabled by emacs -q.  But `eval-after-load' in core files is
> like smoking in an elevator next to a pregnant woman.  It's very
> convenient, maybe even "necessary", from the point of view of the
> user of the feature, but he doesn't bear the pain of debugging a
> Heisenbug that appears for the user of random-package-using-e-a-l.
> 
> I note that all the advocates of e-a-l are package maintainers and UI
> types; all the deprecators are core.  This is an inherent tension, and
> I think it should be resolved in favor of protecting the core.

Nothing wrong with "protecting the core".

The discussion was about the Elisp manual, whose target audience is not just
maintainers of the Emacs core. The help and guidance there are used by a variety
of programmer-users.

Again, there's nothing wrong with (a) giving a general guideline, and also (b)
explaining the issues and giving additional info about contexts where the
guideline might not be something you would want to follow. It's about furthering
understanding, not just rote learning of rules.

Re: eval-after-load not harmful after all (Was: Re: Why js-2mode?)

[Lennart Borgman]

On Tue, Aug 11, 2009 at 2:43 AM, Stephen J. Turnbull<stephen@xemacs.org> wrote:
> Drew Adams writes:
>  > In Icicles, I sometimes `require' other libraries explicitly. But in a few cases
>  > I use `eval-after-load' instead - for exactly the kind of thing that Carsten
>  > described.
>  >
>  > I agree with Daniel, Carsten, and others that `eval-after-load' should not be
>  > demonized.
>
> I have no objection to use of `eval-after-load' in user or site files
> that are disabled by emacs -q.  But `eval-after-load' in core files is
> like smoking in an elevator next to a pregnant woman.  It's very
> convenient, maybe even "necessary", from the point of view of the
> user of the feature, but he doesn't bear the pain of debugging a
> Heisenbug that appears for the user of random-package-using-e-a-l.
>
> I note that all the advocates of e-a-l are package maintainers and UI
> types; all the deprecators are core.  This is an inherent tension, and
> I think it should be resolved in favor of protecting the core.


It is not only hesienbugs that are the problem. It is also package
cooperation, version info regarding this and platform independence.
All these issues tend to be neglected by less experienced developers.
(Naturally, since this is the most difficult part, perhaps.)

Perhaps better infrastructure for this, including support in Emacs,
might help with this.

Re: Why js2-mode in Emacs 23.2?

[Glenn Morris]

Carsten Dominik wrote:

> I'd say this is a good case for using eval-after-load in a normal
> Emacs library.
>
> Here is an example from Org-mode:
>
> (eval-after-load "imenu"
>   '(progn
>      (add-hook 'imenu-after-jump-hook
> 	       (lambda ()
> 		 (if (eq major-mode 'org-mode)
> 		     (org-show-context 'org-goto))))))
[...]
> However, loading imenu just to do this setup is useless for user who
> do use imenu at all.

That would work just fine without eval-after-load or requiring imenu.
A hook does not need to be defined before calling add-hook.

Re: "Font-lock is limited to text matching" is a myth

[Eric M. Ludlam]

On Tue, 2009-08-11 at 00:19 +0200, Lennart Borgman wrote:
> On Tue, Aug 11, 2009 at 12:06 AM, Eric M. Ludlam<eric@siege-engine.com> wrote:
> 
> Hi Eric,
> 
> >  The concept of using the Semantic parser/generator framework for
> > improving font-locking accuracy has come up many times.  No-one to my
> > knowledge has attempted to mix the two.
> 
> 
> Maybe that can easier be done if Semantic parser use
> font-lock/JIT-lock timers and marking to keep track of what need to be
> reparsed? (It is just a wild idea perhaps.)

I'm not certain of how the font/jit lock works.  Semantic works by
tracking edits (after-change-functions) and then on it's own timer, it
coalesces the changes into parsable units.  It then reparses those
units.

Font lock can refontify based on fairly small subsections of a buffer,
such as a single code line, or a comment section.  Semantic's
subsections are the size of functions, variables, and datatypes (ie, the
tags it creates.)

> >  The CONS are that everything in Semantic is set up to parse the entire
> > buffer in one pass, and to parse logical sub-sections only after a full
> > parse has been done.
> 
> 
> So you do a first pass with coarse parsing and then you look in the
> sub-sections for details? Is this strictly necessary? I guess you are
> looking for top level definitions in the first pass?
> 
> Could that pass have its own state and continue upon demand (when an
> item is not recognized) or is such a logic impossible?

It could, but I have not done so.  Tagging information is not generally
needed right away, so just waiting for the user to either ask for it, or
sit idle for a while works pretty well.  The overhead of such an
incremental parser isn't really needed.

> (I guess font-lock/JIT-lock could be improved to help with keeping
> track of what parts of the buffer that have been parsed/maybe
> fontified.)

The needs between the tagging parser and the font-lock parser are
different.  Font lock needs to colorize arbitrary blocks of text, and a
tagging parser needs to parse everything, but only needs the data
periodically.

Converting a tagging parser to a colorizing parser would be challenging
because of these different uses.

> >  I would imagine that the parsing engine in Semantic, if it is deemed
> > critical by the maintainers, will get faster if key components are
> > integrated into the C code.
> 
> Is that part stable?

Yes.  Not much is going on there.

> >  Lastly, as David Engster stated, CEDET has decoration tools that
> > decorate entire tags in some way, such as putting a line on top of
> > functions.  This is a separate decoration activity not related to font
> > lock, and something font lock would not be able to do reliably.
> 
> Why not if it asks the parser?

Font lock runs long before the parser bothers trying to catch up.  Font
lock would needs hooks for after the parser runs.
problems.

While font lock and semantic share a need for a parsing infrastructure,
the where/when of the parsing is quite different.  It is possible to
conceptually mix and match the parsers vs the schedulers.  In practice,
the two tools have their own lengthy histories that will make that
challenging.  Before tackling such a project, it would be wise to take
multi-mode (or similar tool) into account.

Eric

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 10, 2009, at 9:13 PM, Glenn Morris wrote:
> Carsten Dominik wrote:
>>
>>     (add-hook 'imenu-after-jump-hook
> That would work just fine without eval-after-load or requiring imenu.
> A hook does not need to be defined before calling add-hook.

That works for hooks, but not for arbitrary data structures, like,  
say, hs-special-modes-alist. 

Re: eval-after-load not harmful after all (Was: Re: Why js-2mode?)

[Daniel Colascione]

On Aug 10, 2009, at 8:43 PM, Stephen J. Turnbull wrote:

> Drew Adams writes:
>> In Icicles, I sometimes `require' other libraries explicitly. But  
>> in a few cases
>> I use `eval-after-load' instead - for exactly the kind of thing  
>> that Carsten
>> described.
>>
>> I agree with Daniel, Carsten, and others that `eval-after-load'  
>> should not be
>> demonized.
>
> I have no objection to use of `eval-after-load' in user or site files
> that are disabled by emacs -q.  But `eval-after-load' in core files is
> like smoking in an elevator next to a pregnant woman.  It's very
> convenient, maybe even "necessary", from the point of view of the
> user of the feature, but he doesn't bear the pain of debugging a
> Heisenbug that appears for the user of random-package-using-e-a-l.

You haven't actually addressed the reason for using e-a-l and provided  
alternatives, nor have you provided any concrete examples of the harm  
e-a-l might cause. You've just put "necessary" in scare quotes without  
even considering the idea that it might actually *be* necessary.

If packages M needs to work with optional package Q, then there are  
two approaches: either M can use eval-after-load to customize Q, or Q  
can include a list of every possible M with which it can be used. You  
might decry the former, but I don't think the latter is any more  
maintainable. 

Re: "Font-lock is limited to text matching" is a myth

[Stefan Monnier]

> I think Daniel suggested that it is a frame-work that can be used for
> book-keeping of how far parsing has gone (the state must be kept
> elsewhere) and parsing while Emacs is idle.  Don't you think it can be
> used for that?

As it stands, it's not really adapted to it.
I mean, it's OK and there's worse, but it's not great.  It's designed
mostly to keep track of what has been highlighted (without paying too
much attention to potential dependencies between chunks), and it is
designed to rehighlight the parts that change and that are displayed.

For parsing, we generally assume sequential dependencies (e.g. can't
parse the end of the buffer before parsing the beginning), so
font-lock's tracking is not very well adapted.  And parsing may need to
be applied to chunks of text that has not yet and will never
be displayed.

Basically font-lock-syntactic-keywords is the part of font-lock that was
"designed" to handle the parsing side (together with syntax-table), but
it deserves to be improved significantly and it should be moved out of
font-lock since it provides services that are useful/needed by many
other packages, and in buffers where font-lock might not even
be desired.


        Stefan

Re: Why js2-mode in Emacs 23.2?

[Dan Nicolaescu]

Daniel Colascione <danc@merrillpress.com> writes:

  > On Aug 10, 2009, at 9:13 PM, Glenn Morris wrote:
  > > Carsten Dominik wrote:
  > >>
  > >>     (add-hook 'imenu-after-jump-hook
  > > That would work just fine without eval-after-load or requiring imenu.
  > > A hook does not need to be defined before calling add-hook.
  > 
  > That works for hooks, but not for arbitrary data structures, like,
  > say, hs-special-modes-alist. 

For hs-special-modes-alist, if the mode is included in emacs, just add
the initialization where in the definition.  That goes for other similar
structures. 

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 11, 2009, at 12:28 AM, Dan Nicolaescu wrote:

> Daniel Colascione <danc@merrillpress.com> writes:
>
>> On Aug 10, 2009, at 9:13 PM, Glenn Morris wrote:
>>> Carsten Dominik wrote:
>>>>
>>>>    (add-hook 'imenu-after-jump-hook
>>> That would work just fine without eval-after-load or requiring  
>>> imenu.
>>> A hook does not need to be defined before calling add-hook.
>>
>> That works for hooks, but not for arbitrary data structures, like,
>> say, hs-special-modes-alist.
>
> For hs-special-modes-alist, if the mode is included in emacs, just add
> the initialization where in the definition.  That goes for other  
> similar
> structures.
>

Why should hideshow have to include a list of modes? And why should  
the preferred technique differ depending on whether the code is part  
of emacs or not?

Re: Why js2-mode in Emacs 23.2?

[Glenn Morris]

Daniel Colascione wrote:

> That works for hooks, but not for arbitrary data structures, like,  
> say, hs-special-modes-alist. 

Another bad example, since it is autoloaded.

Re: Why js2-mode in Emacs 23.2?

[Dan Nicolaescu]

Daniel Colascione <danc@merrillpress.com> writes:

  > On Aug 11, 2009, at 12:28 AM, Dan Nicolaescu wrote:
  > 
  > > Daniel Colascione <danc@merrillpress.com> writes:
  > >
  > >> On Aug 10, 2009, at 9:13 PM, Glenn Morris wrote:
  > >>> Carsten Dominik wrote:
  > >>>>
  > >>>>    (add-hook 'imenu-after-jump-hook
  > >>> That would work just fine without eval-after-load or requiring
  > >>> imenu.
  > >>> A hook does not need to be defined before calling add-hook.
  > >>
  > >> That works for hooks, but not for arbitrary data structures, like,
  > >> say, hs-special-modes-alist.
  > >
  > > For hs-special-modes-alist, if the mode is included in emacs, just add
  > > the initialization where in the definition.  That goes for other
  > > similar
  > > structures.
  > >
  > 
  > Why should hideshow have to include a list of modes? 

Because it is the cleanest way given the current design?

  > And why should the preferred technique differ depending on whether
  > the code is part of emacs or not?

Because things that are included in emacs can act as being a part of a
whole and take advantage of that to make the code easier to read and
maintain. 

Re: Why js2-mode in Emacs 23.2?

[Daniel Colascione]

On Aug 11, 2009, at 12:39 AM, Dan Nicolaescu wrote:
>>> Why should hideshow have to include a list of modes?
>
> Because it is the cleanest way given the current design?

What makes eval-after-load unclear? It does exactly what it says it's  
going to do.

>> And why should the preferred technique differ depending on whether
>> the code is part of emacs or not?
>
> Because things that are included in emacs can act as being a part of a
> whole and take advantage of that to make the code easier to read and
> maintain.

Being "part of a whole" shouldn't mean being part of a quivering mound  
of hacks. Modularity is a good thing.

Re: "Font-lock is limited to text matching" is a myth

[Steve Yegge]

[-- Attachment #1: Type: text/plain, Size: 18845 bytes --]

Hello all,

Thanks for opening this can of, er, threads.  I was going to ask about
these things myself soon in any case, because it's clear that js2-mode
is not doing a very effective job of surfacing its rich information in
Emacs.  This is partly my fault, but it is also partly due to some
issues with font-lock that I'll describe in nauseating detail.

There are several important ideas being conflated in this thread that
I think need to be teased apart before we can talk responsibly about
any of them.  I've called out the top five conflations in sections
below delimited by roman numerals.

This is all in some sense an elaboration of what Eric Ludlam just
posted, to which I can only add my miserable +1.

Stephen Eilert wrote:
> I do not think that was done without a very good reason (and there's
  a lengthy post explaining it), unless the author is a complete
  masochist.

I don't think of myself that way.  Here, as requested, is a lengthy post
explaining my approach.  For the record, it could have been much
lengthier, and I have lengthy replies ready for all your objections and
concerns.  (Just in case you were wondering.)

I really do want to get this resolved, though.

I. Asynchronous parsing

js2-mode performs both syntactic and (some) semantic analysis.  It
knows, for instance, when you're using a symbol that's not defined in
its file.  js2-mode does not currently understand project structure,
but I'm doing some work in this area, and it may at some point gather
semantic information collected from several files.

Because this analysis requires parsing the entire file at least once
(see my discussion of partial/incremental parsing below), and it may
someday involve looking at symbol tables from other files, it seemed
best to run the parse asynchronously, so as not to interfere with the
user's editing.

One byproduct of having an accurate parser and symbol table is that
you can obtain style runs with relatively small effort, so js2-mode
does its own highlighting.  The downside is that this highlighting
information is unavailable at font-lock time, and it is not available
piecewise -- it's all-or-nothing.

There is a relatively simple alternative that might appease Daniel:
I could have js2-mode simply not do any highlighting by default,
except for errors and warnings.  We'd use whatever highlighting is
provided by espresso-mode, and users would be able to choose between
espresso-highlighting and js2-mode highlighting.  With the former,
they'd get "instantaneous" font-locking, albeit not as rich as what
js2-mode can provide.

This would be trivial to change.  I am actively maintaining js2-mode,
and the only reason I haven't checked in any changes since my initial
commit to the trunk is inexperience:  I'm trying to get a handle on how
many changes people tend to aggregate before checking in a change to
any given mode.  But I have several fixes (including some patches
contributed from users) that are ready to commit, and more on the way.

Errors and warnings would still need to be asynchronous (if they're
enabled).  So, too, would the imenu outline and my in-progress
buffer-based outline, which is somewhat nicer than the IMenu one.

But I think the main objection to js2-mode revolves around its
highlighting, correct?  If so, AND if we can solve the font-lock
integration issues, AND if we can fix the multi-mode issues (II
below), then I'm hopeful that js2-mode might become a reasonable
choice as the default editing mode for JavaScript.

I think espresso-mode is a fine fallback position.  Anything but
java-mode!  The default today is java-mode, and I had no qualms about
replacing it as the default for JavaScript.

Note: diagnostic messages in js2-mode are highlighted using overlays.
I tried using overlays for all highlighting but it was unacceptably
slow and had a tendency to crash Emacs.  But there are usually not
prohibitively many errors and warnings, since the error-recovery
algorithm is somewhat coarse-grained.  So error-reporting works
independently of font-lock.

II. Multi-mode support

JavaScript is especially needful of mumamo (or equivalent) multi-mode
support, because much of the JavaScript in the wild is embedded in
HTML, in template files, even in strings in other languages.

js2-mode does not support mumamo (or mmm-mode, which which I am
currently more familiar) because js2-mode's lexer needs to support
ignoring parts of the buffer.  I do not think this would be very
hard to implement, but I have not done it yet.

If I don't get to it before the next version of Emacs launches, then I
think this should effectively disqualify js2-mode from being the
default JavaScript mode.  It would be an inconsistent user experience
to have one JavaScript mode in .js files and another mode for
JavaScript inside multi-mode-enabled files.

I'm ready to give it a try, though, and I'll ping Lennart offline about
integrating the two somehow.

III. Incremental and partial parsing

Lennart and others have asked whether it is possible for js2-mode to
support partial or incremental parsing.  The short answer is
"incremental: yes; partial: no".

nxml-mode, last I checked, does incremental parsing.  It parses ahead
in the buffer, but then stops and saves its state.  If you jump forward
in the buffer, it resumes and continues the parse until some point
beyond the section you're viewing.

js2-mode could do it this way without much additional effort.  I chose
not to because once you've decided to use background parsing, it
doesn't seem like an especially useful optimization.  But I could see
it being helpful in some cases, such as when you're editing near the
top of a large file -- as long as the whole file isn't encased in some
top-level expression, which unfortunately is often the case in JS.

Partial parsing is a different beast entirely.  The goal of a partial
parser is to re-parse the minimum amount necessary, given some region
that has changed.  I've dug into this a bit, because originally I
wanted to support it in js2-mode.  I even made some progress on an
implementation.

While a few production parsers (for Java and JavaScript) have
implemented partial parsing, the vast majority of them do not support
it -- instead, they re-parse from the top.  They do this because the
incremental benefit of partial parsing is debatable, assuming you're
time- and resource-constrained, as most of us are.

I took a close look at Eclipse and IntelliJ, and even asked some
of their users to characterize the highlighting behavior of the IDE.
Without exception, the IDE users had internalized a ~1000 ms delay
in highlighting and error reporting as part of their normal workflow,
and they uniformly described it as "instant{aneous}" until I made
them time it.

I've been an Emacs user for 20+ years now, and like many I found
the idea of a parsing delay to be somewhere between "undesirable"
and "sickening".  But the majority of programmers today have
apparently learned not to notice delays of ~1sec as long as it
never interferes with their typing or indentation (see IV below).

So after looking at my ~8000 lines of elisp devoted to parsing
JavaScript, I weighed it and decided not to support partial parsing.
It's certainly possible to support it, but I think my time would be
better spent on things that average users are more likely to notice.

YMMV, of course.

The upshot is that if I'm going to support mumamo, it will need
to work within js2-mode's existing full-reparse framework.  I can
think of various ways to make it work, though, and as I mentioned
I'll talk to Lennart about it.

IV.  Indentation

The indentation in js2-mode is broken.  I'll be the first to say it.

It is based on the indentation in Karl Langstrom's mode, which does a
better job for JavaScript than any indenter based on cc-engine, but
that doesn't mean it's a good job.  And it's essentially unconfigurable.

espresso-mode shares this problem, which means that for this
important use case it is not an improvement over js2-mode.

Daniel's objections to js2-mode's non-interaction with font-lock
apply equally to the non-interaction with cc-engine's indentation
configuration system.  The indent configuration for JavaScript should
share as many settings as practical with cc-mode.

I actually made a serious attempt to generate the `c-style-alist'
data structure for js2-mode using the parse tree, but ran into three
issues:

  1) it's much harder than I thought it would be, even with a full
     parse tree available.  I had some 2000 lines of elisp invested
     in it when I pooped out, to be perfectly frank.

  2) `c-style-alist' (like font-lock) does not have enough semantic
      variables to encompass the range of indentation contexts that
      JavaScript programmers care about.  I think we'd need to add
      5-10 more, although it's been 18 months since I looked into it.

  3) indentation in "normal" Emacs modes also runs synchronously as
     the user types.  Waiting 500-800 msec or more for the parse to
     finish is (I think) not acceptable for indentation.  For small
     files the parse time is acceptable, but it would not be generally
     scalable.

#3 is the reason I gave up on #1.  It didn't seem to be worth the
effort to produce an accurate but slow indenter.

I don't know exactly how to solve this problem.  I have lots of
ideas, but it appears there are few low-hanging fruit in this space.

V. Font Lock framework design problems

There seems to be a common misconception flitting about to the
effect that font-lock is perfect and will never need to change.

This is a somewhat paradoxical viewpoint in view of the corpses
littering the path to jit-lock, which include font-lock, fast-lock,
lazy-lock, and vapor-lock.  Each decade we've had a cadre of people
claiming that *-lock meets everyone's needs, and then it gets rewritten
anyway.

So it's hard to understand how it remains such a popular viewpoint.

I'll make yet another attempt to dispel it, since once we're past the
emotional stumbling blocks, font-lock may be able to evolve again.

Va) Inadequate/insufficient style names

There are not enough font-lock faces to represent all the semantic
style runs that are identifiable to "real" language analyzers.
js2-mode makes several semantic distinctions not available in most
Emacs modes, although such distinctions are available in JDEE and
other Cedet-enabled modes, so js2-mode is by no means alone in its
needs.

In addition to the autoloaded font-lock faces, which js2-mode uses
whenever possible, js2-mode defines several new faces, including:

  * function parameters
  * "class" instance members (in JS, prototype and instance props)
  * local variables
  * undeclared variables
  * private members (although I implemented it poorly -- see below)
  * html/xml tags, attr names and delimiters -- used both for html
    in jsdoc comments and for E4X literals
  * doc tags such as those typically found in javadoc/jsdoc comments
  * warnings, errors, and informational diagnostics

I do not expect that this set is all-inclusive -- over time as js2-mode
and similar modes get smarter, they will be able to make other
semantic distinctions that users may wish to customize independently.
Given that Emacs is the most configurable editor on the planet, I do
not see any reason to entertain arguments to the contrary.

Vb) Ad-hoc default faces that are not being autoloaded

There are some modes (e.g. sgml-mode, html-mode, nxml-mode) that
define their own versions of some of the xml/html faces, but it did
not seem right to make js2-mode 'require one of these modes just to
get at ad-hoc "standard" definitions for these faces.

We should define standard faces for xml/html tags and entities, and
for any other faces that are effectively defined by 2 or more modes.

Vc) Additional semantic styles not needed by JavaScript

I have other language modes in progress, and together they define an
ever larger set of semantic styles.  The set of available font-lock
names should try to encompass the _union_ of the needs of most
languages, not the intersection.  There should, for instance, be a
font-lock-symbol-face for languages with distinguished symbols such
as Lisp, Scheme and Ruby.

I think this is relatively easy to fix, provided a little thought
goes into choosing the new faces.  Vd and Ve below should help
clarify why it requires greater than zero thought.

Vd) Composable semantic styles

Some font-lock faces represent "primary" semantic roles, in a vague
way.  For instance, there is a font-lock-function-name-face, and
this is different from font-lock-variable-name-face.  While in some
languages (including JavaScript) the distinction is not necessarily
exact, they can usually be reconciled -- e.g. being a function is
a more "important" property of an identifier than being a variable.

Most of the font-lock faces represent very common primary roles:
strings, comments, keywords, types, preprocessor macros.  But not all.
font-lock-constant face is actually orthogonal to the primary role.
A class or method or parameter can be const or non-const in some
languages.

The semantic notion of public/private/protected/package/friend
visibility is another example.  So is "abstract"/"pure virtual".

Emacs supports composable faces (a style run may have multiple
faces, and the attributes compose according to predefined rules),
but font-lock provides neither consistent nor adequate support for
this notion.

Ve) Ambiguous semantic styles

At least one of the face names is ambiguous -- it's not clear what
font-lock-builtin-face is actually supposed to highlight.  The result
is that different language modes use it for different kinds of
entities.  If you customize the face for one mode, you may wind up
with unsatisfying results in another mode due to the differences
in relative weighting/distribution of semantic types across languages.

As a hypothetical example, someone might enhance python-mode to
use font-lock-builtin-face to highlight True/False/None and possibly
"self", since they're not keywords but they are all handled specially
by the runtime.  (font-lock-type-face might be better for this, but
since they're not really classes, you could argue it either way).
These tokens appear relatively infrequently in Python.  If someone
else were to use it to highlight functions implemented in C in elisp,
there would be a lot more of that face appearing in elisp buffers,
and it might not be easy to choose one face that looks nice in both
situations.

Regardless of the fate of js2-mode, font-lock needs to add more
semantic faces.  By default these new faces might simply inherit face
attributes from their "syntactic parents" -- e.g. the faces for
locals, parameters, instance and static vars might all inherit the
settings for `font-lock-variable-name-face'.  But users should be
able to differentiate among them when the information is available.

Vf) No font-lock interface for setting exact style runs

I could be mistaken here -- if so, please correct me.

My limited understanding of font-lock and its main entry-point
mechanisms such as font-lock-keywords and font-lock-apply-highlight,
all of which use the MATCH-HIGHLIGHT data structure, is that they
are not quite powerful enough for my needs in their current incarnation.

This issue is independent of asynchronous parsing -- I think that
even if my parser were instantaneous, I would still have this issue.

The problem is that I need a way, in a given font-lock redisplay, to
say "highlight the region from X to Y with text properties {Z}".

This use case does not seem like it should be inordinately difficult
to support, but it does not seem to be supported today.

When I assert that it's not possible, I understand that it's
_theoretically_ possible.  Given a JavaScript file with 2500 style
runs, assuming I had that information available at font-lock time, I
could return a matcher that contains 2500 regular expressions, each
one of which is tailored to match one and exactly one region in the
buffer.

In practice, however, I am not aware of a way to do this that is
either clean or efficient.

If this simple feature were supported, I would have a great deal more
incentive to try to get my parsing to be fast enough to work within
the time constraints users expect from font-lock.

Vg) Lack of differentiation between mode- and minor-mode styles

One of the most common complaints from the thousands of users of
js2-mode, most of whom have exercised enough self-restraint to use the
term "work in progress" in preference to "abomination", is that
js2-mode has poor support for minor modes that do their work with
font-lock -- 80-column highlighters being a popular example, although
there are others.

The fundamental problem here is that the font-lock framework does not
differentiate between the mode's syntax highlighting and the keywords
installed by minor modes and by user code.  Instead, it merges them.

As far as I can tell, the officially supported mechanism for
adding additional font-lock patterns is `font-lock-add-keywords'.
This either appends or prepends the keywords to the defaults.

It might be possible to reverse-engineer it, for instance by manually
diffing the buffer's font-lock-defaults and font-lock-keywords and
trying to figure out which ones were added by participants other than
the major mode.  Even if it's possible, it's not clear that it always
works now, and would always work in the future.

For one thing, it's possible (as Daniel observes) to bypass this
mechanism and call font-lock-apply-highlight directly, which makes
the reverse-engineering even more cumbersome and fragile.

(Vf) is the reason (Vg) is a problem for js2-mode.  font-lock-defaults
does not seem to be a very satisfactory way to apply 2000-10000
precise style runs to a buffer, so I do all my own highlighting,
and it doesn't include style-run contributions from minor modes.

I've made some halfhearted attempts to hack around the problem, but
they've proven fragile.  If font-lock were to support (Vf), then I
think (Vg) should "just work".

VI.  Summary

I've called out some of the main integration issues I've encountered.
I've penned several major and minor language modes, not just js2-mode,
and I've chosen to whine here about the problems that could best be
classified as "problem themes".

I'm around, and I'm available for nontrivial work.  If group consensus
is that js2-mode isn't ready yet, I'm happy to keep hacking on it and
taking user patches and feedback until Emacs 24 rolls around.

But it would be nice to have more direct support for modes like mine.
I'm willing to do my end of it, but I'm always oversubscribed, and I've
already signed up to support mouse-enter and mouse-left text props
as part of another js2-mode-related thread.

So a little help would go a long way.

-steve

[-- Attachment #2: Type: text/html, Size: 23973 bytes --]

Re: eval-after-load not harmful after all (Was: Re: Why js-2mode?)

[Leo]

On 2009-08-11 04:06 +0100, Daniel Colascione wrote:
> You haven't actually addressed the reason for using e-a-l and provided
> alternatives, nor have you provided any concrete examples of the harm
> e-a-l might cause. You've just put "necessary" in scare quotes without
> even considering the idea that it might actually *be* necessary.

Imagine the maintainers receive a bug report with the bug hidden
somewhere in after-load-alist and there is no way to identify which
eval-after-load call adds that entry.

It is almost impossible to unload a form in after-load-alist without
restarting Emacs and that is annoying too.

> If packages M needs to work with optional package Q, then there are
> two approaches: either M can use eval-after-load to customize Q, or Q
> can include a list of every possible M with which it can be used. You
> might decry the former, but I don't think the latter is any more
> maintainable.

-- 
Leo's Emacs uptime: 7 days, 9 hours, 6 minutes, 57 seconds

Re: "Font-lock is limited to text matching" is a myth

[Miles Bader]

Steve Yegge <stevey@google.com> writes:
> V. Font Lock framework design problems
>
> There seems to be a common misconception flitting about to the
> effect that font-lock is perfect and will never need to change.

I don't think that's true at all -- I think most of us realize that
font-lock sucks.

On the other hand, most attempts to do ad-hoc highlighting also seem to
suck (this is always my initial assumption).

Since there's a _lot_ to be gained from using a common framework
(consistency, maintainability, etc), it's far preferable that packages
use a standard sucky framework, than their own sucky code.

Thus the tendency to approach any new package that doesn't use font-lock
with skepticism:  It isn't so much a vote of confidence in font-lock, as
it is a recognition that the alternative is _usually_ worse...

If that's not true in your case, then great -- as Stefan said, maybe
it's a good example of an alternative approach, and something that could
be used to experiment with possible future highlighting/parsing/etc
frameworks that do a better job than font-lock.

-Miles

-- 
Defenceless, adj. Unable to attack.

Re: "Font-lock is limited to text matching" is a myth

[Daniel Colascione]

Morning, Steve. It's good to see you jump into the discussion. Thanks  
for all the work you've put into js2-mode.

On Aug 11, 2009, at 2:47 AM, Steve Yegge wrote:
> js2-mode performs both syntactic and (some) semantic analysis.  It
> knows, for instance, when you're using a symbol that's not defined in
> its file.  js2-mode does not currently understand project structure,
> but I'm doing some work in this area, and it may at some point gather
> semantic information collected from several files.

I agree that some kind of project abstraction is certainly in the  
purview of Emacs, and is sorely needed. But a single major-mode is not  
the place for that kind of infrastructure. espresso-mode, for now,  
just looks at all extant buffers in espresso-mode when gathering  
symbol information. That seems like a reasonable approach to use while  
we don't have a project system.

> There is a relatively simple alternative that might appease Daniel:
> I could have js2-mode simply not do any highlighting by default,
> except for errors and warnings.  We'd use whatever highlighting is
> provided by espresso-mode, and users would be able to choose between
> espresso-highlighting and js2-mode highlighting.  With the former,
> they'd get "instantaneous" font-locking, albeit not as rich as what
> js2-mode can provide.

That's an interesting idea, but my concerns are not limited to js2- 
mode's highlighting. js2-mode is a valuable experiment, but I stand by  
my assertion that js2-mode represents a fundamentally wrong way to  
design major modes, and represents a possible future I would like to  
avoid.

I recognize that full parsing can take substantial time, but  
fontification and indentation *must* be synchronous. If the parser  
cannot perform well enough to play a part in fontification and  
indentation, then it must be a separate and optional module.

If you propose relegating js2's parser to error reporting, then  
wouldn't it be better to package it up as a separate, optional minor  
mode? There is *already* a separate and optional full parsing  
framework called CEDET that is powerful, generic, and not tied to a  
major-mode in particular. The right approach is for a given major-mode  
to understand enough of a given language for fontification and  
indentation while leaving more substantial parsing and indexing to  
CEDET (which the user can disable). I recognize that js2's parser may  
work well in its problem domain --- couldn't it just be added to CEDET?

Alternatively, error checking can be offloaded to a separate  
processing tool that regularly scans the buffer. Something like  
Flymake would seem appropriate here.

Even if js2-mode's parser could be made fast enough for synchronous  
fontification, it still would remain an isolated module apart from  
CEDET's infrastructure, and it would still make the mode brittle in  
the face of minor syntactic variations (which might arise, for  
example, from using the C preprocessor on Javascript like we do).  
Rigid parsers with nonterminals and error productions appear  
superficially attractive, but using them for all aspects of a mode not  
only leads to the issues you discuss below, but also prevents that  
mode from being reused for similar languages without the grammar being  
re-worked. It's the wrong approach.

> This would be trivial to change.  I am actively maintaining js2-mode,

My apologies, then. I had merely looked at the lack of commits and  
releases.

> Errors and warnings would still need to be asynchronous (if they're
> enabled).  So, too, would the imenu outline and my in-progress
> buffer-based outline, which is somewhat nicer than the IMenu one.

Have you looked at the espresso imenu support? It heuristically  
captures much of the object structure of a typical Javascript file and  
produces something quite nice, if I say so myself. It's fast enough  
that the parsing operation can be performed synchronously when an  
imenu data structure is required.

> But I think the main objection to js2-mode revolves around its
> highlighting, correct?  If so, AND if we can solve the font-lock
> integration issues, AND if we can fix the multi-mode issues (II
> below), then I'm hopeful that js2-mode might become a reasonable
> choice as the default editing mode for JavaScript.

> I think espresso-mode is a fine fallback position.  Anything but
> java-mode!  The default today is java-mode, and I had no qualms about
> replacing it as the default for JavaScript.

Indeed. java-mode is unacceptable. But it's not quite as unacceptable  
as it appears at first glance: cc-mode would be an excellent platform  
on which to base a Javascript mode, and I only created espresso-mode  
after becoming frustrated time and again in trying to extend cc-mode  
to support Javascript. Unfortunately, cc-mode's core includes some  
baked-in assumptions about the language, that, IIRC, include function  
declarations requiring parameter types. It's a shame, really, because  
Javascript is close enough to C that they could conceivably share a  
mode, whereas something like Perl clearly requires a world unto itself.

> Note: diagnostic messages in js2-mode are highlighted using overlays.
> I tried using overlays for all highlighting but it was unacceptably
> slow and had a tendency to crash Emacs.

I've had the same thought. Just as in aside, overlays seem like a much  
better conceptual fit for fontification than text properties do. To  
this day, I believe the old XEmacs extent system, which was similar to  
overlays, represented a more elegant approach than the semipermanent  
text properties used by GNU Emacs. Nevertheless, text property  
fontification isn't bad enough to warrant the terrible backward  
compatibility problems that would be generated by a switch to overlays.

>
> II. Multi-mode support
>
> JavaScript is especially needful of mumamo (or equivalent) multi-mode
> support, because much of the JavaScript in the wild is embedded in
> HTML, in template files, even in strings in other languages.
>
> js2-mode does not support mumamo (or mmm-mode, which which I am
> currently more familiar) because js2-mode's lexer needs to support
> ignoring parts of the buffer.  I do not think this would be very
> hard to implement, but I have not done it yet.
>
> If I don't get to it before the next version of Emacs launches, then I
> think this should effectively disqualify js2-mode from being the
> default JavaScript mode.  It would be an inconsistent user experience
> to have one JavaScript mode in .js files and another mode for
> JavaScript inside multi-mode-enabled files.

I agree. If I recall correctly, I've also used rather strong language  
to describe the current state of multi-mode support in Emacs. (So it's  
not just js2-mode.) I still believe that some kind of indirect buffer  
solution in the core would be the most effective and elegant approach  
to supporting multiple modes. However, mumamo seems to have improved  
quite a bit, and if it exposes sufficient information to the  
underlying major modes, it shouldn't hard to modify either js2-mode or  
espresso-mode to work with it. (Though I do have to ask how your  
parser will deal with possibly non-contiguous chunks of Javascript.)

>
> I'm ready to give it a try, though, and I'll ping Lennart offline  
> about
> integrating the two somehow.
>
> III. Incremental and partial parsing
>
> Lennart and others have asked whether it is possible for js2-mode to
> support partial or incremental parsing.  The short answer is
> "incremental: yes; partial: no".
>
> nxml-mode, last I checked, does incremental parsing.  It parses ahead
> in the buffer, but then stops and saves its state.  If you jump  
> forward
> in the buffer, it resumes and continues the parse until some point
> beyond the section you're viewing.

espresso does precisely this kind of parsing, as does cc-mode.

> js2-mode could do it this way without much additional effort.

That's good to hear.

> I've been an Emacs user for 20+ years now, and like many I found
> the idea of a parsing delay to be somewhere between "undesirable"
> and "sickening".

That would describe my experience. May I add "maddening" and  
"distracting"?

> But the majority of programmers today have
> apparently learned not to notice delays of ~1sec as long as it
> never interferes with their typing or indentation (see IV below).

That other programmers have resigned themselves to inferior  
fontification is no argument for Emacs to accept it. Asynchronous  
fontification is completely unacceptable for me, and if it were to  
become commonplace and unavoidable in Emacs, I would simply stay with  
older versions.

> So after looking at my ~8000 lines of elisp devoted to parsing
> JavaScript, I weighed it and decided not to support partial parsing.
> It's certainly possible to support it, but I think my time would be
> better spent on things that average users are more likely to notice.

cc-mode has a surprisingly complex and robust reprasing system that  
caches both whitespace runs and syntactic information necessary for  
indentation. It'd be nice to be able to reuse that.

As it is, espresso uses an incremental parsing scheme that feeds into  
the fontification layer.

> IV.  Indentation
>
> The indentation in js2-mode is broken.  I'll be the first to say it.
>
> It is based on the indentation in Karl Langstrom's mode, which does a
> better job for JavaScript than any indenter based on cc-engine, but
> that doesn't mean it's a good job.  And it's essentially  
> unconfigurable.
>
> espresso-mode shares this problem, which means that for this
> important use case it is not an improvement over js2-mode.

Indeed. I've implemented some changes to the indentation system, but  
the basic approach reminds the same.  I should point out, however,  
that cc-mode's indentation, however convenient, is the exception to  
the typical Emacs rule. Most modes have only a few knobs the user may  
tweak to adjust indentation, and users seem happy with those. Also, as  
Alan Mackenzie mentioned in another thread, cc-mode's indentation is a  
maintenance burden. I'm not quite sure moving away from Karl  
Langstrom's indentation approach is worth the trouble right now.

> Daniel's objections to js2-mode's non-interaction with font-lock
> apply equally to the non-interaction with cc-engine's indentation
> configuration system.  The indent configuration for JavaScript should
> share as many settings as practical with cc-mode.
>
> I actually made a serious attempt to generate the `c-style-alist'
> data structure for js2-mode using the parse tree, but ran into three
> issues: [snipped]

I encountered the same problems myself when trying to implement the  
same feature.

> V. Font Lock framework design problems
> There seems to be a common misconception flitting about to the
> effect that font-lock is perfect and will never need to change.

Nobody is making this claim, and it would be a foolish one to make. Of  
course font-lock can be improved. But the fundamental approach is sound.

Actually, there seems to be a common misconception that font-lock is  
an ancient, decrepit mess that's preventing Emacs from striding  
forward into the "modern" world. Far from it: used properly, font-lock  
is flexible and powerful. I'd love to see some improvements, such as  
syntactic keywords being pushed down to a lower level, but the basic  
idea to sound. In one system, one can combine everything from fast,  
efficient keyword fontification to arbitrarily complex schemes that  
depend on elaborate contexts and subtle rules. espresso-mode performs  
this kind of mixing, actually. Font-lock confers many benefits in  
terms of reusability, modularity, and customizability, and it would be  
a waste to replicate it instead of augmenting it. (You have some  
excellent ideas for doing that below.)

Really, those who dislike font-lock have the same mindset as those who  
dislike X11. Like font-lock, X11 is an old, powerful system that  
superficially appears poorly-designed. What detractors ignore is that  
old, mature systems embody years of experience in the problem domain,  
and that attempts at ground-up rewrites typically lead to either a  
system with a reduced feature-set relative to the original, or a mere  
reimplementation of the original system in different terms, and  
without the benefit of the experience embodied in the original system.

> This is a somewhat paradoxical viewpoint in view of the corpses
> littering the path to jit-lock, which include font-lock, fast-lock,
> lazy-lock, and vapor-lock.  Each decade we've had a cadre of people
> claiming that *-lock meets everyone's needs, and then it gets  
> rewritten
> anyway.

I'm not aware of the authors of *any* of these modes making that  
claim. The facilities you mention have all been incremental  
improvements on the basic font-locking idea. Do you really want to  
discourage that kind of experimentation? Also, in defense of these  
modes, jit-lock depends on core Emacs functionality that has not  
always been available, and some of the modes you mentioned would  
doubtlessly had not been written if jit-lock had been available earlier.

> So it's hard to understand how it remains such a popular viewpoint.
>
> I'll make yet another attempt to dispel it, since once we're past the
> emotional stumbling blocks, font-lock may be able to evolve again.
>
> Va) Inadequate/insufficient style names
> Vb) Ad-hoc default faces that are not being autoloaded
> Vc) Additional semantic styles not needed by JavaScript
> Vd) Composable semantic styles

I fully agree with these points. While the default font-lock faces  
have been generally adequate over the years, adding a set of richer  
faces (that perhaps inherit from the traditional ones) would be  
welcome. A composable set of styles is an interesting idea too, and  
it'd be great to see.

> Vf) No font-lock interface for setting exact style runs
>
> I could be mistaken here -- if so, please correct me.

> The problem is that I need a way, in a given font-lock redisplay, to
> say "highlight the region from X to Y with text properties {Z}".
>
> This use case does not seem like it should be inordinately difficult
> to support, but it does not seem to be supported today.

As I detailed in '"Font-lock is limited to text matching" is a myth',  
explicit fontification has essentially always been possible in font- 
lock. cc-mode has used it for over a decade, and today, both espresso- 
mode and nxml use this regrettably poorly-documented facility.

> If this simple feature were supported, I would have a great deal more
> incentive to try to get my parsing to be fast enough to work within
> the time constraints users expect from font-lock.

I've taken pains in espresso-mode to ensure that synchronous  
operations are fast enough to be used interactively, even on large  
files. Were js2's parsing to also become fast and synchronous, some of  
my objections would indeed evaporate.

> Vg) Lack of differentiation between mode- and minor-mode styles
>
> One of the most common complaints from the thousands of users of
> js2-mode, most of whom have exercised enough self-restraint to use the
> term "work in progress" in preference to "abomination", is that
> js2-mode has poor support for minor modes that do their work with
> font-lock -- 80-column highlighters being a popular example, although
> there are others.

As I mentioned earlier, my diction reflects not of js2-mode's  
maturity, but its fundamental structure. I believe it is wrong, and  
"abomination", while incendiary, is correct. I don't want the future  
of Emacs to be chock full of modes like js2.

> For one thing, it's possible (as Daniel observes) to bypass this
> mechanism and call font-lock-apply-highlight directly, which makes
> the reverse-engineering even more cumbersome and fragile.

Quite the opposite, actually.

> (Vf) is the reason (Vg) is a problem for js2-mode.  font-lock-defaults
> does not seem to be a very satisfactory way to apply 2000-10000
> precise style runs to a buffer, so I do all my own highlighting,
> and it doesn't include style-run contributions from minor modes.

When using font-lock-apply-highlights, or its moral equivalents, user  
and minor-mode font locking is automatically composed with the major  
mode's. By using the 'prepend and 'append operators, minor modes and  
users may state the priority of their fontification rules with respect  
to those of the major mode. Niceties like these have grown with Emacs  
for years, and a great deal is lost when a particular major-mode  
attempts to re-implement core functions to account for some imagined,  
or at worst, temporary deficiency.

RE: eval-after-load not harmful after all (Was: Re: Why js-2mode?)

[Stephen J. Turnbull]

Drew Adams writes:
 > sjt wrote:

 > > I note that all the advocates of e-a-l are package maintainers and UI
 > > types; all the deprecators are core.  This is an inherent tension, and
 > > I think it should be resolved in favor of protecting the core.
 > 
 > Nothing wrong with "protecting the core".
 > 
 > The discussion was about the Elisp manual, whose target audience is
 > not just maintainers of the Emacs core. The help and guidance there
 > are used by a variety of programmer-users.
 > 
 > Again, there's nothing wrong with (a) giving a general guideline,
 > and also (b) explaining the issues and giving additional info about
 > contexts where the guideline might not be something you would want
 > to follow.

Yes, there is.  If the manual says "use this feature only when
appropriate," people with a biased view of Emacs (ie,
my-package-centric, as opposed to core-centric) are naturally going to
overuse the feature.  IMO, it's worth leaning against the wind by
documenting the semantics of the feature and the caution against using
it in the core, perhaps with a gloss like "since in the core we can
usually change both libraries to cooperate more closely".  Those who
really want to use the feature will do so anyway, or ask here.

Note that all of the examples given so far of "why I would want to use
eval-after-load" have been pretty dubious, so there's no positive "you
might want to use `eval-after-load' when ..." advice to offer anyway.

Re: "Font-lock is limited to text matching" is a myth

[Miles Bader]

Daniel Colascione <danc@merrillpress.com> writes:
>> Note: diagnostic messages in js2-mode are highlighted using overlays.
>> I tried using overlays for all highlighting but it was unacceptably
>> slow and had a tendency to crash Emacs.
>
> I've had the same thought. Just as in aside, overlays seem like a much
> better conceptual fit for fontification than text properties do.

Why?

[Aside from the "changing text-properties affect buffer-modified status"
silliness, which I think is probably a mistake, but that's easy enough
to work around.]

-miles

-- 
Monday, n. In Christian countries, the day after the baseball game.

Re: eval-after-load not harmful after all (Was: Re: Why js-2mode?)

[Stephen J. Turnbull]

Daniel Colascione writes:

 > You haven't actually addressed the reason for using e-a-l and provided  
 > alternatives, nor have you provided any concrete examples of the harm  
 > e-a-l might cause.

I haven't had to debug such a case in four or five years, and I'd like
to keep setting a new record every morning, just by getting out of
bed.

As a matter of principle, I don't like constructs that change the
behavior of code in a way that that code doesn't anticipate.
Specifically, in order to catch eval-after-load in the act, you need
to know which library invoked it.  Otherwise, by the time you observe
a bug, the behavior has been changed and the entry has been purged
from after-load-alist.  defadvice, by contrast, leaves a trace of its
behavior in the docstring of an adviced function.

 > You've just put "necessary" in scare quotes without even
 > considering the idea that it might actually *be* necessary.

Thanks for playing, but you have failed to read my mind.  In fact, I
quite obviously, though implicitly, have acknowledged the *need* for
the feature by not calling for its removal, indeed, giving a case
where I think it's harmless and useful.

On the other hand, we have yet to see a convincing example of "need"
here.  I don't think your hs-whatever-it-was example does the trick,
Carsten's was definitively exploded by Dan Nicolaescu IIRC.

 > If packages M needs to work with optional package Q, then there are  
 > two approaches: either M can use eval-after-load to customize Q, or Q  
 > can include a list of every possible M with which it can be used. You  
 > might decry the former, but I don't think the latter is any more  
 > maintainable. 

Ah, but we're talking about the core, where there *definitely* is a
third approach: have the two library maintainers negotiate a suitably
general hook in Q that M can use to customize it.  In many, perhaps
most, cases that hook already exists.  Eg, for standardly-defined
modes you can typically do

    (if (featurep 'foo)
        (bar-customize-foo)
      (add-hook 'foo-mode-hook #'bar-customize-foo))

where #'bar-customize-foo is idempotent.  Eg, by wrapping the active
code in

    (unless (get 'bar-customize-foo 'customization-installed-p)
      (bar-customize-foo-internal)
      (put 'bar-customize-foo 'customization-installed-p t))

(`add-one-shot-hook' suffers from the same kind of disabilities that
`eval-after-load' does.)

So there's your alternative.  I don't guarantee that pattern is 100%
general, but between that and use of other, more specific hooks
(existing ones as in Carsten's example, or newly-designed ones), you
can probably restrict cases in the core to very nearly zero.

I think the onus is on those who wish to use eval-after-load in the
core, or even to document "appropriate uses" outside the core, to
define what those appropriate uses might be.  Note that the approach
of coooperatively defining hooks is theoretically available outside
the core, too, although coordination is more expensive then.

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Tue, Aug 11, 2009 at 4:37 PM, Miles Bader<miles@gnu.org> wrote:

> [Aside from the "changing text-properties affect buffer-modified status"
> silliness, which I think is probably a mistake, but that's easy enough
> to work around.]

There are needs for it sometimes, but I always forget that it does.
And there is no really easy, standard way to avoid the buffer
modifying.

I have two alternative suggestions:

1) Make a macro available for avoiding it.

2) Add an extra argument to put-text-property DONT-MODIFY-BUFFER and
friends. Or make new versions of them.

Probably 2 will be faster.

Re: "Font-lock is limited to text matching" is a myth

[Chong Yidong]

Daniel Colascione <danc@merrillpress.com> writes:

>> There is a relatively simple alternative that might appease Daniel:
>> I could have js2-mode simply not do any highlighting by default,
>> except for errors and warnings.  We'd use whatever highlighting is
>> provided by espresso-mode, and users would be able to choose between
>> espresso-highlighting and js2-mode highlighting.  With the former,
>> they'd get "instantaneous" font-locking, albeit not as rich as what
>> js2-mode can provide.
>
> That's an interesting idea, but my concerns are not limited to js2-
> mode's highlighting.

You don't properly elaborate what these additional concerns are.

I don't use either mode, but from reading the discussion it seems to me
that the optimal solution is to use espresso mode's code for font-lock
and indentation, and js2-mode's code for the commands that need to be
better aware of semantic information (show/hide elements, error
reporting, marking defuns, etc.)

> There is *already* a separate and optional full parsing framework
> called CEDET that is powerful, generic, and not tied to a major-mode
> in particular. The right approach is for a given major-mode to
> understand enough of a given language for fontification and
> indentation while leaving more substantial parsing and indexing to
> CEDET (which the user can disable). I recognize that js2's parser may
> work well in its problem domain --- couldn't it just be added to
> CEDET?

I don't think this is a necessary condition.  We can freely move the
code around the Emacs tree once it's already in there; what's more
important is to have something working.

Re: "Font-lock is limited to text matching" is a myth

[Daniel Colascione]

On Aug 11, 2009, at 10:37 AM, Miles Bader wrote:
> Daniel Colascione <danc@merrillpress.com> writes:
>>> Note: diagnostic messages in js2-mode are highlighted using  
>>> overlays.
>>> I tried using overlays for all highlighting but it was unacceptably
>>> slow and had a tendency to crash Emacs.
>>
>> I've had the same thought. Just as in aside, overlays seem like a  
>> much
>> better conceptual fit for fontification than text properties do.
>
> Why?

AIUI, text properties are considered *part* of the document being  
edited, which is why modifying text properties modifies the buffer  
modified flag, why they appear in the undo list, and so on, whereas  
overlays are for transient, regenerable information *about* the text  
in the document. Fontification is definitely regenerable, transient  
information, and so should use overlays. I'm also not particularly  
fond of fontification being carried around when I kill and yank.

But like you said, these are minor issues and can be worked around.  
It's no big deal, and text properties perform much better on GNU  
Emacs. (IIRC, each overlay actually uses a pair of markers. It's no  
wonder that using hundreds or thousands of them is slow.)

> [Aside from the "changing text-properties affect buffer-modified  
> status"
> silliness, which I think is probably a mistake, but that's easy enough
> to work around.]

Don't forget the undo entries too. Actually, the full list comes from  
font-lock.el:

(defmacro save-buffer-state (varlist &rest body)
     "Bind variables according to VARLIST and eval BODY restoring  
buffer state."
     (declare (indent 1) (debug let))
     (let ((modified (make-symbol "modified")))
       `(let* ,(append varlist
		      `((,modified (buffer-modified-p))
			(buffer-undo-list t)
			(inhibit-read-only t)
			(inhibit-point-motion-hooks t)
			(inhibit-modification-hooks t)
			deactivate-mark
			buffer-file-name
			buffer-file-truename))
	 (progn
	   ,@body)
	 (unless ,modified
	   (restore-buffer-modified-p nil)))))

Re: "Font-lock is limited to text matching" is a myth

[Daniel Colascione]

On Aug 11, 2009, at 10:49 AM, Lennart Borgman wrote:

> On Tue, Aug 11, 2009 at 4:37 PM, Miles Bader<miles@gnu.org> wrote:
>
>> [Aside from the "changing text-properties affect buffer-modified  
>> status"
>> silliness, which I think is probably a mistake, but that's easy  
>> enough
>> to work around.]
>
> There are needs for it sometimes, but I always forget that it does.
> And there is no really easy, standard way to avoid the buffer
> modifying.
>
> I have two alternative suggestions:
>
> 1) Make a macro available for avoiding it.

See the macro I included in reply to the OP. You can also use c-save- 
buffer-state.

Re: "Font-lock is limited to text matching" is a myth

[Daniel Colascione]

On Aug 11, 2009, at 10:50 AM, Chong Yidong wrote:

> Daniel Colascione <danc@merrillpress.com> writes:
>
>>> There is a relatively simple alternative that might appease Daniel:
>>> I could have js2-mode simply not do any highlighting by default,
>>> except for errors and warnings.  We'd use whatever highlighting is
>>> provided by espresso-mode, and users would be able to choose between
>>> espresso-highlighting and js2-mode highlighting.  With the former,
>>> they'd get "instantaneous" font-locking, albeit not as rich as what
>>> js2-mode can provide.
>>
>> That's an interesting idea, but my concerns are not limited to js2-
>> mode's highlighting.
>
> You don't properly elaborate what these additional concerns are.

I thought I discussed them well enough in the remainder of the  
message, but I'll summarize them here:

- full parsers belong in the CEDET infrastructure where they can be  
accessed via a common API
- if this API is not rich enough, it ought to be extended
- normal major modes shouldn't rely on rigid grammars because that  
makes them brittle. Anything that needs such a grammar should be an  
optional and preferably generic minor-mode

(I still think it'd be worthwhile to include the JS2 parser in CEDET,  
which supports parsers not necessarily generated with wisent or bison.  
It seems robust in its problem domain.)

> I don't use either mode, but from reading the discussion it seems to  
> me
> that the optimal solution is to use espresso mode's code for font-lock
> and indentation, and js2-mode's code for the commands that need to be
> better aware of semantic information (show/hide elements, error
> reporting, marking defuns, etc.)

That could work, but keep in mind that espresso already has a fairly  
robust and substantial parsing system that doesn't rely on a full  
grammar, and that can still reliably mark functions and generate  
multilevel imenu structures.

In terms of featureset, I don't see the differences between (espresso 
+CEDET+some-sort-of-flymake-lint) and (js2) to be very large, though  
what differences there are should certainly be dealt with in whatever  
ends up being the default for Javascript.

>> There is *already* a separate and optional full parsing framework
>> called CEDET that is powerful, generic, and not tied to a major-mode
>> in particular. The right approach is for a given major-mode to
>> understand enough of a given language for fontification and
>> indentation while leaving more substantial parsing and indexing to
>> CEDET (which the user can disable). I recognize that js2's parser may
>> work well in its problem domain --- couldn't it just be added to
>> CEDET?
>
> I don't think this is a necessary condition.  We can freely move the
> code around the Emacs tree once it's already in there; what's more
> important is to have something working.

We already have not one, but two working solutions. And granted, I'm  
not a core developer, but from where I'm sitting it seems code is  
seldom moved around once committed.

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Tue, Aug 11, 2009 at 4:53 PM, Daniel Colascione<danc@merrillpress.com> wrote:

> Don't forget the undo entries too. Actually, the full list comes from
> font-lock.el:
>
> (defmacro save-buffer-state (varlist &rest body)
>    "Bind variables according to VARLIST and eval BODY restoring buffer
> state."
>    (declare (indent 1) (debug let))
>    (let ((modified (make-symbol "modified")))
>      `(let* ,(append varlist
>                      `((,modified (buffer-modified-p))
>                        (buffer-undo-list t)
>                        (inhibit-read-only t)
>                        (inhibit-point-motion-hooks t)
>                        (inhibit-modification-hooks t)
>                        deactivate-mark
>                        buffer-file-name
>                        buffer-file-truename))
>         (progn
>           ,@body)
>         (unless ,modified
>           (restore-buffer-modified-p nil)))))

And JIT lock takes care of errors too:

(defmacro jit-with-buffer-unmodified (&rest body)
  "Eval BODY, preserving the current buffer's modified state."
  (declare (debug t))
  (let ((modified (make-symbol "modified")))
    `(let ((,modified (buffer-modified-p)))
       (unwind-protect
           (progn ,@body)
         (unless ,modified
           (restore-buffer-modified-p nil))))))

(defmacro with-buffer-prepared-for-jit-lock (&rest body)
  "Execute BODY in current buffer, overriding several variables.
Preserves the `buffer-modified-p' state of the current buffer."
  (declare (debug t))
  `(mumamo-jit-with-buffer-unmodified
    (let ((buffer-undo-list t)
          (inhibit-read-only t)
          (inhibit-point-motion-hooks t)
          (inhibit-modification-hooks t)
          deactivate-mark
          buffer-file-name
          buffer-file-truename)
      ,@body)))

Re: eval-after-load not harmful after all

[Stefan Monnier]

>> I note that all the advocates of e-a-l are package maintainers and UI
>> types; all the deprecators are core.  This is an inherent tension, and
>> I think it should be resolved in favor of protecting the core.

Actually, I think "tension" is the wrong word.  The core just want to
send a message, nothing more.  The main message is "eval-after-load and
advice should ideally not be necessary: we happily support them and
won't be offended at all if you use them, but if you do need them,
please understand that it is evidence of a shortcoming in some package
that should be reported so we can fix it".

> The discussion was about the Elisp manual, whose target audience is
> not just maintainers of the Emacs core.  The help and guidance there
> are used by a variety of programmer-users.

And that's why we want this message in the manual.


        Stefan

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Tue, Aug 11, 2009 at 5:06 PM, Daniel Colascione<danc@merrillpress.com> wrote:

> In terms of featureset, I don't see the differences between
> (espresso+CEDET+some-sort-of-flymake-lint) and (js2) to be very large,

Completion?

Re: "Font-lock is limited to text matching" is a myth

[Daniel Colascione]

On Aug 11, 2009, at 11:11 AM, Lennart Borgman wrote:

> On Tue, Aug 11, 2009 at 5:06 PM, Daniel Colascione<danc@merrillpress.com 
> > wrote:
>
>> In terms of featureset, I don't see the differences between
>> (espresso+CEDET+some-sort-of-flymake-lint) and (js2) to be very  
>> large,
>
> Completion?

Isn't that CEDET's job? If it is not, espresso has a lexical database  
of its own. (Try typing M-.) It wouldn't be hard to wire that up to  
completion.

Re: "Font-lock is limited to text matching" is a myth

[Miles Bader]

Daniel Colascione <danc@merrillpress.com> writes:
> AIUI, text properties are considered *part* of the document being
> edited, which is why modifying text properties modifies the buffer
> modified flag, why they appear in the undo list, and so on, whereas
> overlays are for transient, regenerable information *about* the text  in
> the document. Fontification is definitely regenerable, transient
> information, and so should use overlays.

I agree that it would be nice to have "unrecorded" text-properties.  As
Lennart mentioned, maybe just another flag option to `put-text-property'
(etc), meaning "don't put this change in the undo list, or change
buffer-modification state", would be enough (I haven't looked at the
details).

However, overlays aren't just "unrecorded text-properties", they are
also different in other ways -- in particular, each overlay is a
separate lisp object, with an identity, which can carry around its own
property list, does not coalesce with other overlays, can be moved, etc.

My impression is that for many fontification uses, while the
"unrecorded" nature of overlays would be very useful, their "identity"
nature would not be useful or would even pose a problem.

Overlays seem better suited to other uses, where you might want to move
around an overlay, or delete it easily given just a handle, etc.
[I don't think most fontification uses are like that.]

-Miles

-- 
`The suburb is an obsolete and contradictory form of human settlement'

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Tue, Aug 11, 2009 at 5:16 PM, Daniel Colascione<danc@merrillpress.com> wrote:
> On Aug 11, 2009, at 11:11 AM, Lennart Borgman wrote:
>
>> On Tue, Aug 11, 2009 at 5:06 PM, Daniel Colascione<danc@merrillpress.com>
>> wrote:
>>
>>> In terms of featureset, I don't see the differences between
>>> (espresso+CEDET+some-sort-of-flymake-lint) and (js2) to be very large,
>>
>> Completion?
>
> Isn't that CEDET's job? If it is not, espresso has a lexical database of its
> own. (Try typing M-.) It wouldn't be hard to wire that up to completion.

Yes, but you said "lint"... ;-)

CEDET must have something to say and lint does not give it much to
say. (Sorry for the noise.)

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Tue, Aug 11, 2009 at 5:56 PM, Stephen J. Turnbull<stephen@xemacs.org> wrote:
> position).  So, for a very silly example, I could copy the "for" from
> a buffer containing a python program "for x in interator:" to the
> first "for" in this sentence, and I really don't want the highlighting
> coming along with it.

I consider this a bug. Pasting should by default be done after
removing the text properties. There should IMO be a separate command
paste-with-text-properties for the current behaviour (if deemed
needed).

Re: "Font-lock is limited to text matching" is a myth

[Stephen J. Turnbull]

Trimming the CC list, which may be getting close to Mailman's
limit for potential spam....

Miles Bader writes:

 > > I've had the same thought. Just as in aside, overlays seem like a much
 > > better conceptual fit for fontification than text properties do.
 > 
 > Why?

To me, in theory, to the extent that "fontification" means "syntax
highlighting", syntax is almost entirely about position in the token
stream, rather than the text composing the token (even keywords, which
in many languages are a syntax error if they are in the wrong
position).  So, for a very silly example, I could copy the "for" from
a buffer containing a python program "for x in interator:" to the
first "for" in this sentence, and I really don't want the highlighting
coming along with it.  The highlighting should stay where it is, eg if
I substitute "while" for "for".

In practice, this only ever bothers me with Gnus, when I cut and paste
from a Gnus buffer to a message buffer.  (I told you the example was
silly!)  Also, if I use something like pending-delete mode, select a
"for" keyword and overwrite it with "while", for example, IIRC text
properties happen to DTRT.

 > [Aside from the "changing text-properties affect buffer-modified status"

*boggle*

Re: "Font-lock is limited to text matching" is a myth

[Miles Bader]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:
> To me, in theory, to the extent that "fontification" means "syntax
> highlighting", syntax is almost entirely about position in the token
> stream, rather than the text composing the token (even keywords, which
> in many languages are a syntax error if they are in the wrong
> position).

However, being able to add properties to strings, and paste them into
buffer (and display them as messages, and ...) is _very_ useful.  It's
quite often far more convenient to construct propertized text outside
the buffer and just insert it.

Cut-and-paste has its own issues (and there is some mechanism in Emacs
to deal with them, e.g., `yank-excluded-properties'), but they seem to
be more connected with that particular application, than with the
underlying low-level abstractions.

-Miles

-- 
"1971 pickup truck; will trade for guns"

Re: "Font-lock is limited to text matching" is a myth

[Stefan Monnier]

> I. Asynchronous parsing

BTW, I'm interested in adding core-Emacs support for such parsing, so if
you have any ideas about it, please share them.  The way I see it, there
should be a table-based parsing engine written in C running in
a spearate thread, so the question are "what should the tables look
like?", "what should the output look like?", "what should the engine
look like?", "how should the asynchronous code synchronize with the rest
of Emacs?".  Any help along this effort would be welcome.

> There is a relatively simple alternative that might appease Daniel:
> I could have js2-mode simply not do any highlighting by default,
> except for errors and warnings.  We'd use whatever highlighting is
> provided by espresso-mode, and users would be able to choose between
> espresso-highlighting and js2-mode highlighting.  With the former,
> they'd get "instantaneous" font-locking, albeit not as rich as what
> js2-mode can provide.

Bringing espresso-mode and js2-mode closer together would be good
(e.g. by merging them into a single mode with customization options
allowing to choose between different ways to do highlighting, imenu,
etc...).
Especially on the indentation side since it seems they ahre some of
their history.

> Errors and warnings would still need to be asynchronous (if they're
> enabled).  So, too, would the imenu outline and my in-progress
> buffer-based outline, which is somewhat nicer than the IMenu one.

Please try and make sure that the imenu-style info you provide is
provided in a compatible way (e.g. so that which-func-mode and other
users of imenu info work).  "Make sure" might mean "report any
obstruction along the way", of course.

> Note: diagnostic messages in js2-mode are highlighted using overlays.
> I tried using overlays for all highlighting but it was unacceptably
> slow and had a tendency to crash Emacs.

Overlays should be used parsimoniously (their algorithmic complexity is
poor).  They're basically somewhere between O(N) and O(N^2), whereas
text-properties are O(log N) as god intended.  So for the few expected
errors, overlays are OK, but for font-lock-style highlighting, it'll
kill performance in very short order.

> I took a close look at Eclipse and IntelliJ, and even asked some
> of their users to characterize the highlighting behavior of the IDE.
> Without exception, the IDE users had internalized a ~1000 ms delay
> in highlighting and error reporting as part of their normal workflow,
> and they uniformly described it as "instant{aneous}" until I made
> them time it.

;-)
There used to be a defer-mode for font-lock (I actually hacked one
myself before the official one came out).

> I've been an Emacs user for 20+ years now, and like many I found
> the idea of a parsing delay to be somewhere between "undesirable"
> and "sickening".  But the majority of programmers today have
> apparently learned not to notice delays of ~1sec as long as it
> never interferes with their typing or indentation (see IV below).

Yes, you mostly notice it when you previously used a system without such
a delay.  But I agree that a 1s delay is not problematic (it even
sometimes avoids intermediate states that are undesirable).

> It is based on the indentation in Karl Langstrom's mode, which does a
> better job for JavaScript than any indenter based on cc-engine, but
> that doesn't mean it's a good job.  And it's essentially unconfigurable.

> espresso-mode shares this problem, which means that for this
> important use case it is not an improvement over js2-mode.

In this case, if espresso-mode gets installed, I'll ask you guys to get
together and merge your two indentation codes (it can be as simple as
"pick one", I don't care; I just don't want to have two diverging
copies of the same code base and feature).

> Daniel's objections to js2-mode's non-interaction with font-lock
> apply equally to the non-interaction with cc-engine's indentation
> configuration system.  The indent configuration for JavaScript should
> share as many settings as practical with cc-mode.

I'm not too fond of cc-mode's indentation code and configuration,
actually, so I don't mind if js2-mode doesn't share anything with it
(tho I won't oppose a change in this respect either).

>   3) indentation in "normal" Emacs modes also runs synchronously as
>      the user types.  Waiting 500-800 msec or more for the parse to
>      finish is (I think) not acceptable for indentation.  For small
>      files the parse time is acceptable, but it would not be generally
>      scalable.

Agreed.  Do you happen to know who other IDEs do about it?

> There seems to be a common misconception flitting about to the
> effect that font-lock is perfect and will never need to change.

I can guarantee you that those misconceptions are not shared by the
authors and maintainers of font-lock.

> Va) Inadequate/insufficient style names

[ Putting on my functional programmer hat here. ]
All you're saying here is that your languages have too many concepts.

[ Putting on my Emacs maintainer hat again. ]
highlighting should be about helping the user understand his code:
highlighting every character with a different color is the way to
get there.  You may want to help him find structure (e.g. make
function/method declaration stand out), you may want to help him not get
confused (highlight strings and comments differently), you may want to
attract his attention to weird things (undeclared variables, ...), but
I highly doubt that highlighting function parameters differently from
local variables will help her in any way.

This said, the set of default faces deserves a rethink as well as some
additions, yes.

> languages, not the intersection.  There should, for instance, be a
> font-lock-symbol-face for languages with distinguished symbols such
> as Lisp, Scheme and Ruby.

What for?

> Vf) No font-lock interface for setting exact style runs
[...]
> The problem is that I need a way, in a given font-lock redisplay, to
> say "highlight the region from X to Y with text properties {Z}".

I'm not sure I understand the problem.  What's wrong with
put-text-property?

> When I assert that it's not possible, I understand that it's
> _theoretically_ possible.  Given a JavaScript file with 2500 style
> runs, assuming I had that information available at font-lock time, I
> could return a matcher that contains 2500 regular expressions, each
> one of which is tailored to match one and exactly one region in the
> buffer.

Just place in font-lock-keywords a MATCHER that is a function whose code
walks the list of your "runs" (checking which of your runs are within
(point) and LIMIT) and uses add-text-properties on them; and finally
returns nil.

> In practice, however, I am not aware of a way to do this that is
> either clean or efficient.

That seems about as efficient as it can get within the constraints
of Elisp.  And it looks pretty clean to me.

> Vg) Lack of differentiation between mode- and minor-mode styles
[...]
> As far as I can tell, the officially supported mechanism for
> adding additional font-lock patterns is `font-lock-add-keywords'.
> This either appends or prepends the keywords to the defaults.

Yes, this sucks.  It should be replaced by a more declarative interface.

> But it would be nice to have more direct support for modes like mine.

We're pretty far from it, I think, but the intention is definitely there.


        Stefan

Re: "Font-lock is limited to text matching" is a myth

[Stephen J. Turnbull]

Lennart Borgman writes:

 > I consider this a bug.

Why?  It is the *defining* characteristic of a *text* property that it
is part of the text, and gets copied with its associated text.

 > Pasting should by default be done after
 > removing the text properties. There should IMO be a separate command
 > paste-with-text-properties for the current behaviour (if deemed
 > needed).

IMO (and O is all it is), based on experience with XEmacs, the world
would be a much better place if overlays were used in a lot of places
where text-properties are currently used, because they have better
behavior by default.  (In XEmacs, in programs it doesn't matter
because extents can be overlay-like or text-property-like along about
5 dimensions which can be chosen independently.  However, although
there is a fair amount of need for variations that don't match either
exactly, the bundles of behaviors called "text property" and
"overlay", respectively, do seem to be the most useful by far.  So
these are the variants supported by interactive commands and
convenience functions.)

OTOH, as Miles points out, it is often the case that it is useful to
construct a string complete with text properties, and then paste it
repeatedly.  Yes, I *do* want "e.g." to be italicized every time I
paste it.

Re: "Font-lock is limited to text matching" is a myth

[Stephen J. Turnbull]

Miles Bader writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > > To me, in theory, to the extent that "fontification" means "syntax
 > > highlighting", syntax is almost entirely about position in the token
 > > stream, rather than the text composing the token (even keywords, which
 > > in many languages are a syntax error if they are in the wrong
 > > position).
 > 
 > However, being able to add properties to strings, and paste them into
 > buffer (and display them as messages, and ...) is _very_ useful.

True.  But that's just saying "copying text properties is useful *when
adding text*".  I didn't think that's what Daniel meant; AIUI, he was
referring to syntax highlighting of *existing* text in the buffer; and
that is the context where I think overlays are more natural.

Re: "Font-lock is limited to text matching" is a myth

[Stefan Monnier]

> C preprocessor on Javascript like we do).  Rigid parsers with nonterminals
> and error productions appear  superficially attractive, but using them for
> all aspects of a mode not  only leads to the issues you discuss below, but
> also prevents that  mode from being reused for similar languages without the
> grammar being  re-worked. It's the wrong approach.

I've had a fairly long exchange with CEDET's author several years ago,
where I made a similar point.  I clearly fully agree: indentation should
work backward from the indentation point and look at as little of the
buffer as possible.  This ensures that if some weird thing is going on
elsewhere in the buffer, it won't interfere.

IOW, the parsing infrastructure (currently limited to syntax-tables and
Lisp) should be able to parse backwards.  I'm leaning towards
operator-precedence-grammars for now.

> enough to warrant the terrible backward  compatibility problems that would
> be generated by a switch to overlays.

The problem is performance (and no, it's not intrinsic to overlays, but
it's what we have, so until someone revampts the implementation it's
what we have to live with).

> That other programmers have resigned themselves to inferior
> fontification is no argument for Emacs to accept it.  Asynchronous
> fontification is completely unacceptable for me, and if it were to
> become commonplace and unavoidable in Emacs, I would simply stay with
> older versions.

I partly agree.  I imagine that it'd be OK for the fontification to take
place in 2 steps: one part on-the-fly and another part with a delay
(this other part could be a refinement which could include
warnings/error messages, but also updates to other parts of the code).


        Stefan

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Tue, Aug 11, 2009 at 7:00 PM, Stephen J. Turnbull<stephen@xemacs.org> wrote:
> Lennart Borgman writes:
>
>  > I consider this a bug.
>
> Why?  It is the *defining* characteristic of a *text* property that it
> is part of the text, and gets copied with its associated text.


I was thinking about the properties set by font-lock. They are dynamic
and belong to the context in the buffer they came from.

I thought that JIT-lock would be confused. But it looks like that is
taken care of now so then I do not see any problems with the current
default. Sorry for wasting time.

Re: "Font-lock is limited to text matching" is a myth

[joakim]

Daniel Colascione <danc@merrillpress.com> writes:


> - full parsers belong in the CEDET infrastructure where they can be
> accessed via a common API
> - if this API is not rich enough, it ought to be extended
> - normal major modes shouldn't rely on rigid grammars because that
> makes them brittle. Anything that needs such a grammar should be an
> optional and preferably generic minor-mode
>
> (I still think it'd be worthwhile to include the JS2 parser in CEDET,
> which supports parsers not necessarily generated with wisent or bison.
> It seems robust in its problem domain.)

Small piece of information: CEDET already includes a Javascript parser
that I provided some time ago.


Joakim Verona

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Tue, Aug 11, 2009 at 8:04 PM, <joakim@verona.se> wrote:
> Daniel Colascione <danc@merrillpress.com> writes:
>
>
>> - full parsers belong in the CEDET infrastructure where they can be
>> accessed via a common API
>> - if this API is not rich enough, it ought to be extended
>> - normal major modes shouldn't rely on rigid grammars because that
>> makes them brittle. Anything that needs such a grammar should be an
>> optional and preferably generic minor-mode
>>
>> (I still think it'd be worthwhile to include the JS2 parser in CEDET,
>> which supports parsers not necessarily generated with wisent or bison.
>> It seems robust in its problem domain.)
>
> Small piece of information: CEDET already includes a Javascript parser
> that I provided some time ago.


What benefits do the different approaches (your CEDET Semantic
Javascript parser and JS2, maybe also a theoretical sub-process rhino
(or similar) parser) have?

Re: "Font-lock is limited to text matching" is a myth

[Edward O'Connor]

>> languages, not the intersection.  There should, for instance, be a
>> font-lock-symbol-face for languages with distinguished symbols such
>> as Lisp, Scheme and Ruby.
>
> What for?

I imagine Steve doesn't mean all Lisp symbols, but rather keywords
(currently font locked with font-lock-builtin-face). I believe they're
the closest analogue to Ruby's symbols.


Ted

Re: "Font-lock is limited to text matching" is a myth

[joakim]

Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Tue, Aug 11, 2009 at 8:04 PM, <joakim@verona.se> wrote:
>> Daniel Colascione <danc@merrillpress.com> writes:
>>
>>
>>> - full parsers belong in the CEDET infrastructure where they can be
>>> accessed via a common API
>>> - if this API is not rich enough, it ought to be extended
>>> - normal major modes shouldn't rely on rigid grammars because that
>>> makes them brittle. Anything that needs such a grammar should be an
>>> optional and preferably generic minor-mode
>>>
>>> (I still think it'd be worthwhile to include the JS2 parser in CEDET,
>>> which supports parsers not necessarily generated with wisent or bison.
>>> It seems robust in its problem domain.)
>>
>> Small piece of information: CEDET already includes a Javascript parser
>> that I provided some time ago.
>
>
> What benefits do the different approaches (your CEDET Semantic
> Javascript parser and JS2, maybe also a theoretical sub-process rhino
> (or similar) parser) have?

I havent looked into what JS2 does. Cedet provides interfaces to hook in
parsers of varying kinds. If you use this interface, you are a cedet
parser, and you get the benefits that all the cedet tools work: semantic
navigation etc. currently there are no sub-process parsers implemeted
AFAIK, but there are sub-process tag generators.


>
-- 
Joakim Verona

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Tue, Aug 11, 2009 at 3:50 AM, Eric M. Ludlam<eric@siege-engine.com> wrote:
> On Tue, 2009-08-11 at 00:19 +0200, Lennart Borgman wrote:
>> On Tue, Aug 11, 2009 at 12:06 AM, Eric M. Ludlam<eric@siege-engine.com> wrote:
>>
>> Hi Eric,
>>
>> >  The concept of using the Semantic parser/generator framework for
>> > improving font-locking accuracy has come up many times.  No-one to my
>> > knowledge has attempted to mix the two.
>>
>>
>> Maybe that can easier be done if Semantic parser use
>> font-lock/JIT-lock timers and marking to keep track of what need to be
>> reparsed? (It is just a wild idea perhaps.)
>
> I'm not certain of how the font/jit lock works.  Semantic works by
> tracking edits (after-change-functions) and then on it's own timer, it
> coalesces the changes into parsable units.  It then reparses those
> units.


This is precisely what font/jit lock does too.

More precisely this is done by setting the text property 'fontified to
nil (if I remember correctly). JIT-lock will then pick up this and
refontify those parts.


> Font lock can refontify based on fairly small subsections of a buffer,
> such as a single code line, or a comment section.  Semantic's
> subsections are the size of functions, variables, and datatypes (ie, the
> tags it creates.)


Font/jit lock may also need bigger sections. In the case of mumamo it
does, for example. A change like adding a new major mode chunk
boundary may make it necessary to refontify the whole rest of the
buffer.


>> So you do a first pass with coarse parsing and then you look in the
>> sub-sections for details? Is this strictly necessary? I guess you are
>> looking for top level definitions in the first pass?
>>
>> Could that pass have its own state and continue upon demand (when an
>> item is not recognized) or is such a logic impossible?
>
> It could, but I have not done so.  Tagging information is not generally
> needed right away, so just waiting for the user to either ask for it, or
> sit idle for a while works pretty well.  The overhead of such an
> incremental parser isn't really needed.


The benefits coudl be that Emacs behaved much more smoothly. Both
because the background parsing could be interrupted and restarted
easier and because it could be easier to handle precomutation
(perhaps).

The overhead might perhaps be rather small if you could say "please
give me a JIT framework that computes these values: ... (list of
functions)... and runs according to scheme ...(JIT computation
style)..."? Just loose ideas of course, but...


> The needs between the tagging parser and the font-lock parser are
> different.  Font lock needs to colorize arbitrary blocks of text, and a
> tagging parser needs to parse everything, but only needs the data
> periodically.


A very good point that maybe can be translated to strucures in a JIT framework.


> Converting a tagging parser to a colorizing parser would be challenging
> because of these different uses.


They might use similar JIT frameworks but not the same timers (but
maybe cooperating dito).


>> >  I would imagine that the parsing engine in Semantic, if it is deemed
>> > critical by the maintainers, will get faster if key components are
>> > integrated into the C code.
>>
>> Is that part stable?
>
> Yes.  Not much is going on there.


Stefan seems positive to getting this kind of things into C code. To
me it seems very reasonable too.


>> >  Lastly, as David Engster stated, CEDET has decoration tools that
>> > decorate entire tags in some way, such as putting a line on top of
>> > functions.  This is a separate decoration activity not related to font
>> > lock, and something font lock would not be able to do reliably.
>>
>> Why not if it asks the parser?
>
> Font lock runs long before the parser bothers trying to catch up.  Font
> lock would needs hooks for after the parser runs.
> problems.


Perhaps it can be handled like this:

- The parser puts a tag on the parts where it wants special faces.
- It also sets 'fontified to nil on those regions. This will tell JIT
lock to refontify those parts.
- Then font-lock keywords functions supplied by the parser can set the
actual fontification when called by font/JIT-lock.

This will avoid the problem that the special parser fontification gets
thrown away when scrolling that Joakim mentioned.


> While font lock and semantic share a need for a parsing infrastructure,
> the where/when of the parsing is quite different.  It is possible to
> conceptually mix and match the parsers vs the schedulers.  In practice,
> the two tools have their own lengthy histories that will make that
> challenging.  Before tackling such a project, it would be wise to take
> multi-mode (or similar tool) into account.

Yes.

Thanks for the good explanations.


> Eric
>

Re: "Font-lock is limited to text matching" is a myth

[Steve Yegge]

[-- Attachment #1: Type: text/plain, Size: 6735 bytes --]

On Tue, Aug 11, 2009 at 9:04 AM, Stefan Monnier <monnier@iro.umontreal.ca>wrote:

>
> Bringing espresso-mode and js2-mode closer together would be good
> (e.g. by merging them into a single mode with customization options
> allowing to choose between different ways to do highlighting, imenu,
> etc...).


I've now had a chance to take a quick look at espresso-mode, and it
looks like it's most likely a better choice for inclusion in Emacs, assuming
it works well.  I haven't had a chance to use it for actual work yet, but I
will at the earliest opportunity.

js2-mode was a fairly quick/dirty mode wrapped around a parser that was
written to feed an interpreter, not to feed an editing mode.  espresso-mode
is clearly designed from the ground up to be a good Emacs mode for JS.

I like the idea of making js2-mode a minor mode that can supply parse
errors and warnings asynchronously if enabled.  That's its core strength in
any case.  If Daniel's amenable I'd be happy to start working with him in
that direction.


> Especially on the indentation side since it seems they ahre some of
> their history.
>

I'm guessing that indentation needs a lot more work to make it
configurable enough to satisfy the needs of companies like mine, where
the coding style guides would specify indentation rules in pixels/points
if they could get away with it.

It would be nice if someone would dive in and enhance cc-engine to
handle JavaScript constructs.  I may give it another go at some point
if nobody else does.



> > Daniel's objections to js2-mode's non-interaction with font-lock
> > apply equally to the non-interaction with cc-engine's indentation
> > configuration system.  The indent configuration for JavaScript should
> > share as many settings as practical with cc-mode.
>
> I'm not too fond of cc-mode's indentation code and configuration,
> actually, so I don't mind if js2-mode doesn't share anything with it
> (tho I won't oppose a change in this respect either).
>

I'd like to hear more about the objections.  I realize it's horribly
complex,
but I've looked at other configurable indenters and they always wind up
being too complex for most users as well -- gigantic manuals, custom
minilanguages, the works.


>
> >   3) indentation in "normal" Emacs modes also runs synchronously as
> >      the user types.  Waiting 500-800 msec or more for the parse to
> >      finish is (I think) not acceptable for indentation.  For small
> >      files the parse time is acceptable, but it would not be generally
> >      scalable.
>
> Agreed.  Do you happen to know who other IDEs do about it?
>

Yes.  Eclipse has a fast, inaccurate parser that runs inline as you type,
and a slow, accurate one that lags behind by a few hundred ms.  (They
handle name resolution this way as well, except the fast one runs with
the slow parser, and the slow one can take minutes to hours.)

Eclipse uses the fast/inaccurate parser for both fontification and
indentation.  The slower parser is used when (for instance) you want to
reformat a block of code -- something like a cross between indent-region
and fill-region for source code.  I'm not an Eclipse user myself, so I'm
not familiar with all the ins and outs, but this is the basic approach they
take.


> > Va) Inadequate/insufficient style names
>
> [ Putting on my functional programmer hat here. ]
> All you're saying here is that your languages have too many concepts.
>

Yes, well, of course.  But they're not _my_ languages now, are they?


>
> [ Putting on my Emacs maintainer hat again. ]
> highlighting should be about helping the user understand his code:
> highlighting every character with a different color is the way to
> get there.


I'm not 100% sure I follow this statement.  Do you mean "not the way"?


> You may want to help him find structure (e.g. make
> function/method declaration stand out), you may want to help him not get
> confused (highlight strings and comments differently), you may want to
> attract his attention to weird things (undeclared variables, ...), but
> I highly doubt that highlighting function parameters differently from
> local variables will help her in any way.
>

We don't know this, and in fact cannot know it a priori, since there are new
languages appearing all the time.  And templates with mixed languages
complicate things further.

I think it's best not to be in the business of dictating or even advising
taste.
We should focus on making things flexible enough for people to make the
distinctions they wish to make.


>
> This said, the set of default faces deserves a rethink as well as some
> additions, yes.
>
> > languages, not the intersection.  There should, for instance, be a
> > font-lock-symbol-face for languages with distinguished symbols such
> > as Lisp, Scheme and Ruby.
>
> What for?


I meant for quoted symbols -- e.g. I color 'foo, :foo and #'foo differently
in my elisp code.  They all have colors from the same region of rgb space,
but they're different enough that I can tell keyword args from
function-quoted
symbols with just a glance from far away.  I find this helpful.  Others'
MMV.


>
> > Vf) No font-lock interface for setting exact style runs
> [...]
> > The problem is that I need a way, in a given font-lock redisplay, to
> > say "highlight the region from X to Y with text properties {Z}".
>
> I'm not sure I understand the problem.  What's wrong with
> put-text-property?
>

font-lock biffs my properties.


>
> > When I assert that it's not possible, I understand that it's
> > _theoretically_ possible.  Given a JavaScript file with 2500 style
> > runs, assuming I had that information available at font-lock time, I
> > could return a matcher that contains 2500 regular expressions, each
> > one of which is tailored to match one and exactly one region in the
> > buffer.
>
> Just place in font-lock-keywords a MATCHER that is a function whose code
> walks the list of your "runs" (checking which of your runs are within
> (point) and LIMIT) and uses add-text-properties on them; and finally
> returns nil.


This is different from what Daniel advised, and neither approach is very
well documented (if at all).  I will try them both and report back when I
can.


> > Vg) Lack of differentiation between mode- and minor-mode styles
> [...]
> > As far as I can tell, the officially supported mechanism for
> > adding additional font-lock patterns is `font-lock-add-keywords'.
> > This either appends or prepends the keywords to the defaults.
>
> Yes, this sucks.  It should be replaced by a more declarative interface.
>

A simple workaround for now might be to keep pointers to the originals
and the extras in an alist that remembers who added which keywords.

-steve


>
>
>        Stefan
>

[-- Attachment #2: Type: text/html, Size: 10219 bytes --]

Re: "Font-lock is limited to text matching" is a myth

[Eric M. Ludlam]

On Tue, 2009-08-11 at 12:04 -0400, Stefan Monnier wrote:
> > I. Asynchronous parsing
> 
> BTW, I'm interested in adding core-Emacs support for such parsing, so if
> you have any ideas about it, please share them.  The way I see it, there
> should be a table-based parsing engine written in C running in
> a separate thread, so the question are "what should the tables look
> like?", "what should the output look like?", "what should the engine
> look like?", "how should the asynchronous code synchronize with the rest
> of Emacs?".  Any help along this effort would be welcome.

Hi,

  I don't think you could define a way to parse some things, like C++
with pre-processor support, via table declaration.  There will always be
some decision that needs to be made that the table designer misses out
on.  For example, to selectively exclude a block of code in #ifdef
MYSYM ... #endif.

  I do think that if there was a set of Emacs Lisp functions that were
considered "thread safe" that parser authors like myself would find a
way to restrict ourselves to that list in those situations where the
special built-in parser needs to make such decisions.  It might even be
a special set of new functions, like 'parser-cons', or 'parser-car',
though in writing it, it does feel icky.

  The project I was going to pick up after the current CEDET
merge/release stuff is related to this, though I was going to build a
separate process and communicate via pipe/socket/whatever with it. This
is because I'm seeing situations where the tag database is so large that
the whole machine slows down.  In this case it is not related to parser
speed, just the data structure size. Imagine a Lisp data structure for
the entirety of the Linux kernel's symbol space.  Such a subprocess
would also enable background parsing of files not currently in buffers
of the active Emacs.

  Let me summarize.  I think CEDET has managed to use timers to overcome
asynchronous parsing problems for the purpose of TAGGING files, and may
not be able to take advantage of asynchronous table-driven parsing
system.  There are, however, different problems CEDET faces that could
take advantage of asynchronous behaviors, but mostly in the way you can
asynchronously run "make TAGS" for etags.  As such, table driven
asynchronous parsing system could focus on a narrower set of
requirements that are not as rigorous for purposes of colorizing, and
have a synchronous form for when other kinds of logic are needed.

  As far as how to define tables for a parsing system written in C, an
old-school solution is to just use the flex/bison engines under the
Emacs Lisp API.  There are a lot of new parser generator systems though,
and I don't really know what the best one might be.  Defining some new
system for parsing using a more modern technique that has enough hooks
for tag generation would be easy to integrate into CEDET, and would
obsolete nothing.

  One of the hairier parts of the CEDET parser is the lexical analyzer.
I remember jumping through hoops trying to squeak out every stray
(if ..) or function call from the core loop, while keeping things
declarative and flexible enough to handle all the various languages.
Even so, the variable-length lexical token still ended up shuttling text
strings around in some situations where I would have preferred
references to a buffer.  Major performance was gained by treating
parenthetical groups as single lex tokens, expanding only if actually
needed.  Tagging parsers can then skip parsing function bodies, for
example, which provides a nice speed boost, but is not so good for
colorizing.

  As far as making this asynchronous, however, there aren't too many
languages that can use nothing but the core lexical types.

  Once you have a lexical token stream, David Ponce's wisent parser (a
port of bison) is top-notch, and very effective.  Moving bits into C may
not provide a huge speed boost, and would certainly not be separable
from lisp code that needs to be synchronous as far as TAGGING is
concerned.  It might be possible to define simplified code blocks such
as (FONT $1 'function) and get away with making asynchronous parsers.
I'm not sure if that is sufficient.

  Error handling in bison is a bit confusing for the uninitiated, and
hard to get right.  CEDET solved this by making the parsers implicitly
iterative at the framework level.  That means when you enter the parser
engine, it will return nil, or a single tag.  If it gets a nil, it skips
that lexical token, and tries again on the next one.  This makes it
robust to "junk" between tags, or between fcn arguments, or between
commands.

  After babbling for a while, I would guess that Stefan is probably
asking for help identifying something like a syntax table.  I think
lexical analysis is common between all the parser generator frameworks,
and has the potential to make a data structure larger than the buffer it
was derived from.  Deriving a push/pop lexical analyzer structure that
shares data with the buffer text, or can even cache itself on top of the
buffer so it doesn't need to "analyze" the whole thing over and over
would be a great first step for any parsing system.

Eric

Re: "Font-lock is limited to text matching" is a myth

[Miles Bader]

"Eric M. Ludlam" <eric@siege-engine.com> writes:
> As far as how to define tables for a parsing system written in C, an
> old-school solution is to just use the flex/bison engines under the
> Emacs Lisp API.  There are a lot of new parser generator systems
> though, and I don't really know what the best one might be.
>
> One of the hairier parts of the CEDET parser is the lexical analyzer.

Slightly off-topic, but I'm a huge fan of "LPeg" [1], which is a
pattern-matching library for Lua, based on Parsing Expression Grammars
(PEGs).

I've always wished for something like LPeg in elisp, and since Lua is at
heart quite lisp-like (despite the very different syntax), I think it
could work very well.  Maybe it wouldn't be too hard to adapt LPeg's
core to elisp (it's licensed under the BSD license).

[There's a popular implementation technique for PEGs called "packrat
parsers", and many PEG libraries use that technique -- however
apparently packrat parsers have some serious problems in practice, so
LPeg uses a different technique.  See [2] for a discussion of this, and
of the LPeg implementation in detail.]

Some nice things about LPeg:

  (1) It's very fast.

  (2) It's very concise; for typical usage, it's essentially like
      writing a parser in yacc or whatever.

  (3) It makes it trivial to insert code and hooks at any point in the
      parse; not just "actions", but code that can determine how the
      parsing happens.  This give a _huge_ amount of flexibility.

  (4) It's very easy to "think about", despite the flexibility and
      presence of arbitrary code driving parsing, because it works kind
      of like a recursive descent parser, operating greedily (but
      provides mechanisms to do automatic backtracking when necessary).
       
  (5) Because it's so fast and flexible, typical practice is to _not_
      have a separate lexical analyzer, but just do lexical analysis in
      the parser.  This easier and more convenient, and also makes it
      easier to use parser information in lexical analysis (e.g., the
      famous "typedef" vs. "id" issue in C parsers).

  (6) It's very small -- the entire implementation (core engine and Lua
      interface) is only 2000 lines of C.

[The standard way to use LPeg in Lua uses Lua's ability to easily
overload standard operators, giving them LPeg-specific meanings when
invoked on first-class "pattern" objects.  That can't be done in elisp,
but I think a more lispy approach should be easy.]

[1] http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html

[2] http://www.inf.puc-rio.br/~roberto/docs/peg.pdf

-Miles

-- 
Zeal, n. A certain nervous disorder afflicting the young and inexperienced.

Re: "Font-lock is limited to text matching" is a myth

[Xah Lee]

[-- Attachment #1: Type: text/plain, Size: 3428 bytes --]

i very much second this! PEG's the next level of regex, and i expect it to
replace regex in some sense in the coming years for the whole field of text
processing.

there are currently 2 of them in elisp as far as i know:

 * http://www.emacswiki.org/cgi-bin/wiki/ParserCompiler (2008) by Mike
Mattie.

 * http://www.emacswiki.org/emacs/ParsingExpressionGrammars (2008) by Helmut
Eller.

it'd be much better if PEG is integrated from the ground up in elisp,
possibly implemented in C or from other libs for speed. I imagine functions
that takes a regex can have a version with PEG.

  Xah

On Tue, Aug 11, 2009 at 11:43 PM, Miles Bader <miles@gnu.org> wrote:

> "Eric M. Ludlam" <eric@siege-engine.com> writes:
> > As far as how to define tables for a parsing system written in C, an
> > old-school solution is to just use the flex/bison engines under the
> > Emacs Lisp API.  There are a lot of new parser generator systems
> > though, and I don't really know what the best one might be.
> >
> > One of the hairier parts of the CEDET parser is the lexical analyzer.
>
> Slightly off-topic, but I'm a huge fan of "LPeg" [1], which is a
> pattern-matching library for Lua, based on Parsing Expression Grammars
> (PEGs).
>
> I've always wished for something like LPeg in elisp, and since Lua is at
> heart quite lisp-like (despite the very different syntax), I think it
> could work very well.  Maybe it wouldn't be too hard to adapt LPeg's
> core to elisp (it's licensed under the BSD license).
>
> [There's a popular implementation technique for PEGs called "packrat
> parsers", and many PEG libraries use that technique -- however
> apparently packrat parsers have some serious problems in practice, so
> LPeg uses a different technique.  See [2] for a discussion of this, and
> of the LPeg implementation in detail.]
>
> Some nice things about LPeg:
>
>  (1) It's very fast.
>
>  (2) It's very concise; for typical usage, it's essentially like
>      writing a parser in yacc or whatever.
>
>  (3) It makes it trivial to insert code and hooks at any point in the
>      parse; not just "actions", but code that can determine how the
>      parsing happens.  This give a _huge_ amount of flexibility.
>
>  (4) It's very easy to "think about", despite the flexibility and
>      presence of arbitrary code driving parsing, because it works kind
>      of like a recursive descent parser, operating greedily (but
>      provides mechanisms to do automatic backtracking when necessary).
>
>  (5) Because it's so fast and flexible, typical practice is to _not_
>      have a separate lexical analyzer, but just do lexical analysis in
>      the parser.  This easier and more convenient, and also makes it
>      easier to use parser information in lexical analysis (e.g., the
>      famous "typedef" vs. "id" issue in C parsers).
>
>  (6) It's very small -- the entire implementation (core engine and Lua
>      interface) is only 2000 lines of C.
>
> [The standard way to use LPeg in Lua uses Lua's ability to easily
> overload standard operators, giving them LPeg-specific meanings when
> invoked on first-class "pattern" objects.  That can't be done in elisp,
> but I think a more lispy approach should be easy.]
>
> [1] http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html
>
> [2] http://www.inf.puc-rio.br/~roberto/docs/peg.pdf
>
> -Miles
>
> --
> Zeal, n. A certain nervous disorder afflicting the young and inexperienced.
>
>
>

[-- Attachment #2: Type: text/html, Size: 4670 bytes --]

Re: "Font-lock is limited to text matching" is a myth

[Chong Yidong]

Steve Yegge <stevey@google.com> writes:

> I've now had a chance to take a quick look at espresso-mode, and it
> looks like it's most likely a better choice for inclusion in Emacs,
> assuming it works well.
>
> I like the idea of making js2-mode a minor mode that can supply parse
> errors and warnings asynchronously if enabled.  That's its core
> strength in any case.  If Daniel's amenable I'd be happy to start
> working with him in that direction.

Good, thanks for taking the time to look at this.

Would it be possible to call the javascript mode in Emacs
javascript-mode?  Is there any historical reason?

asynchronous parsing (was: "Font-lock is limited to text matching" is a myth)

[Ted Zlatanov]

On Tue, 11 Aug 2009 12:04:20 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> I. Asynchronous parsing
SM> BTW, I'm interested in adding core-Emacs support for such parsing, so if
SM> you have any ideas about it, please share them.  The way I see it, there
SM> should be a table-based parsing engine written in C running in
SM> a spearate thread, so the question are "what should the tables look
SM> like?", "what should the output look like?", "what should the engine
SM> look like?", "how should the asynchronous code synchronize with the rest
SM> of Emacs?".  Any help along this effort would be welcome.

Maybe it would make sense to make this engine the first implementation
of an asynchronous process in Emacs.  We know we need general-purpose
asynchronous processes, it's been discussed many times.  Doing it with a
parser, a read-only process with internal state, would be a good start.
The difference from your suggestion is that rather than implementing a
one-off, the goal becomes the first cut of a general facility.

I bring this up because the need for asynchronous processes keeps coming
up in Gnus discussions...

Ted

Re: "Font-lock is limited to text matching" is a myth

[Lennart Borgman]

On Wed, Aug 12, 2009 at 3:48 PM, Chong Yidong<cyd@stupidchicken.com> wrote:

> Would it be possible to call the javascript mode in Emacs
> javascript-mode?  Is there any historical reason?

There have been some variations of "javascript-mode", like there has
been for "css-mode".

I think the main difficulty might be that some people might be loading
an old version of it. That is a real problem that has caused a lot of
trouble that I have seen for example in mumamo. Better error/warning
messages would be useful for taking care of this (maybe something
along the lines that Juri suggested, collecting messages and showing
the more important of them after a command has finished / Emacs has
started).

Customiizations too, of course, but I hope those will be taken care of
by the javascript/espresso maintainers. (They will probably take care
of the needs people have seen and redirect them to useful Emacs
functions or implement them.)

Re: asynchronous parsing

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 11 Aug 2009 12:04:20 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>
>>> I. Asynchronous parsing
> SM> BTW, I'm interested in adding core-Emacs support for such parsing, so if
> SM> you have any ideas about it, please share them.  The way I see it, there
> SM> should be a table-based parsing engine written in C running in
> SM> a spearate thread, so the question are "what should the tables look
> SM> like?", "what should the output look like?", "what should the engine
> SM> look like?", "how should the asynchronous code synchronize with the rest
> SM> of Emacs?".  Any help along this effort would be welcome.
>
> Maybe it would make sense to make this engine the first implementation
> of an asynchronous process in Emacs.  We know we need general-purpose
> asynchronous processes, it's been discussed many times.  Doing it with a
> parser, a read-only process with internal state, would be a good start.
> The difference from your suggestion is that rather than implementing a
> one-off, the goal becomes the first cut of a general facility.
>
> I bring this up because the need for asynchronous processes keeps coming
> up in Gnus discussions...

Sounds very interesting, how would that work?


> Ted
>
>
-- 
Joakim Verona

Re: asynchronous parsing

[Ted Zlatanov]

On Wed, 12 Aug 2009 19:16:15 +0200 joakim@verona.se wrote: 

j> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Tue, 11 Aug 2009 12:04:20 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>> 
>>>> I. Asynchronous parsing
SM> BTW, I'm interested in adding core-Emacs support for such parsing, so if
SM> you have any ideas about it, please share them.  The way I see it, there
SM> should be a table-based parsing engine written in C running in
SM> a spearate thread, so the question are "what should the tables look
SM> like?", "what should the output look like?", "what should the engine
SM> look like?", "how should the asynchronous code synchronize with the rest
SM> of Emacs?".  Any help along this effort would be welcome.
>> 
>> Maybe it would make sense to make this engine the first implementation
>> of an asynchronous process in Emacs.  We know we need general-purpose
>> asynchronous processes, it's been discussed many times.  Doing it with a
>> parser, a read-only process with internal state, would be a good start.
>> The difference from your suggestion is that rather than implementing a
>> one-off, the goal becomes the first cut of a general facility.
>> 
>> I bring this up because the need for asynchronous processes keeps coming
>> up in Gnus discussions...

j> Sounds very interesting, how would that work?

The same way Stefan's suggestion would work: a separate thread running
this parser.  The difference is that we don't just do one thread for the
parser and code especially for that purpose, but instead think of a
general thread facility the parser can use.  Once we prototype the
facility, we'll know better what works and what doesn't.

The specifics of the parser, if that's what you're curious about, are
not important to me (I don't plan to write such parsers or configure
them myself).  I just need the general threading facility for Gnus work.

Ted

Re: asynchronous parsing

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 12 Aug 2009 19:16:15 +0200 joakim@verona.se wrote: 
>
> j> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> On Tue, 11 Aug 2009 12:04:20 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 
>>> 
>>>>> I. Asynchronous parsing
> SM> BTW, I'm interested in adding core-Emacs support for such parsing, so if
> SM> you have any ideas about it, please share them.  The way I see it, there
> SM> should be a table-based parsing engine written in C running in
> SM> a spearate thread, so the question are "what should the tables look
> SM> like?", "what should the output look like?", "what should the engine
> SM> look like?", "how should the asynchronous code synchronize with the rest
> SM> of Emacs?".  Any help along this effort would be welcome.
>>> 
>>> Maybe it would make sense to make this engine the first implementation
>>> of an asynchronous process in Emacs.  We know we need general-purpose
>>> asynchronous processes, it's been discussed many times.  Doing it with a
>>> parser, a read-only process with internal state, would be a good start.
>>> The difference from your suggestion is that rather than implementing a
>>> one-off, the goal becomes the first cut of a general facility.
>>> 
>>> I bring this up because the need for asynchronous processes keeps coming
>>> up in Gnus discussions...
>
> j> Sounds very interesting, how would that work?
>
> The same way Stefan's suggestion would work: a separate thread running
> this parser.  The difference is that we don't just do one thread for the
> parser and code especially for that purpose, but instead think of a
> general thread facility the parser can use.  Once we prototype the
> facility, we'll know better what works and what doesn't.

Its this I'm interested in.

> The specifics of the parser, if that's what you're curious about, are
> not important to me (I don't plan to write such parsers or configure
> them myself).  I just need the general threading facility for Gnus work.

That would indeed be great.

> Ted
>
>
-- 
Joakim Verona

Re: "Font-lock is limited to text matching" is a myth

[Steve Yegge]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

On Wed, Aug 12, 2009 at 6:48 AM, Chong Yidong <cyd@stupidchicken.com> wrote:

> Steve Yegge <stevey@google.com> writes:
>
> > I've now had a chance to take a quick look at espresso-mode, and it
> > looks like it's most likely a better choice for inclusion in Emacs,
> > assuming it works well.
> >
> > I like the idea of making js2-mode a minor mode that can supply parse
> > errors and warnings asynchronously if enabled.  That's its core
> > strength in any case.  If Daniel's amenable I'd be happy to start
> > working with him in that direction.
>
> Good, thanks for taking the time to look at this.
>
> Would it be possible to call the javascript mode in Emacs
> javascript-mode?  Is there any historical reason?
>

+1 -- I was just going to suggest that.  "espresso" evokes Java,
and "js2" (es4) is now moribund.  I'll make sure we fix it up once
we've integrated the modes.  (We'll probably use js- as the package
prefix, for readability and convenience.)

-steve

[-- Attachment #2: Type: text/html, Size: 1478 bytes --]

Re: asynchronous parsing

[Stefan Monnier]

> Maybe it would make sense to make this engine the first implementation
> of an asynchronous process in Emacs.  We know we need general-purpose
> asynchronous processes, it's been discussed many times.  Doing it with a
> parser, a read-only process with internal state, would be a good start.

Am I really so transparent?


        Stefan

Re: asynchronous parsing

[Ted Zlatanov]

On Wed, 12 Aug 2009 22:51:44 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Maybe it would make sense to make this engine the first implementation
>> of an asynchronous process in Emacs.  We know we need general-purpose
>> asynchronous processes, it's been discussed many times.  Doing it with a
>> parser, a read-only process with internal state, would be a good start.

SM> Am I really so transparent?

Hmm, maybe I should keep quiet about your nefarious plans next time.

Ted

Re: "Font-lock is limited to text matching" is a myth

[Stefan Monnier]

>> I'm not too fond of cc-mode's indentation code and configuration,
>> actually, so I don't mind if js2-mode doesn't share anything with it
>> (tho I won't oppose a change in this respect either).

> I'd like to hear more about the objections.  I realize it's horribly
> complex, but I've looked at other configurable indenters and they
> always wind up being too complex for most users as well -- gigantic
> manuals, custom minilanguages, the works.

I have two reasons not to like it: the horribly complex indentation
code, and the horribly complex compilation and configuration "stuff"
(very non-standard with very dubious benefits).
But I don't claim there's something better out there to fulfill the
same requirements.

> Yes.  Eclipse has a fast, inaccurate parser that runs inline as you type,
> and a slow, accurate one that lags behind by a few hundred ms.  (They
> handle name resolution this way as well, except the fast one runs with
> the slow parser, and the slow one can take minutes to hours.)

> Eclipse uses the fast/inaccurate parser for both fontification and
> indentation.  The slower parser is used when (for instance) you want to
> reformat a block of code -- something like a cross between indent-region
> and fill-region for source code.  I'm not an Eclipse user myself, so I'm
> not familiar with all the ins and outs, but this is the basic approach they
> take.

Thanks.  That makes a lot of sense.

>> [ Putting on my functional programmer hat here. ]
>> All you're saying here is that your languages have too many concepts.
> Yes, well, of course.  But they're not _my_ languages now, are they?

They're yours in the sense that you care for them and (presumably)
use them.

>> [ Putting on my Emacs maintainer hat again. ]
>> highlighting should be about helping the user understand his code:
>> highlighting every character with a different color is the way to
>> get there.
> I'm not 100% sure I follow this statement.  Do you mean "not the way"?

Yes, the "not" somehow escaped, sorry.

>> You may want to help him find structure (e.g. make
>> function/method declaration stand out), you may want to help him not get
>> confused (highlight strings and comments differently), you may want to
>> attract his attention to weird things (undeclared variables, ...), but
>> I highly doubt that highlighting function parameters differently from
>> local variables will help her in any way.

> We don't know this, and in fact cannot know it a priori, since there are new
> languages appearing all the time.  And templates with mixed languages
> complicate things further.

In Emacs we generally work on a need basis (pretty much like eXtreme
Programming advocates, although the relationship doesn't go much
further I believe).  It's very easy for a major mode to add its own face
if the needs comes up.  Until now, I haven't come across any language
where highlighting function parameters specially would be useful.

> I think it's best not to be in the business of dictating or even
> advising taste.  We should focus on making things flexible enough for
> people to make the distinctions they wish to make.

We'll have to agree to disagree on this one.

>> > Vf) No font-lock interface for setting exact style runs
>> [...]
>> > The problem is that I need a way, in a given font-lock redisplay, to
>> > say "highlight the region from X to Y with text properties {Z}".
>> I'm not sure I understand the problem.  What's wrong with
>> put-text-property?
> font-lock biffs my properties.

That's another problem.  If you do it from font-lock-keywords, then it
works just fine.

>> Just place in font-lock-keywords a MATCHER that is a function whose code
>> walks the list of your "runs" (checking which of your runs are within
>> (point) and LIMIT) and uses add-text-properties on them; and finally
>> returns nil.
> This is different from what Daniel advised, and neither approach is
> very well documented (if at all).  I will try them both and report
> back when I can.

I believe it is the same as what Daniel advised, actually.  It's not
well documented simply because there's not much to document: the MATCHER
is a function so it can do anything it wants.

>> > Vg) Lack of differentiation between mode- and minor-mode styles
>> [...]
>> > As far as I can tell, the officially supported mechanism for
>> > adding additional font-lock patterns is `font-lock-add-keywords'.
>> > This either appends or prepends the keywords to the defaults.
>> Yes, this sucks.  It should be replaced by a more declarative interface.

> A simple workaround for now might be to keep pointers to the originals
> and the extras in an alist that remembers who added which keywords.

If you assume that there are no coincidences (same keywords added by
different (minor)modes), then as long as each (minor)mode adds and
removes his own, then it's not too terrible.  But, yes, occasionally
it's a pain (when you want to reconstruct the whole thing for one
reason or another).  Mostly it means that you have to remember what
you've added so that you can properly remove it.


        Stefan

Re: eval-after-load not harmful after all

[Leo]

On 2009-08-11 16:08 +0100, Stefan Monnier wrote:
>>> I note that all the advocates of e-a-l are package maintainers and UI
>>> types; all the deprecators are core.  This is an inherent tension, and
>>> I think it should be resolved in favor of protecting the core.
>
> Actually, I think "tension" is the wrong word.  The core just want to
> send a message, nothing more.  The main message is "eval-after-load and
> advice should ideally not be necessary: we happily support them and
> won't be offended at all if you use them, but if you do need them,
> please understand that it is evidence of a shortcoming in some package
> that should be reported so we can fix it".

Emacs -Q already has a non-nil after-load-alist value. I think this is a
really bad example.

Is there any way to cleanly unload an eval-after-load form? For example,
if a package eval-after-load something that interferes with a user's
personal setting, what to do?

-- 
Leo's Emacs uptime: 12 days, 21 hours, 31 minutes, 19 seconds

Re: eval-after-load not harmful after all

[Lennart Borgman]

On Sun, Aug 16, 2009 at 11:43 PM, Leo<sdl.web@gmail.com> wrote:

> Emacs -Q already has a non-nil after-load-alist value. I think this is a
> really bad example.


I think at least for CUA-mode and semantic-mode they instead good
examples on that eval-after-load can be useful even within Emacs. If I
understand it right these checks if an old version of the modes have
been loaded before the one that is now bundled with Emacs. Can you see
a better way to do this? I can't.


> Is there any way to cleanly unload an eval-after-load form? For example,
> if a package eval-after-load something that interferes with a user's
> personal setting, what to do?


Can't you just delete the entry in after-load-alist? Or do you want to
do it afterwards? I can see no way to do that in the general case.

Re: eval-after-load not harmful after all

[Leo]

On 2009-08-17 01:34 +0100, Lennart Borgman wrote:
> On Sun, Aug 16, 2009 at 11:43 PM, Leo<sdl.web@gmail.com> wrote:
>
>> Emacs -Q already has a non-nil after-load-alist value. I think this is a
>> really bad example.
>
>
> I think at least for CUA-mode and semantic-mode they instead good
> examples on that eval-after-load can be useful even within Emacs. If I
> understand it right these checks if an old version of the modes have
> been loaded before the one that is now bundled with Emacs. Can you see
> a better way to do this? I can't.

I just found out those are added from within simple.el. But they look to me
more like a last minute hack before release although I am not sure.

What those e-v-l's do is pretty little and they don't belong there. If
there's a bug in, say, bbdb 2.29, should a new entry be added? After all
if a package is installed by a user, it is the user's responsibility.

>> Is there any way to cleanly unload an eval-after-load form? For example,
>> if a package eval-after-load something that interferes with a user's
>> personal setting, what to do?
>
> Can't you just delete the entry in after-load-alist? Or do you want to
> do it afterwards? I can see no way to do that in the general case.

Each entry could involve multiple e-v-l's. There's extra care that needs
to be taken.

-- 
Leo's Emacs uptime: 13 days, 11 hours, 21 minutes, 36 seconds

Re: eval-after-load not harmful after all

[Lennart Borgman]

On Mon, Aug 17, 2009 at 1:44 PM, Leo<sdl.web@gmail.com> wrote:

>> I think at least for CUA-mode and semantic-mode they instead good
>> examples on that eval-after-load can be useful even within Emacs. If I
>> understand it right these checks if an old version of the modes have
>> been loaded before the one that is now bundled with Emacs. Can you see
>> a better way to do this? I can't.
>
> I just found out those are added from within simple.el. But they look to me
> more like a last minute hack before release although I am not sure.


I am afraid you are totally misunderstanding this. You will see that
if you try to answer my question above.


> What those e-v-l's do is pretty little and they don't belong there. If
> there's a bug in, say, bbdb 2.29, should a new entry be added? After all
> if a package is installed by a user, it is the user's responsibility.


Yes, but that does not mean you should not try to help.

And these eval-after-load has probably avoided a lot of work for those
involved in Emacs development.

Re: eval-after-load not harmful after all

[Leo]

On 2009-08-17 12:55 +0100, Lennart Borgman wrote:
>> I just found out those are added from within simple.el. But they look to me
>> more like a last minute hack before release although I am not sure.
>
> I am afraid you are totally misunderstanding this. You will see that
> if you try to answer my question above.

There's hardly room for misunderstanding. I admit that e-v-l is probably
the only tool available at the moment for dealing with this problem but
it is by no means the optimal way.

-- 
Leo's Emacs uptime: 13 days, 12 hours, 9 minutes, 59 seconds

Re: eval-after-load not harmful after all

[Lennart Borgman]

On Mon, Aug 17, 2009 at 2:26 PM, Leo<sdl.web@gmail.com> wrote:
> On 2009-08-17 12:55 +0100, Lennart Borgman wrote:
>>> I just found out those are added from within simple.el. But they look to me
>>> more like a last minute hack before release although I am not sure.
>>
>> I am afraid you are totally misunderstanding this. You will see that
>> if you try to answer my question above.
>
> There's hardly room for misunderstanding. I admit that e-v-l is probably
> the only tool available at the moment for dealing with this problem but
> it is by no means the optimal way.


If you think so, then try to suggest another way. If you can then I am
admittedly misunderstanding this.

BTW I think nxml-mode also should be checked this way. Could this
please be added to Emacs?

Re: "Font-lock is limited to text matching" is a myth

[Stefan Monnier]

> Slightly off-topic, but I'm a huge fan of "LPeg" [1], which is a
> pattern-matching library for Lua, based on Parsing Expression Grammars
> (PEGs).

FWIW, the "parsing machine" used by LPEG (at least the one described in
their SP&E article) is very similar to the one used in src/regex.c.
So if someone is up to it, it shouldn't be too difficult to extend
Emacs's regexes to something like LPEG.


        Stefan