web-mode.el

web-mode.el

[Bois Francois-Xavier]

Dear Emacsians,

I've written web-mode.el, a major mode for PHP/HTML templates files.
It aims to natively indent and syntaxcolor according to the "context"
of (point) (which can be in an HTML part or in a PHP/JavaScript/CSS
bloc).
As it is my first experience with emacs lisp programming, I would be
really grateful to receive your feedbacks on any aspect of this mode
(core programming, function/var naming, conventions, etc.). For many
aspects, I wonder if I use hacks or correct ways to do the job (ex. I
look at the face name in my indent function to know if (point) is in a
comment or a string).
The mode is available here:
http://fxbois.free.fr/lisp/web-mode.el

I will also publish a web page which will look like this:
http://fxbois.free.fr/lisp/web-mode.html

Perhaps you also have comments about it.

I will release this mode on github after reading your comments/advices.

Thank you in advance.

François-Xavier Bois

ps: sorry for my poor english

Re: web-mode.el

[Lennart Borgman]

On Tue, Jun 12, 2012 at 12:24 AM, Bois Francois-Xavier <fxbois@gmail.com> wrote:
>
> Dear Emacsians,
>
> I've written web-mode.el, a major mode for PHP/HTML templates files.
> It aims to natively indent and syntaxcolor according to the "context"
> of (point) (which can be in an HTML part or in a PHP/JavaScript/CSS
> bloc).
> As it is my first experience with emacs lisp programming, I would be
> really grateful to receive your feedbacks on any aspect of this mode
> (core programming, function/var naming, conventions, etc.). For many
> aspects, I wonder if I use hacks or correct ways to do the job (ex. I
> look at the face name in my indent function to know if (point) is in a
> comment or a string).
> The mode is available here:
> http://fxbois.free.fr/lisp/web-mode.el
>
> I will also publish a web page which will look like this:
> http://fxbois.free.fr/lisp/web-mode.html
>
> Perhaps you also have comments about it.

Dear Bois, have you compared the functionality in web-mode.el with
that you can get with nXthml? (See EmacsWiki for links.)

Re: web-mode.el

[Bois Francois-Xavier]

Dear Lennart

Yes before starting this project I tried almost all other alternatives
and none of them really fully worked for me. That's why I started
web-mode.el ; I use it now almost every day for a few months.

Features that were really important for me are :
 - indentation (in particular HTML tags, HTML attributes, PHP arrays
or function calls spanning on multiple lines)
 - coherent syntax highlighting (according to the context)
 - centralization (config)
 - HTML element navigation
 - maintainability, evolutivity
 - speed



On Tue, Jun 12, 2012 at 1:37 PM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Tue, Jun 12, 2012 at 12:24 AM, Bois Francois-Xavier <fxbois@gmail.com> wrote:
>>
>> Dear Emacsians,
>>
>> I've written web-mode.el, a major mode for PHP/HTML templates files.
>> It aims to natively indent and syntaxcolor according to the "context"
>> of (point) (which can be in an HTML part or in a PHP/JavaScript/CSS
>> bloc).
>> As it is my first experience with emacs lisp programming, I would be
>> really grateful to receive your feedbacks on any aspect of this mode
>> (core programming, function/var naming, conventions, etc.). For many
>> aspects, I wonder if I use hacks or correct ways to do the job (ex. I
>> look at the face name in my indent function to know if (point) is in a
>> comment or a string).
>> The mode is available here:
>> http://fxbois.free.fr/lisp/web-mode.el
>>
>> I will also publish a web page which will look like this:
>> http://fxbois.free.fr/lisp/web-mode.html
>>
>> Perhaps you also have comments about it.
>
> Dear Bois, have you compared the functionality in web-mode.el with
> that you can get with nXthml? (See EmacsWiki for links.)



-- 
François-Xavier BOIS
KerniX Software
15, rue Cels
75014 PARIS
tel : 01 53 98 73 43
fax : 01 53 98 73 45

Re: web-mode.el

[Lennart Borgman]

On Tue, Jun 12, 2012 at 1:54 PM, Bois Francois-Xavier <fxbois@gmail.com> wrote:
> Dear Lennart
>
> Thank you for your mail.
> Yes before starting this project I tried almost all alternative and
> none of them really fully worked for me.

I am, as you probably know, the author of nXhtml. It includes MuMaMo,
a general framework for editing files where different Emacs' major
modes can be useful for different parts of the files. (Which I call
chunks there.)

This approach can not be implemented in a perfect way as Emacs works
now. However implementing special solutions for different problems
would in my opinion be just too much work. Maintenance would take very
much time.

So I wonder if you perhaps could tell what problems you saw in nXhtml
(in this regard) and look into if you could fix them?

> That's why I started this
> project ; I use it now almost every day for a few months and I would
> like to share it now.
> Features that were really important for me were :
>  - indentation (in particular, HTML elements, HTML attributes, PHP
> arrays on multilines)
>  - cohérent syntax highlighting (according to the context)
>  - HTML element navigation
>  - maintainability
>  - speed
>
> Have you tried this mode ?
>
> On Tue, Jun 12, 2012 at 1:37 PM, Lennart Borgman
> <lennart.borgman@gmail.com> wrote:
>> On Tue, Jun 12, 2012 at 12:24 AM, Bois Francois-Xavier <fxbois@gmail.com> wrote:
>>>
>>> Dear Emacsians,
>>>
>>> I've written web-mode.el, a major mode for PHP/HTML templates files.
>>> It aims to natively indent and syntaxcolor according to the "context"
>>> of (point) (which can be in an HTML part or in a PHP/JavaScript/CSS
>>> bloc).
>>> As it is my first experience with emacs lisp programming, I would be
>>> really grateful to receive your feedbacks on any aspect of this mode
>>> (core programming, function/var naming, conventions, etc.). For many
>>> aspects, I wonder if I use hacks or correct ways to do the job (ex. I
>>> look at the face name in my indent function to know if (point) is in a
>>> comment or a string).
>>> The mode is available here:
>>> http://fxbois.free.fr/lisp/web-mode.el
>>>
>>> I will also publish a web page which will look like this:
>>> http://fxbois.free.fr/lisp/web-mode.html
>>>
>>> Perhaps you also have comments about it.
>>
>> Dear Bois, have you compared the functionality in web-mode.el with
>> that you can get with nXthml? (See EmacsWiki for links.)
>
>
>
> --
> François-Xavier BOIS
> KerniX Software
> 15, rue Cels
> 75014 PARIS
> tel : 01 53 98 73 43
> fax : 01 53 98 73 45

Re: web-mode.el

[Bois Francois-Xavier]

I will try to report you the problems I encounter(ed) but now that I
ve invested so much time in web-mode.el, I don't think I will find the
time to fix them.

On Tue, Jun 12, 2012 at 3:24 PM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Tue, Jun 12, 2012 at 1:54 PM, Bois Francois-Xavier <fxbois@gmail.com> wrote:
>> Dear Lennart
>>
>> Thank you for your mail.
>> Yes before starting this project I tried almost all alternative and
>> none of them really fully worked for me.
>
> I am, as you probably know, the author of nXhtml. It includes MuMaMo,
> a general framework for editing files where different Emacs' major
> modes can be useful for different parts of the files. (Which I call
> chunks there.)
>
> This approach can not be implemented in a perfect way as Emacs works
> now. However implementing special solutions for different problems
> would in my opinion be just too much work. Maintenance would take very
> much time.
>

Re: web-mode.el

[Chong Yidong]

Bois Francois-Xavier <fxbois@gmail.com> writes:

> I've written web-mode.el, a major mode for PHP/HTML templates files.
> It aims to natively indent and syntaxcolor according to the "context"
> of (point) (which can be in an HTML part or in a PHP/JavaScript/CSS
> bloc).

Thanks, this looks interesting.  The problem with nXhtml mode (and
php-mode) is that we've had a hell of a time trying to get the copyright
assignments to get them incorporated into Emacs, and at this point I've
given up.  So if you have a newly-written mode that provides similar
functionality, and are willing to make a copyright assignment to the
FSF, I might be interested in including it in Emacs.

I took a brief glance through your code.  Here are a few comments:

The name should be something like php-template-mode.el rather than
web-mode.el.

  (defvar web-mode-variable-name-face 'web-mode-variable-name-face
    "Face name to use for variable names.")

Don't do this.  Just define the variable with defface, and use the face
name (a symbol) directly wherever you need the face.  (This is mentioned
in the Elisp manual node "Faces".)

The major mode should derive from either prog-mode or text-mode (not
sure which is better), rather than fundamental mode.

  (defun web-mode-reload ()
    "Reload web-mode."
    (interactive)
    (unload-feature 'web-mode)
    (web-mode))

I'm not sure what this is for, but it's probably not necessary.

Re: web-mode.el

[Bois Francois-Xavier]

On Wed, Jun 13, 2012 at 9:45 AM, Chong Yidong <cyd@gnu.org> wrote:
> Bois Francois-Xavier <fxbois@gmail.com> writes:
>
>> I've written web-mode.el, a major mode for PHP/HTML templates files.
>> It aims to natively indent and syntaxcolor according to the "context"
>> of (point) (which can be in an HTML part or in a PHP/JavaScript/CSS
>> bloc).
>
> Thanks, this looks interesting.  The problem with nXhtml mode (and
> php-mode) is that we've had a hell of a time trying to get the copyright
> assignments to get them incorporated into Emacs, and at this point I've
> given up.  So if you have a newly-written mode that provides similar
> functionality, and are willing to make a copyright assignment to the
> FSF, I might be interested in including it in Emacs.

I would be very happy to have the mode included. What would be the
steps to have it in ?

> I took a brief glance through your code.  Here are a few comments:
>
> The name should be something like php-template-mode.el rather than
> web-mode.el.

The mode will perhaps take care of other languages. Adding Ruby/Java
(JSP) to the mode would be quite easy now. So I would prefer to have a
generic name.

>  (defvar web-mode-variable-name-face 'web-mode-variable-name-face
>    "Face name to use for variable names.")
>
> Don't do this.  Just define the variable with defface, and use the face
> name (a symbol) directly wherever you need the face.  (This is mentioned
> in the Elisp manual node "Faces".)

I will look into this

>
> The major mode should derive from either prog-mode or text-mode (not
> sure which is better), rather than fundamental mode.

Ok. But I have a question. Is it a convention ? Is it necessary ?
I used to think that deriving form a very simple mode would make my mode faster.


>  (defun web-mode-reload ()
>    "Reload web-mode."
>    (interactive)
>    (unload-feature 'web-mode)
>    (web-mode))

It was for me while I was developing.

> I'm not sure what this is for, but it's probably not necessary.

Re: web-mode.el

[Steinar Bang]

>>>>> Bois Francois-Xavier <fxbois@gmail.com>:
> On Wed, Jun 13, 2012 at 9:45 AM, Chong Yidong <cyd@gnu.org> wrote:

>> The major mode should derive from either prog-mode or text-mode (not
>> sure which is better), rather than fundamental mode.

> Ok. But I have a question. Is it a convention ? Is it necessary ?

Your mode will then inherit settings for text-mode.  Ie. it will run
text-mode-hook and respect all text-mode variables.

For me this is currently just setting auto-fill-mode to "1", but it used
to include iso-accents-mode.  It is convenient to set common settings
like this for all modes used to write human readable text (text-mode,
XML mode(s), email buffers, and version control commit buffers).

Re: web-mode.el

[Lennart Borgman]

On Wed, Jun 13, 2012 at 9:45 AM, Chong Yidong <cyd@gnu.org> wrote:
>
> Bois Francois-Xavier <fxbois@gmail.com> writes:
>
> > I've written web-mode.el, a major mode for PHP/HTML templates files.
> > It aims to natively indent and syntaxcolor according to the "context"
> > of (point) (which can be in an HTML part or in a PHP/JavaScript/CSS
> > bloc).
>
> Thanks, this looks interesting.  The problem with nXhtml mode (and
> php-mode) is that we've had a hell of a time trying to get the copyright
> assignments to get them incorporated into Emacs, and at this point I've
> given up.

The problem is with php-mode.el and some other major modes
nXhtml/MuMaMo may use. Nearly all of nXhtml is written by me so when
it comes to copyright there is no problem with that.

As I have said before, there are technical problems with mumamo.el
which can not be solved currently. Some very tough changes have to be
made to Emacs to make it better. Those changes will also make
mumamo.el a lot simpler. So I have left mumamo.el in the state it is.
It works reasonably well in the current situation, but it is not at
all perfect. (I have not tested with Emacs 24 yet.)

Does web-mode.el incorporate a new mode for PHP? Then it would be
interesting to test this with mumamo.el.

Re: web-mode.el

[Bois Francois-Xavier]

>
> Does web-mode.el incorporate a new mode for PHP? Then it would be
> interesting to test this with mumamo.el.

It is not really a new mode for PHP. There is a dedicated function for
syntax hightlight (with dedicated font lock keywords) and dedicated
cases in the main indent function. It is the same with HTML, CSS,
JavaScript.

Re: web-mode.el

[Lennart Borgman]

On Wed, Jun 13, 2012 at 12:49 PM, Bois Francois-Xavier <fxbois@gmail.com> wrote:
>>
>> Does web-mode.el incorporate a new mode for PHP? Then it would be
>> interesting to test this with mumamo.el.
>
> It is not really a new mode for PHP. There is a dedicated function for
> syntax hightlight (with dedicated font lock keywords) and dedicated
> cases in the main indent function. It is the same with HTML, CSS,
> JavaScript.

It would be very nice if those functions for PHP would be available as
a major mode. Then it could immediately be used in a lot of other
combinations with nXhtml/MuMaMo.

I guess you have written the parsing functions so that they do not
cross the chunk boundaries. Or, do I misunderstand that? If you have
done that then perhaps a major mode for PHP could use that and make it
available in some way to multi major modes frameworks?

Re: web-mode.el

[Bois Francois-Xavier]

> It would be very nice if those functions for PHP would be available as
> a major mode. Then it could immediately be used in a lot of other
> combinations with nXhtml/MuMaMo.
>
> I guess you have written the parsing functions so that they do not
> cross the chunk boundaries. Or, do I misunderstand that? If you have
> done that then perhaps a major mode for PHP could use that and make it
> available in some way to multi major modes frameworks?

But I think you would be disappointed by the few cases I handle for
PHP. I have written a mode for templates. In an HTML template I don't
expect people to declare function, declare classes, use exceptions
etc. All those cases are not properly handled by my mode. I don'
really "parse" PHP code ... it is more rules like ... "if the previous
line ends with { then I indent". By contrast, the parsing is much
stronger/powerful for HTML.

Re: web-mode.el

[Lennart Borgman]

On Wed, Jun 13, 2012 at 1:43 PM, Bois Francois-Xavier <fxbois@gmail.com> wrote:
>> It would be very nice if those functions for PHP would be available as
>> a major mode. Then it could immediately be used in a lot of other
>> combinations with nXhtml/MuMaMo.
>>
>> I guess you have written the parsing functions so that they do not
>> cross the chunk boundaries. Or, do I misunderstand that? If you have
>> done that then perhaps a major mode for PHP could use that and make it
>> available in some way to multi major modes frameworks?
>
> But I think you would be disappointed by the few cases I handle for
> PHP. I have written a mode for templates. In an HTML template I don't
> expect people to declare function, declare classes, use exceptions
> etc. All those cases are not properly handled by my mode. I don'
> really "parse" PHP code ... it is more rules like ... "if the previous
> line ends with { then I indent". By contrast, the parsing is much
> stronger/powerful for HTML.

Thanks Bois, I see. This kind of limitations is one of the reasons I
think a general multi mode framework should be used. The the full
power of the Emacs' major modes can be used. (But see my previous
messages for the current status of this.)

Re: web-mode.el

[Stefan Monnier]

> mumamo.el a lot simpler. So I have left mumamo.el in the state it is.
> It works reasonably well in the current situation, but it is not at
> all perfect. (I have not tested with Emacs 24 yet.)

Still waiting to see it in GNU ELPA, by the way.


        Stefan

Re: web-mode.el

[Lennart Borgman]

On Wed, Jun 13, 2012 at 2:30 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> mumamo.el a lot simpler. So I have left mumamo.el in the state it is.
>> It works reasonably well in the current situation, but it is not at
>> all perfect. (I have not tested with Emacs 24 yet.)
>
> Still waiting to see it in GNU ELPA, by the way.

Yes, my bad (or my bad time). It looks like things would develop a
little bit easier if I could find some time to add it.

Re: web-mode.el

[Dmitry Gutov]

Hi Chong,

> Thanks, this looks interesting.  The problem with nXhtml mode (and
> php-mode) is that we've had a hell of a time trying to get the copyright
> assignments to get them incorporated into Emacs, and at this point I've
> given up.

Is there a particular reason that multi-mode is not in the list?

http://www.loveshack.ukfsn.org/emacs/multi-mode.el

I reviewed the packages dealing with this problem recently, and this one
looks most well-written, if probably outdated.
And if we believe the copyright notice, copyright assignments are
already in place. Provided it's a viable candidate, I might see what I
can do to bring it up-to-date.

-- Dmitry

Re: web-mode.el

[Lennart Borgman]

On Wed, Jun 13, 2012 at 12:37 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> Hi Chong,
>
>> Thanks, this looks interesting.  The problem with nXhtml mode (and
>> php-mode) is that we've had a hell of a time trying to get the copyright
>> assignments to get them incorporated into Emacs, and at this point I've
>> given up.
>
> Is there a particular reason that multi-mode is not in the list?
>
> http://www.loveshack.ukfsn.org/emacs/multi-mode.el
>
> I reviewed the packages dealing with this problem recently, and this one
> looks most well-written, if probably outdated.
> And if we believe the copyright notice, copyright assignments are
> already in place. Provided it's a viable candidate, I might see what I
> can do to bring it up-to-date.

Please see my comments about mumamo.el. It is not possible to write
something that works for all cases with the current Emacs. nXhtml
includes a rather large test-suite where you can see some of the
problems. It should be rather easy to change this test suite to run
with for example multi-mode.el. If you do that I think you will get a
better view of what works and what does not work.

However it might not be worth the trouble. The real problem lies in
the Emacs C core. Parsing functions can currently not be stopped from
parsing things outside of the major mode chunk they belong too.
(mumamo.el goes a long way to try to address this problems as far as
possible. That makes the code quite complicated. A rewrite of the C
core makes things very simple. In addition to this rewrite of the
scheduling functions to add suitable tools for handling chunk finding
might be necessary, but that is much simpler.)§

Re: web-mode.el

[Dmitry Gutov]

Hi François,

> As it is my first experience with emacs lisp programming, I would be
> really grateful to receive your feedbacks on any aspect of this mode
> (core programming, function/var naming, conventions, etc.). For many
> aspects, I wonder if I use hacks or correct ways to do the job (ex. I
> look at the face name in my indent function to know if (point) is in a
> comment or a string).

To address the last point, you'd usually call (syntax-ppss) and
examine the returned value. `M-x describe-function parse-partial-sexp`
for its format.

> I will also publish a web page which will look like this:
> http://fxbois.free.fr/lisp/web-mode.html
>
> Perhaps you also have comments about it.

I don't write PHP, so I'm definitely not going to use it, but you may be
interested in this addition to mmm-mode I wrote last weekend:

https://github.com/purcell/mmm-mode/blob/master/mmm-erb.el

Basically, it provides syntax highlighting and smart indentation,
supporting CSS, JS and ERB/EJS subregions. And utility commands already
present in html-mode.

It reuses submodes' indent-line-functions, so extending it to support
PHP, for example, shouldn't be much harder than writing a few regexps
for `mmm-add-group` and `mmm-add-classes` and a scanning function like
`mmm-erb-scan-erb`. That's provided, of course, we'll have some php-mode
package to work with.

mmm-mode is rather heavy, so speed might become an issue with large
files, but it should have room for optimization.

-- Dmitry

Re: web-mode.el

[Richard Riley]

Bois Francois-Xavier <fxbois@gmail.com> writes:

> Dear Emacsians,
>
> I've written web-mode.el, a major mode for PHP/HTML templates files.
> It aims to natively indent and syntaxcolor according to the "context"
> of (point) (which can be in an HTML part or in a PHP/JavaScript/CSS
> bloc).
> As it is my first experience with emacs lisp programming, I would be
> really grateful to receive your feedbacks on any aspect of this mode
> (core programming, function/var naming, conventions, etc.). For many
> aspects, I wonder if I use hacks or correct ways to do the job (ex. I
> look at the face name in my indent function to know if (point) is in a
> comment or a string).
> The mode is available here:
> http://fxbois.free.fr/lisp/web-mode.el
>
> I will also publish a web page which will look like this:
> http://fxbois.free.fr/lisp/web-mode.html
>
> Perhaps you also have comments about it.
>
> I will release this mode on github after reading your comments/advices.
>
> Thank you in advance.
>
> François-Xavier Bois
>
> ps: sorry for my poor english
>

just fyi : The git link in the web-mode source doesn't work:-

https://github.com/fxbois/web-mode

I would be interested to understand more fully what this offers that
nxhtml does not. Also, in the auto-mode-alist example here

http://fxbois.free.fr/lisp/web-mode.html

did you really mean psp as opposed to php?

regards

r.

Re: web-mode.el

[Dmitry Gutov]

Lennart Borgman <lennart.borgman@gmail.com> writes:

 > On Wed, Jun 13, 2012 at 12:37 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
 >>
 >> Hi Chong,
 >>
 >>> Thanks, this looks interesting. The problem with nXhtml mode (and
 >>> php-mode) is that we've had a hell of a time trying to get the 
copyright
 >>> assignments to get them incorporated into Emacs, and at this point I've
 >>> given up.
 >>
 >> Is there a particular reason that multi-mode is not in the list?
 >>
 >> http://www.loveshack.ukfsn.org/emacs/multi-mode.el
 >>
 >> I reviewed the packages dealing with this problem recently, and this one
 >> looks most well-written, if probably outdated.
 >> And if we believe the copyright notice, copyright assignments are
 >> already in place. Provided it's a viable candidate, I might see what I
 >> can do to bring it up-to-date.
 >
 > Please see my comments about mumamo.el. It is not possible to write
 > something that works for all cases with the current Emacs. nXhtml
 > includes a rather large test-suite where you can see some of the
 > problems. It should be rather easy to change this test suite to run
 > with for example multi-mode.el. If you do that I think you will get a
 > better view of what works and what does not work.

I'll look at the tests, thanks.

 > However it might not be worth the trouble. The real problem lies in
 > the Emacs C core. Parsing functions can currently not be stopped from
 > parsing things outside of the major mode chunk they belong too.
 > (mumamo.el goes a long way to try to address this problems as far as
 > possible. That makes the code quite complicated. A rewrite of the C
 > core makes things very simple. In addition to this rewrite of the
 > scheduling functions to add suitable tools for handling chunk finding
 > might be necessary, but that is much simpler.)=C2=A7

 From what I know, mmm-mode counteracts this problem effectively enough,
narrowing buffer to chunks during fontification, binding
font-lock-dont-widen to t, etc.
I'm also using narrowing during indentation, but some modes (like js)
use (widen) in indent-line-function. It should be somewhat fixable by
applying and removing syntax-table text property to text before the
chunk, but may make things even slower.

Anyway, you've seen the list of my immediate gripes about how MuMaMo
works for ERB, and I'm inclined to fault chunk detection and indentation
logic in your code rather than facilities provided by Emacs.

That's not to say that some sort of built-in support wouldn't help, of
course.

-- Dmitry

Re: web-mode.el

[Lennart Borgman]

On Thu, Jun 14, 2012 at 2:33 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
>
> > However it might not be worth the trouble. The real problem lies in
> > the Emacs C core. Parsing functions can currently not be stopped from
> > parsing things outside of the major mode chunk they belong too.
> > (mumamo.el goes a long way to try to address this problems as far as
> > possible. That makes the code quite complicated. A rewrite of the C
> > core makes things very simple. In addition to this rewrite of the
> > scheduling functions to add suitable tools for handling chunk finding
> > might be necessary, but that is much simpler.)=C2=A7
>
> From what I know, mmm-mode counteracts this problem effectively enough,
> narrowing buffer to chunks during fontification, binding
> font-lock-dont-widen to t, etc.

I am afraid that is a total misunderstanding. Narrowing etc does not
help because it does not affect all the functions involved.

> Anyway, you've seen the list of my immediate gripes about how MuMaMo
> works for ERB, and I'm inclined to fault chunk detection

Chunk detection must logically be done from the start of the file. In
the first version of MuMaMo I tried to do chunk detection from
anywhere in the file. That worked for simple cases but not for more
complicated cases so I rewrote this several years ago.

I have no idea about what trouble you have had. Perhaps you could
describe them? (Maybe there are still pieces of code I have forgotten
to rewrite?)

> and indentation logic in your code

Indentation is surpricingly hard actually. One problem is that you
have to do some language specific parsing (which the major mode
normally does). Another problem is getting this to work with chunks
and across chunks. There are several alternatives for this in
mumamo.el - and unfortunately I think there might some trouble there.

I suspect you have been looking at some of the easier cases. Please
take a look at the test cases. There should be more, but I think I
have written test cases for some of the troublesome cases.

> rather than facilities provided by Emacs.

See my first comment above.

Re: web-mode.el

[Dmitry Gutov]

On 14.06.2012 5:49, Lennart Borgman wrote:
> On Thu, Jun 14, 2012 at 2:33 AM, Dmitry Gutov<dgutov@yandex.ru>  wrote:
>>
>> Lennart Borgman<lennart.borgman@gmail.com>  writes:
>>
>>
>>> However it might not be worth the trouble. The real problem lies in
>>> the Emacs C core. Parsing functions can currently not be stopped from
>>> parsing things outside of the major mode chunk they belong too.
>>> (mumamo.el goes a long way to try to address this problems as far as
>>> possible. That makes the code quite complicated. A rewrite of the C
>>> core makes things very simple. In addition to this rewrite of the
>>> scheduling functions to add suitable tools for handling chunk finding
>>> might be necessary, but that is much simpler.)=C2=A7
>>
>>  From what I know, mmm-mode counteracts this problem effectively enough,
>> narrowing buffer to chunks during fontification, binding
>> font-lock-dont-widen to t, etc.
>
> I am afraid that is a total misunderstanding. Narrowing etc does not
> help because it does not affect all the functions involved.

I must've mixed it up with multi-mode. mmm-mode does not indeed call 
narrow-to-region there, and instead calls respective 
font-lock-fontify-region-function's with chunk start and end positions.

https://github.com/purcell/mmm-mode/blob/master/mmm-region.el#L793

>> Anyway, you've seen the list of my immediate gripes about how MuMaMo
>> works for ERB, and I'm inclined to fault chunk detection
>
> Chunk detection must logically be done from the start of the file. In
> the first version of MuMaMo I tried to do chunk detection from
> anywhere in the file. That worked for simple cases but not for more
> complicated cases so I rewrote this several years ago.
>
> I have no idea about what trouble you have had. Perhaps you could
> describe them? (Maybe there are still pieces of code I have forgotten
> to rewrite?)

https://bugs.launchpad.net/nxhtml/+bug/946464

>> and indentation logic in your code
>
> Indentation is surpricingly hard actually. One problem is that you
> have to do some language specific parsing (which the major mode
> normally does). Another problem is getting this to work with chunks
> and across chunks. There are several alternatives for this in
> mumamo.el - and unfortunately I think there might some trouble there.

One important thing that I'm almost certain your code doesn't do is set 
indent-line-function in all chunks to the same value.
This function, when called, should save (- (current-column) 
(current-indentation)) value, to restore it after indentation if it's 
positive.
Then it should go (back-to-indentation), look in what chunk the point is 
now, and act accordingly.

> I suspect you have been looking at some of the easier cases. Please
> take a look at the test cases. There should be more, but I think I
> have written test cases for some of the troublesome cases.

I posit that I've managed to make the examples from the bug above work 
better then they do with eruby-html-mumamo-mode, using lesser (I think?) 
amount of code.
Nothing is perfect, of course, but all templates in my current projects 
I've tried yet have shown no problems.

Re: web-mode.el

[Stefan Monnier]

> I'm also using narrowing during indentation, but some modes (like js)
> use (widen) in indent-line-function. It should be somewhat fixable by

I don't think it's worthwhile trying to make a "multi-major-mode" that
works with any major mode, no matter what wicked thing it does.
So it's OK to impose a few conventions that the major mode needs
to obey.  The important and difficult part is to figure what those
conventions need to be.

Of course, tweaking some parts of the C mode might help.  E.g. we could
maybe extend syntax-tables slightly so that chunk boundaries are marked
with a special syntax-table value which then makes forward-comment skip
over "other language" chunks, so that any indentation code which
consistently ignores comments would then work in multi-major-mode.


        Stefan

Re: web-mode.el

[Vitalie Spinu]

  >> Stefan Monnier <monnier@iro.umontreal.ca>
  >> on Wed, 13 Jun 2012 23:23:54 -0400 wrote:

  >> I'm also using narrowing during indentation, but some modes (like js)
  >> use (widen) in indent-line-function. It should be somewhat fixable by

  > I don't think it's worthwhile trying to make a "multi-major-mode" that
  > works with any major mode, no matter what wicked thing it does.
  > So it's OK to impose a few conventions that the major mode needs
  > to obey.  The important and difficult part is to figure what those
  > conventions need to be.

I also think that multi-major-mode is too much work into a wrong
direction. Major mode with chunk switching modes is a dead end
also. Modes are becoming heavier and more complex, and one can never
know reliably what is happening during the initialization. Besides that,
this approach is extremely expensive. Mumamo has almost 10000 lines and
it doesn't look like it's the end of it.

Dave Love wrote his 400 lines multi-mode.el in 2003, which is based on
indirect buffers (http://www.loveshack.ukfsn.org/emacs/multi-mode.el).
It still needs tweaking, but it's lightening fast and doesn't require
mode swapping. It happened that I've started improving it here
(https://github.com/vitoshka/litprog) as part of a general literate
programing framework project a couple of weeks ago.

I bet a working *minor*-web-mode can be written in less than 100 lines
only by specifying chunk delimiters regexps and a mode detection
function.

  > Of course, tweaking some parts of the C mode might help.  E.g. we could
  > maybe extend syntax-tables slightly so that chunk boundaries are marked
  > with a special syntax-table value which then makes forward-comment skip
  > over "other language" chunks, so that any indentation code which
  > consistently ignores comments would then work in multi-major-mode.

In litprog-mode.el chunk delimiters already marked as comments to avoid
issues with parse-partial-sexp (they even have a separate mode). I guess
this works well enough for most of the cases. To achieve full generality
one would need to mark whole "irrelevant" chunks as comments. For
example in js-mode buffer all html, php etc chunks are comments. In
html-mode buffer js, php are comments, etc. 

Dmitry suggested collecting test cases. That would be great for
comparing multi modes head to head. Where can I find really difficult
ones?

Best, 
Vitalie.

Re: web-mode.el

[Lennart Borgman]

On Fri, Jun 15, 2012 at 10:34 AM, Vitalie Spinu <spinuvit@gmail.com> wrote:
>
> Dmitry suggested collecting test cases. That would be great for
> comparing multi modes head to head. Where can I find really difficult
> ones?

I have been collecting some test cases from the bug reports to
nXhtml/MuMaMo. There are unit tests for most of them.

Those tests are distributed with nXhtml (and there are even menu
entries to run them). Please look at the versions in the repository:

  http://bazaar.launchpad.net/~nxhtml/nxhtml/main/files/head:/tests/

Perhaps you want to try to generalize the unit tests to work for other
multi major mode frameworks too?

Re: web-mode.El

[Vitalie Spinu]

  >> Lennart Borgman <lennart.borgman@gmail.com>
  >> on Fri, 15 Jun 2012 11:23:31 +0200 wrote:

  > On Fri, Jun 15, 2012 at 10:34 AM, Vitalie Spinu <spinuvit@gmail.com> wrote:
  >> 
  >> Dmitry suggested collecting test cases. That would be great for
  >> comparing multi modes head to head. Where can I find really difficult
  >> ones?

  > I have been collecting some test cases from the bug reports to
  > nXhtml/MuMaMo. There are unit tests for most of them.

  > Those tests are distributed with nXhtml (and there are even menu
  > entries to run them). Please look at the versions in the repository:

  >   http://bazaar.launchpad.net/~nxhtml/nxhtml/main/files/head:/tests/

Wow! That's huge collection of examples and test suites.  Thanks. 

  > Perhaps you want to try to generalize the unit tests to work for other
  > multi major mode frameworks too?

That's a good idea. Will look into it.

Vitalie.

Re: web-mode.El

[Lennart Borgman]

On Fri, Jun 15, 2012 at 11:41 AM, Vitalie Spinu <spinuvit@gmail.com> wrote:
>  >> Lennart Borgman <lennart.borgman@gmail.com>
>  >> on Fri, 15 Jun 2012 11:23:31 +0200 wrote:
>
>  > On Fri, Jun 15, 2012 at 10:34 AM, Vitalie Spinu <spinuvit@gmail.com> wrote:
>  >>
>  >> Dmitry suggested collecting test cases. That would be great for
>  >> comparing multi modes head to head. Where can I find really difficult
>  >> ones?
>
>  > I have been collecting some test cases from the bug reports to
>  > nXhtml/MuMaMo. There are unit tests for most of them.
>
>  > Those tests are distributed with nXhtml (and there are even menu
>  > entries to run them). Please look at the versions in the repository:
>
>  >   http://bazaar.launchpad.net/~nxhtml/nxhtml/main/files/head:/tests/
>
> Wow! That's huge collection of examples and test suites.  Thanks.
>
>  > Perhaps you want to try to generalize the unit tests to work for other
>  > multi major mode frameworks too?
>
> That's a good idea. Will look into it.
>
> Vitalie.

Forgot to say two things:

- The number in the test file names corresponds to bug reports here:
https://bugs.launchpad.net/nxhtml

- I have not written unit tests for all this files (because of lack of time).

Re: web-mode.el

[Dmitry Gutov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
 >> I'm also using narrowing during indentation, but some modes (like js)
 >> use (widen) in indent-line-function. It should be somewhat fixable by
 >
 > I don't think it's worthwhile trying to make a "multi-major-mode" that
 > works with any major mode, no matter what wicked thing it does.
 > So it's OK to impose a few conventions that the major mode needs
 > to obey.  The important and difficult part is to figure what those
 > conventions need to be.

I agree in principle. And since fontification of chunks can already be
done reliably enough (if not with best performance), I think indentation
function conventions would be more important. Shall we start with
collecting cases?

So far I have two problems without obvious solutions:

1) js-mode uses (widen) at the beginning of js-indent-line, and then calls
(syntax-ppss).

2) sgml-indent-line calls sgml-parse-tag-backward, which does
(re-search-backward "[<>]"), finds "<" and performs simple regexp check.
Thus, <% if a < 3 %> breaks indentation on following lines, until first
closing tag.

So it's 3 built-in functions as candidates for special interaction
with new syntax-table value.

 > Of course, tweaking some parts of the C mode might help.  E.g. we could
 > maybe extend syntax-tables slightly so that chunk boundaries are marked
 > with a special syntax-table value which then makes forward-comment skip
 > over "other language" chunks, so that any indentation code which
 > consistently ignores comments would then work in multi-major-mode.

Did you mean to write "C code" in the first sentence?

As far as I can see, enhanced (forward-comment) won't help much with
indentation.

And crossing two boundaries won't necessarily mean that we're in a chunk
in the same mode (or we're limited to supporting only two modes in the
same buffer), so that means syntax-table values can't be trivial.
Cons cell (mode-above . mode-below), and related code will need
to consider movement direction?

Re: web-mode.el

[Lennart Borgman]

On Thu, Jun 14, 2012 at 4:24 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> I'm also using narrowing during indentation, but some modes (like js)
>>> use (widen) in indent-line-function. It should be somewhat fixable by
>>
>> I don't think it's worthwhile trying to make a "multi-major-mode" that
>> works with any major mode, no matter what wicked thing it does.
>> So it's OK to impose a few conventions that the major mode needs
>> to obey.  The important and difficult part is to figure what those
>> conventions need to be.
>
> I agree in principle. And since fontification of chunks can already be
> done reliably enough (if not with best performance), I think indentation
> function conventions would be more important. Shall we start with
> collecting cases?

I am not sure I agree. The way I have proposed to do it is to change
the low level functions reading the buffer so that you can lie to them
and say that the chunks in other major modes are just whitespace+line
feeds. It would be reliable (I think), but it requires much effort so
it still may not be worth it.

I took a look at what I had done so far (it was some time ago now). I
have not touched the C code. I have only made some changes in
mumamo.el towards this goal.

As Dmitry says, fontification is rather simple (but you have to be
careful to make it fast). Indentation is harder.

First of all I have tried to write down the logic to use. I have
rewritten that part a few times. Even this was harder then I expected.
You can see my current comments here in mumamo-indent-line-function-1
(at line 7915 currently):

  http://bazaar.launchpad.net/~nxhtml/nxhtml/main/view/head:/util/mumamo.el

Second I have moved towards doing indentation the way it should have
been done if the changes in the C code were done. I do this by copying
the code to a mirror buffer, replacing everything in "other" chunks
with whitespace+line feeds.

It is easy to get things a bit wrong when you do this of course. I am
not sure if the problem you have seen, Dmitry, is because (1) failed
to copy correctly, (2) the idea is somehow buggy, or (3) I just did
not implement all cases or did not finish implementing them.


> So far I have two problems without obvious solutions:
>
> 1) js-mode uses (widen) at the beginning of js-indent-line, and then calls
> (syntax-ppss).

There are several modes for javascript. And of course for html too;
nxml-mode is one. Stopping nxml-mode from crossing the chunks
boundaries is not easy.


>> Of course, tweaking some parts of the C mode might help.  E.g. we could
>> maybe extend syntax-tables slightly so that chunk boundaries are marked
>> with a special syntax-table value which then makes forward-comment skip
>> over "other language" chunks, so that any indentation code which
>> consistently ignores comments would then work in multi-major-mode.

I have tried to implement something like that in mumamo.el, but I
think it is not working correctly anymore. Something might have
changed since I did that.

Re: web-mode.el

[Dmitry Gutov]

On 14.06.2012 20:54, Lennart Borgman wrote:
> On Thu, Jun 14, 2012 at 4:24 PM, Dmitry Gutov<dgutov@yandex.ru>  wrote:
>> Stefan Monnier<monnier@iro.umontreal.ca>  writes:
>>>> I'm also using narrowing during indentation, but some modes (like js)
>>>> use (widen) in indent-line-function. It should be somewhat fixable by
>>>
>>> I don't think it's worthwhile trying to make a "multi-major-mode" that
>>> works with any major mode, no matter what wicked thing it does.
>>> So it's OK to impose a few conventions that the major mode needs
>>> to obey.  The important and difficult part is to figure what those
>>> conventions need to be.
>>
>> I agree in principle. And since fontification of chunks can already be
>> done reliably enough (if not with best performance), I think indentation
>> function conventions would be more important. Shall we start with
>> collecting cases?
>
> I am not sure I agree. The way I have proposed to do it is to change
> the low level functions reading the buffer so that you can lie to them
> and say that the chunks in other major modes are just whitespace+line
> feeds. It would be reliable (I think), but it requires much effort so
> it still may not be worth it.

Fully isolating the chunks might not be the best course of action, even 
if it were not too hard to do.
For example, if we want to adjust indentation inside <% if %> <% end%> 
blocks, we need to be able to actually look inside them while we're in 
the primary mode chunk (between those two).

> I took a look at what I had done so far (it was some time ago now). I
> have not touched the C code. I have only made some changes in
> mumamo.el towards this goal.
>
> As Dmitry says, fontification is rather simple (but you have to be
> careful to make it fast). Indentation is harder.
>
> First of all I have tried to write down the logic to use. I have
> rewritten that part a few times. Even this was harder then I expected.
> You can see my current comments here in mumamo-indent-line-function-1
> (at line 7915 currently):
>
>    http://bazaar.launchpad.net/~nxhtml/nxhtml/main/view/head:/util/mumamo.el
>
> Second I have moved towards doing indentation the way it should have
> been done if the changes in the C code were done. I do this by copying
> the code to a mirror buffer, replacing everything in "other" chunks
> with whitespace+line feeds.
>
> It is easy to get things a bit wrong when you do this of course. I am
> not sure if the problem you have seen, Dmitry, is because (1) failed
> to copy correctly, (2) the idea is somehow buggy, or (3) I just did
> not implement all cases or did not finish implementing them.

I don't think I understand the code enough to answer that, but I'm 
inclined to fault implementation (not necessarily of the stage you 
described above). For example, one of the problems shows that 
indentation function behaves differently, depending on where inside the 
(same) line the point is.

As an aside, this bit from the docstring sounds not very useful, maybe 
even harmful:

"- However first non-empty line indentation in a chunk when going
in is special if prev-prev chunk is on same mumamo-depth and
have the same major mode. Then indent relative last non-empty
line in prev-prev chunk."

>> So far I have two problems without obvious solutions:
>>
>> 1) js-mode uses (widen) at the beginning of js-indent-line, and then calls
>> (syntax-ppss).
>
> There are several modes for javascript. And of course for html too;
> nxml-mode is one. Stopping nxml-mode from crossing the chunks
> boundaries is not easy.

js2-mode's indentation function is very similar to js-mode, since 
they're both derived from an javascript-mode. It doesn't work in 
multi-mode context, though, and while fixing that shouldn't be too hard, 
I'm not sure it's worth spending time on, because js2-mode's advantages 
over js-mode are most evident with larger files, not small snippets.

I have yet to look into using nxml as primary mode. Do you have any 
suggestions how that could be made easier?

>>> Of course, tweaking some parts of the C mode might help.  E.g. we could
>>> maybe extend syntax-tables slightly so that chunk boundaries are marked
>>> with a special syntax-table value which then makes forward-comment skip
>>> over "other language" chunks, so that any indentation code which
>>> consistently ignores comments would then work in multi-major-mode.
>
> I have tried to implement something like that in mumamo.el, but I
> think it is not working correctly anymore. Something might have
> changed since I did that.

Could you show where this is implemented?

Re: web-mode.el

[Lennart Borgman]

On Fri, Jun 15, 2012 at 5:24 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> Fully isolating the chunks might not be the best course of action, even if
> it were not too hard to do.
> For example, if we want to adjust indentation inside <% if %> <% end%>
> blocks, we need to be able to actually look inside them while we're in the
> primary mode chunk (between those two).

Can you give an example of what you mean here?

> I don't think I understand the code enough to answer that, but I'm inclined
> to fault implementation (not necessarily of the stage you described above).
> For example, one of the problems shows that indentation function behaves
> differently, depending on where inside the (same) line the point is.

Eh, yes, that is of course just a bug (but maybe not a very simple one).

> As an aside, this bit from the docstring sounds not very useful, maybe even
> harmful:
>
> "- However first non-empty line indentation in a chunk when going
> in is special if prev-prev chunk is on same mumamo-depth and
> have the same major mode. Then indent relative last non-empty
> line in prev-prev chunk."

Why do you think this is wrong?

> js2-mode's indentation function is very similar to js-mode, since they're
> both derived from an javascript-mode. It doesn't work in multi-mode context,
> though, and while fixing that shouldn't be too hard, I'm not sure it's worth
> spending time on, because js2-mode's advantages over js-mode are most
> evident with larger files, not small snippets.

Is not this is a general problem, not something specific to js2-mode?

> I have yet to look into using nxml as primary mode. Do you have any
> suggestions how that could be made easier?

There are such multi major modes in nXhtml. However they does not
always behave as I expected.

>>>> Of course, tweaking some parts of the C mode might help.  E.g. we could
>>>> maybe extend syntax-tables slightly so that chunk boundaries are marked
>>>> with a special syntax-table value which then makes forward-comment skip
>>>> over "other language" chunks, so that any indentation code which
>>>> consistently ignores comments would then work in multi-major-mode.
>>
>>
>> I have tried to implement something like that in mumamo.el, but I
>> think it is not working correctly anymore. Something might have
>> changed since I did that.
>
>
> Could you show where this is implemented?

It just stores the value of syntax-ppss cache at the end of a chunk.
When handling the next chunk of the same time this value is picked up
again.

Re: web-mode.el

[Dmitry Gutov]

On 16.06.2012 5:43, Lennart Borgman wrote:
> On Fri, Jun 15, 2012 at 5:24 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>>
>> Fully isolating the chunks might not be the best course of action, even if
>> it were not too hard to do.
>> For example, if we want to adjust indentation inside <% if %> <% end%>
>> blocks, we need to be able to actually look inside them while we're in the
>> primary mode chunk (between those two).
>
> Can you give an example of what you mean here?

I was just arguing a general point. If your proposal was to be able to 
call various functions manually while lying to them or their callees, 
then go back to seeing the whole buffer "truly", this example doesn't apply.

>> As an aside, this bit from the docstring sounds not very useful, maybe even
>> harmful:
>>
>> "- However first non-empty line indentation in a chunk when going
>> in is special if prev-prev chunk is on same mumamo-depth and
>> have the same major mode. Then indent relative last non-empty
>> line in prev-prev chunk."
>
> Why do you think this is wrong?

If mumamo-depth doesn't take into account nesting level of plain html 
tags, then this is going to be wrong when nesting levels are different.

If it does take into account those, this shouldn't be a problem, but I 
still don't understand why you need this mechanic at all.

>> js2-mode's indentation function is very similar to js-mode, since they're
>> both derived from an javascript-mode. It doesn't work in multi-mode context,
>> though, and while fixing that shouldn't be too hard, I'm not sure it's worth
>> spending time on, because js2-mode's advantages over js-mode are most
>> evident with larger files, not small snippets.
>
> Is not this is a general problem, not something specific to js2-mode?

js2-mode is doing things differently from most major modes: 1) it reads 
all buffer contents, scanning through every buffer character one by one 
while parsing, 2) its fontification is all manual, performed by setting 
'font-lock-face and 'syntax-table text properties during parsing.
And parsing is scheduled directly from after-change-functions, not 
through jit-lock.

>> I have yet to look into using nxml as primary mode. Do you have any
>> suggestions how that could be made easier?
>
> There are such multi major modes in nXhtml. However they does not
> always behave as I expected.

I was asking specifically about nxml, because my initial attempt to make 
it work with erb-style tags in mmm-mode had failed.

Re: web-mode.el

[Lennart Borgman]

On Sat, Jun 16, 2012 at 3:19 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 16.06.2012 5:43, Lennart Borgman wrote:
>>
>
> I was just arguing a general point. If your proposal was to be able to call
> various functions manually while lying to them or their callees, then go
> back to seeing the whole buffer "truly", this example doesn't apply.

I see. Yes, that was my proposal. I imagined something like I
mentioned in a previous example i. e.:

  (with-chunks chunk-list
     ... here you lie...
    )


>>> As an aside, this bit from the docstring sounds not very useful, maybe
>>> even
>>> harmful:
>>>
>>> "- However first non-empty line indentation in a chunk when going
>>> in is special if prev-prev chunk is on same mumamo-depth and
>>> have the same major mode. Then indent relative last non-empty
>>> line in prev-prev chunk."
>>
>>
>> Why do you think this is wrong?
>
>
> If mumamo-depth doesn't take into account nesting level of plain html tags,
> then this is going to be wrong when nesting levels are different.
>
> If it does take into account those, this shouldn't be a problem, but I still
> don't understand why you need this mechanic at all.

The reason you may need it is that for example chunks at a deeper
level may have html code in them. They may be writing html code.
MuMaMo tries to make it possible to fontify and indent this html code
separately then.


> js2-mode is doing things differently from most major modes: 1) it reads all
> buffer contents, scanning through every buffer character one by one while
> parsing, 2) its fontification is all manual, performed by setting
> 'font-lock-face and 'syntax-table text properties during parsing.
> And parsing is scheduled directly from after-change-functions, not through
> jit-lock.

That is because it is a full parser. I think we will see more such
examples now that CEDET is finally inside Emacs.


>>> I have yet to look into using nxml as primary mode. Do you have any
>>> suggestions how that could be made easier?
>>
>>
>> There are such multi major modes in nXhtml. However they does not
>> always behave as I expected.
>
>
> I was asking specifically about nxml, because my initial attempt to make it
> work with erb-style tags in mmm-mode had failed.

Yes, I was trying to answer that. The reason it fails is because
nxml-mode (like js2-mode) contains a full parser. I have been thinking
that rewriting it is very difficult, but maybe Stefan's proposal (the
change to "syntax-ppss") is a  rather easy road.

Re: web-mode.el

[Dmitry Gutov]

On 16.06.2012 17:30, Lennart Borgman wrote:
>>>> As an aside, this bit from the docstring sounds not very useful, maybe
>>>> even harmful:
>>>>
>>>> "- However first non-empty line indentation in a chunk when going
>>>> in is special if prev-prev chunk is on same mumamo-depth and
>>>> have the same major mode. Then indent relative last non-empty
>>>> line in prev-prev chunk."
>>>
>>>
>>> Why do you think this is wrong?
>>
>>
>> If mumamo-depth doesn't take into account nesting level of plain html tags,
>> then this is going to be wrong when nesting levels are different.
>>
>> If it does take into account those, this shouldn't be a problem, but I still
>> don't understand why you need this mechanic at all.
>
> The reason you may need it is that for example chunks at a deeper
> level may have html code in them. They may be writing html code.
> MuMaMo tries to make it possible to fontify and indent this html code
> separately then.

What if the difference between mumamo-depth of pre-pre chunk and this 
one is 1 or -1, can you indent correctly in this case? If so, why the 
difference of 0 is a special case? Is this just a speed optimization?

>> js2-mode is doing things differently from most major modes: 1) it reads all
>> buffer contents, scanning through every buffer character one by one while
>> parsing, 2) its fontification is all manual, performed by setting
>> 'font-lock-face and 'syntax-table text properties during parsing.
>> And parsing is scheduled directly from after-change-functions, not through
>> jit-lock.
>
> That is because it is a full parser. I think we will see more such
> examples now that CEDET is finally inside Emacs.

As long as we don't try to use js2-mode as primary mode, the "full 
parser" thing won't be a problem, since it respects narrowing.
The only thing to change is the way parsing is invoked.

>>>> I have yet to look into using nxml as primary mode. Do you have any
>>>> suggestions how that could be made easier?
>>>
>>>
>>> There are such multi major modes in nXhtml. However they does not
>>> always behave as I expected.
>>
>>
>> I was asking specifically about nxml, because my initial attempt to make it
>> work with erb-style tags in mmm-mode had failed.
>
> Yes, I was trying to answer that. The reason it fails is because
> nxml-mode (like js2-mode) contains a full parser. I have been thinking
> that rewriting it is very difficult, but maybe Stefan's proposal (the
> change to "syntax-ppss") is a  rather easy road.

I don't think this changes much for nxml-mode: it uses the parser from 
xmltok.el, which doesn't employ any of the -sexp commands.

Re: web-mode.el

[Lennart Borgman]

On Tue, Jun 19, 2012 at 3:18 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> What if the difference between mumamo-depth of pre-pre chunk and this one is
> 1 or -1, can you indent correctly in this case? If so, why the difference of
> 0 is a special case? Is this just a speed optimization?

Yes, 0 is a special case. All chunks that "belongs to" the main major
mode (this could be for example html-mode) in the file has
mumamo-depth 0. Sub-chunks (for example a js or css chunk) has
mumamo-depth 1, i.e. 0+1.

The sub-chunks could themselves have sub-chunks which then have a +1
greater mumamo-depth.

If a pre-pre chunk has the same depth as this one then the code in
this one could be a continuation of the code in the pre-pre chunk.
(This could be the case in for example a file which html+php code.) So
the user probably expects those chunks to be indented together.


>> That is because it is a full parser. I think we will see more such
>> examples now that CEDET is finally inside Emacs.
>
>
> As long as we don't try to use js2-mode as primary mode, the "full parser"
> thing won't be a problem, since it respects narrowing.

That is nice, but we can't always expect it to work that way. At least not yet.


>> Yes, I was trying to answer that. The reason it fails is because
>> nxml-mode (like js2-mode) contains a full parser. I have been thinking
>> that rewriting it is very difficult, but maybe Stefan's proposal (the
>> change to "syntax-ppss") is a  rather easy road.
>
>
> I don't think this changes much for nxml-mode: it uses the parser from
> xmltok.el, which doesn't employ any of the -sexp commands.

I thought that parser perhaps could use the -sexp commands for testing.

Re: web-mode.el

[Dmitry Gutov]

On 19.06.2012 5:56, Lennart Borgman wrote:
> On Tue, Jun 19, 2012 at 3:18 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>>
>> What if the difference between mumamo-depth of pre-pre chunk and this one is
>> 1 or -1, can you indent correctly in this case? If so, why the difference of
>> 0 is a special case? Is this just a speed optimization?
>
> Yes, 0 is a special case. All chunks that "belongs to" the main major
> mode (this could be for example html-mode) in the file has
> mumamo-depth 0. Sub-chunks (for example a js or css chunk) has
> mumamo-depth 1, i.e. 0+1.
>
> The sub-chunks could themselves have sub-chunks which then have a +1
> greater mumamo-depth.
>
> If a pre-pre chunk has the same depth as this one then the code in
> this one could be a continuation of the code in the pre-pre chunk.
> (This could be the case in for example a file which html+php code.) So
> the user probably expects those chunks to be indented together.

Does that mean that

<div>
   <% if foo %>
     <span>Hello!</span>
   <% end %>
</div>

should be indented as

<div><!-- prev-prev chunk %>
   <% if foo %>
   <span>Hello!</span><!-- current chunk -->
   <% end %>
</div>

?

>>> Yes, I was trying to answer that. The reason it fails is because
>>> nxml-mode (like js2-mode) contains a full parser. I have been thinking
>>> that rewriting it is very difficult, but maybe Stefan's proposal (the
>>> change to "syntax-ppss") is a  rather easy road.
>>
>>
>> I don't think this changes much for nxml-mode: it uses the parser from
>> xmltok.el, which doesn't employ any of the -sexp commands.
>
> I thought that parser perhaps could use the -sexp commands for testing.

I think this will be a performance hit. When you just check sexp status 
at important points, that's fine, but here you'd have to call 
`syntax-ppss' each time you're scanning a token.

To look at it another way, an indent-line-function skipping comments is 
normal. A parser, on the other hand, parses comments, too.

Re: web-mode.el

[Stefan Monnier]

> 1) js-mode uses (widen) at the beginning of js-indent-line, and then calls
> (syntax-ppss).

Depending on how multi-major-mode works, widen is not necessarily
a problem.  Clearly, it would need to hook into syntax-ppss.

> 2) sgml-indent-line calls sgml-parse-tag-backward, which does
> (re-search-backward "[<>]"), finds "<" and performs simple regexp check.
> Thus, <% if a < 3 %> breaks indentation on following lines, until first
> closing tag.

I think we can treat this as a bug in sgml-indent-line, which should try
and use syntax-ppss or something like that instead of regexps.

>> Of course, tweaking some parts of the C mode might help.  E.g. we could
>> maybe extend syntax-tables slightly so that chunk boundaries are marked
>> with a special syntax-table value which then makes forward-comment skip
>> over "other language" chunks, so that any indentation code which
>> consistently ignores comments would then work in multi-major-mode.
> Did you mean to write "C code" in the first sentence?

Yes, indeed, sorry.

> As far as I can see, enhanced (forward-comment) won't help much with
> indentation.

It would, if you combine it with parse-sexp-ignore-comments.

> And crossing two boundaries won't necessarily mean that we're in a chunk
> in the same mode (or we're limited to supporting only two modes in the
> same buffer), so that means syntax-table values can't be trivial.
> Cons cell (mode-above . mode-below), and related code will need
> to consider movement direction?

Something like (mode-above . mode-below) is what I had in mind, yes.


        Stefan

Re: web-mode.el

[Dmitry Gutov]

On 14.06.2012 21:28, Stefan Monnier wrote:
>> 1) js-mode uses (widen) at the beginning of js-indent-line, and then calls
>> (syntax-ppss).
>
> Depending on how multi-major-mode works, widen is not necessarily
> a problem.  Clearly, it would need to hook into syntax-ppss.

That's good.

>> 2) sgml-indent-line calls sgml-parse-tag-backward, which does
>> (re-search-backward "[<>]"), finds "<" and performs simple regexp check.
>> Thus,<% if a<  3 %>  breaks indentation on following lines, until first
>> closing tag.
>
> I think we can treat this as a bug in sgml-indent-line, which should try
> and use syntax-ppss or something like that instead of regexps.

I wonder how that could be fixed exactly. parse-partial-sexp doesn't 
look helpful, because it works with single characters, and sgml is 
concerned with full tags. It also has to handle unclosed tags like <br>, 
some closing tags are optional, and HTML 4 has self-closing tags.

>>> Of course, tweaking some parts of the C mode might help.  E.g. we could
>>> maybe extend syntax-tables slightly so that chunk boundaries are marked
>>> with a special syntax-table value which then makes forward-comment skip
>>> over "other language" chunks, so that any indentation code which
>>> consistently ignores comments would then work in multi-major-mode.

>> As far as I can see, enhanced (forward-comment) won't help much with
>> indentation.
>
> It would, if you combine it with parse-sexp-ignore-comments.

If parse-partial-sexp just starts from (point-min), and then skips over 
"comments", it will never visit submode regions this way, no?

Another thing to consider - having "visibility" into previous chunks of 
the same submode may be more harmful than useful in some cases.
For example, when indenting the chunk in the middle,

   <% if true %>
     <div><%=
       some_text
     %></div>
   <% end %>

the chunk will see the "if" statement, and decide that it should be 
indented only 2 spaces from. And since it does do some indenting, I 
can't with my current heuristic decide that it should be instead 
indented relative to the line 2.

This is kinda handwavy, the example is contrived, the heuristic could be 
improved (suggestions welcome), but this could become more important for 
some other combinations of modes.

Also, if I have an extra paren inside some <script> tag, I, as a user, 
might be surprised to see JS code code inside another <script> tag down 
below indent weirdly.

>> And crossing two boundaries won't necessarily mean that we're in a chunk
>> in the same mode (or we're limited to supporting only two modes in the
>> same buffer), so that means syntax-table values can't be trivial.
>> Cons cell (mode-above . mode-below), and related code will need
>> to consider movement direction?
>
> Something like (mode-above . mode-below) is what I had in mind, yes.
>
>
>          Stefan
>
>

Re: web-mode.el

[Stefan Monnier]

>>> 2) sgml-indent-line calls sgml-parse-tag-backward, which does
>>> (re-search-backward "[<>]"), finds "<" and performs simple regexp check.
>>> Thus,<% if a<  3 %>  breaks indentation on following lines, until first
>>> closing tag.
>> I think we can treat this as a bug in sgml-indent-line, which should try
>> and use syntax-ppss or something like that instead of regexps.
> I wonder how that could be fixed exactly. parse-partial-sexp doesn't look
> helpful, because it works with single characters, and sgml is concerned with
> full tags. It also has to handle unclosed tags like <br>, some closing tags
> are optional, and HTML 4 has self-closing tags.

I think just checking after the regexp-match whether the match was found
within a "comment" should do the trick, assuming we can get syntax-ppss
(or some extension thereof) to treat "other modes" as comments.

> If parse-partial-sexp just starts from (point-min), and then skips over
> "comments", it will never visit submode regions this way, no?

That's why we'd need to hook into syntax-ppss to run parse-partial-sexp
on a chunk-by-chunk basis, maybe.  Also parse-sexp-ignore-comments also
affects (for|back)ward-sexp, as well as up-list, which are important
building blocks for indentation algorithms.

> Another thing to consider - having "visibility" into previous chunks of the
> same submode may be more harmful than useful in some cases.

That's OK: the low-level code can't know those things, but the
higher-level code which handles the various chunks can treat different
chunks differently.  E.g. using narrow-to-region for chunks which need
to ignore previous text.


        Stefan

Re: web-mode.el

[Lennart Borgman]

On Sat, Jun 16, 2012 at 8:17 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>>>> 2) sgml-indent-line calls sgml-parse-tag-backward, which does
>>>> (re-search-backward "[<>]"), finds "<" and performs simple regexp check.
>>>> Thus,<% if a<  3 %>  breaks indentation on following lines, until first
>>>> closing tag.
>>> I think we can treat this as a bug in sgml-indent-line, which should try
>>> and use syntax-ppss or something like that instead of regexps.
>> I wonder how that could be fixed exactly. parse-partial-sexp doesn't look
>> helpful, because it works with single characters, and sgml is concerned with
>> full tags. It also has to handle unclosed tags like <br>, some closing tags
>> are optional, and HTML 4 has self-closing tags.
>
> I think just checking after the regexp-match whether the match was found
> within a "comment" should do the trick, assuming we can get syntax-ppss
> (or some extension thereof) to treat "other modes" as comments.

So this is similar to a light version of my suggestion. Just change
one function instead of all functions that are reading the buffer. I
am beginning to like the idea ;-)

"syntax-ppss" then need a table of chunks. And there must be a way to
quickly switch that table.

Could it look like a macro, something like "with-chunks"? Will that
take care of "inner calls" to syntax-ppss in a correct way?

Re: web-mode.el

[Dmitry Gutov]

On 16.06.2012 10:17, Stefan Monnier wrote:
>>>> 2) sgml-indent-line calls sgml-parse-tag-backward, which does
>>>> (re-search-backward "[<>]"), finds "<" and performs simple regexp check.
>>>> Thus,<% if a<  3 %>  breaks indentation on following lines, until first
>>>> closing tag.
>>> I think we can treat this as a bug in sgml-indent-line, which should try
>>> and use syntax-ppss or something like that instead of regexps.
>> I wonder how that could be fixed exactly. parse-partial-sexp doesn't look
>> helpful, because it works with single characters, and sgml is concerned with
>> full tags. It also has to handle unclosed tags like <br>, some closing tags
>> are optional, and HTML 4 has self-closing tags.
>
> I think just checking after the regexp-match whether the match was found
> within a "comment" should do the trick, assuming we can get syntax-ppss
> (or some extension thereof) to treat "other modes" as comments.

That should work. Looks like currently it looks for comments with 
(search-backward), too.

>> If parse-partial-sexp just starts from (point-min), and then skips over
>> "comments", it will never visit submode regions this way, no?
>
> That's why we'd need to hook into syntax-ppss to run parse-partial-sexp
> on a chunk-by-chunk basis, maybe.  Also parse-sexp-ignore-comments also
> affects (for|back)ward-sexp, as well as up-list, which are important
> building blocks for indentation algorithms.
>
>> Another thing to consider - having "visibility" into previous chunks of the
>> same submode may be more harmful than useful in some cases.
>
> That's OK: the low-level code can't know those things, but the
> higher-level code which handles the various chunks can treat different
> chunks differently.  E.g. using narrow-to-region for chunks which need
> to ignore previous text.

Yes, well, except js-indent-line calls (widen) first thing, so 
(narrow-to-region) will not help.

Re: web-mode.el

[Lennart Borgman]

On Sat, Jun 16, 2012 at 3:27 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> Yes, well, except js-indent-line calls (widen) first thing, so
> (narrow-to-region) will not help.

That is one of the things I have been struggling with in mumamo.el. I
have tried to "advice" `widen', but it does not work any more.

So some way to handle `widen' is necessary too.

Re: web-mode.el

[Stefan Monnier]

> Yes, well, except js-indent-line calls (widen) first thing, so
> (narrow-to-region) will not help.

As mentioned earlier, we get to choose the supported interface, so if
`widen' can't be supported, we just need to change js-indent-line.
I wouldn't worry about it.


        Stefan

Re: web-mode.el

[Dmitry Gutov]

On 18.06.2012 5:49, Stefan Monnier wrote:
>> Yes, well, except js-indent-line calls (widen) first thing, so
>> (narrow-to-region) will not help.
>
> As mentioned earlier, we get to choose the supported interface, so if
> `widen' can't be supported, we just need to change js-indent-line.
> I wouldn't worry about it.

I though the idea was to do something or other while retaining backward 
compatibility. From what I understand, `widen' is used to counteract the 
situation when a user calls `narrow-to-region' interactively.
If supporting this is not a priority, okay.

It occurs to me that, once the supported interface is established, 
introducing new syntax-table value becomes not strictly necessary 
because, for example, a version of Lennart's `with-chunks' macro can be 
implemented in Lisp by applying whitespace or comment-starter/ender 
syntax to foreign chunks around the body calls (idea from mmm-noweb).
Is that right?

Re: web-mode.el

[Stefan Monnier]

> compatibility. From what I understand, `widen' is used to counteract the
> situation when a user calls `narrow-to-region' interactively.

Yes, and that is a problem that needs fixing: as mentioned elsewhere on
emacs-devel several months (years?) ago, Emacs needs to distinguish
between "narrow as a way to make the rest invisible" and "narrow to
pretend the rest doesn't exist at all".  C-x n n does the first, while
the second was used by Rmail, is used by Info, and could be used by
a multi-major-mode.

> It occurs to me that, once the supported interface is established,
> introducing new syntax-table value becomes not strictly necessary because,
> for example, a version of Lennart's `with-chunks' macro can be implemented
> in Lisp by applying whitespace or comment-starter/ender syntax to foreign
> chunks around the body calls (idea from mmm-noweb).
> Is that right?

That's right as long as you're willing to dynamically modify the buffer
(with syntax-table text-properties) every time you move from one chunk
to another.

If you have many chunks, that can be a problem.  This can happen if you
push the idea of multi-major-mode a bit further and consider for example
that comments and strings are really "text-mode chunks" embedded in
"some other mode chunks".


        Stefan

Re: web-mode.el

[Lennart Borgman]

On Tue, Jun 19, 2012 at 5:09 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>
>> It occurs to me that, once the supported interface is established,
>> introducing new syntax-table value becomes not strictly necessary because,
>> for example, a version of Lennart's `with-chunks' macro can be implemented
>> in Lisp by applying whitespace or comment-starter/ender syntax to foreign
>> chunks around the body calls (idea from mmm-noweb).
>> Is that right?
>
> That's right as long as you're willing to dynamically modify the buffer
> (with syntax-table text-properties) every time you move from one chunk
> to another.
>
> If you have many chunks, that can be a problem.

My idea with (with-chunks chunk-table ...) that chunk-table contains
this information instead. Then it can be cashed by the multi major
mode framework.

Re: web-mode.el

[Dmitry Gutov]

On 19.06.2012 7:09, Stefan Monnier wrote:
>> It occurs to me that, once the supported interface is established,
>> introducing new syntax-table value becomes not strictly necessary because,
>> for example, a version of Lennart's `with-chunks' macro can be implemented
>> in Lisp by applying whitespace or comment-starter/ender syntax to foreign
>> chunks around the body calls (idea from mmm-noweb).
>> Is that right?
>
> That's right as long as you're willing to dynamically modify the buffer
> (with syntax-table text-properties) every time you move from one chunk
> to another.

Or just every time I want to indent a line, instead. And if I always use 
`narrow-to-region' before performing indentation within a submode, the 
syntax-table text property (unless it's 'whitespace') isn't going to 
affect the result, so I might as well assign text properties only once 
when the buffer is parsed for chunks, "commenting out" the submodes.

> If you have many chunks, that can be a problem.  This can happen if you
> push the idea of multi-major-mode a bit further and consider for example
> that comments and strings are really "text-mode chunks" embedded in
> "some other mode chunks".

Yes, but what follows is defining the supported interface and patching 
existing packages becomes first priority, and implementing low-level 
support in C code - only second, which means more time to get the latter 
right.

For example, I'd like to see a use case for handling submode regions of 
the same mode as though they're contiguous (as oppsed to isolated), in 
the context of `syntax-ppss'. If there isn't, it might simplify the 
implementation.

On 19.06.2012 14:39, Lennart Borgman wrote:
 > My idea with (with-chunks chunk-table ...) that chunk-table contains
 > this information instead. Then it can be cashed by the multi major
 > mode framework.

You might want to consider how syntax-ppss cache works. Since 
text-property change in considered buffer modification, 
`syntax-ppss-flush-cache' will be called from `before-change-functions', 
invalidating the cache entries with positions after the edit position, 
and keeping the ones before it.
With your proposal, it would need to somehow keep track of the 
chunk-table, too.

Re: web-mode.el

[Lennart Borgman]

On Wed, Jun 20, 2012 at 4:01 AM, Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 19.06.2012 14:39, Lennart Borgman wrote:
>> My idea with (with-chunks chunk-table ...) that chunk-table contains
>> this information instead. Then it can be cashed by the multi major
>> mode framework.
>
> You might want to consider how syntax-ppss cache works. Since text-property
> change in considered buffer modification, `syntax-ppss-flush-cache' will be
> called from `before-change-functions', invalidating the cache entries with
> positions after the edit position, and keeping the ones before it.
> With your proposal, it would need to somehow keep track of the chunk-table,
> too.

Yes, that would be nice. (I do not remember if I corrected my self,
but a parameter to syntax-ppss corresponding to a function to get this
would be enough.)