good "modern" example code for a programming-language mode?

good "modern" example code for a programming-language mode?

[Miles Bader]

Does anyone have any suggestions for an Emacs programming-language
mode implementation that makes good use of "modern" Emacs features
(font-lock, syntax-ppss, etc) to do indentation/syntax-highlighting
(for a language that has at least somewhat complicated features like
multiple-line comments with multiple-character comment delimiters),
etc?

I'd like to read some code to see how these features are used in practice.

Thanks,

-Miles

-- 
Cat is power.  Cat is peace.

Re: good "modern" example code for a programming-language mode?

[Stefan Monnier]

> Does anyone have any suggestions for an Emacs programming-language
> mode implementation that makes good use of "modern" Emacs features
> (font-lock, syntax-ppss, etc) to do indentation/syntax-highlighting
> (for a language that has at least somewhat complicated features like
> multiple-line comments with multiple-character comment delimiters),
> etc?

> I'd like to read some code to see how these features are used in practice.

I've used octave-mod.el as a testbed for the new SMIE stuff, so it's
probably worth looking at it.  The sad thing is that outline-mod.el
included a lot more to start with, and some of hit has been replaced by
the SMIE features, but not all of it (some more could be removed,
I guess), so it's fairly large and I don't recommend to treat all of
it as a good example.

Maybe a cleaner example for SMIE would be modula2.el since that mode did
not provide any feature (or close enough), so it's still very lean.


        Stefan

Re: good "modern" example code for a programming-language mode?

[Michael Welsh Duggan]

Miles Bader <miles@gnu.org> writes:

> Does anyone have any suggestions for an Emacs programming-language
> mode implementation that makes good use of "modern" Emacs features
> (font-lock, syntax-ppss, etc) to do indentation/syntax-highlighting
> (for a language that has at least somewhat complicated features like
> multiple-line comments with multiple-character comment delimiters),
> etc?
>
> I'd like to read some code to see how these features are used in
> practice.

I don't have a good answer for you, but I'd like to see more new modes
based the semantic library.  It just seems like the right thing to do.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: good "modern" example code for a programming-language mode?

[Miles Bader]

Michael Welsh Duggan <md5i@md5i.com> writes:
> I don't have a good answer for you, but I'd like to see more new modes
> based the semantic library.  It just seems like the right thing to do.

Are there _any_ modes using it to handle "basic" language support
(indentation, highlighting)....?

-Miles

-- 
Laughter, n. An interior convulsion, producing a distortion of the features
and accompanied by inarticulate noises. It is infectious and, though
intermittent, incurable.

Re: good "modern" example code for a programming-language mode?

[Miles Bader]

Thanks Stefan, those look like great sources of code to copy, er, I mean
inspiration...

I didn't even know about SMIE, but it looks very useful indeed!

-Miles

-- 
80% of success is just showing up.  --Woody Allen

Re: good "modern" example code for a programming-language mode?

[David Engster]

Miles Bader writes:
> Michael Welsh Duggan <md5i@md5i.com> writes:
>> I don't have a good answer for you, but I'd like to see more new modes
>> based the semantic library.  It just seems like the right thing to do.
>
> Are there _any_ modes using it to handle "basic" language support
> (indentation, highlighting)....?

No. I also don't think that Semantic is well suited for that.

As for highlighting, there's already the semantic-decorate minor mode
which will highlight the semantical expressions found in the
buffer. When activating it, you'll already see that Semantic doesn't
parse the contents of functions, for example. It only parses (parts of)
the contents when you're within that function, to determine the
variables/types/etc. in the local scope.

Regarding indentation: this is much more a matter of taste than
semantics or syntax. If there was a direct connection between semantical
analysis and indentation, why were all those wars fought for the
"correct" indentation of C, for instance? Just looking at c-style-alist
will make that clear. There's also a very good write-up regarding this
issue from Steve Yegge when he dealt with indentation of Javascript:

http://steve-yegge.blogspot.com/2008/03/js2-mode-new-javascript-mode-for-emacs.html

-David

Re: good "modern" example code for a programming-language mode?

[Andy Moreton]

On Wed 16 Feb 2011, David Engster wrote:

> Miles Bader writes:
>> Michael Welsh Duggan <md5i@md5i.com> writes:
>>> I don't have a good answer for you, but I'd like to see more new modes
>>> based the semantic library.  It just seems like the right thing to do.
>>
>> Are there _any_ modes using it to handle "basic" language support
>> (indentation, highlighting)....?
>
> No. I also don't think that Semantic is well suited for that.
>
> As for highlighting, there's already the semantic-decorate minor mode
> which will highlight the semantical expressions found in the
> buffer. When activating it, you'll already see that Semantic doesn't
> parse the contents of functions, for example. It only parses (parts of)
> the contents when you're within that function, to determine the
> variables/types/etc. in the local scope.

Semantic seems quite heavyweight, and is not that well documented. the
manul reads more as describing a box of parts, but doesn't really
provide much help in assembling them into something useful. It would
benefit from being split into a user guide and and an internals manual,
with some worked examples for both.

> Regarding indentation: this is much more a matter of taste than
> semantics or syntax. If there was a direct connection between semantical
> analysis and indentation, why were all those wars fought for the
> "correct" indentation of C, for instance? Just looking at c-style-alist
> will make that clear.

Broaden your thoughts: there are other languages beyond C. In some of
them (e.g. python, haskell) indentation is semantically significant.

    AndyM

Re: good "modern" example code for a programming-language mode?

[Stefan Monnier]

> Regarding indentation: this is much more a matter of taste than
> semantics or syntax. If there was a direct connection between semantical
> analysis and indentation, why were all those wars fought for the
> "correct" indentation of C, for instance?

OTOH there is a clear connection between syntax and indentation.
My experience (which lead me to write SMIE) is that all indentation
styles for all languages share a lot of common principles, so that
indentation can be (partly) automatically derived from the language's
grammar.  And indeed, if you give a valid grammar to SMIE, it will give
you a valid indentation function in return.  Of course, the indentation
style it uses will most likely not fully match the user's expectations,
so SMIE provides a hook to tweak the default indentation rules for the
various possible situations.

E.g. for Prolog, the tweaks I've needed until now are limited to:

(defun prolog-smie-rules (kind token)
  (pcase (cons kind token)
    (`(:elem . basic) prolog-indent-width)
    (`(:after . ".") 0) ;; To work around smie-closer-alist.
    (`(:after . ,(or `":-" `"->" `"-->")) prolog-indent-width))

and for /bin/sh it currently looks like:

(defun sh-smie-rules (kind token)
  (pcase (cons kind token)
    (`(:elem . basic) sh-indentation)
    (`(:after . "case-)") (or sh-indentation smie-indent-basic))
    (`(:before . ,(or `"(" `"{" `"["))
     (if (smie-rule-hanging-p) (smie-rule-parent)))
    ;; FIXME: Maybe this handling of ;; should be made into
    ;; a smie-rule-terminator function that takes the substitute ";" as arg.
    (`(:before . ,(or `";;" `";&" `";;&"))
     (if (and (smie-rule-bolp) (looking-at ";;?&?[ \t]*\\(#\\|$\\)"))
         (cons 'column (smie-indent-keyword ";"))
       (smie-rule-separator kind)))
    (`(:after . ,(or `";;" `";&" `";;&"))
     (with-demoted-errors
       (smie-backward-sexp token)
       (cons 'column (current-column))))))

You can check what's used in Modula-2 and Octave by looking for
"smie-rules".

For SML it is a good bit more complex, for some reason (even though SML
was my original inspiration for SMIE).  Maybe it's because the
language's syntax tends to introduce a lot more indentation than is
practical, so people have come up with all kinds of conventions for
cases where indentation can be reduced without losing the property that
indentation reflects internal structure.

> Just looking at c-style-alist will make that clear.

Indeed, for some reason C indentation is a lot more contentious than
most other languages.  I haven't yet tried to use SMIE for C-like
languages, so I'm not sure how well it would accommodate all the
nitty-gritty details of each one's personal favorite indentation style.


        Stefan

Re: good "modern" example code for a programming-language mode?

[David Kastrup]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> Semantic seems quite heavyweight, and is not that well documented. the
> manul reads more as describing a box of parts, but doesn't really
> provide much help in assembling them into something useful. It would
> benefit from being split into a user guide and and an internals
> manual, with some worked examples for both.

Semantic would benefit from actually being included in Emacs.  At the
current point of time, it is impossible to actually create a
Semantic-based mode since only some stripped-down runtime support is
present, not corresponding to the included documentation.

That situation was supposed to be temporary.

-- 
David Kastrup

Re: good "modern" example code for a programming-language mode?

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

> and for /bin/sh it currently looks like:
>
> (defun sh-smie-rules (kind token)
>   (pcase (cons kind token)
>     (`(:elem . basic) sh-indentation)
>     (`(:after . "case-)") (or sh-indentation smie-indent-basic))
>     (`(:before . ,(or `"(" `"{" `"["))
>      (if (smie-rule-hanging-p) (smie-rule-parent)))
>     ;; FIXME: Maybe this handling of ;; should be made into
>     ;; a smie-rule-terminator function that takes the substitute ";" as arg.
>     (`(:before . ,(or `";;" `";&" `";;&"))
>      (if (and (smie-rule-bolp) (looking-at ";;?&?[ \t]*\\(#\\|$\\)"))
>          (cons 'column (smie-indent-keyword ";"))
>        (smie-rule-separator kind)))
>     (`(:after . ,(or `";;" `";&" `";;&"))
>      (with-demoted-errors
>        (smie-backward-sexp token)
>        (cons 'column (current-column))))))

Do you have the SMIE-based indentation of sh in some (private) branch?
I can't find the above function in sh-script.el...

Bye,
Tassilo

Re: good "modern" example code for a programming-language mode?

[David Engster]

Stefan Monnier writes:
>> Regarding indentation: this is much more a matter of taste than
>> semantics or syntax. If there was a direct connection between semantical
>> analysis and indentation, why were all those wars fought for the
>> "correct" indentation of C, for instance?
>
> OTOH there is a clear connection between syntax and indentation.

Yes, there is a connection, but it's only covering a certain amount of
indentation cases, depending on the language. Let's take comments, for
example. Just looking at the doc-string from c-lineup-C-comments, I
wonder what a grammar could help you here?  Or indenting continuation
lines; at least in Semantic those are usually dealt with in the lexer,
so the grammar doesn't even "see" those. In the end, you're still
looking around with some regexps to see where you're at.

> My experience (which lead me to write SMIE) is that all indentation
> styles for all languages share a lot of common principles, so that
> indentation can be (partly) automatically derived from the language's
> grammar.  And indeed, if you give a valid grammar to SMIE, it will
> give you a valid indentation function in return.  Of course, the
> indentation style it uses will most likely not fully match the user's
> expectations, so SMIE provides a hook to tweak the default indentation
> rules for the various possible situations.

This is really interesting. So you're generating something like
c-offsets-alist from a grammar?

> Indeed, for some reason C indentation is a lot more contentious than
> most other languages.  I haven't yet tried to use SMIE for C-like
> languages, so I'm not sure how well it would accommodate all the
> nitty-gritty details of each one's personal favorite indentation style.

C-like languages are an extreme case, for sure. I'm not saying a
semantic analysis won't help. It is absolutely possible to use Semantic
as an aid for indentation or highlighting (I use it for that in
minimap.el, for example). I just wanted to make clear that it's not
possible to base font-locking or indentation solely on Semantic, mainly
because it doesn't parse the whole buffer (usually not even close).

-David

Re: good "modern" example code for a programming-language mode?

[Stefan Monnier]

> Do you have the SMIE-based indentation of sh in some (private) branch?

Yes.  And it uses tweaks in smie.el.

> I can't find the above function in sh-script.el...

It's not ready to replace the previous code.


        Stefan

Re: good "modern" example code for a programming-language mode?

[Stefan Monnier]

>> OTOH there is a clear connection between syntax and indentation.
> Yes, there is a connection, but it's only covering a certain amount of
> indentation cases, depending on the language. Let's take comments, for
> example. Just looking at the doc-string from c-lineup-C-comments, I
> wonder what a grammar could help you here?  Or indenting continuation
> lines; at least in Semantic those are usually dealt with in the lexer,
> so the grammar doesn't even "see" those. In the end, you're still
> looking around with some regexps to see where you're at.

Of course.  Note that the grammar does not have to be the same as the
one used by the compiler, so it may make continuation-line markers and
comments "visible" (tho it'd probably be difficult since those can
typically appear anywhere).

> This is really interesting. So you're generating something like
> c-offsets-alist from a grammar?

Kind of.

> C-like languages are an extreme case, for sure. I'm not saying a
> semantic analysis won't help. It is absolutely possible to use Semantic
> as an aid for indentation or highlighting (I use it for that in
> minimap.el, for example). I just wanted to make clear that it's not
> possible to base font-locking or indentation solely on Semantic, mainly
> because it doesn't parse the whole buffer (usually not even close).

Agreed.  Most of the issue is that Semantic is designed for other tasks,
so it made different trade-offs.  E.g. using it for font-lock could
prove problematic since the parser would then have to be run much
more eagerly.


        Stefan

Re: good "modern" example code for a programming-language mode?

[Miles Bader]

David Engster <deng@randomsample.de> writes:
>>> I don't have a good answer for you, but I'd like to see more new modes
>>> based the semantic library.  It just seems like the right thing to do.
>>
>> Are there _any_ modes using it to handle "basic" language support
>> (indentation, highlighting)....?
>
> No. I also don't think that Semantic is well suited for that.

Yeah, Stefan's SMIE is looking like a much better fit right now.

-Miles

-- 
Alliance, n. In international politics, the union of two thieves who have
their hands so deeply inserted in each other's pockets that they cannot
separately plunder a third.