Re: Handling extensions of programming languages

[Matt Armstrong <matt@rfc20.org>]

haj@posteo.de (Harald Jörg) writes:

> today I'm looking for advice or hints how to deal with a task for
> CPerl mode which might have been solved for other programming
> languages: How to handle extensions of the language.

[...]

> Background: In Perl, adding new syntax to the language is easy enough
> so that many developers have done this

[...]

> My first approach was to keep all the code in one place and evaluate
> all the font-lock and indenting variables at runtime, as buffer-local
> variables, for the different versions.  This works to some extent for
> highlightingq, but fails if an extension needs different logic for
> indentation.

I'm not an expert in this topic it pertains to Emacs itself, but I've
always editor and development tools interesting and so have paid
attention to these issues over the years.

Very good Emacs support for languages with flexible syntax, which have a
high level of faithfulness to the language, or even "perfect"
faithfulness, all seem to rely on tools native to the language and
external to Emacs, usually by way of some sort of external server.

Examples: SLIME and Sly for Common Lisp, https://www.racket-mode.com/
for Racket, and, to a lesser degree of functionality, every language
with LSP support, especially C++ (which is known to be effectively
impossible to parse faithfully without what amounts to an entire
compiler frontend).  Indentation (formatting) source code is part of the
LSP protocol.  The common theme seems to be using the
interpreter/compiler itself to parse, without relying on the editor to
understand the code deeply.

For a different approach, you have examples of complete or nearly
complete parsers written in Emacs Lisp.  There is at least one parser
for Javascript that was at one time fully compliant with the language
standard to the point of providing a full parse tree to Lisp
(https://elpa.gnu.org/packages/js2-mode.html).  The CEDIT package has
some complex parser technology.  cc-mode for the C family of languages
is surprisingly good.  The drawback here is that, by design, any syntax
extensions and local mini-DSLs, etc., must also have parsers written in
Emacs Lisp.  You see this issue with js2-mode, where it lags the current
language standard a bit.

(info "(ccmode)Custom Macros") is an example of how cc-mode supports a
limited form of syntax extension.

I think most modes in Emacs Lisp take a pragmatic approach, using
heuristics that get the job done most of the time without being too
computationally expensive.  The SMIE package is a generalization of this
idea, see (info "(elisp)SMIE").

I am not aware of anything like SMIE that allows for languages
extensions to be "plugged in" in a general way.

In languages that support 'embeddng' other languages in sub-sections of
code (e.g. CSS or PHP in HTML), the kinds of approaches seen at
https://www.emacswiki.org/emacs/MultipleModes have been tried.