Macros considered harmful (was: Tree-sitter integration on feature/tree-sitter (severe performance issues together with linum-mode))

[Stefan Monnier <monnier@iro.umontreal.ca>]

Clément Pit-Claudel [2022-09-05 19:53:26] wrote:
> On 8/20/22 07:14, Stefan Monnier wrote:
>> FWIW, I think specifying the highlighting rules with something akin to:
>>      (defvar <foo> '<rules>)
>> is a mistake.  It should go through some kind of macro, such as (maybe):
>>      (defvar <foo> (tree-sitter-rules <rules>))
>> which can thus do any preprocessing we may want, such as pre-compiling
>> queries.  It also helps evolve the syntax since we can more easily warn
>> about obsolete uses, etc...
>> I've had a "rewrite font-lock.el so the rules go through a macro" in my
>> todo list for ages.
>
> We do things this way in Flycheck, but we've been bitten a few times by the
> way this pattern interacts with `with-eval-after-load`, so I would be
> careful about adopting this pattern in tree-sitter (unless we expect it to
> be preloaded).

Very good point.  It would introduce a strong compile-time dependency on
the tree-sitter package, which could be a serious annoyance for random
`foo-mode` packages which want to keep working in the absence of
tree-sitter.

Hmm...

[ I guess we'd have to make that macro lightweight and independent from
  tree-sitter itself and put it into a separate package distributed via
  GNU ELPA, so packages can feel free to depend on it.
  Hmm...  Another problem.  ]

> In fact, I think your suggestion back then was to *not* use a macro?

Indeed,

[ Lua eshews macros altogether for those kinds of reasons, AFAIU.  ]

Admittedly, another way around these kinds of problems is to teach the
compiler how to deal with an unknown macro.  I.e. something like
(declare-macro my-foo ...) so that if the compiler see (my-foo ...) but
`my-foo` can't be macroexpanded (because the macro is not yet defined),
it doesn't incorrectly compile it into a function call, but instead
residualizes it into something like a call to `eval`.  Making it
interact correctly with lexical scoping could be tricky (I guess the
simplest solution would be to residualize the whole toplevel expression
in which the macro call was found).


        Stefan