Re: Emacs contributions, C and Lisp

[David Engster <deng@randomsample.de>]

Mario Lang writes:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> > To figure out just what Emacs needs, that's the task I am talking
>>>> > about.
>>>> If you want to support things like completions, refactoring, symbol
>>>> searches, etc., we need full access to the AST from inside Emacs.
>>> With all due respect, it is so important to avoid the full AST
>>> that I'm not going to give up on it just because someone claims
>>> that is necessary.
>>
>> Give up on what, exactly?
>>
>> Richard, this is absurd!
>> I'm convinced there is a misunderstanding.
>
> While I also think this attempt to cage data is rather absurd, I think
> there is likely another misunderstanding.  What Clang currently seems to
> do for editor integration seems to match Richard's vision.
>
> While they have functionality to dump the full AST, clang-format,
> clang-query, clang-rename and the completion interface don't expose a
> full AST at all. The idea behind the Clang tools is that they parse
> the translation units on their own, and do manipulations on the data
> internally.  You pass parameters to these tools, like a position in a
> file you want to see completions for, or a range of locations to
> reformat, or a matcher expression to search for particular AST
> constructs.

There is one important point that needs to be mentioned: all those
tools, as well as the clang binary itself, are not really
"external". They all simply call functions from libclang. You probably
know this, but this has already confused people here.

Therefore, to do what libclang does, you would have to implement all
that functionality inside GCC. I don't doubt that it is possible - it's
just software, after all. However, you would need experienced GCC
hackers to do that, since you need to hook yourself deep into the parser
to do what libclang does. AFAIK, this is not possible with a mere GCC
plugin.

Unfortunately, GCC hackers are a rare breed. I'm not one of them, and I
don't plan to become one. So my goal is to get GCCs internal data into a
hacking-friendly environment, which essentially means "out of GCC". My
understanding is that due to clang's modular architecture and because it
is implemented as a library, it is much easier to hook yourself into it
than it is in GCC.

For instance, implementing something like completions inside GCC is not
trivial at all. There was actually a patch which tried to implement this
(search in their mailing list for 'gccsense'). It was realized by
introducing a "phantom token" into the lexer, and when the parser would
reach that token, it would spill out the possible completions based on
the current AST. This patch was rejected because it is arguably a hack
and would be very hard to maintain. I actually pointed Richard to this
patch over two years ago; he wanted to investigate, and I never heard
back.

> If GNU were to match that functionality, it would not rely on exporting
> a full AST.  Walking a full AST in elisp just to do completion seems
> like a performance hog anyway.

Walking the full AST in Elisp is out of the question, yes. Some stuff
must be done inside the plugin. For instance, matching certain
expressions in the AST could be done there. And whatever Emacs does must
be done lazily, meaning only looking at those parts of the AST that are
currently in scope; and with heavy caching, of course. Fortunately, much
of that infrastructure is already present in CEDET.

I would also envision external tools. For instance, CEDET already relies
on GNU Global for finding references. One could extend GNU Global to use
GCC's AST instead of parsing the files itself, and introduce new queries
specific to C++.

> So the approach *they* are taking is to write external tools that do the
> various refactoring jobs.

Again: not "really" external. It's all libclang.

> These tools can easily be called from *any* editor, not needing to
> duplicate the various refactoring features everywhere.  I think this
> approach is rather good, and it does *not* rely on fully exporting any
> AST to the various editors, because, as already mentioned, the goal is
> to *not* duplicate refactoring features in the various editing
> environments.
>
> If GNU were to match this, the work would go into tools based on GCC.

But those tools would need the AST! This is precisely what Richard does
not want. So the only solution is to do it inside GCC. I'm not opposed
to that approach at all, but it's not like this is a new idea. GCC is
very hard to hack on, and my guess is that there's simply no manpower to
implement this.

-David