Re: access to parser stack in SMIE

[Stefan Monnier <monnier@iro.umontreal.ca>]

Your problem is one I also bumped into for the Modula-2 and Pascal modes
(can't remember how I "solved" them nor to what extent the "solution"
works).

> When all the tokens are properly disambiguated, SMIE can traverse
> correctly from package "begin" to "package". But we can't do that while
> disambiguating "begin"; that's circular.

Actually, in some cases, it can be made to work: to disambiguate "begin"
setup a loop that calls smie-backward-sexp repeatedly (starting from the
position just before the "begin", of course) checking after each call
whether the result lets us decide which of the two begins we're
dealing with.

Depending on the particular language's grammar this may not work because
"checking whether the result lets us decide" may not be possible.
But the recursion does terminate at least.

> The solution I found is to deliberately call `smie-forward-sexp'
> starting at the beginning of the buffer.

Right: the only way the parsing-stack can be used reliably is if we
always parse from the beginning.

> It might also make sense to incorporate the refined keyword cache
> mechanism into smie.

Right, if we want to make the stack visible, then we also need to
implement the cache.

Note that such a "forward full-parse with cache" approach has several
downsides:
- potential performance impact on long buffers.
- risk of the cache going out of sync.
- parse errors far away in an unrelated (earlier) part of the buffer
  can prevent proper local indentation.  Parse errors can occur for lots
  of reasons (e.g. temporarily incorrect code, incomplete parser, or
  use of "exotic" language extension, use of a preprocessor, ...).

That doesn't mean it's not workable, but those downsides had better come
with some significant upside.  One significant upside is that by only
parsing forward we can use any other parsing technology, such as LALR,
GLR, PEG, ...

I.e. I think it's an interesting direction but it would be another package.


        Stefan