Re: access to parser stack in SMIE

[Stefan Monnier <monnier@IRO.UMontreal.CA>]

>> Actually, in some cases, it can be made to work: to disambiguate "begin"
>> setup a loop that calls smie-backward-sexp repeatedly (starting from the
>> position just before the "begin", of course) checking after each call
>> whether the result lets us decide which of the two begins we're
>> dealing with.
> In the Ada use case I presented (package ... begin), that ends up going
> all the way back to package at the beginning of the buffer, encountering
> more "begin" tokens on the way; it's much more efficient to start there
> in the first place.

In your example, we might indeed end up scanning the buffer 3 times
(once for the normal scan, once to disambiguate package's `begin', and
one more time (in various chunks) to disambiguate the `begin's of the
nested functions).

But I wonder now: can a "begin" that comes right after a "function
... end;" be a begin-open?  I don't think so (but my Ada is rusty and
outdated).  So you might really not need to scan that far back.

>> Right, if we want to make the stack visible, then we also need to
>> implement the cache.
> Ok. Although different languages may want to cache different things, so
> I'm not sure that would really be common code.

We'd make smie.el cache the whole stack state at various buffer
positions (just like we do for the syntax-ppss cache).

>> - risk of the cache going out of sync.
> Yes. I've already got an interactive `ada-indent-invalidate-cache' to
> handle that, but the user will have to be aware. So far, the cache has
> not gotten out of sync, but I haven't used this to write real
> code yet.

I don't think there is that much to worry about in general.

> Switching to another parsing technology just for the forward full-parse
> means supporting a second grammar; that's a lot of work. It might be
> simpler to switch to only forward full-parse (which I think is what you
> are suggesting).

Yes, if you end up doing a full forward parse in most cases anyway,
there's little point doing extra work to support backward parsing.

> If I switch to only using forward full-parse, I'd have to do things
> differently in the indentation rules as well. The key information they
> need need is the location of the statement start for every indentation
> point. So the forward parse could store the relevant statement start
> (and maybe other stuff) with every token.

Indeed.

> Hmm.  The actual change to smie I'm asking for is very small (if we
> leave out the cache); perhaps you could put that in, with a large "use
> at your own risk" comment?

Indeed, just exposing the stack is not that bad, and lets you solve your
problem.  But it's kind of ugly.  Maybe I could instead provide
a function that lets you query the particular part of the stack that
you're interested in (that would make it easier to adapt to a new
format of the stack, for example).

Could you describe which part of the stack you need to know?


        Stefan