Re: Tokenizing

[Eric Ludlam <ericludlam@gmail.com>]

  On 09/20/2014 12:40 PM, Vladimir Kazanov wrote:
>> Tokenizing the whole buffer after any change is easily fast enough (on
>> modern hardware), even on a 7000 line buffer. Semantic parsing gets a
>> lot slower.
>>
> This is what I do right now in my prototype of a smarter Python mode.
> The tokenizing process itself is usually fast enough. But parsing is
> more complicated, and may take some time to rebuild the parse tree.
> Incremental approach is a natural step here.
>

I caught only the tail of this thread, so I apologize if I refer to the 
incorrect thing.

A year ago or so we were talking about ada-mode, a modified parser, and 
how it might integrate with CEDET/Semantic on the CEDET mailing list.  
Is it still 'wisi', a different flavor of 'wisent' ?

If calls into your parser are being handled by parts of CEDET/Semantic 
for creating tags, then there is an incremental parser that you can 
enable that works with two of the other parser types included.  The act 
of tagging the buffer enables it to track edits, and only reparse the 
bits that change.  It handles incomplete code, and waits for it to be 
completed.  There are many other possible states as well.

I certainly recommend trying to take advantage of that if the parsing 
you refer to is somehow related to Semantic.  I'll be happy to help you 
figure out how to make your parser work in that framework.

If this thread isn't about that, then carry on... :)

Eric