Re: forward-sexp when on a floating point number

[Stefan Monnier <monnier@iro.umontreal.ca>]

>> Lexing via FSM is "standard" in the world of computer languages, so I'm
>> pretty sure it'd be good/useful to add such functionality to Emacs's core.
> If it can provide identical behavior, simpler and more efficiently, I'm happy
> to swap out what we have in core.

My suggestion is to *extend* syntax-table, such that (aref
<syntax-table> <char>) can return another syntax-table (IOW another
state in the FSM, because the char we just considered is part of a token
but that token isn't complete yet).

Major modes could use it or not.

> Yes, we control core, but that doesn't mean it should be the first place we
> look to make changes when brewing new ideas.

I largely agree.  The main reason why it didn't turn out that way for
many of the features I added is that I wanted to make use of them, and
since most of the packages I work on (and use) are in core, I couldn't
make use of those new features in them until that new feature is
in core.

It's also part of the motivation to try and bring GNU ELPA and core
closer together (either by exporting core packages to GNU ELPA like we
have now, or by including GNU ELPA packages into core like we want to
do but still haven't done).


        Stefan


PS: Lexing via FSM is harder than I make it out to be, of course, since
multi-char tokens introduce the question of how to figure out with which
state to start lexing (e.g. if we start a command from the middle of
a token), as well as how to "tokenize backward" ("single-char tokens"
(like we have now) can trivially be parsed with the same FSM going
forward and backward, but that's not true for the more general case).


        Stefan