Re: forward-sexp when on a floating point number

[Marcin Borkowski <mbork@mbork.pl>]

On 2016-01-18, at 22:34, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:

>> FWIW, I did something like this for (La)TeX here:
>> https://github.com/mbork/tex-plus.el (note: this is still WiP, and
>> contains more than just tokenizing TeX).  Either TeX token syntax is
>> weird and/or difficult, either I really suck at writing lexers/parsers,
>> or this is indeed a (potentially) difficult problem.  (Probably all
>> three.)
>
> Don't know about the middle one, but the other two indeed apply.  In the
> general case, the esiest way to solve this problem is probably to go
> back to a safe earlier state and then lex forward from there.

Yes, in case of TeX, it is relatively easy (assuming the user did not
play around with catcodes - in such a case, /anything/ can happen (see
e.g. https://tug.org/TUGboat/tb19-4/tb61carl.pdf - try it!).

> A "safe earlier state" could be "right after a character which can only
> appear at the end of a token" [ of course, there's no guarantee that such
> a character exists ].

In TeX (under normal catcode regime), it's just like that: newline.

> Also "going back and then lex forward" implies a potential serious
> performance problem, so it would require some form of caching.

Exactly, and I wanted to avoid that.  Frankly, I had no idea how to do
this anyway; now I'm a bit wiser, and I guess I could run an idle timer
to update the cache, and have a global variable for keeping the state
(cache synchronized or not), set to "not in sync" after any
text-changing command.  Of course, any command that actually needs
a cache would have to recreate it, too.  Would that be a good plan?

>         Stefan

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Faculty of Mathematics and Computer Science
Adam Mickiewicz University