bug#36432: 26.2; SMIE does not request forward tokens when point is at point-max

[Eli Zaretskii <eliz@gnu.org>]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Sat, 29 Jun 2019 14:01:52 +0100
> Cc: 36432@debbugs.gnu.org
> 
> On 29/06/2019, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Sat, 29 Jun 2019 15:42:27 +0300
> >> From: Eli Zaretskii <eliz@gnu.org>
> >> Cc: 36432@debbugs.gnu.org
> >>
> >> How would Emacs know this is the case?  Why don't you also override
> >> point-max to make it consistent with those illusory characters?
> >
> > Or actually insert those characters at EOB, but make them invisible?
> 
> I think I'd like to avoid doing that unless I do it for all of the
> virtual tokens. I don't know how to do that without it impacting the
> underlying source code. Is there more documentation about doing it
> that way? It'd be a drastic change in how the lexer is written.

Why? does the lexer only pay attention to visible characters?

> BTW I had a look through smie.el and I can't see anywhere obvious
> where (eobp) or (point-max) are called that would lead to the bug I'm
> seeing.

It's likely in lower-level code.  Like I said: this assumption is
everywhere in Emacs.