bug#4030: forward-sexp parses character literal ?; as comment

[Stefan Monnier <monnier@iro.umontreal.ca>]

>>> We could try to mark any `?;' or `?"' sequences appropriately when
>>> fontifying though.
>> We could/should/will improve the syntax parsing to handle those
>> things properly.  But it's a non-trivial amount of work, especially
>> since every major mode has similar issues but needs different
>> extra functionality.
> But fontification is mode-specific.  So it would be sufficient to look
> for ?s that are not within strings or comments and followed by a
> semicolon, paren or double-quote and mark that appropriately (obviously
> the parser is derailed at that time but it still might help people spot
> the bug).

Yes, but we need to do that even on chunks of code that have not yet
been (and may never be) displayed and in buffers where font-lock
is disabled.  IOW I'm not talking about fontification but about parsing.

>>>> I'll also point out that an "Unbalanced parentheses" error from deep
>>>> inside Customize is not a very helpful error message (especially as it
>>>> does not indicate in which buffer the unbalanced parentheses were
>>>> found); but perhaps Customize should be adapted to cope if forward-sexp
>>>> cannot easily be fixed.
>>> Getting good diagnostics after a parsing error is hard.
>> 
>> Agreed, but that doesn't mean we shouldn't intend to do better: the
>> current behavior (signalling an internal error to the user) is a bug
>> that needs to be fixed.

> If the problem comes from the unprotected

>       (save-excursion (forward-sexp (buffer-size)))) ; Test for scan errors.

> call in `custom-save-delete' we could simply wrap it as in the patch
> below.

Pretty much, yes, tho the error message should give more info (the OP
complained about lack of info in the error message).

> But do we really have to scan the buffer in the first place?

Don't know.  Maybe not, indeed.  Maybe it's just to detect the "too many
closing parens" case as well (i.e. rather than silently ignore trailing
code).


        Stefan