bug#4030: forward-sexp parses character literal ?; as comment

[Stefan Monnier <monnier@iro.umontreal.ca>]

>> While the workaround is good (and documented in the Ubuntu bug as well),
>> the ability of Customize depends on this code working correctly, and it
>> should handle any nominally well-formed .emacs file.  Perhaps there are
>> other pieces of code which rely on forward-sexp et alii for Emacs Lisp
>> parsing as well.

> Using a backslash _is_ the canonical way for handling this problem.  If
> some part of Emacs puts such a semicolon into an Elisp buffer without
> escaping it, then that part of Emacs is wrong and has to be fixed.

Indeed.

> If you manually insert such a construct, then you are on your own
> (just as when within a string you put a left paren in column zero).

Pretty much, yes.

> We could try to mark any `?;' or `?"' sequences appropriately when
> fontifying though.

We could/should/will improve the syntax parsing to handle those
things properly.  But it's a non-trivial amount of work, especially
since every major mode has similar issues but needs different
extra functionality.

>> I'll also point out that an "Unbalanced parentheses" error from deep
>> inside Customize is not a very helpful error message (especially as it
>> does not indicate in which buffer the unbalanced parentheses were
>> found); but perhaps Customize should be adapted to cope if forward-sexp
>> cannot easily be fixed.
> Getting good diagnostics after a parsing error is hard.

Agreed, but that doesn't mean we shouldn't intend to do better: the
current behavior (signalling an internal error to the user) is a bug
that needs to be fixed.

>> It appears that src/syntax.c could perhaps be adapted to take into
>> account character literals as well as quoted strings, but I am not
>> familiar enough with Emacs internals to tell whether this is really a
>> feasible approach.

Yes, clearly it deserves improvement, but that's nontrivial.


        Stefan