Re: JSON parsing of Extended ASCII

[Eli Zaretskii <eliz@gnu.org>]

> Date: Tue, 10 May 2022 19:18:37 +0200
> From: Felix Weilbach <felix.weilbach@t-online.de>
> 
> I'm working on source code that has Extended ASCII characters inside 
> comments in it.

What do you mean by "Extended ASCII"?  Is this some encoding different
from UTF-8, like Latin-1 or Windows codepage 1251?  Or is it something
else?

> This is unfortunate because LSP-mode crashes if it 
> encounters an invalid UTF-8 string. I know this mailing list is not 
> about LSP-mode, but I could track down the issue to the function 
> json_stringn() that gets called in lisp_to_json() inside json.c. Visual 
> Studio Code can cope with these files fine and their LSP implementation 
> has no problem with these files. Therefore I think Emacs should handle 
> this situation as well. I want to find a solution to this problem and 
> implement it. What do you think about this problem? Have you ideas for a 
> solution? Should a potential solution be implemented inside Emacs or 
> LSP-mode?
> 
> I think there could be implemented a function inside Emacs that tries to 
> convert Extended ASCII to UTF-8 and then encodes it to JSON.

Strings held inside Emacs are stored in (a superset of) UTF-8 anyway,
so this problem should not exist for you.  IOW, the conversion to
UTF-8 is done automatically when non-ASCII string is read into Emacs.
So I think you should provide more details about your use case
specifics, because it is likely that the problem is not where you
think it is.

In any case, Emacs and json.c require UTF-8 because that's what
libjansson, the library we use for JSON parsing, enforces.  It won't
help if we pass non-UTF-8 strings to the library, because it won't
work.