JSON parsing of Extended ASCII

JSON parsing of Extended ASCII

[Felix Weilbach]

Hello,


I'm working on source code that has Extended ASCII characters inside 
comments in it. This is unfortunate because LSP-mode crashes if it 
encounters an invalid UTF-8 string. I know this mailing list is not 
about LSP-mode, but I could track down the issue to the function 
json_stringn() that gets called in lisp_to_json() inside json.c. Visual 
Studio Code can cope with these files fine and their LSP implementation 
has no problem with these files. Therefore I think Emacs should handle 
this situation as well. I want to find a solution to this problem and 
implement it. What do you think about this problem? Have you ideas for a 
solution? Should a potential solution be implemented inside Emacs or 
LSP-mode?

I think there could be implemented a function inside Emacs that tries to 
convert Extended ASCII to UTF-8 and then encodes it to JSON.


Kind regards

Felix Weilbach

Re: JSON parsing of Extended ASCII

[Eli Zaretskii]

> Date: Tue, 10 May 2022 19:18:37 +0200
> From: Felix Weilbach <felix.weilbach@t-online.de>
> 
> I'm working on source code that has Extended ASCII characters inside 
> comments in it.

What do you mean by "Extended ASCII"?  Is this some encoding different
from UTF-8, like Latin-1 or Windows codepage 1251?  Or is it something
else?

> This is unfortunate because LSP-mode crashes if it 
> encounters an invalid UTF-8 string. I know this mailing list is not 
> about LSP-mode, but I could track down the issue to the function 
> json_stringn() that gets called in lisp_to_json() inside json.c. Visual 
> Studio Code can cope with these files fine and their LSP implementation 
> has no problem with these files. Therefore I think Emacs should handle 
> this situation as well. I want to find a solution to this problem and 
> implement it. What do you think about this problem? Have you ideas for a 
> solution? Should a potential solution be implemented inside Emacs or 
> LSP-mode?
> 
> I think there could be implemented a function inside Emacs that tries to 
> convert Extended ASCII to UTF-8 and then encodes it to JSON.

Strings held inside Emacs are stored in (a superset of) UTF-8 anyway,
so this problem should not exist for you.  IOW, the conversion to
UTF-8 is done automatically when non-ASCII string is read into Emacs.
So I think you should provide more details about your use case
specifics, because it is likely that the problem is not where you
think it is.

In any case, Emacs and json.c require UTF-8 because that's what
libjansson, the library we use for JSON parsing, enforces.  It won't
help if we pass non-UTF-8 strings to the library, because it won't
work.

Re: JSON parsing of Extended ASCII

[Stefan Monnier]

Felix Weilbach [2022-05-10 19:18:37] wrote:
> I'm working on source code that has Extended ASCII characters inside
> comments in it. This is unfortunate because LSP-mode crashes if it
> encounters an invalid UTF-8 string.

Please report it as a bug.  It might be a bug in LSP-mode, or maybe in
Emacs, or maybe in your LSP server, so as always, provide as many
details as possible, starting with the actual error message (or even
better a backtrace) as well as steps to (try and) reproduce the problem.


        Stefan