Re: String encoding in json.c

[Philipp Stephani <p.stephani2@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4727 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Sa., 23. Dez. 2017 um 16:53 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sat, 23 Dec 2017 15:31:06 +0000
> > Cc: emacs-devel@gnu.org
> >
> >  The coding operations are "expensive no-ops" except when they aren't,
> >  and that is exactly when we need their 'expensive" parts.
> >
> > In which case are they not no-ops?
>
> When the input is not a valid UTF-8 sequence.  When that happens, we
> produce a special representation of such raw bytes instead of
> signaling EILSEQ and refusing to decode the input.  Encoding (if and
> when it is done) then performs the opposite conversion, producing the
> same single raw byte in the output stream.  This allows Emacs to
> manipulate text that included invalid sequences without crashing,
> because all the low-level primitives that walk buffer text and strings
> by characters assume the internal representation of each character is
> valid.
>

OK, thanks for the refresher. I was aware of the single byte
representation, but forgot how exactly it's handled during coding.


>
> > Using utf-8-unix as encoding seems to keep the encoding intact.
>
> First, you forget about decoding.


OK, let's treat encoding and decoding separately.

- We encode Lisp strings when passing them to Jansson. Jansson only accepts
UTF-8 strings and fails (with proper error reporting, not crashing) when
encountering non-UTF-8 strings. I think encoding can only make a difference
here for strings that contain sequences of bytes that are themselves valid
UTF-8 code unit sequences, such as "Ä\xC3\x84". This string is encoded as
"\xC3\x84\xC3\x84" using utf-8-unix. (Note how this is a case where
encoding and decoding are not inverses of each other.) Without encoding,
the string contents will be \xC3\x84 plus two invalid 5-byte sequences. I
think it's not obvious at all which interpretation is correct; after all,
"Ä\xC3\x84" is not equal to "ÄÄ", but the two strings now result in the
same JSON representation. This could be at least surprising, and I'd argue
that the other behavior (raising an error) would be more correct and more
obvious.

- We decode UTF-8 strings after receiving them from Jansson. Jansson
guarantees to only ever emit well-formed UTF-8. Given that for well-formed
UTF-8 strings, the UTF-8 representation and the Emacs representation are
one and the same, we don't need decoding.



>   And second, encoding keeps the
> encoding intact precisely because it is not a no-op: raw bytes are
> held in buffer and string text as special multibyte sequences, not as
> single bytes, so just copying them to output instead of encoding will
> produce non-UTF-8 multibyte sequences.
>

That's the correct behavior, I think. JSON values must be valid Unicode
strings, and raw bytes are not.


>
> > I've spot-checked some other code where we interface with external
> libraries, namely dbusbind.c and
> > gnutls.c. In no cases I've found explicit coding operations (except for
> filenames, where the situation is
> > different); these files always use SDATA directly. dbusbind.c even has
> the comment
> >
> >   /* We need to send a valid UTF-8 string.  We could encode `object'
> >      but by not encoding it, we guarantee it's valid utf-8, even if
> >      it contains eight-bit-bytes.  Of course, you can still send
> >      manually-crafted junk by passing a unibyte string.  */
>
> If gnutls.c and dbusbind.c don't encode and decode text that comes
> from and goes to outside, then they are buggy.


Not necessarily. As mentioned, the internal encoding of multibyte strings
is even mentioned in the Lisp reference; and the above comment indicates
that it's OK to use that information at least within the Emacs codebase.
BTW, that comment was added by Stefan in
commit e454a4a330cc6524cf0d2604b4fafc32d5bda795, where he removed an
explicit encoding step.


>   (At least for
> gnutls.c, I think you are mistaken, because the encoding/decoding is
> in process.c, see, e.g., read_process_output.)
>

Some parts are definitely encoded, but for example, there is c_hostname in
Fgnutls_boot, which doesn't encode the user-supplied string.


>
> > It's the *current* json.c (and emacs-module.c) that's inconsistent
> > with the rest of the codebase.
>
> Well, I disagree with that conclusion.  Just look at all the calls to
> decode_coding_*, encode_coding_*, DECODE_SYSTEM, ENCODE_SYSTEM, etc.,
> and you will see where we do that.
>

We obviously do *some* encoding/decoding. But when interacting with
third-party libraries, we seem to leave it out pretty frequently, if those
libraries use UTF-8 as well.

[-- Attachment #2: Type: text/html, Size: 6152 bytes --]