bug#30076: [PATCH] web: Recognize JSON content type as text.

bug#30076: [PATCH] web: Recognize JSON content type as text.

[Arun Isaac]

* module/web/response.scm (text-content-type?): Recognize JSON content
  type as text.
---
 module/web/response.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/module/web/response.scm b/module/web/response.scm
index 06e1c6dc1..679304c4d 100644
--- a/module/web/response.scm
+++ b/module/web/response.scm
@@ -184,6 +184,7 @@ reason phrase for the response's code."
 represents a textual type such as `text/plain'."
   (let ((type (symbol->string type)))
     (or (string-prefix? "text/" type)
+        (string-suffix? "/json" type)
         (string-suffix? "/xml" type)
         (string-suffix? "+xml" type))))
 
-- 
2.15.1

bug#30076: [PATCH] web: Recognize JSON content type as text.

[Mark H Weaver]

Hi Arun,

Arun Isaac <arunisaac@systemreboot.net> writes:
> * module/web/response.scm (text-content-type?): Recognize JSON content
>   type as text.

While this would seem reasonable at first glance, it seems to me that
this will result in JSON texts with non-ASCII characters being
mishandled in many cases.

Within Guile, 'text-content-type?' is currently used in two places:

* 'decode-response-body' in (web client), and
* 'response-body-port' in (web response).

In both places, if 'text-content-type?' returns true, the encoding of
the response is assumed to be "ISO-8859-1" if not otherwise specified by
an explicit 'charset' parameter.  This is what RFC 2616 specifies for
text/plain, although RFC 6657 would change the default to US-ASCII, as
it was in RFC 2046, and maybe we should look into that.

However, things are quite different for the application/json MIME type,
as specified in RFCs 4627 and 7159.  Those RFCs specify that JSON text
"SHALL" (i.e. MUST) be encoded in Unicode (UTF-8, UTF-16 or UTF-32),
that the default encoding is UTF-8, and furthermore that no charset
parameter is defined for application/json.

So, we can expect at least some conforming implementations to omit the
'charset' parameter, and yet in that case we must assume that the
encoding is Unicode, and most definitely not ISO-8859-1.

RFC 4627 makes the additional interesting observation (in section 3,
"encoding") that since the first two characters of JSON text will always
be ASCII, and since UTF-8/UTF-16/UTF-32 are the only valid encodings for
JSON text, we can reliably determine the encoding by looking at the
pattern of nul bytes in the first four octets:

           00 00 00 xx  UTF-32BE
           00 xx 00 xx  UTF-16BE
           xx 00 00 00  UTF-32LE
           xx 00 xx 00  UTF-16LE
           xx xx xx xx  UTF-8

Given that any of these encodings above are possible, and that there is
no 'charset' parameter defined for "application/json", it seems to me
that we have no choice but to be prepared to auto-detect the encoding,
as described in RFC 4627 section 3 if the 'charset' parameter is
missing.

What do you think?

      Mark

bug#30076: [PATCH] web: Recognize JSON content type as text.

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:
> RFC 4627 makes the additional interesting observation (in section 3,
> "encoding") that since the first two characters of JSON text will always
> be ASCII,

Sorry, it turns out that's no longer the case.  RFC 4627 specified that
a JSON text must be either an object or array, but in RFC 7159 a JSON
text can be any JSON value.  So only the first character is guaranteed
to be ASCII.

Having looked into this a bit more, I wonder if Guile should even try to
set the port encoding itself.  As far as I can tell, there's no way to
know the encoding of the response payload in the general case, without
knowledge of the specific MIME media type.  We could teach Guile about
"application/json", but if we follow that path, it would lead to us
teaching Guile's web library about more media types over time, but we
cannot hope to know about all of them.

The 'charset' parameter is not universal.  Whether it is a valid
parameter, and how its value is to be interpreted, depends on the media
type.  For "application/json", technically there is no 'charset'
parameter at all.

Since it's not feasible for Guile to reliably choose the right encoding
for arbitrary media types, perhaps it would be better for Guile to
explicitly say that it's the application programmer's job to set the
encoding of the port, if it contains textual data.

What do you think?

      Mark

bug#30076: [PATCH] web: Recognize JSON content type as text.

[Arun Isaac]

> Having looked into this a bit more, I wonder if Guile should even try to
> set the port encoding itself.  As far as I can tell, there's no way to
> know the encoding of the response payload in the general case, without
> knowledge of the specific MIME media type.  We could teach Guile about
> "application/json", but if we follow that path, it would lead to us
> teaching Guile's web library about more media types over time, but we
> cannot hope to know about all of them.

> Since it's not feasible for Guile to reliably choose the right encoding
> for arbitrary media types, perhaps it would be better for Guile to
> explicitly say that it's the application programmer's job to set the
> encoding of the port, if it contains textual data.

"application/json" is common enough that it would be convenient for the
application programmer to have Guile know about it. But, as a Guile
maintainer, this is your call. I don't have strong opinions this way or
that.