url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

[Iñigo Serna]

Hi,

I'm developing a package that uses `url-retrieve` to connect to a HTTP server passsing a string query as a json object.

It mostly works, except when I need to pass a non-ASCII string, where it fails with "Multibyte text in HTTP request" error [1] raised in function `url-http-create-request` in url-http.el.
This happens even if I encode the query string with UTF-8.

I guess this is related to bug #23750, that in v25.1 times added a check to make this error happen.
If I remove that "fix" the code works without any problems.

I also tried emacs-request package [3], it works using curl backend and fails when using url-retrieve backend with the same error.

I tried emacs versions 26.3 (from Fedora 32), and last commits from 27.x and 28.x branches.
The whole function is at [4].


I reread several times the bug information and I can't fully understand the need of that check, IMO it corresponds to the program developer, not to the API.
Anyway, is there anything I could do? Any way to pass data as a UTF-8 string?


Thanks in advance,
Iñigo Serna


[1]
Debugger entered--Lisp error: (error "Multibyte text in HTTP request: POST /jsonrpc.js H...")
signal(error ("Multibyte text in HTTP request: POST /jsonrpc.js H..."))
error("Multibyte text in HTTP request: %s" "POST /jsonrpc.js HTTP/1.1\15\nMIME-Version: 1.0\15\nConn...")
url-http-create-request()
url-http(#s(url :type "http" :user nil :password nil :host "nas" :portspec 9002 :filename "/jsonrpc.js" :target nil :attributes nil :fullness t :silent t :use-cookies nil :asynchronous nil) #f(compiled-function (&rest ignored) #<bytecode -0x113049c86a7cc012>) (nil))
url-retrieve-internal("http://nas:9002/jsonrpc.js" #f(compiled-function (&rest ignored) #<bytecode -0x113049c86a7cc012>) (nil) t t)
url-retrieve("http://nas:9002/jsonrpc.js" #f(compiled-function (&rest ignored) #<bytecode -0x113049c86a7cc012>) nil t t)
url-retrieve-synchronously("http://nas:9002/jsonrpc.js" t t 1)


[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23750

[3] https://github.com/tkf/emacs-request

[4] 
(defun lms2--cmd (query &optional playerid)
  "Sync HTTP request QUERY to LMS server.  PLAYERID is optional.
QUERY is a string."
  (setq playerid (or playerid lms2--default-playerid ""))
  (let* ((json-array-type 'list)
         (url-request-method "POST")
         (url-request-extra-headers '(("Content-Type" . "application/json; charset=utf-8")))
         (url-request-data (json-encode-alist `[(method . "slim.request") (params . [,playerid ,(split-string query)])]))
         (response (with-current-buffer (url-retrieve-synchronously (concat lms2-url "/jsonrpc.js") t t 1)
                     (goto-char (point-min))
                     (re-search-forward "^$")
                     (delete-region (point) (point-min))
                     (decode-coding-string (buffer-string) 'utf-8))))
    (unless (string= "" response)
      (alist-get 'result (json-read-from-string response)))))

(lms2--cmd (encode-coding-string "artists 0 10 search:Björk" 'utf-8))

Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

[Dmitry Gutov]

Hi!

On 14.06.2020 00:08, Iñigo Serna wrote:
> This happens even if I encode the query string with UTF-8.

What about encoding the body of the request?

Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

[Iñigo Serna]

Hi Dmitry,

First of all, thanks for your time.

On 14 June 2020 at 00:12 +02, Dmitry Gutov <dgutov@yandex.ru> wrote:

> What about encoding the body of the request?

Do you mean execute `encode-coding-string` outside `json-encode`?
Uhmm… It works!

         (url-request-data (encode-coding-string
                            (json-encode-alist `[(method . "slim.request") (params . [,playerid ,(split-string query)])])
                            'utf-8))


Thanks again for the help,
Iñigo

Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

[Dmitry Gutov]

On 14.06.2020 02:24, Iñigo Serna wrote:
> First of all, thanks for your time.
> 
> On 14 June 2020 at 00:12 +02, Dmitry Gutov<dgutov@yandex.ru>  wrote:
> 
>> What about encoding the body of the request?
> Do you mean execute `encode-coding-string` outside `json-encode`?
> Uhmm… It works!
> 
>           (url-request-data (encode-coding-string
>                              (json-encode-alist `[(method . "slim.request") (params . [,playerid ,(split-string query)])])
>                              'utf-8))
> 
> 
> Thanks again for the help,

Cheers!

Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

[Stefan Monnier]

>          (url-request-data (encode-coding-string
>                             (json-encode-alist `[(method . "slim.request") (params . [,playerid ,(split-string query)])])
>                             'utf-8))

This might beg for some better documentation about the string returned
by `json-encode`.  After all, AFAIK the JSON format requires utf-8
encoding, so it would make a lot of sense for `json-encode` to do the
utf-8 encoding directly and make sure it always returns a unibyte string.


        Stefan

Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

[Yuri Khan]

On Sun, 14 Jun 2020 at 07:21, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> This might beg for some better documentation about the string returned
> by `json-encode`.  After all, AFAIK the JSON format requires utf-8
> encoding, so it would make a lot of sense for `json-encode` to do the
> utf-8 encoding directly and make sure it always returns a unibyte string.

An application might not want to send the encoded string over the
network right away. It might want to insert it into a buffer.

It would make a lot of sense for people who deal with files, sockets
and network protocols to also deal with encoding, while people who
deal with strings and buffers use JSON-serialized strings
transparently.

Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

[Stefan Monnier]

>> This might beg for some better documentation about the string returned
>> by `json-encode`.  After all, AFAIK the JSON format requires utf-8
>> encoding, so it would make a lot of sense for `json-encode` to do the
>> utf-8 encoding directly and make sure it always returns a unibyte string.
>
> An application might not want to send the encoded string over the
> network right away. It might want to insert it into a buffer.

I didn't suggest to change the behavior, only to document it because
both outputs can make sense, so common sense is not sufficient to know
what should be returned.


        Stefan

Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

[Yuri Khan]

On Sun, 14 Jun 2020 at 20:49, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> >> This might beg for some better documentation about the string returned
> >> by `json-encode`.  After all, AFAIK the JSON format requires utf-8
> >> encoding, so it would make a lot of sense for `json-encode` to do the
> >> utf-8 encoding directly and make sure it always returns a unibyte string.
> >
> > An application might not want to send the encoded string over the
> > network right away. It might want to insert it into a buffer.
>
> I didn't suggest to change the behavior, only to document it because
> both outputs can make sense, so common sense is not sufficient to know
> what should be returned.

Oh, I see. Yes please. More libraries should take care to document
their behavior wrt string encodings.