all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* url-retrieve "Multibyte text in HTTP request" error when passing UTF-8
@ 2020-06-13 21:08 Iñigo Serna
  2020-06-13 22:12 ` Dmitry Gutov
  0 siblings, 1 reply; 8+ messages in thread
From: Iñigo Serna @ 2020-06-13 21:08 UTC (permalink / raw)
  To: emacs-devel

Hi,

I'm developing a package that uses `url-retrieve` to connect to a HTTP server passsing a string query as a json object.

It mostly works, except when I need to pass a non-ASCII string, where it fails with "Multibyte text in HTTP request" error [1] raised in function `url-http-create-request` in url-http.el.
This happens even if I encode the query string with UTF-8.

I guess this is related to bug #23750, that in v25.1 times added a check to make this error happen.
If I remove that "fix" the code works without any problems.

I also tried emacs-request package [3], it works using curl backend and fails when using url-retrieve backend with the same error.

I tried emacs versions 26.3 (from Fedora 32), and last commits from 27.x and 28.x branches.
The whole function is at [4].


I reread several times the bug information and I can't fully understand the need of that check, IMO it corresponds to the program developer, not to the API.
Anyway, is there anything I could do? Any way to pass data as a UTF-8 string?


Thanks in advance,
Iñigo Serna


[1]
Debugger entered--Lisp error: (error "Multibyte text in HTTP request: POST /jsonrpc.js H...")
signal(error ("Multibyte text in HTTP request: POST /jsonrpc.js H..."))
error("Multibyte text in HTTP request: %s" "POST /jsonrpc.js HTTP/1.1\15\nMIME-Version: 1.0\15\nConn...")
url-http-create-request()
url-http(#s(url :type "http" :user nil :password nil :host "nas" :portspec 9002 :filename "/jsonrpc.js" :target nil :attributes nil :fullness t :silent t :use-cookies nil :asynchronous nil) #f(compiled-function (&rest ignored) #<bytecode -0x113049c86a7cc012>) (nil))
url-retrieve-internal("http://nas:9002/jsonrpc.js" #f(compiled-function (&rest ignored) #<bytecode -0x113049c86a7cc012>) (nil) t t)
url-retrieve("http://nas:9002/jsonrpc.js" #f(compiled-function (&rest ignored) #<bytecode -0x113049c86a7cc012>) nil t t)
url-retrieve-synchronously("http://nas:9002/jsonrpc.js" t t 1)


[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23750

[3] https://github.com/tkf/emacs-request

[4] 
(defun lms2--cmd (query &optional playerid)
  "Sync HTTP request QUERY to LMS server.  PLAYERID is optional.
QUERY is a string."
  (setq playerid (or playerid lms2--default-playerid ""))
  (let* ((json-array-type 'list)
         (url-request-method "POST")
         (url-request-extra-headers '(("Content-Type" . "application/json; charset=utf-8")))
         (url-request-data (json-encode-alist `[(method . "slim.request") (params . [,playerid ,(split-string query)])]))
         (response (with-current-buffer (url-retrieve-synchronously (concat lms2-url "/jsonrpc.js") t t 1)
                     (goto-char (point-min))
                     (re-search-forward "^$")
                     (delete-region (point) (point-min))
                     (decode-coding-string (buffer-string) 'utf-8))))
    (unless (string= "" response)
      (alist-get 'result (json-read-from-string response)))))

(lms2--cmd (encode-coding-string "artists 0 10 search:Björk" 'utf-8))



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8
  2020-06-13 21:08 url-retrieve "Multibyte text in HTTP request" error when passing UTF-8 Iñigo Serna
@ 2020-06-13 22:12 ` Dmitry Gutov
  2020-06-13 23:24   ` Iñigo Serna
  0 siblings, 1 reply; 8+ messages in thread
From: Dmitry Gutov @ 2020-06-13 22:12 UTC (permalink / raw)
  To: Iñigo Serna, emacs-devel

Hi!

On 14.06.2020 00:08, Iñigo Serna wrote:
> This happens even if I encode the query string with UTF-8.

What about encoding the body of the request?



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8
  2020-06-13 22:12 ` Dmitry Gutov
@ 2020-06-13 23:24   ` Iñigo Serna
  2020-06-13 23:27     ` Dmitry Gutov
  2020-06-14  0:20     ` Stefan Monnier
  0 siblings, 2 replies; 8+ messages in thread
From: Iñigo Serna @ 2020-06-13 23:24 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: emacs-devel

Hi Dmitry,

First of all, thanks for your time.

On 14 June 2020 at 00:12 +02, Dmitry Gutov <dgutov@yandex.ru> wrote:

> What about encoding the body of the request?

Do you mean execute `encode-coding-string` outside `json-encode`?
Uhmm… It works!

         (url-request-data (encode-coding-string
                            (json-encode-alist `[(method . "slim.request") (params . [,playerid ,(split-string query)])])
                            'utf-8))


Thanks again for the help,
Iñigo



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8
  2020-06-13 23:24   ` Iñigo Serna
@ 2020-06-13 23:27     ` Dmitry Gutov
  2020-06-14  0:20     ` Stefan Monnier
  1 sibling, 0 replies; 8+ messages in thread
From: Dmitry Gutov @ 2020-06-13 23:27 UTC (permalink / raw)
  To: Iñigo Serna; +Cc: emacs-devel

On 14.06.2020 02:24, Iñigo Serna wrote:
> First of all, thanks for your time.
> 
> On 14 June 2020 at 00:12 +02, Dmitry Gutov<dgutov@yandex.ru>  wrote:
> 
>> What about encoding the body of the request?
> Do you mean execute `encode-coding-string` outside `json-encode`?
> Uhmm… It works!
> 
>           (url-request-data (encode-coding-string
>                              (json-encode-alist `[(method . "slim.request") (params . [,playerid ,(split-string query)])])
>                              'utf-8))
> 
> 
> Thanks again for the help,

Cheers!



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8
  2020-06-13 23:24   ` Iñigo Serna
  2020-06-13 23:27     ` Dmitry Gutov
@ 2020-06-14  0:20     ` Stefan Monnier
  2020-06-14  8:33       ` Yuri Khan
  1 sibling, 1 reply; 8+ messages in thread
From: Stefan Monnier @ 2020-06-14  0:20 UTC (permalink / raw)
  To: Iñigo Serna; +Cc: emacs-devel, Dmitry Gutov

>          (url-request-data (encode-coding-string
>                             (json-encode-alist `[(method . "slim.request") (params . [,playerid ,(split-string query)])])
>                             'utf-8))

This might beg for some better documentation about the string returned
by `json-encode`.  After all, AFAIK the JSON format requires utf-8
encoding, so it would make a lot of sense for `json-encode` to do the
utf-8 encoding directly and make sure it always returns a unibyte string.


        Stefan




^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8
  2020-06-14  0:20     ` Stefan Monnier
@ 2020-06-14  8:33       ` Yuri Khan
  2020-06-14 13:49         ` Stefan Monnier
  0 siblings, 1 reply; 8+ messages in thread
From: Yuri Khan @ 2020-06-14  8:33 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Dmitry Gutov, Iñigo Serna, Emacs developers

On Sun, 14 Jun 2020 at 07:21, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> This might beg for some better documentation about the string returned
> by `json-encode`.  After all, AFAIK the JSON format requires utf-8
> encoding, so it would make a lot of sense for `json-encode` to do the
> utf-8 encoding directly and make sure it always returns a unibyte string.

An application might not want to send the encoded string over the
network right away. It might want to insert it into a buffer.

It would make a lot of sense for people who deal with files, sockets
and network protocols to also deal with encoding, while people who
deal with strings and buffers use JSON-serialized strings
transparently.



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8
  2020-06-14  8:33       ` Yuri Khan
@ 2020-06-14 13:49         ` Stefan Monnier
  2020-06-14 17:30           ` Yuri Khan
  0 siblings, 1 reply; 8+ messages in thread
From: Stefan Monnier @ 2020-06-14 13:49 UTC (permalink / raw)
  To: Yuri Khan; +Cc: Dmitry Gutov, Iñigo Serna, Emacs developers

>> This might beg for some better documentation about the string returned
>> by `json-encode`.  After all, AFAIK the JSON format requires utf-8
>> encoding, so it would make a lot of sense for `json-encode` to do the
>> utf-8 encoding directly and make sure it always returns a unibyte string.
>
> An application might not want to send the encoded string over the
> network right away. It might want to insert it into a buffer.

I didn't suggest to change the behavior, only to document it because
both outputs can make sense, so common sense is not sufficient to know
what should be returned.


        Stefan




^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: url-retrieve "Multibyte text in HTTP request" error when passing UTF-8
  2020-06-14 13:49         ` Stefan Monnier
@ 2020-06-14 17:30           ` Yuri Khan
  0 siblings, 0 replies; 8+ messages in thread
From: Yuri Khan @ 2020-06-14 17:30 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: Dmitry Gutov, Iñigo Serna, Emacs developers

On Sun, 14 Jun 2020 at 20:49, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> >> This might beg for some better documentation about the string returned
> >> by `json-encode`.  After all, AFAIK the JSON format requires utf-8
> >> encoding, so it would make a lot of sense for `json-encode` to do the
> >> utf-8 encoding directly and make sure it always returns a unibyte string.
> >
> > An application might not want to send the encoded string over the
> > network right away. It might want to insert it into a buffer.
>
> I didn't suggest to change the behavior, only to document it because
> both outputs can make sense, so common sense is not sufficient to know
> what should be returned.

Oh, I see. Yes please. More libraries should take care to document
their behavior wrt string encodings.



^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2020-06-14 17:30 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-06-13 21:08 url-retrieve "Multibyte text in HTTP request" error when passing UTF-8 Iñigo Serna
2020-06-13 22:12 ` Dmitry Gutov
2020-06-13 23:24   ` Iñigo Serna
2020-06-13 23:27     ` Dmitry Gutov
2020-06-14  0:20     ` Stefan Monnier
2020-06-14  8:33       ` Yuri Khan
2020-06-14 13:49         ` Stefan Monnier
2020-06-14 17:30           ` Yuri Khan

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.