HTTP POST requests with url.el

HTTP POST requests with url.el

[Tassilo Horn]

Hi,

I'm rewriting my emms-lastfm plugin to use url.el instead of the
unmaintained http-{get,post}.el. But I don't get the track submission
via HTTP POST working.

Here's my code:

--8<---------------cut here---------------start------------->8---
(defun emms-lastfm-submit-track ()
  "Submits the current track (`emms-lastfm-current-track') to
last.fm."
  (interactive)
  (let* ((artist (emms-track-get emms-lastfm-current-track 'info-artist))
         (title  (emms-track-get emms-lastfm-current-track 'info-title))
         (album  (emms-track-get emms-lastfm-current-track 'info-album))
         (musicbrainz-id "")
         (track-length (number-to-string
                        (emms-track-get emms-lastfm-current-track
                                        'info-playing-time)))
         (date (format-time-string "%Y-%m-%d %H:%M:%S" (current-time) t))
         (url-http-attempt-keepalives nil)
         (url-request-method "POST")
         (url-request-data (concat "u=" emms-lastfm-username
                                   "&s=" (md5 (concat
                                               (md5 emms-lastfm-password)
                                               emms-lastfm-md5-challenge))
                                   "&a[0]=" artist
                                   "&t[0]=" title
                                   "&b[0]=" album
                                   "&m[0]=" musicbrainz-id
                                   "&l[0]=" track-length
                                   "&i[0]=" date "\r\n")))
    (setq emms-lastfm-buffer
          (url-retrieve emms-lastfm-submit-url
                        'emms-lastfm-submission-sentinel))))

(defun emms-lastfm-submission-sentinel ()
  "Is called after a track submission to last.fm was made."
  (save-excursion
    (set-buffer emms-lastfm-buffer)
    (if (re-search-forward "OK" nil t 2)
        (message "\"%s\" submitted..."
                 (emms-track-description emms-lastfm-current-track))
      ;; TODO: Inform the user what went wrong.
      (message "Song couldn't be submitted"))))
--8<---------------cut here---------------end--------------->8---

When the request is done I get this in *URL-DEBUG*:

,----
| http -> Contacting host: 62.216.251.205:80
| http -> Marking connection as busy: 62.216.251.205:80 #<process 62.216.251.205>
| http -> Request is: 
| POST /protocol_1.1 HTTP/1.1
| MIME-Version: 1.0
| Connection: close
| Extension: Security/Digest Security/SSL
| Host: 62.216.251.205
| Accept-charset: utf-8;q=1, iso-8859-1;q=0.5, iso-8859-15;q=0.5, windows-1252;q=0.5, big5;q=0.5, iso-2022-jp;q=0.5, shift_jis;q=0.5, iso-8859-2;q=0.5, iso-8859-3;q=0.5, iso-8859-4;q=0.5, iso-8859-5;q=0.5, iso-8859-7;q=0.5, iso-8859-8;q=0.5, iso-8859-9;q=0.5, gb2312;q=0.5, euc-jp;q=0.5, euc-kr;q=0.5, tis-620;q=0.5, iso-8859-14;q=0.5, windows-1251;q=0.5, koi8-r;q=0.5, koi8-u;q=0.5, viscii;q=0.5, hz-gb-2312;q=0.5, iso-2022-cn-ext;q=0.5, iso-2022-cn;q=0.5, iso-2022-jp-2;q=0.5, iso-2022-kr;q=0.5, utf-16;q=0.5, utf-16be;q=0.5, utf-16le;q=0.5
| Accept: */*
| User-Agent: URL/Emacs (i686-pc-linux-gnu; X11)
| Content-length: 172
| 
| u=heimdall80&s=7728cf8409b2888f4a5077c26480798f&a[0]=Leonard Cohen&t[0]=Waiting For The Miracle (Edited)&b[0]=Natural Born Killers&m[0]=&l[0]=223&i[0]=2006-10-20 17:09:14
| 
| http -> Calling after change function `url-http-wait-for-headers-change-function' for `#<process 62.216.251.205>'
| http -> url-http-wait-for-headers-change-function ( *http 62.216.251.205:80*)
| http -> Saw end of headers... ( *http 62.216.251.205:80*)
| http -> url-http-parse-response called in ( *http 62.216.251.205:80*)
| http -> No content-length, being dumb.
| http -> url-http-end-of-document-sentinel in buffer ( *http 62.216.251.205:80*)
| http -> Marking connection as free: 62.216.251.205:80 #<process 62.216.251.205>
| http -> url-http-parse-headers called in ( *http 62.216.251.205:80*)
| http -> url-http-parse-response called in ( *http 62.216.251.205:80*)
| http -> Parsed HTTP headers: class=2 status=200
| http -> Finished parsing HTTP headers: t
| http -> Marking connection as free: 62.216.251.205:80 #<process 62.216.251.205>
| http -> Activating callback in buffer ( *http 62.216.251.205:80*)
`----

But the server response tells me there was something wrong with my post
data.

,----
| HTTP/1.1 200 OK
| Server: Apache-Coyote/1.1
| Pragma: no-cache
| Cache-Control: no-cache
| Content-Type: text/plain;charset=ISO-8859-1
| Date: Fri, 20 Oct 2006 17:09:14 GMT
| Connection: close
| 
| FAILED Plugin bug: Not all request variables are set - no POST
| parameters.  INTERVAL 1
`----

Does anyone know what I'm doing wrong, or how I can find the bug?

Bye and thanks,
Tassilo
-- 
[Emacs] is written in Lisp, which is the only computer language that is
beautiful.  -- Neal Stephenson, _In the Beginning was the Command Line_

Re: HTTP POST requests with url.el

[Stefan Monnier]

>          (url-request-data (concat "u=" emms-lastfm-username
>                                    "&s=" (md5 (concat
>                                                (md5 emms-lastfm-password)
>                                                emms-lastfm-md5-challenge))
>                                    "&a[0]=" artist
>                                    "&t[0]=" title
>                                    "&b[0]=" album
>                                    "&m[0]=" musicbrainz-id
>                                    "&l[0]=" track-length
>                                    "&i[0]=" date "\r\n")))

> Does anyone know what I'm doing wrong, or how I can find the bug?

Could it be that the data needs to be be url-encoded (SPC -> %20)?


        Stefan

Re: HTTP POST requests with url.el

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

>>          (url-request-data (concat "u=" emms-lastfm-username
>>                                    "&s=" (md5 (concat
>>                                                (md5 emms-lastfm-password)
>>                                                emms-lastfm-md5-challenge))
>>                                    "&a[0]=" artist
>>                                    "&t[0]=" title
>>                                    "&b[0]=" album
>>                                    "&m[0]=" musicbrainz-id
>>                                    "&l[0]=" track-length
>>                                    "&i[0]=" date "\r\n")))
>
>> Does anyone know what I'm doing wrong, or how I can find the bug?
>
> Could it be that the data needs to be be url-encoded (SPC -> %20)?

I stole the url-encoding function from http-get.el:

--8<---------------cut here---------------start------------->8---
(defun http-url-encode (str content-type)
  "URL encode STR using CONTENT-TYPE as the coding system."
  (apply 'concat
   (mapcar (lambda (c)
       (if (or (and (>= c ?a) (<= c ?z))
         (and (>= c ?A) (<= c ?Z))
         (and (>= c ?0) (<= c ?9)))
           (string c)
         (format "%%%02x" c)))
     (encode-coding-string str content-type))))
--8<---------------cut here---------------end--------------->8---

So I tried sending this:

u%3dheimdall80%26s%3d75e64782c62944cf3a0bf850a2c45d7c%26a%5b0%5d%3dLeonard%20Cohen%26t%5b0%5d%3dThe%20Future%20%28Edited%29%26b%5b0%5d%3dNatural%20Born%20Killers%26m%5b0%5d%3d%26l%5b0%5d%3d229%26i%5b0%5d%3d2006%2d10%2d20%2019%3a29%3a33

But it fails with the same error.

Another idea?

Bye and thanks,
Tassilo
-- 
A child of five could understand this! Fetch me a child of five!

Re: HTTP POST requests with url.el

[Tassilo Horn]

Hi,

I finally got it. The url-encoding seems to be done by `url-retrieve',
but I had to set 

  (url-request-extra-headers
          '(("Content-type" . "application/x-www-form-urlencoded")))

Now it works like a charm.

Thanks a lot,
Tassilo
-- 
[Emacs] is written in Lisp, which is the only computer language that is
beautiful.  -- Neal Stephenson, _In the Beginning was the Command Line_

Re: HTTP POST requests with url.el

[Andreas Seltenreich]

Stefan Monnier writes:

>> Does anyone know what I'm doing wrong, or how I can find the bug?
>
> Could it be that the data needs to be be url-encoded (SPC -> %20)?

Additionally, a proper "Content-type" header is missing. I.e.,

    (url-request-extra-headers
     '(("Content-type" . "application/x-www-form-urlencoded")))

regards,
andreas

Re: HTTP POST requests with url.el

[Stefan Monnier]

>>> Does anyone know what I'm doing wrong, or how I can find the bug?
>> Could it be that the data needs to be be url-encoded (SPC -> %20)?

> Additionally, a proper "Content-type" header is missing. I.e.,

>     (url-request-extra-headers
>      '(("Content-type" . "application/x-www-form-urlencoded")))

BTW, I think the URL library could use a "url-send" function or something
like that which makes it easier to send.  Or at least add something to the
URL manual to explain how to do it.
Patches welcome,


        Stefan