help with URL module needed

help with URL module needed

[Paul Pogonyshev]

I'm trying to login to Wikipedia from within Emacs using the URL
module (to be able to edit and save articles in Emacs.)  However,
I'm having problems with sending POST requests (to transmit login
information.)  Here is what I get:

[in *URL-DEBUG* buffer]:

http -> Contacting host: [[my proxy]]
http -> Marking connection as busy: [[my proxy]] #<process ...>
http -> Request is: 
POST http://ru.wikipedia.org/w/wiki.phtml?title=%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5:Userlogin&action=submit HTTP/1.1 
MIME-Version: 1.0 
Connection: close 
Extension: Security/Digest Security/SSL 
Host: ru.wikipedia.org 
Accept-charset: utf-8;q=1, iso-8859-1;q=0.5, big5;q=0.5, iso-2022-jp;q=0.5, shift_jis;q=0.5, iso-8859-15;q=0.5, iso-8859-2;q=0.5, iso-8859-3;q=0.5, iso-8859-4;q=0.5, iso-8859-5;q=0.5, iso-8859-7;q=0.5, iso-8859-8;q=0.5, iso-8859-9;q=0.5, gb2312;q=0.5, euc-jp;q=0.5, euc-kr;q=0.5, tis-620;q=0.5, iso-8859-14;q=0.5, windows-1252;q=0.5, windows-1251;q=0.5, koi8-r;q=0.5, koi8-u;q=0.5, viscii;q=0.5, hz-gb-2312;q=0.5, iso-2022-cn-ext;q=0.5, iso-2022-cn;q=0.5, iso-2022-jp-2;q=0.5, iso-2022-kr;q=0.5, utf-16;q=0.5, utf-16be;q=0.5, utf-16le;q=0.5 
Accept: */* 
User-Agent: URL/Emacs (i686-pc-linux-gnu; X11) 
Content-Type: application/x-www-form-urlencoded 
Content-length: 84 
 
wpName=Paul+Pogonyshev&wpLoginattempt=Login&wpPassword=xxxxxxxxxx&wpRetype=&wpEmail= 


The response from proxy is (HTML is stripped):

HTTP/1.0 411 Length Required
Server: squid/2.5.STABLE2
Mime-Version: 1.0
Date: Fri, 26 Nov 2004 15:20:13 GMT
Content-Type: text/html
Content-Length: 2107
Expires: Fri, 26 Nov 2004 15:20:13 GMT
X-Squid-Error: ERR_INVALID_REQ 0
X-Cache: MISS from [[my proxy]]
Proxy-Connection: close

While trying to process the request:

[see above]

The following error was encountered:

Invalid Request

Some aspect of the HTTP Request is invalid.  Possible problems:

* Missing or unknown request method
* Missing URL
* Missing HTTP Identifier (HTTP/1.0)
* Request is too large
* Content-Length missing for POST or PUT requests
* Illegal character in hostname; underscores are not allowed


The URL etc. are all correct.  If I don't send any POST data, the request
is passed to ru.wikipedia.org just fine.  The problem is only with POST
data, but I don't understand where it is.

Can anyone help here?

Paul

Re: help with URL module needed

[Masatake YAMATO]

> I'm trying to login to Wikipedia from within Emacs using the URL
> module (to be able to edit and save articles in Emacs.)  However,
> I'm having problems with sending POST requests (to transmit login
> information.)  Here is what I get:
> 
> [in *URL-DEBUG* buffer]:

Which version of emacs do you use?
If you use the source code in the CVS repository, could you tell me
the date?
How can I reproduce the error?

Re: help with URL module needed

[Paul Pogonyshev]

> > I'm trying to login to Wikipedia from within Emacs using the URL
> > module (to be able to edit and save articles in Emacs.)  However,
> > I'm having problems with sending POST requests (to transmit login
> > information.)  Here is what I get:
> >
> > [in *URL-DEBUG* buffer]:
>
> Which version of emacs do you use?
> If you use the source code in the CVS repository, could you tell me
> the date?

CVS of November 14.

> How can I reproduce the error?

I made more experiments and narrowed down conditions that lead to error.
You need an account in the English Wikipedia domain (en.wikipedia.org).

First enter valid "wpName" and "wpPassword" for your account in the code
below and try to login.  Here it fails with "411 Length Required" HTTP
code.  Now replace your password with _any_ invalid password (i.e. change
one single symbol.) and try to login again.  Here, it succeeds with
"200 OK" (though the login itself fails, of course.)

So, to summarize: if you try to login with valid name/password, HTTP
request fails for a reason I don't understand.  If you try to login with
invalid password, HTTP request succeeds, but login fails (as it should.)
This looks so weird to me that I'm not sure you can reproduce it.

Does this all make any sense?

Paul

P.S. Thi, I tried to remove "\r\n" (in `url-http.el'), but it didn't
     make any difference.


(require 'url)

(defun wikipedia-login ()
  (interactive)
  (let ((domain                    "en")
	(url-request-method        "POST")
	(url-request-extra-headers '(("Content-Type" . "application/x-www-form-urlencoded")))
	(url-request-data
	 (wikipedia-build-post-data '(("wpName"         . "")
				      ("wpLoginattempt" . "Login")
				      ("wpPassword"     . "")
				      ("wpRetype"       . "")
				      ("wpEmail"        . ""))))
	(url-debug	     t))
    (url-retrieve (concat "http://" domain ".wikipedia.org/w/wiki.phtml?title=Special:Userlogin&action=submit")
		  (lambda () (pop-to-buffer (current-buffer))))))

(defun wikipedia-build-post-data (post-data-alist)
  (mapconcat (lambda (association)
	       (concat (car association) "="
		       (url-hexify-string (cdr association))))
	     post-data-alist "&"))

Re: help with URL module needed

[Paul Pogonyshev]

I wrote:
> > > I'm trying to login to Wikipedia from within Emacs using the URL
> > > module (to be able to edit and save articles in Emacs.)  However,
> > > I'm having problems with sending POST requests (to transmit login
> > > information.)  Here is what I get:
> > >
> > > [in *URL-DEBUG* buffer]:
> >
> > Which version of emacs do you use?
> > If you use the source code in the CVS repository, could you tell me
> > the date?
>
> CVS of November 14.
>
> > How can I reproduce the error?
>
> I made more experiments and narrowed down conditions that lead to error.
> You need an account in the English Wikipedia domain (en.wikipedia.org).

I'll assume the lack of response is due to nobody's having an account at
Wikipedia ;)

So, I'll ease it.  Evaluate these two functions and type
`M-x wikipedia-login RET':

(defun wikipedia-login ()
  (interactive)
  (let ((domain                    "en")
	(url-request-method        "POST")
	(url-request-extra-headers '(("Content-Type" . "application/x-www-form-urlencoded")))
	(url-request-data
	 (wikipedia-build-post-data '(("wpName"         . "Testy Test")
				      ("wpLoginattempt" . "Login")
				      ("wpPassword"     . "test")
				      ("wpRetype"       . "")
				      ("wpEmail"        . ""))))
	(url-debug	     t))
    (url-retrieve (concat "http://" domain ".wikipedia.org/w/wiki.phtml?title=Special:Userlogin&action=submit")
		  (lambda () (pop-to-buffer (current-buffer))))))

(defun wikipedia-build-post-data (post-data-alist)
  (mapconcat (lambda (association)
	       (concat (car association) "="
		       (url-hexify-string (cdr association))))
	     post-data-alist "&"))

You should get a ``411 Length Required''.

Now go back and change the (valid) password to anything else, evaluate
the new function and type `M-x wikipedia-login RET' again.  You should
get ``200 OK''.

Now, I agree that this doesn't really look like an URL module problem.
But can anyone confirm this weird behaviour and explain me (or at least
guess) what's going wrong?

Paul

Re: help with URL module needed

[Andreas Schwab]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

> So, I'll ease it.  Evaluate these two functions and type
> `M-x wikipedia-login RET':
>
> (defun wikipedia-login ()
>   (interactive)
>   (let ((domain                    "en")
> 	(url-request-method        "POST")
> 	(url-request-extra-headers '(("Content-Type" . "application/x-www-form-urlencoded")))
> 	(url-request-data
> 	 (wikipedia-build-post-data '(("wpName"         . "Testy Test")
> 				      ("wpLoginattempt" . "Login")
> 				      ("wpPassword"     . "test")
> 				      ("wpRetype"       . "")
> 				      ("wpEmail"        . ""))))
> 	(url-debug	     t))
>     (url-retrieve (concat "http://" domain ".wikipedia.org/w/wiki.phtml?title=Special:Userlogin&action=submit")
> 		  (lambda () (pop-to-buffer (current-buffer))))))
>
> (defun wikipedia-build-post-data (post-data-alist)
>   (mapconcat (lambda (association)
> 	       (concat (car association) "="
> 		       (url-hexify-string (cdr association))))
> 	     post-data-alist "&"))
>
> You should get a ``411 Length Required''.

The problem is that url-http-create-request does not account for the
trailing CRLF in the Content-Length, or rather that it sends the trailing
CRLF in the first place.

I've checked in this patch to fix it:

2004-11-28  Andreas Schwab  <schwab@suse.de>

	* url-http.el (url-http-create-request): Don't add newline after
	the request data.

--- lisp/url/url-http.el	19 Apr 2004 09:48:36 +0200	1.5
+++ lisp/url/url-http.el	28 Nov 2004 20:06:35 +0100	
@@ -255,15 +255,16 @@ request.
 	   (if ref-url (concat
 			"Referer: " ref-url "\r\n"))
 	   extra-headers
-	   ;; Any data
+	   ;; Length of data
 	   (if url-request-data
 	       (concat
 		"Content-length: " (number-to-string
 				    (length url-request-data))
-		"\r\n\r\n"
-		url-request-data))
+		"\r\n"))
 	   ;; End request
-	   "\r\n"))
+	   "\r\n"
+	   ;; Any data
+	   url-request-data))
     (url-http-debug "Request is: \n%s" request)
     request))
 

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: help with URL module needed

[Paul Pogonyshev]

Andreas Schwab wrote:
> Paul Pogonyshev <pogonyshev@gmx.net> writes:
> > So, I'll ease it.  Evaluate these two functions and type
> > `M-x wikipedia-login RET':
> >
> > (defun wikipedia-login ()
> >   (interactive)
> >   (let ((domain                    "en")
> > 	(url-request-method        "POST")
> > 	(url-request-extra-headers '(("Content-Type" .
> > "application/x-www-form-urlencoded"))) (url-request-data
> > 	 (wikipedia-build-post-data '(("wpName"         . "Testy Test")
> > 				      ("wpLoginattempt" . "Login")
> > 				      ("wpPassword"     . "test")
> > 				      ("wpRetype"       . "")
> > 				      ("wpEmail"        . ""))))
> > 	(url-debug	     t))
> >     (url-retrieve (concat "http://" domain
> > ".wikipedia.org/w/wiki.phtml?title=Special:Userlogin&action=submit")
> > (lambda () (pop-to-buffer (current-buffer))))))
> >
> > (defun wikipedia-build-post-data (post-data-alist)
> >   (mapconcat (lambda (association)
> > 	       (concat (car association) "="
> > 		       (url-hexify-string (cdr association))))
> > 	     post-data-alist "&"))
> >
> > You should get a ``411 Length Required''.
>
> The problem is that url-http-create-request does not account for the
> trailing CRLF in the Content-Length, or rather that it sends the trailing
> CRLF in the first place.

Unfortunately, this doesn't help here (I recompiled Emacs and verified that
"\r\n" is not appended after POST data anymore.)  I still get ``200 OK'' with
invalid password and ``411 Length Required'' with the valid one.

Do you mean that you cannot reproduce this?

Paul

Re: help with URL module needed

[Andreas Schwab]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

> Unfortunately, this doesn't help here (I recompiled Emacs and verified that
> "\r\n" is not appended after POST data anymore.)  I still get ``200 OK'' with
> invalid password and ``411 Length Required'' with the valid one.

I have no idea why it does not work for you.  It does fix the bug for me,
using your testcase.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: help with URL module needed

[Paul Pogonyshev]

Andreas Schwab wrote:
> Paul Pogonyshev <pogonyshev@gmx.net> writes:
> > Unfortunately, this doesn't help here (I recompiled Emacs and verified
> > that "\r\n" is not appended after POST data anymore.)  I still get ``200
> > OK'' with invalid password and ``411 Length Required'' with the valid
> > one.
>
> I have no idea why it does not work for you.  It does fix the bug for me,
> using your testcase.

That's bad :(

Can you maybe help me indirectly?  Do you know how I could catch a request
that is sent by one of the browsers?  (I have Konqueror, Mozilla and Lynx
here, but all quite old.)  Maybe there is a generic way to log everything
sent by HTTP?

Then I would be able to compare working and non-working requests...

Paul

Re: help with URL module needed

[Alex Schroeder]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

> Can you maybe help me indirectly?  Do you know how I could catch a request
> that is sent by one of the browsers?  (I have Konqueror, Mozilla and Lynx
> here, but all quite old.)  Maybe there is a generic way to log everything
> sent by HTTP?

I use firefox and install the Live HTTP Headers extension for this
kind of debugging.

http://livehttpheaders.mozdev.org/

Alex.
-- 
.O.  http://www.emacswiki.org/alex/
..O  Schroeder's fifth law:
OOO  Never accept more work than you can handle in one night of hacking.

Re: help with URL module needed

[Andreas Schwab]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

> Can you maybe help me indirectly?  Do you know how I could catch a request
> that is sent by one of the browsers?  (I have Konqueror, Mozilla and Lynx
> here, but all quite old.)  Maybe there is a generic way to log everything
> sent by HTTP?

For example tcpdump, ethereal, or a transparent proxy.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: help with URL module needed

[Paul Pogonyshev]

Hmm...

I think I found a bug in URL module that _might_ be related to the weird problem
I'm experiencing.

My `url-cookie-storage' variable has cookies listed for my *proxy domain*, not
(ru|en).wikipedia.org!  As the result, the cookies are not sent, as URL module
seems to correctly compare domains (i.e. it doesn't compare with the proxy.)

Can you please look at this?

Andreas Schwab wrote:
> Paul Pogonyshev <pogonyshev@gmx.net> writes:
> > Can you maybe help me indirectly?  Do you know how I could catch a
> > request that is sent by one of the browsers?  (I have Konqueror, Mozilla
> > and Lynx here, but all quite old.)  Maybe there is a generic way to log
> > everything sent by HTTP?
>
> For example tcpdump, ethereal, or a transparent proxy.

Thanks.

In case anyone is interested, this way of dumping requests seems particularly
easy (for localhost only): use tcputils (ftp://ftp.lysator.liu.se/pub/unix/tcputils/)
and do

	tcplisten 80

Then all you need is to hack the <form action="..." ...> a little to make it
send the request to 127.0.0.1.

Paul

Re: help with URL module needed

[Paul Pogonyshev]

I wrote:
> Hmm...
>
> I think I found a bug in URL module that _might_ be related to the weird
> problem I'm experiencing.
>
> My `url-cookie-storage' variable has cookies listed for my *proxy domain*,
> not (ru|en).wikipedia.org!  As the result, the cookies are not sent, as URL
> module seems to correctly compare domains (i.e. it doesn't compare with the
> proxy.)

So, do you care to apply this patch that fixes the _bug_?

2004-11-30  Paul Pogonyshev  <pogonyshev@gmx.net>

	* url-http.el (url-http-handle-cookies): Bind `url-current-object'
	to `url-http-cookies-sources' so that cookies are attributed to
	proper domain.
	(url-http): Make local variable `url-http-cookies-sources'.


--- url-http.el	28 Nov 2004 23:58:24 +0200	1.6
+++ url-http.el	30 Nov 2004 21:44:25 +0200	
@@ -342,7 +342,8 @@ This allows us to use `mail-fetch-field'
 The buffer must already be narrowed to the headers, so mail-fetch-field will
 work correctly."
   (let ((cookies (mail-fetch-field "Set-Cookie" nil nil t))
-	(cookies2 (mail-fetch-field "Set-Cookie2" nil nil t)))
+	(cookies2 (mail-fetch-field "Set-Cookie2" nil nil t))
+	(url-current-object url-http-cookies-sources))
     (and cookies (url-http-debug "Found %d Set-Cookie headers" (length cookies)))
     (and cookies2 (url-http-debug "Found %d Set-Cookie2 headers" (length cookies2)))
     (while cookies
@@ -1043,7 +1044,8 @@ CBARGS as the arguments."
 		       url-http-process
 		       url-http-method
 		       url-http-extra-headers
-		       url-http-data))
+		       url-http-data
+		       url-http-cookies-sources))
 	  (set (make-local-variable var) nil))
 
 	(setq url-http-method (or url-request-method "GET")
@@ -1055,7 +1057,10 @@ CBARGS as the arguments."
 	      url-http-chunked-counter 0
 	      url-callback-function callback
 	      url-callback-arguments cbargs
-	      url-http-after-change-function 'url-http-wait-for-headers-change-function)
+	      url-http-after-change-function 'url-http-wait-for-headers-change-function
+	      url-http-cookies-sources (if (boundp 'proxy-object)
+					   proxy-object
+					 url-current-object))
 
 	(set-process-buffer connection buffer)
 	(set-process-sentinel connection 'url-http-end-of-document-sentinel)

Re: help with URL module needed

[Andreas Schwab]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

> 	* url-http.el (url-http-handle-cookies): Bind `url-current-object'
> 	to `url-http-cookies-sources' so that cookies are attributed to
> 	proper domain.
> 	(url-http): Make local variable `url-http-cookies-sources'.

Thanks, I've installed that.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: help with URL module needed

[Thien-Thi Nguyen]

   From: Andreas Schwab <schwab@suse.de>
   Date: Sun, 28 Nov 2004 23:59:38 +0100

   Paul Pogonyshev <pogonyshev@gmx.net> writes:

   > Unfortunately, this doesn't help here (I recompiled Emacs and
   > verified that "\r\n" is not appended after POST data anymore.)  I
   > still get ``200 OK'' with invalid password and ``411 Length
   > Required'' with the valid one.

   I have no idea why it does not work for you.  It does fix the bug for
   me, using your testcase.

pardon me if this is now resolved; my isp (where i read emacs-devel)
just switched things around and i suspect i've lost some mail...

i think you have to omit CRLF only for POST data w/ content-type
application/x-www-form-urlencoded -- in Guile-WWW[1], for example, there
is special-case handling for this, while other POST data is handled more
or less "normally".

ok, that's the extent of my understanding of this wrinkle.

thi


[1] http://www.glug.org/people/ttn/software/guile-www/
    (see file "http.scm")

Re: help with URL module needed

[Andreas Schwab]

Thien-Thi Nguyen <ttn@agora-net.com> writes:

> i think you have to omit CRLF only for POST data w/ content-type
> application/x-www-form-urlencoded -- in Guile-WWW[1], for example, there
> is special-case handling for this, while other POST data is handled more
> or less "normally".

Should Content-Length include the trainling CRLF in the other cases?

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: help with URL module needed

[Thien-Thi Nguyen]

   From: Andreas Schwab <schwab@suse.de>
   Date: Tue, 30 Nov 2004 01:35:52 +0100

   Should Content-Length include the trainling CRLF in the other cases?

yes, i believe so (from trial and error in the wild, away from the http
specs).  to sum up, if these conditions are met:

 - content-type is application/x-www-form-urlencoded
 - POST data is exactly one line (this is usually the case for this
   content-type since urlencoding "flattens" embedded newlines)

then the final CRLF should be omitted, and the content-length set to
the length of the line in bytes after urlencoding.  i've found this to
work w/ both HTTP 1.0 and 1.1 servers, and have since become uncurious
about the rationale.  now i'm curious again, but still not enough to dig
into the specs.

thi

Re: help with URL module needed

[Mark A. Hershberger]

[-- Attachment #1.1: Type: text/plain, Size: 1554 bytes --]


Since (length url-request-data) returns characters in the string
instead of bytes, shouldn't it be (string-bytes url-request-data)
instead?

Additionally, what happens if you post using Unicode?

Does the following patch help any?

--- url-http.el	28 Nov 2004 19:13:50 -0000	1.6
+++ url-http.el	30 Nov 2004 17:16:09 -0000
@@ -259,7 +259,7 @@
 	   (if url-request-data
 	       (concat
 		"Content-length: " (number-to-string
-				    (length url-request-data))
+				    (string-bytes url-request-data))
 		"\r\n"))
 	   ;; End request
 	   "\r\n"
@@ -1060,6 +1060,9 @@
 	(set-process-buffer connection buffer)
 	(set-process-sentinel connection 'url-http-end-of-document-sentinel)
 	(set-process-filter connection 'url-http-generic-filter)
+	(set-process-coding-system connection
+				   (detect-coding-string url-request-data t)
+				   url-request-coding-system)
 	(process-send-string connection (url-http-create-request url))))
     buffer))
 
--- url-vars.el	19 Oct 2004 21:37:18 -0000	1.8
+++ url-vars.el	30 Nov 2004 17:16:09 -0000
@@ -218,6 +218,8 @@
 
 (defvar url-request-data nil "Any data to send with the next request.")
 
+(defvar url-request-coding-system 'binary "The coding system to use for the request.")
+
 (defvar url-request-extra-headers nil
   "A list of extra headers to send with the next request.
 Should be an assoc list of headers/contents.")
 
 

-- 
A choice between one man and a shovel, or a dozen men with teaspoons
is clear to me, and I'm sure it is clear to you also.
    -- Zimran Ahmed <http://www.winterspeak.com/>

[-- Attachment #1.2: Type: application/pgp-signature, Size: 188 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: help with URL module needed

[Thien-Thi Nguyen]

Paul Pogonyshev <pogonyshev@gmx.net> writes:

   The URL etc. are all correct.  If I don't send any POST data, the
   request is passed to ru.wikipedia.org just fine.  The problem is only
   with POST data, but I don't understand where it is.

   Can anyone help here?

i see the post data includes a trailing CRLF.  some servers seem to get
confused given application/x-www-form-urlencoded and post data ending w/
CRLF.  can you arrange to not include a CRLF?  is the content-length
(sans CRLF) correct?

thi