bug#22744: 24.5; url-retrieve callback is not invoked when http response content is empty

bug#22744: 24.5; url-retrieve callback is not invoked when http response content is empty

[Shiyao Ma]

[-- Attachment #1: Type: text/plain, Size: 3941 bytes --]

A working snippet is here:

https://bpaste.net/show/d2bd28c10f46

The callback paste-callback-bpaste is not invoked.

Turning on `url-debug', we can observe that the post response is a mere
http header with status 302, and no *data content*.


Regards.



**************************



In GNU Emacs 24.5.1 (x86_64-apple-darwin14.4.0, NS apple-appkit-1348.17)
 of 2015-08-11 on Mango
Configured using:
 `configure --prefix=/usr/local/Cellar/emacs/24.5
 --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs/24.5/share/info/emacs --without-dbus
 --without-gnutls --with-ns --disable-ns-self-contained'

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  elisp-slime-nav-mode: t
  eldoc-mode: t
  which-key-mode: t
  async-bytecomp-package-mode: t
  evil-jumper-mode: t
  evil-leader-mode: t
  evil-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  evil-local-mode: t
  override-global-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
ad-handle-definition: `evil-mode' got redefined
Source file
`/Users/Eddie/.emacs.d/lisp/bundle/dpaste_de-20131015.525/dpaste_de.el'
newer than byte-compiled file
ad-handle-definition: `find-tag-noselect' got redefined
Loading term/xterm...done
6am is refreshing

Load-path shadows:
/Users/Eddie/.emacs.d/lisp/bundle/helm-20160121.2157/helm-multi-match hides
/Users/Eddie/.emacs.d/lisp/bundle/helm-core-20160121.2157/helm-multi-match

Features:
(shadow sort my-gnus mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mail-utils
xterm elisp-slime-nav help-mode etags eldoc my-conf my-lisp my-org
finder-inf which-key my-paste paste url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util mailcap dpaste_de
web s ucs-normalize rx time-stamp dash browse-url json url-parse
auth-source gnus-util time-date mm-util mail-prsvr password-cache
url-vars my-tmux emux my-helm helm helm-source helm-multi-match helm-lib
dired helm-config helm-easymenu cl-macs gv async-bytecomp async my-evil
evil-jumper evil-leader evil evil-integration undo-tree diff evil-maps
evil-commands evil-command-window evil-types evil-search evil-ex
evil-macros evil-repeat evil-states evil-core advice help-fns
evil-common windmove thingatpt rect evil-digraphs evil-vars ring
my-utils my-graphics use-package diminish bind-key easy-mmode info
easymenu eieio byte-opt bytecomp byte-compile cl-extra cconv eieio-core
package epg-config edmacro kmacro cl-loaddefs cl-lib tooltip electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel ns-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
cocoa ns multi-tty emacs)

Memory information:
((conses 16 314308 152941)
 (symbols 48 29606 97)
 (miscs 40 32 184)
 (strings 32 47915 121314)
 (string-bytes 1 1321209)
 (vectors 16 42457)
 (vector-slots 8 705376 93653)
 (floats 8 109 567)
 (intervals 56 217 74)
 (buffers 960 12))

[-- Attachment #2: Type: text/html, Size: 4428 bytes --]

bug#22744: 24.5; url-retrieve callback is not invoked when http response content is empty

[Lars Ingebrigtsen]

Shiyao Ma <i@introo.me> writes:

> A working snippet is here:
>
> https://bpaste.net/show/d2bd28c10f46
>
> The callback paste-callback-bpaste is not invoked.
>
> Turning on `url-debug', we can observe that the post response is a mere
> http header with status 302, and no *data content*.

A 302 is a redirection, and URL follows redirections, and do not call
the callback before we get to the "real" page (or it fails).

Is there nothing else returned from the server?  Like a Location header?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22744: 24.5; url-retrieve callback is not invoked when http response content is empty

[Shiyao Ma]

[-- Attachment #1: Type: text/plain, Size: 260 bytes --]

To add more detail,

The url-debug output:
https://bpaste.net/show/2945ab6727be


A sample post response:
https://bpaste.net/show/25777c5fde69
No response content, the result is merely the http headers.


Also, url-retrieve doesn't follow the 302 redirection.

[-- Attachment #2: Type: text/html, Size: 469 bytes --]

bug#22744: 24.5; url-retrieve callback is not invoked when http response content is empty

[Lars Ingebrigtsen]

Shiyao Ma <i@introo.me> writes:

> A sample post response:
> https://bpaste.net/show/25777c5fde69
> No response content, the result is merely the http headers.

So you do a POST, and then you get a 302 redirect (which results in a
GET)?  I think I remember somebody working on this issue a few weeks
ago...

Oh, yeah:

commit 46dfdd831b817ef9e281350043bd4231f2dc5acc
Author: Nicolas Petton <nicolas@petton.fr>
Date:   Thu Feb 4 21:43:42 2016 +0100

    Do not ignore redirections of 301, 302 and 307 status codes
    
    The current version of HTTP/1.1 (RFC 7231) no longer requires
    confirmation on 301, 302 or 307 status codes, therefore we do not have
    to ignore redirects for other requests than GET and HEAD.
    
    * lisp/url/url-http.el (url-http-parse-headers): Do not ignore 301, 302
    and 307 redirects for other requests than GET and HEAD.

So this probably works in emacs-25 now.  Could you download (from git)
and test?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22744: 24.5; url-retrieve callback is not invoked when http response content is empty

[Shiyao Ma]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

I installed from the git head.


For the callback status, it's:
(:error (error http 405) :redirect https://bpaste.net/show/9f7b83e91ebc)


For the callback buffer, it's:
HTTP/1.1 405 METHOD NOT ALLOWED
Server: nginx
Date: Sun, 21 Feb 2016 10:45:21 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Allow: HEAD, OPTIONS, GET
X-Cache-Hits: 0
X-Cache-Age: 0
X-Cache-Status: MISS

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>



For the url-debug output, it's:
https://bpaste.net/show/f5b3e99df8fd




*****

So from the url-debug output, the logic is doing wrong. On Line#30, it's
doing a *redirect* with post, resulting an HTTP 405.



BTW, possible to do nothing other than firing up the callback when
receiving the HTTP 302  on Line#32 ?


Regards.

[-- Attachment #2: Type: text/html, Size: 1289 bytes --]

bug#22744: 24.5; url-retrieve callback is not invoked when http response content is empty

[Lars Ingebrigtsen]

Shiyao Ma <i@introo.me> writes:

> I installed from the git head.
>
> For the callback status, it's:
> (:error (error http 405) :redirect https://bpaste.net/show/9f7b83e91ebc)
>
> For the callback buffer, it's:
> HTTP/1.1 405 METHOD NOT ALLOWED
> Server: nginx

[...]

> So from the url-debug output, the logic is doing wrong. On Line#30, it's doing
> a *redirect* with post, resulting an HTTP 405.

So that web site wants the redirect to be done with GET instead of POST?
I don't know what the standard says should happen here...

Nicolas, could you look into this?  The issue is that when POST-ing to
bpaste.net, it 302 redirects...

> BTW, possible to do nothing other than firing up the callback when receiving
> the HTTP 302 on Line#32 ?

No, the callbacks are fired after landing on the final URL after all the
redirects.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22744: 24.5; url-retrieve callback is not invoked when http response content is empty

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> So from the url-debug output, the logic is doing wrong. On Line#30, it's doing
>> a *redirect* with post, resulting an HTTP 405.
>
> So that web site wants the redirect to be done with GET instead of POST?
> I don't know what the standard says should happen here...

This was apparently fixed in 2018 like this:

           ('found			; 302
	    ;; 302 Found was ambiguously defined in the standards, but
	    ;; it's now recommended that it's treated like 303 instead
	    ;; of 307, since that's what most servers expect.
	    (setq url-http-method "GET"
		  url-http-data nil))

To I think it's fixed, and I'm closing this bug report.  If you're
seeing a problem here, please reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no