bug#35658: 27.0.50; Problems with chunked/gzipped content in url-http

bug#35658: 27.0.50; Problems with chunked/gzipped content in url-http

[Lars Ingebrigtsen]

To reproduce, say:

(url-retrieve-synchronously "https://www.facebook.com/verkstedetbar/events/?ref=page_internal")

This now appears to spin until Facebook closes the connection on their
side.

Looking at the data, it appears that Facebook outputs chunked gzipped
data, which is pretty normal.  What's not normal is that what's in the
http buffer is

HTTP/1.1 200 OK
Content-Encoding: gzip
[...]
Date: Thu, 09 May 2019 17:45:53 GMT
Transfer-Encoding: chunked
Connection: keep-alive

3^_\213^H^@^@^@^@^@^@...

(I've transliterated the binary data into what it looks like in the
buffer.)

I.e., there's not chunking header before that gzip file start.  Somehow,
either Emacs is removing the leading

4af^M

and replacing it with "3", or Facebook is outputting something odd.  I
tend to believe it's the former, since curl (and other programs)
understand the chunked data perfectly.

It's most strange...

If we switch gzip http off by saying

(setq url-mime-encoding-string nil)

then the chunked headers are correct and everything works.



In GNU Emacs 27.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
 of 2019-05-02 built on sandy
Repository revision: d4fa998c3142b5ae13664295dcf2136397b05f5a
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description: Ubuntu 18.04.2 LTS


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35658: 27.0.50; Problems with chunked/gzipped content in url-http

[Lars Ingebrigtsen]

I've poked at this some more, and it does seem like url-http is doing
something odd to the data.  When I try the same in the with-fetched-url
branch, I get the expected:

Transfer-Encoding: chunked
Connection: keep-alive

327
^_\213^H^@^@^@...

So some thing is doing ... something to the data as it arrives from
Facebook.  url-http is working in a multibyte buffer, so I guess there
are many possible reasons for this happening...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35658: 27.0.50; Problems with chunked/gzipped content in url-http

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So some thing is doing ... something to the data as it arrives from
> Facebook.  url-http is working in a multibyte buffer, so I guess there
> are many possible reasons for this happening...

Nothing so complicated -- if I remove the contents of
url-http-chunked-encoding-after-change-function, the text is all there,
so it's doing something very wonky.

What seems to happen is that Facebook will output the first character of
the chunked header first, and then the rest afterwards.  This somehow
makes that function delete parts of the header before it's gotten the
complete header, which again means that it'll never find the real size
of the data.

And it's a function that's virtually impossible to debug in any sensible
manner, because everything happens async and the function depends on
many buffer-local variables...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#35658: 27.0.50; Problems with chunked/gzipped content in url-http

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> What seems to happen is that Facebook will output the first character of
> the chunked header first, and then the rest afterwards. 

Yup.  Pushed a fix.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no