bug#22921: 25.1.50; Wishlist: There should be a new URL interface

bug#22921: 25.1.50; Wishlist: There should be a new URL interface

[Lars Magne Ingebrigtsen]

The current interfaces to the URL library aren't very handy.  It's
`url-retrieve' and `url-retrieve-synchronously'.  The main problem is
that they're inconsistent and difficult to interact with.

For instance, most of the "inputs" to the functions are dynamically
bound specially variables like `url-request-method', and that sucks.
The buffers that are returned are inconsistent based on the URL
method -- HTTP buffers come with the headers first, while ftp buffers do
not.

So: `with-url-buffer'.

Typical usage would be

(with-url-buffer ("https://fsf.org"
                  :verbose t
                  :request-method "POST"
                  :timeout 10)
  (do-stuff-with-the-result-here))

That is, the callback is in the form.  There are no headers in the
buffer, and point is at point-min when the body is called.  Headers are
accessed with a new form like `(url-header "date")', and the headers are
available in a buffer-local variable called `url-headers'.

Upon completion of the body form, the buffer is killed, unless a
parameter :inhibit-kill-buffer is given.

Synchronous and asynchronous operation looks identical, but synchronous
operation happens when you say :synchronous t.

Implementation-wise, this happens with lexical bindings switched on, and
the body form is turned into a closure that's evaluated upon completion.

`with-url-buffer' will not support the exotic, never-actually-used
protocols like news:, irc:, imap:, etc, but only http:, https:, ftp:
and file:.



In GNU Emacs 25.1.50.23 (x86_64-unknown-linux-gnu, GTK+ Version 3.4.2)
 of 2016-03-06 built on stories
Repository revision: 20cc4182598c6c724f4556faf6bc322c9e58ad1d
Windowing system distributor 'The X.Org Foundation', version 11.0.11204000
System Description:	Debian GNU/Linux 7.9 (wheezy)

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GSETTINGS NOTIFY GNUTLS
LIBXML2 FREETYPE LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LANG: en_US
  locale-coding-system: iso-latin-1-unix

Major mode: Group

Minor modes in effect:
  gnus-topic-mode: t
  gnus-undo-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t

Recent messages:
Wrote /home/larsi/.newsrc
Saving /home/larsi/.newsrc.eld...
Saving file /home/larsi/.newsrc.eld...
Wrote /home/larsi/.newsrc.eld
Saving /home/larsi/.newsrc.eld...done
No more unseen articles
No more unread articles
next-line: End of buffer [2 times]
(No changes need to be saved)
Making completion list...

Load-path shadows:
/home/larsi/src/clock.el/clock hides /home/larsi/lisp/clock
/home/larsi/src/cddb.el/expect hides /home/larsi/lisp/expect
/home/larsi/src/pvr.el/pvr hides /home/larsi/lisp/pvr
~/pgnus/contrib/vcard hides /home/larsi/lisp/vcard
/home/larsi/src/cddb.el/captitle hides /home/larsi/lisp/captitle
~/lisp/zenirc-2.112/src/zenirc-example hides /home/larsi/lisp/zenirc-example
/home/larsi/lisp/dom hides /home/larsi/src/emacs/trunk/lisp/dom
~/pgnus/contrib/compface hides /home/larsi/src/emacs/trunk/lisp/image/compface

Features:
(shadow emacsbug sendmail binhex canlock ecomplete flow-fill eieio-opt
speedbar sb-image ezimage dframe find-func thingatpt view sgml-mode
misearch multi-isearch tabify imenu man vc vc-dispatcher copyright
vc-cvs eww shr-color color gnus-html url-queue help-fns url-cache
gnus-picon sort gnus-cite smiley ansi-color mm-archive gnus-async
gnus-dup qp gnus-ml gmane spam-gmane dns mm-url disp-table gnus-fun
gnus-mdrtn gnus-topic pop3 nndoc nnmbox nndraft utf-7 nnmh nnml nnfolder
network-stream nsm starttls nnir spam-report spam spam-stat gnus-uu yenc
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art
mm-uu mml2015 mm-view mml-smime smime dig nntp gnus-cache gnus-sum
gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source
utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range message
format-spec rfc822 mml mml-sec epa epg mailabbrev gmm-utils mailheader
gnus-win gnus nnheader gnus-util rmail rmail-loaddefs mail-utils movie
mkv shr svg imdb dom pvr debug debbugs-gnu easy-mmode derived debbugs
soap-client mm-decode mm-bodies mm-encode url-http tls gnutls url-auth
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums url-gw puny url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util mailcap warnings rng-xsd rng-dt rng-util xsd-regexp xml ido seq
flyspell ispell benchmark w3m browse-url doc-view subr-x dired
dired-loaddefs image-mode timezone w3m-hist w3m-fb w3m-ems wid-edit
w3m-ccl ccl w3m-favicon w3m-image w3m-proc w3m-util add-log mail-extr
mm-util mail-prsvr jka-compr cl finder-inf package epg-config
url-handlers url-parse auth-source cl-seq eieio byte-opt bytecomp
byte-compile cl-extra help-mode easymenu cconv eieio-core cl-macs gv
eieio-loaddefs cl-loaddefs pcase cl-lib password-cache url-vars
time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev
obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote inotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 4019121 362371)
 (symbols 48 170790 312)
 (miscs 40 674 1974)
 (strings 32 317931 33454)
 (string-bytes 1 13992070)
 (vectors 16 43895)
 (vector-slots 8 1764946 243016)
 (floats 8 7065 3818)
 (intervals 56 929585 49118)
 (buffers 976 59)
 (heap 1024 212624 72649))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22921: 25.1.50; Wishlist: There should be a new URL interface

[Stefan Kangas]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So: `with-url-buffer'.

What's the current status of your with-fetched-url branch?  Is it
ready to merge into master?  It would be nice to get it in so that we
can start playing around with it.

Best regards,
Stefan Kangas

bug#22921: 25.1.50; Wishlist: There should be a new URL interface

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> So: `with-url-buffer'.
>
> What's the current status of your with-fetched-url branch?  Is it
> ready to merge into master?  It would be nice to get it in so that we
> can start playing around with it.

It should be (more or less) ready to go.  I've been meaning to merge it
ever since Emacs 28 started, but I haven't found time yet.

If I remember, there may be some quirks with certain auth schemes that
needs fixing, but that can be done after the merge.  But I need to block
out, say, a week where I can respond to bug reports in a speedy manner,
and I haven't found that week yet...

But it'll happen sometime within a month or to.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37577: 27.0.50; GNU Emacs / eww inconsistent in showing images

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> What's the current status of your with-fetched-url branch?  Is it
>> ready to merge into master?  It would be nice to get it in so that we
>> can start playing around with it.
>
> It should be (more or less) ready to go.  I've been meaning to merge it
> ever since Emacs 28 started, but I haven't found time yet.

Hmm, maybe we can have it in Emacs 29?  :-)

bug#22921: bug#37577: 27.0.50; GNU Emacs / eww inconsistent in showing images

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>>> What's the current status of your with-fetched-url branch?  Is it
>>> ready to merge into master?  It would be nice to get it in so that we
>>> can start playing around with it.
>>
>> It should be (more or less) ready to go.  I've been meaning to merge it
>> ever since Emacs 28 started, but I haven't found time yet.
>
> Hmm, maybe we can have it in Emacs 29?  :-)

:-)

It got put on the back burner because I started thinking about the proxy
stuff that url.el supports but I hadn't implemented for with-url yet.

Because I think that's the wrong level to do it.  Emacs should support a
general network proxy thing, that would allow all network connections to
be proxied.  (Or selectively.)

And the more I looked at this, the more apparently it became that Emacs'
network stuff should just be totally reimplemented, because it's so full
of...  oddnesses geared toward processes.  And processes and network
connections are very different things.

So it's a Big Project.  :-/

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37577: 27.0.50; GNU Emacs / eww inconsistent in showing images

[Unknown]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]


GNU Emacs seems to be inconsistent when showing images fetched via
http(s). Here is an example:

M-x eww

Paste in a URL of a webpage with many pictures, eg
https://moving-pictures.ingebrigtsen.no/2019/10/01/songs-from-the-second-floor-2/

Expected result is all images (eventually) load. Instead I see gray
boxes where the images should appear. This even after waiting 10
minutes. In Firefox I can get all images in seconds.

Here is what my Emacs window looks like:


[-- Attachment #2: emacs.png --]
[-- Type: image/png, Size: 63157 bytes --]

[-- Attachment #3: Type: text/plain, Size: 3881 bytes --]


It seems that the fewer images there are, the more likely they are to
appear. But it is very inconsistent, in my experience.

I don't expect Emacs to be optimized to fetch a lot of images
efficiently in the way, say, a dedicated web browser might be, but not
showing all images eventually, is suboptimimal.


In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.11)
 of 2019-09-21 built on tullinup
Repository revision: fea4a73300aa1da0a80548081874ced0c8df00f3
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12000000
System Description: Debian GNU/Linux bullseye/sid

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Contacting host: moving-pictures.ingebrigtsen.no:443
Quit
Configured using:
 'configure --without-pop'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY INOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM THREADS LIBSYSTEMD PDUMPER LCMS2
GMP

Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: eww

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort hashcash mail-extr emacsbug message dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config mm-decode mm-bodies
mm-encode mailabbrev gmm-utils mailheader sendmail cl-extra help-mode
gnutls network-stream url-http mail-parse rfc2231 url-gw nsm rmc
url-cache url-auth eww easymenu mm-url gnus nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums time-date mail-utils wid-edit
mm-util mail-prsvr thingatpt url-queue url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars mailcap shr text-property-search
puny svg xml seq byte-opt gv bytecomp byte-compile cconv dom browse-url
format-spec cl-loaddefs cl-lib tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 114744 5561)
 (symbols 48 9806 1)
 (strings 32 41653 2181)
 (string-bytes 1 3091594)
 (vectors 16 15376)
 (vector-slots 8 195918 11516)
 (floats 8 81 719)
 (intervals 56 1009 0)
 (buffers 992 13))

-- 
 "And all you can talk about is money and fairytales          Adam Sjøgren
  of eternal economic growth. How dare you!"             asjo@koldfront.dk

bug#37577: 27.0.50; GNU Emacs / eww inconsistent in showing images

[Lars Ingebrigtsen]

Adam Sjøgren <asjo@koldfront.dk> writes:

> Paste in a URL of a webpage with many pictures, eg
> https://moving-pictures.ingebrigtsen.no/2019/10/01/songs-from-the-second-floor-2/
>
> Expected result is all images (eventually) load. Instead I see gray
> boxes where the images should appear. This even after waiting 10
> minutes. In Firefox I can get all images in seconds.

shr (via url-queue) uses a very primitive timeout mechanism for
image-loading, and it aborts a lot of downloads prematurely.  And as you
say, the more images it's downloading in parallel, the more likely it is
that it drops the image.

This will work a lot better once the with-fetched-url branch has been
merged into Emacs, but that won't happen until Emacs 28, which is
probably still weeks away.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37577: 27.0.50; GNU Emacs / eww inconsistent in showing images

[Unknown]

Lars writes:

> This will work a lot better once the with-fetched-url branch has been
> merged into Emacs, but that won't happen until Emacs 28, which is
> probably still weeks away.

Nice! I just merged the branch locally, and I am trying it out (after
resolving merge conflict semi-randomly).

It seems to work a lot better - there are still a couple of images
missing on long pages, but it's far from the around half I had before.

Thanks!

bug#37577: 27.0.50; GNU Emacs / eww inconsistent in showing images

[Lars Ingebrigtsen]

Adam Sjøgren <asjo@koldfront.dk> writes:

> It seems to work a lot better - there are still a couple of images
> missing on long pages, but it's far from the around half I had before.

Great; thanks for testing.

I've now merged this bug with bug#22921, which is the bug report about
the new URL interface.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no