bug#22921: 25.1.50; Wishlist: There should be a new URL interface

bug#22921: 25.1.50; Wishlist: There should be a new URL interface

[Lars Magne Ingebrigtsen]

The current interfaces to the URL library aren't very handy.  It's
`url-retrieve' and `url-retrieve-synchronously'.  The main problem is
that they're inconsistent and difficult to interact with.

For instance, most of the "inputs" to the functions are dynamically
bound specially variables like `url-request-method', and that sucks.
The buffers that are returned are inconsistent based on the URL
method -- HTTP buffers come with the headers first, while ftp buffers do
not.

So: `with-url-buffer'.

Typical usage would be

(with-url-buffer ("https://fsf.org"
                  :verbose t
                  :request-method "POST"
                  :timeout 10)
  (do-stuff-with-the-result-here))

That is, the callback is in the form.  There are no headers in the
buffer, and point is at point-min when the body is called.  Headers are
accessed with a new form like `(url-header "date")', and the headers are
available in a buffer-local variable called `url-headers'.

Upon completion of the body form, the buffer is killed, unless a
parameter :inhibit-kill-buffer is given.

Synchronous and asynchronous operation looks identical, but synchronous
operation happens when you say :synchronous t.

Implementation-wise, this happens with lexical bindings switched on, and
the body form is turned into a closure that's evaluated upon completion.

`with-url-buffer' will not support the exotic, never-actually-used
protocols like news:, irc:, imap:, etc, but only http:, https:, ftp:
and file:.



In GNU Emacs 25.1.50.23 (x86_64-unknown-linux-gnu, GTK+ Version 3.4.2)
 of 2016-03-06 built on stories
Repository revision: 20cc4182598c6c724f4556faf6bc322c9e58ad1d
Windowing system distributor 'The X.Org Foundation', version 11.0.11204000
System Description:	Debian GNU/Linux 7.9 (wheezy)

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GSETTINGS NOTIFY GNUTLS
LIBXML2 FREETYPE LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LANG: en_US
  locale-coding-system: iso-latin-1-unix

Major mode: Group

Minor modes in effect:
  gnus-topic-mode: t
  gnus-undo-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t

Recent messages:
Wrote /home/larsi/.newsrc
Saving /home/larsi/.newsrc.eld...
Saving file /home/larsi/.newsrc.eld...
Wrote /home/larsi/.newsrc.eld
Saving /home/larsi/.newsrc.eld...done
No more unseen articles
No more unread articles
next-line: End of buffer [2 times]
(No changes need to be saved)
Making completion list...

Load-path shadows:
/home/larsi/src/clock.el/clock hides /home/larsi/lisp/clock
/home/larsi/src/cddb.el/expect hides /home/larsi/lisp/expect
/home/larsi/src/pvr.el/pvr hides /home/larsi/lisp/pvr
~/pgnus/contrib/vcard hides /home/larsi/lisp/vcard
/home/larsi/src/cddb.el/captitle hides /home/larsi/lisp/captitle
~/lisp/zenirc-2.112/src/zenirc-example hides /home/larsi/lisp/zenirc-example
/home/larsi/lisp/dom hides /home/larsi/src/emacs/trunk/lisp/dom
~/pgnus/contrib/compface hides /home/larsi/src/emacs/trunk/lisp/image/compface

Features:
(shadow emacsbug sendmail binhex canlock ecomplete flow-fill eieio-opt
speedbar sb-image ezimage dframe find-func thingatpt view sgml-mode
misearch multi-isearch tabify imenu man vc vc-dispatcher copyright
vc-cvs eww shr-color color gnus-html url-queue help-fns url-cache
gnus-picon sort gnus-cite smiley ansi-color mm-archive gnus-async
gnus-dup qp gnus-ml gmane spam-gmane dns mm-url disp-table gnus-fun
gnus-mdrtn gnus-topic pop3 nndoc nnmbox nndraft utf-7 nnmh nnml nnfolder
network-stream nsm starttls nnir spam-report spam spam-stat gnus-uu yenc
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art
mm-uu mml2015 mm-view mml-smime smime dig nntp gnus-cache gnus-sum
gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source
utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range message
format-spec rfc822 mml mml-sec epa epg mailabbrev gmm-utils mailheader
gnus-win gnus nnheader gnus-util rmail rmail-loaddefs mail-utils movie
mkv shr svg imdb dom pvr debug debbugs-gnu easy-mmode derived debbugs
soap-client mm-decode mm-bodies mm-encode url-http tls gnutls url-auth
mail-parse rfc2231 rfc2047 rfc2045 ietf-drums url-gw puny url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util mailcap warnings rng-xsd rng-dt rng-util xsd-regexp xml ido seq
flyspell ispell benchmark w3m browse-url doc-view subr-x dired
dired-loaddefs image-mode timezone w3m-hist w3m-fb w3m-ems wid-edit
w3m-ccl ccl w3m-favicon w3m-image w3m-proc w3m-util add-log mail-extr
mm-util mail-prsvr jka-compr cl finder-inf package epg-config
url-handlers url-parse auth-source cl-seq eieio byte-opt bytecomp
byte-compile cl-extra help-mode easymenu cconv eieio-core cl-macs gv
eieio-loaddefs cl-loaddefs pcase cl-lib password-cache url-vars
time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev
obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote inotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 4019121 362371)
 (symbols 48 170790 312)
 (miscs 40 674 1974)
 (strings 32 317931 33454)
 (string-bytes 1 13992070)
 (vectors 16 43895)
 (vector-slots 8 1764946 243016)
 (floats 8 7065 3818)
 (intervals 56 929585 49118)
 (buffers 976 59)
 (heap 1024 212624 72649))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22921: 25.1.50; Wishlist: There should be a new URL interface

[Stefan Kangas]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So: `with-url-buffer'.

What's the current status of your with-fetched-url branch?  Is it
ready to merge into master?  It would be nice to get it in so that we
can start playing around with it.

Best regards,
Stefan Kangas

bug#22921: 25.1.50; Wishlist: There should be a new URL interface

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> So: `with-url-buffer'.
>
> What's the current status of your with-fetched-url branch?  Is it
> ready to merge into master?  It would be nice to get it in so that we
> can start playing around with it.

It should be (more or less) ready to go.  I've been meaning to merge it
ever since Emacs 28 started, but I haven't found time yet.

If I remember, there may be some quirks with certain auth schemes that
needs fixing, but that can be done after the merge.  But I need to block
out, say, a week where I can respond to bug reports in a speedy manner,
and I haven't found that week yet...

But it'll happen sometime within a month or to.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37577: 27.0.50; GNU Emacs / eww inconsistent in showing images

[Stefan Kangas]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> What's the current status of your with-fetched-url branch?  Is it
>> ready to merge into master?  It would be nice to get it in so that we
>> can start playing around with it.
>
> It should be (more or less) ready to go.  I've been meaning to merge it
> ever since Emacs 28 started, but I haven't found time yet.

Hmm, maybe we can have it in Emacs 29?  :-)

bug#22921: bug#37577: 27.0.50; GNU Emacs / eww inconsistent in showing images

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>>> What's the current status of your with-fetched-url branch?  Is it
>>> ready to merge into master?  It would be nice to get it in so that we
>>> can start playing around with it.
>>
>> It should be (more or less) ready to go.  I've been meaning to merge it
>> ever since Emacs 28 started, but I haven't found time yet.
>
> Hmm, maybe we can have it in Emacs 29?  :-)

:-)

It got put on the back burner because I started thinking about the proxy
stuff that url.el supports but I hadn't implemented for with-url yet.

Because I think that's the wrong level to do it.  Emacs should support a
general network proxy thing, that would allow all network connections to
be proxied.  (Or selectively.)

And the more I looked at this, the more apparently it became that Emacs'
network stuff should just be totally reimplemented, because it's so full
of...  oddnesses geared toward processes.  And processes and network
connections are very different things.

So it's a Big Project.  :-/

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no