with-url

with-url

[Lars Ingebrigtsen]

I've basically implemented the previously discussed `with-url' macro.
Doc string below.  I'll push it to a feature branch tomorrow.

But I have some style questions I'm not quite sure about what the right
approach is.

First of all, the basic form looks like this:

  (with-url (headers "http://fsf.org/")
    (message "The size of the FSF front page is %s" (buffer-size)))

And you can cram all the parameters as keywords after the URL.  For
instance, if you put :method "POST" in there, it'll use POST instead of
GET, and if you put :wait t in there, everything will be done
synchronously instead of asynchronously.  (Which is often nice when
testing code.)

First question: Should the `headers' variable binding be explicit or
implicit?  In Common Lisp there's a culture for having them all be very
explicit, but in Emacs we have a culture for hiding variables in
buffer-local variables.  One is nice and readable, and the other is nice
and short.

  (with-url (headers "http://fsf.org/")
    (message "Content type is %s" (url-header headers 'content-type)))

==> Content type is text/html;charset=utf-8

vs

  (with-url ("http://fsf.org/")
    (message "Content type is %s" (url-header 'content-type)))

The latter looks nicer, no?  The problem with that is this:

  (with-url ("http://fsf.org/")
    (with-url ("http://goole.org/")
      (message "Content type is %s" (url-header 'content-type))))

But I wanted the first one!

  (with-url ("http://fsf.org/")
    (let ((buffer (current-buffer)))
      (with-url ("http://goole.org/")
        (message "Content type is %s" (with-current-buffer buffer
                                        (url-header 'content-type))))))

*waugh*

And, yes, working with URL-ey stuff, you're often nesting these calls,
because the first call gives you some information, and then you want to
do another request...  But do you often refer back to the headers of the
previous request?  I don't know?

Opinions?


The second question is about error handling.  URL requests will fail,
either because the page doesn't exist, or the host is down, or it times
out.  (Yes, there are timeouts in the new interface.)  So in real life,
these calls should always look like...

  (with-url (headers "http://fsf.org/")
    (if (url-header-ok-p headers)
        (message "Content type is %s" (url-header headers 'content-type)
        (message "I guess I'll do something else"))))

Although looking at `url-retrieve' calls in the Emacs code base, many
callers don't bother because error handling is boring.  And, besides,
it'll indent the code so much.  :-)

An alternative would be to say something like...  er...

  (with-url (headers "http://fsf.org/")
    (ok
      (message "Content type is %s" (url-header headers 'content-type)))
    (error
      (message "I guess I'll do something else")))

That is, make the with-url have two body forms -- one which is called
when things went OK, and one when it didn't.  (And marked by those two
symbols.)

If the body doesn't have those symbols, then it's all an `ok' form and
no error callbacks.  Or

  (with-url (headers "http://fsf.org/")
    (error
      (message "I guess I'll do something else"))
    (message "Content type is %s" (url-header headers 'content-type)))

That is, you can put the error handling form at the start if the first
symbol is `error' in that form.  Well, it can't be `error', but perhaps
`url-error'.  Or...

Opinions?  :-)



----

with-url is a Lisp macro in ‘../../with-url/lisp/url/with-url.el’.

(with-url (HEADER-VARIABLE URL &key WAIT TIMEOUT READ-TIMEOUT SILENT
INHIBIT-COOKIES INHIBIT-CACHE HEADERS (METHOD "GET") DATA
(DATA-CHARSET 'utf-8) DATA-ENCODING) &body BODY)

Retrieve URL and execute BODY with point in a buffer with the response.

Example:

  (with-url (headers "http://fsf.org/")
    (message "The size of the FSF front page is %s" (buffer-size)))

The buffer is killed after BODY has exited.

HEADER-VARIABLE is bound to a structure that contains the response
headers and status.  These can be accessed with ‘url-header’ like this:

  (url-header headers "Content-Type")

Case is not significant.

Additional keywords can be given to ‘with-url’ to alter its operation.

:wait t
Normal ‘with-url’ operation is asynchronous.  If this parameter is given,
the retrieval will be synchronous instead.

:timeout SECONDS
Give up after approximately SECONDS seconds and execute BODY.

:read-timeout SECONDS
If no data has been received for the last SECONDS seconds, give
up and execute BODY.

:silent t
Issue no messages during operation.

:inhibit-cookies t
Neither send nor store cookies.

:headers ALIST
Add ALIST to the headers sent over to the server.  This should typically
look like

  (("User-Agent" "Emacs"))

If the header name is the same as one of the automatically
generated headers, the value from this list will override the
automatically generated header.  To disable the header
completely, use nil as the value.

Additional elements in this alist are interpreted as the
charset (defaulting to utf-8) and the encoding method (defaulting
to url-encode).

:method GET/POST/etc
The method to use for retrieving an HTTP(S) resource.  This defaults
to GET, and other popular values are POST, UPDATE and PUT.

:data STRING
Data to include in the body of the HTTP(S) request when using
POST, UPDATE or PUT.

:data-charset CHARSET
What charset this data should be interpreted as.  This defaults
to UTF-8.

:data-encoding ENCODING
When using the posting methods, the data is usually encoded in
some fashion.  Popular encodings are ‘url-form’ and ‘base64’.

[back]

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Stefan Monnier]

>   (with-url ("http://fsf.org/")
>     (message "Content type is %s" (url-header 'content-type)))

Why not

    (with-url ("http://fsf.org/")
      (let ((buffer (current-buffer)))
        (with-url ("http://goole.org/")
          (message "Content type is %s"
                   (url-header 'content-type buffer)))))


-- Stefan

Re: with-url

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Why not
>
>     (with-url ("http://fsf.org/")
>       (let ((buffer (current-buffer)))
>         (with-url ("http://goole.org/")
>           (message "Content type is %s"
>                    (url-header 'content-type buffer)))))

And the buffer parameter would be optional and default to the current
buffer?  Yeah, that sounds quite nice...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 29 Dec 2016 01:41:02 +0100
> 
> First of all, the basic form looks like this:
> 
>   (with-url (headers "http://fsf.org/")
>     (message "The size of the FSF front page is %s" (buffer-size)))

I'm probably missing something, but why do you need a macro?  The
description seems to say that BODY is executed with point in a buffer
with the response, so why not just set up that buffer and return with
it as the current buffer?  Or maybe even introduce a function very
similar to url-retrieve, but without the mess with encoding/unibyte
etc.?

As for the syntax, this macro looks unusual to me, certainly wrt other
with-SOMETHING macros, in that it accepts a list of arguments of
variable length and contents.  But that's me.

A few comments to the doc string:

> :headers ALIST
> Add ALIST to the headers sent over to the server.  This should typically
> look like
> 
>   (("User-Agent" "Emacs"))

What about unibyte/multibyte issue in the headers?  Should that be
encoded by the caller?

> Additional elements in this alist are interpreted as the
> charset (defaulting to utf-8) and the encoding method (defaulting
> to url-encode).

Please don't use "charset" when you really mean "coding-system".  In
Emacs parlance, "charset" means something very different (and "utf-8"
is not a charset in that meaning), so this will spread confusion.

Also, why should the default be UTF-8?  Isn't the system locale's
codeset a better default?  The URL doesn't have to be on another
machine, right?

> :method GET/POST/etc
> The method to use for retrieving an HTTP(S) resource.  This defaults
> to GET, and other popular values are POST, UPDATE and PUT.

Does thus mean this macro is only for HTTP/HTTPS?

> :data-charset CHARSET
> What charset this data should be interpreted as.  This defaults
> to UTF-8.

Once again, not "charset".  And this is not about interpretation, this
is about encoding and/or decoding.

Thanks.

Re: with-url

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>>   (with-url (headers "http://fsf.org/")
>>     (message "The size of the FSF front page is %s" (buffer-size)))
>
> I'm probably missing something, but why do you need a macro?  The
> description seems to say that BODY is executed with point in a buffer
> with the response, so why not just set up that buffer and return with
> it as the current buffer?

The body is executed synchronously or as a callback later, depending on
whether you put a :wait t in the parameter list or not.  I don't see how
that's possibly without using a macro.  (I mean, without radically
rewriting the call if you want a synchronous/asynchronous execution.)

> As for the syntax, this macro looks unusual to me, certainly wrt other
> with-SOMETHING macros, in that it accepts a list of arguments of
> variable length and contents.  But that's me.

We haven't done that much in Emacs Lisp, but it's a common idiom in
Common Lisp, and I think it's kinda nice.

  (with-open-file (s "/tmp/foo"
                   :direction :output
                   :if-exists :rename)
    (princ :foo s))

> A few comments to the doc string:
>
>> :headers ALIST
>> Add ALIST to the headers sent over to the server.  This should typically
>> look like
>> 
>>   (("User-Agent" "Emacs"))
>
> What about unibyte/multibyte issue in the headers?  Should that be
> encoded by the caller?

I think that question is answered by your next comment?  :-)

>> Additional elements in this alist are interpreted as the
>> charset (defaulting to utf-8) and the encoding method (defaulting
>> to url-encode).
>
> Please don't use "charset" when you really mean "coding-system".  In
> Emacs parlance, "charset" means something very different (and "utf-8"
> is not a charset in that meaning), so this will spread confusion.

True, I'll reword that.

> Also, why should the default be UTF-8?  Isn't the system locale's
> codeset a better default?  The URL doesn't have to be on another
> machine, right?

No, but utf-8 is the normal default in network communication these days.

>> :method GET/POST/etc
>> The method to use for retrieving an HTTP(S) resource.  This defaults
>> to GET, and other popular values are POST, UPDATE and PUT.
>
> Does thus mean this macro is only for HTTP/HTTPS?

No, it'll work for file: and ftp:, too.  I should probably group the
HTTP(S)-only options.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Please don't use "charset" when you really mean "coding-system".  In
>> Emacs parlance, "charset" means something very different (and "utf-8"
>> is not a charset in that meaning), so this will spread confusion.
>
> True, I'll reword that.

(But in a networking environment, the Emacs terminology is very
confusing.  You have `coding system' where all standards say `charset',
and in addition you have all the standards talking about `encoding',
which is things like base64 and url-encoding.  So the parameter list, if
changed, will talk about :data-coding-system and
:data-transfer-encoding, which is probably not as helpful as it should
be.

And note that utf-8 is a perfectly good charset in standards: "Charset"
means "encoded character set" in RFCs.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 29 Dec 2016 17:35:07 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >>   (with-url (headers "http://fsf.org/")
> >>     (message "The size of the FSF front page is %s" (buffer-size)))
> >
> > I'm probably missing something, but why do you need a macro?  The
> > description seems to say that BODY is executed with point in a buffer
> > with the response, so why not just set up that buffer and return with
> > it as the current buffer?
> 
> The body is executed synchronously or as a callback later, depending on
> whether you put a :wait t in the parameter list or not.  I don't see how
> that's possibly without using a macro.

Well, url-retrieve uses a callback.  Won't that fit the bill here?

Re: with-url

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 29 Dec 2016 17:40:09 +0100
> 
> (But in a networking environment, the Emacs terminology is very
> confusing.  You have `coding system' where all standards say `charset',
> and in addition you have all the standards talking about `encoding',
> which is things like base64 and url-encoding.  So the parameter list, if
> changed, will talk about :data-coding-system and
> :data-transfer-encoding, which is probably not as helpful as it should
> be.

How about :coding-system and :transfer-encoding instead?

> And note that utf-8 is a perfectly good charset in standards: "Charset"
> means "encoded character set" in RFCs.)

We have mime-charset to match that, but in that case you will need to
convert the value to a coding-system internally, as the conversion is
not a trivial one.

I think this eventually boils down to the question whether most users
of this feature feel more at home with network-style "charset" or with
Emacs-style "coding-system".  I don't know the answer.

Re: with-url

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> The body is executed synchronously or as a callback later, depending on
>> whether you put a :wait t in the parameter list or not.  I don't see how
>> that's possibly without using a macro.
>
> Well, url-retrieve uses a callback.  Won't that fit the bill here?

This is also a callback in the asynchronous case, but it's not in the
synchronous case.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> (But in a networking environment, the Emacs terminology is very
>> confusing.  You have `coding system' where all standards say `charset',
>> and in addition you have all the standards talking about `encoding',
>> which is things like base64 and url-encoding.  So the parameter list, if
>> changed, will talk about :data-coding-system and
>> :data-transfer-encoding, which is probably not as helpful as it should
>> be.
>
> How about :coding-system and :transfer-encoding instead?

It's the coding system for the data, not the rest of what's transferred,
so you need the `data' in there...  The headers are each encoded
according to various other conventions (the Host is punycode, the
User-Agent is ... er...  I haven't looked up the standard, but I'm
guessing url-encode).  

> I think this eventually boils down to the question whether most users
> of this feature feel more at home with network-style "charset" or with
> Emacs-style "coding-system".  I don't know the answer.

I think calling the parameters :data-coding-system (etc) and then
clarifying in the doc string that it's the charset (if you squint at it)
is OK...  I was just kinda digressing: I think the nomenclature
`coding-system' in Emacs is unfortunate, but it's what we're stuck with.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 29 Dec 2016 19:52:18 +0100
> 
> > How about :coding-system and :transfer-encoding instead?
> 
> It's the coding system for the data, not the rest of what's transferred,
> so you need the `data' in there...  The headers are each encoded
> according to various other conventions (the Host is punycode, the
> User-Agent is ... er...  I haven't looked up the standard, but I'm
> guessing url-encode).  

If there could be different encodings involved, you need to allow
the caller to specify each one of them separately.

> I think the nomenclature `coding-system' in Emacs is unfortunate

It's too late for that ship.

Re: with-url

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> If there could be different encodings involved, you need to allow
> the caller to specify each one of them separately.

Yes, as the doc string says, you can specify a coding system and a
transfer encoding per header.

(The vast majority of the callers don't need to, since the defaults are
sane.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Lars Ingebrigtsen]

Nobody had any opinions on the error handling part?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 30 Dec 2016 15:12:28 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If there could be different encodings involved, you need to allow
> > the caller to specify each one of them separately.
> 
> Yes, as the doc string says, you can specify a coding system and a
> transfer encoding per header.

The doc string says that, but I find it hard to understand how to do
that, exactly.

> (The vast majority of the callers don't need to, since the defaults are
> sane.)

That is generally true for everything Emacs does with encoding and
decoding these days.

Re: with-url

[Dmitry Gutov]

On 29.12.2016 03:41, Lars Ingebrigtsen wrote:

> That is, you can put the error handling form at the start if the first
> symbol is `error' in that form.  Well, it can't be `error', but perhaps
> `url-error'.  Or...
>
> Opinions?  :-)

I think you're trying to magic it up too much, by adding error handling 
forms to the macro. Personally, I'd prefer to use condition-case to 
handle errors.

Of course, that will require a new error type (or several) to be defined.

That usage conflicts with asynchronous requests, though. More on that 
later (*).

Another thing to note is that a 404 status should be an error for some 
applications, but others would prefer to treat all statuses the same, 
avoid following redirects, etc, and parse the response headers and body 
anyway. So whether `with-url' should signal an error in those cases, 
should probably be governed by another keyword argument. In "serious" 
libraries, in e.g. Ruby, we make this choice using wrappers, like 
client-side middleware.

Something else I take an issue with, is with `with-url' being a macro. 
:) That violates The First Rule of Macro Club.

I think we can do without, by using one of the more traditional control 
flow solutions. First of all, the fact that `url-retrieve' uses a 
callback is one of its lesser problems. The arguments that the callback 
gets passed are a much bigger one. So we would stop at fixing just that.

The other options are:

1) Return a Promise value. Emacs could use a standardized Promise 
structure in other places, too.

2) Basically require to use the newly-introduced threads for asynchrony. 
The library would suspend the current thread and resume it when 
something happens (a response or an error). Someone should investigate 
the validity of this approach, but the good news is, it can be layered 
on top of (1) later. This option can also be compatible with using 
`condition-case' to handle errors (*).

Re: with-url

[Stefan Monnier]

> 1) Return a Promise value. Emacs could use a standardized Promise structure
> in other places, too.

We have url-future (which got a "url-" prefix because we wanted to
experiment with it before making it global).


        Stefan

Re: with-url

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> (But in a networking environment, the Emacs terminology is very
>> confusing.  You have `coding system' where all standards say `charset',
>> and in addition you have all the standards talking about `encoding',
>> which is things like base64 and url-encoding.  So the parameter list, if
>> changed, will talk about :data-coding-system and
>> :data-transfer-encoding, which is probably not as helpful as it should
>> be.
>
> How about :coding-system and :transfer-encoding instead?

Thinking about it a bit more, I'm reverting the working to "charset"
again.  That's what this is about, since the string in question will end
up in a HTTP header saying charset="foo".

Emacs coding systems are (vaguely) supersets of the charset concept, so
it's all good.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Thinking about it a bit more, I'm reverting the working to "charset"
> again.

That should be "wording", not "working".  My speeling is getting wrse
all teh time.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> I think you're trying to magic it up too much, by adding error
> handling forms to the macro. Personally, I'd prefer to use
> condition-case to handle errors.

Nope.  You should only signal an error if you have an error, and getting
a 404 isn't an error.  

> Something else I take an issue with, is with `with-url' being a
> macro. :) That violates The First Rule of Macro Club.

Yes, the first rule is "use a macro when it makes sense".

> 1) Return a Promise value. Emacs could use a standardized Promise
> structure in other places, too.

Emacs could, but Emacs doesn't.

> 2) Basically require to use the newly-introduced threads for
> asynchrony.

Using the new threads is a trivial rewrite and doesn't affect the
signature of the macro.

Anyway, I think the best option here would be to just have an
:ignore-errors parameter to `with-url'.  In that case, the body will
just be evaluated if we get non-error response.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 21 Jan 2017 16:31:53 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> (But in a networking environment, the Emacs terminology is very
> >> confusing.  You have `coding system' where all standards say `charset',
> >> and in addition you have all the standards talking about `encoding',
> >> which is things like base64 and url-encoding.  So the parameter list, if
> >> changed, will talk about :data-coding-system and
> >> :data-transfer-encoding, which is probably not as helpful as it should
> >> be.
> >
> > How about :coding-system and :transfer-encoding instead?
> 
> Thinking about it a bit more, I'm reverting the working to "charset"
> again.  That's what this is about, since the string in question will end
> up in a HTTP header saying charset="foo".

Once again, may I suggest mime-charset instead?  We already use this
elsewhere in Emacs, and using it here will prevent confusion with
Emacs charsets.

> Emacs coding systems are (vaguely) supersets of the charset concept

No, they aren't.

Re: with-url

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Once again, may I suggest mime-charset instead?  We already use this
> elsewhere in Emacs, and using it here will prevent confusion with
> Emacs charsets.

The charsets in MIME are the same as the charsets in all other relevant
RFC.  It is clear to somebody talking HTTP what a charset is, even if
that person has never heard of MIME.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 21 Jan 2017 17:32:05 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Once again, may I suggest mime-charset instead?  We already use this
> > elsewhere in Emacs, and using it here will prevent confusion with
> > Emacs charsets.
> 
> The charsets in MIME are the same as the charsets in all other relevant
> RFC.  It is clear to somebody talking HTTP what a charset is, even if
> that person has never heard of MIME.

I want to cater to Emacs users and developers as well, and I think
mime-charset is a good compromise.  Once again, we have a
:mime-charset property of coding systems, so using that terminology
will I think provide good mnemonic hints both to protocol experts and
to Emacs developers.

Re: with-url

[Dmitry Gutov]

On 21.01.2017 18:44, Lars Ingebrigtsen wrote:

> Nope.  You should only signal an error if you have an error, and getting
> a 404 isn't an error.

Whether 404 is an error or not, depends on an application. And I'm 
saying that as someone with a few years of experience writing web 
applications that do HTTP requests themselves.

Consider the case where an application calls a function foo (which 
implements some sort of abstraction layer), and the said function uses 
with-url.

If foo is supposed to be general purpose (used in different 
applications), it would have to handle errors in a general way, so that 
any of its callers could still handle any kinds of errors, without being 
a macro. I'd rather with-url (or its replacement) was an example of 
doing that already.

>> Something else I take an issue with, is with `with-url' being a
>> macro. :) That violates The First Rule of Macro Club.
>
> Yes, the first rule is "use a macro when it makes sense".

Not really. And macros make sense the most when we add a control flow 
construct, or some sort of definition syntax for a new facility. Neither 
of which is really the case here.

You're basically joining a library function with a control flow 
construct together.

>> 2) Basically require to use the newly-introduced threads for
>> asynchrony.
>
> Using the new threads is a trivial rewrite and doesn't affect the
> signature of the macro.

It could simplify it, by allowing to treat asynchronous code like 
synchronous one.

> Anyway, I think the best option here would be to just have an
> :ignore-errors parameter to `with-url'.  In that case, the body will
> just be evaluated if we get non-error response.

I'm rather more worried about being able to handle errors in an 
asynchronous fashion.

You could say 404 is not an error, but handling it in a similar way to 
all other "actual" errors like "can't resolve the hostname" and 
"connection timeout" can be pretty handy.

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Not really. And macros make sense the most when we add a control flow
> construct, or some sort of definition syntax for a new
> facility. Neither of which is really the case here.
>
> You're basically joining a library function with a control flow
> construct together.

It's a control flow construct in much the same way `with-temp-file' is.

> You could say 404 is not an error, but handling it in a similar way to
> all other "actual" errors like "can't resolve the hostname" and
> "connection timeout" can be pretty handy.

Yes, none of these should signal an error from the library.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Whether 404 is an error or not, depends on an application. And I'm
> saying that as someone with a few years of experience writing web
> applications that do HTTP requests themselves.

(Yes, I have, of course, no such experience.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Dmitry Gutov]

On 21.01.2017 23:32, Lars Ingebrigtsen wrote:

> It's a control flow construct in much the same way `with-temp-file' is.

with-temp-file has no error handling section. If anything unexpected 
happens during setup or teardown, you can be sure it'll signal a proper 
error. The callers can't miss it, and they're free to handle different 
errors via condition-case.

>> You could say 404 is not an error, but handling it in a similar way to
>> all other "actual" errors like "can't resolve the hostname" and
>> "connection timeout" can be pretty handy.
>
> Yes, none of these should signal an error from the library.

How will the callers handle failure, then?

Re: with-url

[Dmitry Gutov]

On 21.01.2017 23:34, Lars Ingebrigtsen wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> Whether 404 is an error or not, depends on an application. And I'm
>> saying that as someone with a few years of experience writing web
>> applications that do HTTP requests themselves.
>
> (Yes, I have, of course, no such experience.)

It's basically impossible to argue from experience that "certain classes 
of applications" don't exist. Maybe you just haven't seen them, or 
haven't tried the different approaches.

It is, on the other hand, possible to argue from experience that 
"certain classes of applications" exist.

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> It's basically impossible to argue from experience that "certain
> classes of applications" don't exist.

It's a good thing that nobody has done that, then.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> How will the callers handle failure, then?

In the body of the form they can examine the status, of course.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Dmitry Gutov]

On 21.01.2017 23:49, Lars Ingebrigtsen wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> How will the callers handle failure, then?
>
> In the body of the form they can examine the status, of course.

Will they examine the resolved hostname and the absence of a connection 
timeout as well?

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Will they examine the resolved hostname and the absence of a
> connection timeout as well?

I'm not sure what you're talking about, if anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Dmitry Gutov]

On 22.01.2017 00:12, Lars Ingebrigtsen wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> Will they examine the resolved hostname and the absence of a
>> connection timeout as well?
>
> I'm not sure what you're talking about, if anything.

You said "none of these should signal an error from the library" in 
response to my enumerations of different kinds of errors: 404, hostname 
resolution failure, connection timeout.

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 22.01.2017 00:12, Lars Ingebrigtsen wrote:
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>
>>> Will they examine the resolved hostname and the absence of a
>>> connection timeout as well?
>>
>> I'm not sure what you're talking about, if anything.
>
> You said "none of these should signal an error from the library" in
> response to my enumerations of different kinds of errors: 404,
> hostname resolution failure, connection timeout.

Yes...  but you asked about how code would examine "the absence of a
connection timeout".  I don't know what you mean.  If there's no timeout
(and there's no other errors), `url-errorp' says nil.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Dmitry Gutov]

On 22.01.2017 00:23, Lars Ingebrigtsen wrote:

>> You said "none of these should signal an error from the library" in
>> response to my enumerations of different kinds of errors: 404,
>> hostname resolution failure, connection timeout.
>
> Yes...  but you asked about how code would examine "the absence of a
> connection timeout".  I don't know what you mean.

That's exactly my point. You said that none of these should be errors, 
and then only replied how the callers will find out about the error 
status, not about the other kinds of errors.

> If there's no timeout
> (and there's no other errors), `url-errorp' says nil.

So, in the Brave New World the callers will continue to use url-errorp? 
Error handling C-style?

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 22.01.2017 00:23, Lars Ingebrigtsen wrote:
>
>>> You said "none of these should signal an error from the library" in
>>> response to my enumerations of different kinds of errors: 404,
>>> hostname resolution failure, connection timeout.
>>
>> Yes...  but you asked about how code would examine "the absence of a
>> connection timeout".  I don't know what you mean.
>
> That's exactly my point. You said that none of these should be errors,
> and then only replied how the callers will find out about the error
> status, not about the other kinds of errors.

"The absence of a connection timeout" is not an error.  A connection
timeout is an error, of course, and the caller can access that error
like all the other errors.

None of these errors are control-flow-level errors, but are expected
behaviour.  You don't signal Lisp-level errors for expected behaviour.
That's what Java programmers did in the 90s, and they've learned their
lessons.

Well, most of them.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Dmitry Gutov]

On 22.01.2017 00:37, Lars Ingebrigtsen wrote:

> "The absence of a connection timeout" is not an error.  A connection
> timeout is an error, of course, and the caller can access that error
> like all the other errors.

Via a global variable?

> None of these errors are control-flow-level errors, but are expected
> behaviour.

Try evaluating (insert-file-contents "/abc/def").

It signals file-error.

How is that different, conceptually speaking, from a 404 status 
(application level error) or domain resolution error (IP level error)?

> That's what Java programmers did in the 90s, and they've learned their
> lessons.

I can assure you, they are still doing that. Just like many other 
programmer using certain other object-oriented languages, when 
implementing business logic.

The lesson to be learned here is, it pretty much works.

> Well, most of them.

Ones that switched to Go?

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Via a global variable?

No.  I suggested a bound variable in the form executed,

(with-url (status "http:...")
  (url-error status)
  ...)

but Stefan suggested a buffer-local variable instead.  I went with the
latter today.

>> None of these errors are control-flow-level errors, but are expected
>> behaviour.
>
> Try evaluating (insert-file-contents "/abc/def").
>
> It signals file-error.
>
> How is that different, conceptually speaking, from a 404 status
> (application level error) or domain resolution error (IP level error)?

The network is transient.  These events are expected.  That the file
you've just examined isn't there is a different class of error.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[John Mastro]

>>> Something else I take an issue with, is with `with-url' being a
>>> macro. :) That violates The First Rule of Macro Club.
>>
>> Yes, the first rule is "use a macro when it makes sense".
>
> Not really. And macros make sense the most when we add a control flow
> construct, or some sort of definition syntax for a new facility.
> Neither of which is really the case here.

I apologize if this has already been discussed, but how about
implementing this as a new function that improves on the current
situation but offers a traditional interface, plus a macro as an
optional convenience layer atop it?

        John

Re: with-url

[Yuri Khan]

On Sun, Jan 22, 2017 at 5:07 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> The network is transient.  These events are expected.  That the file
> you've just examined isn't there is a different class of error.

I work on the server-side team of a network interaction where all of
the errors mentioned upthread can and do occur: DNS failure,
connection timeout, connection refusal, 4xx and 5xx status codes, *and
also* 3xx codes with redirects to unexpected locations and 2xx codes
with an unexpected Content-Type or malformed content.

We constantly have to remind our client-side team that they need to be
prepared for all of those, and that their QA must test each of those.
Whenever they forget to do that, horrible things happen, up to and
including application startup failure requiring reinstallation.

A good HTTP client framework *should* signal errors and let the
willing client catch them, rather than leave it to the client to
explicitly check for every error condition. Myers’ Razor: Make
interfaces easy to use correctly and hard to use incorrectly.

The only class of HTTP client applications that I know where the
expected way to process failure responses is the same as for success
responses is the web browser. And I don’t envision *very* many of
these written in Elisp.

Re: with-url

[Dmitry Gutov]

On 22.01.2017 01:07, Lars Ingebrigtsen wrote:
> I suggested a bound variable in the form executed,
>
> (with-url (status "http:...")
>   (url-error status)
>   ...)
>
> but Stefan suggested a buffer-local variable instead.  I went with the
> latter today.

Will `status' only contain HTTP status, or will it contain any other 
errors as well?

> The network is transient.  These events are expected.  That the file
> you've just examined isn't there is a different class of error.

Apparently we might have to agree to disagree here.

Re: with-url

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Will `status' only contain HTTP status, or will it contain any other
> errors as well?

It's an object that can be interrogated through a number of functions,
and gives you access to headers, results and errors.

> Apparently we might have to agree to disagree here.

Says you!

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Lars Ingebrigtsen]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> A good HTTP client framework *should* signal errors and let the
> willing client catch them, rather than leave it to the client to
> explicitly check for every error condition. Myers’ Razor: Make
> interfaces easy to use correctly and hard to use incorrectly.

It depends on the language.  If the only way you have to report back
different classes of responses is through the signalling system, then
that's what you're apt to do.  (Also see: Java, the 90s.)

Anyway, it's all rather moot, since this is, of course, an asynchronous
library function, and you can't really send any Lisp-level signals in
that context.  So `with-url' uses the mechanisms that virtually all
other async HTTP libraries uses: Gives the callback a status object to
interrogate.

But discussion on programming philosophies is fun, anyway.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Dmitry Gutov]

On 22.01.2017 17:46, Lars Ingebrigtsen wrote:

> It depends on the language.  If the only way you have to report back
> different classes of responses is through the signalling system, then
> that's what you're apt to do.  (Also see: Java, the 90s.)

There's nothing stopping an arbitrary piece of Java code from using 
complex return values, or even "global variables", to do that.

> Anyway, it's all rather moot, since this is, of course, an asynchronous
> library function, and you can't really send any Lisp-level signals in
> that context.

That's why I mentioned threads.

Re: with-url

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I want to cater to Emacs users and developers as well, and I think
> mime-charset is a good compromise.  Once again, we have a
> :mime-charset property of coding systems, so using that terminology
> will I think provide good mnemonic hints both to protocol experts and
> to Emacs developers.

We also have a couple of handful of variables and function that just say
"charset" when it's in the context of network communications, like
`mail-parse-charset' and `message-posting-charset'.

I think the likelihood of any Emacs programmer that's interested in
networking libraries being confused by a parameter called :charset is
less than their being confused by a parameter called :mime-charset.

You come to this from a different angle, and is perhaps not all that
interested in networking protocols?  In that case, I understand your
hesitancy, but I...  think I'm right.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Lars Ingebrigtsen]

Anyway, progress on the with-url branch has been much more rapid than I
thought it would be, and it "basically works" now: You can use it to
browse the web.

What's left of the main part is to finish implementing the header/data
encoding framework and do some quality assurance on form submission.

I've reused some components of the old URL library, like the cookie and
cache bits, but I just discovered that the url-cache library is
basically ... er...  not very good.  That is, it doesn't implement
anything like a traditional web browser cache, with lifetime management
and heeding the required caching headers.  (In addition, it's not very,
er, privacy conscious, as it'll basically record every site you've been
to, and every thing you've looked at, in plain text.)  So I'll be
implementing a new cache, too.  *sigh*

My plan is then to merge this with master in a few weeks (well, not
really a merge, but more of a patch set), and then it'll be in Emacs
26.1.  And then we can mark url-retrieve and all the rest as obsolete in
Emacs 27, after with-url has grown the bits that people feel they need
to do the switchover.

In particular, with-url does not implement schemes like imap:, irc:,
news: and ldap: and all the other things that nobody uses any more.
(And most of which have been broken in Emacs for something like a
decade, I think: See the bug tracker for details.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 22 Jan 2017 18:13:53 +0300
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> > Anyway, it's all rather moot, since this is, of course, an asynchronous
> > library function, and you can't really send any Lisp-level signals in
> > that context.
> 
> That's why I mentioned threads.

With the current implementation of concurrency, threads cannot
usefully signal anything: if a thread raises a signal that is not
caught, it simply dies.  And you cannot catch such a signal from
another thread.

So your proposal would mean that the thread which runs the async code
sets some variable, and the application in the main thread then
signals an error based on that variable's value.  is that what you had
in mind?

Re: with-url

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 22 Jan 2017 16:55:01 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I want to cater to Emacs users and developers as well, and I think
> > mime-charset is a good compromise.  Once again, we have a
> > :mime-charset property of coding systems, so using that terminology
> > will I think provide good mnemonic hints both to protocol experts and
> > to Emacs developers.
> 
> We also have a couple of handful of variables and function that just say
> "charset" when it's in the context of network communications, like
> `mail-parse-charset' and `message-posting-charset'.
> 
> I think the likelihood of any Emacs programmer that's interested in
> networking libraries being confused by a parameter called :charset is
> less than their being confused by a parameter called :mime-charset.
> 
> You come to this from a different angle, and is perhaps not all that
> interested in networking protocols?  In that case, I understand your
> hesitancy, but I...  think I'm right.  :-)

Is there anything I can do to convince you to reconsider?

Re: with-url

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 22 Jan 2017 17:04:35 +0100
> 
> My plan is then to merge this with master in a few weeks (well, not
> really a merge, but more of a patch set), and then it'll be in Emacs
> 26.1.

Please add some documentation before you land this.

Thanks.

Re: with-url

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Please add some documentation before you land this.

Yes, of course.  It'll be replete with examples.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Dmitry Gutov]

On 22.01.2017 19:36, Eli Zaretskii wrote:

> With the current implementation of concurrency, threads cannot
> usefully signal anything: if a thread raises a signal that is not
> caught, it simply dies.  And you cannot catch such a signal from
> another thread.

The idea is that it would be caught by the caller in the same thread.

> So your proposal would mean that the thread which runs the async code
> sets some variable, and the application in the main thread then
> signals an error based on that variable's value.  is that what you had
> in mind?

The model would function somehow like this:

We're in a certain thread. Elisp code calls with-url, the internals make 
a request, _suspend the current thread_, and when the request is 
complete, the low-level code updates some variable and resumes the 
thread. After that, the code in that thread checks the result of the 
request, and, depending on it, may signal an error.

If the request can't be sent for some reason, so there's no point in 
suspending the thread, the internals can signal an error right away.

Point is, the errors is signaled in the same thread that made the 
request. And the execution looks synchronous from the point of view of 
the high-level code in that thread.

Not sure how/if that would work if the request is made from the main 
thread. Maybe that kind of code would require a minor rewrite.

Re: not with-url

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The construct sounds useful, but the name is not clear.
Would you please rename it before installing it?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: with-url

[Ted Zlatanov]

On Tue, 03 Jan 2017 12:47:15 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> 1) Return a Promise value. Emacs could use a standardized Promise structure
>> in other places, too.

SM> We have url-future (which got a "url-" prefix because we wanted to
SM> experiment with it before making it global).

I'm not sure if Stefan is in favor, so I'm either seconding or thirding.

If you feel url-future is not right for `with-url', how about
`with-url-future' with different call semantics but the same internals?

Ted

Re: with-url

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> If you feel url-future is not right for `with-url', how about
> `with-url-future' with different call semantics but the same internals?

I'm completely indifferent to the presence of a "future" version of
`with-url'.  :-)  I'm sure one can be trivially made from `with-url' if
suddenly all Emacs web programmers become future-crazy.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: with-url

[Ted Zlatanov]

On Fri, 27 Jan 2017 00:08:33 +0100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> If you feel url-future is not right for `with-url', how about
>> `with-url-future' with different call semantics but the same internals?

LI> I'm completely indifferent to the presence of a "future" version of
LI> `with-url'.  :-)  I'm sure one can be trivially made from `with-url' if
LI> suddenly all Emacs web programmers become future-crazy.

In that case I'll look into it once you merge your with-url branch.

Ted