more url-utils?

more url-utils?

[joakim]

I feel the url functions have become really useful with the addition of
the xml parser and shr, but we can do better still.

Would the defun below be suitable to include in url-util?
You nearly always want to work with the data part of the document.
The code snippet below occurs a lot in different packages.

A wrapper that downloads a file(like curl or wget) would be nice also.

(defun org-feed-skip-http-headers (buffer)
  "Remove HTTP headers from BUFFER, and return it.
Assumes headers are indeed present!"
  (with-current-buffer buffer
    (widen)
    (goto-char (point-min))
    (search-forward "\n\n")
    (delete-region (point-min) (point))
    buffer))
-- 
Joakim Verona

Re: more url-utils?

[Lars Magne Ingebrigtsen]

joakim@verona.se writes:

> Would the defun below be suitable to include in url-util?
> You nearly always want to work with the data part of the document.

Perhaps binding something like `url-return-data-only', and then having
url-retrieve strip the headers before returning the buffer would be
nice.

My guess is that more than 90% of the users of `url-retrieve' don't care
about the headers, so having it do the stripping would be convenient.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: more url-utils?

[joakim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> joakim@verona.se writes:
>
>> Would the defun below be suitable to include in url-util?
>> You nearly always want to work with the data part of the document.
>
> Perhaps binding something like `url-return-data-only', and then having
> url-retrieve strip the headers before returning the buffer would be
> nice.
>
> My guess is that more than 90% of the users of `url-retrieve' don't care
> about the headers, so having it do the stripping would be convenient.

Yes, normaly you only want the data, and an exception if that doesnt
happen properly. So, your proposal is good. Except a function argument
would expose the interface better IMHO. (But I understand why in
practice a binding might be more feasible. Hmm.)

-- 
Joakim Verona

Re: more url-utils?

[Lars Magne Ingebrigtsen]

joakim@verona.se writes:

> Yes, normaly you only want the data, and an exception if that doesnt
> happen properly. So, your proposal is good. Except a function argument
> would expose the interface better IMHO. (But I understand why in
> practice a binding might be more feasible. Hmm.)

For backwards compatibility (i.e., for out-of-bzr users) a binding might
be preferable (perhaps), but you're right, an extra optional parameter
`data-only-p' would make more sense.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: more url-utils?

[Dimitri Fontaine]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> My guess is that more than 90% of the users of `url-retrieve' don't care
> about the headers, so having it do the stripping would be convenient.

Right.  As a user of it, I concur.

Now maybe a new function url-delete-headers would be good enough.  It
would do the delete in current-buffer, and avoid code that currently
looks like that:

    ;; prune HTTP headers before save
    (goto-char (point-min))
    (re-search-forward "^$" nil 'move)
    (forward-char)
    (delete-region (point-min) (point))

Regards,
-- 
dim

Re: more url-utils?

[Jason Rumney]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> joakim@verona.se writes:
>
>> Would the defun below be suitable to include in url-util?
>> You nearly always want to work with the data part of the document.
>
> Perhaps binding something like `url-return-data-only', and then having
> url-retrieve strip the headers before returning the buffer would be
> nice.

It would be even nicer if the headers were parsed into a buffer local alist.

Re: more url-utils?

[Ted Zlatanov]

On Sun, 15 May 2011 22:26:13 +0800 Jason Rumney <jasonr@gnu.org> wrote: 

JR> It would be even nicer if the headers were parsed into a buffer
JR> local alist.

Yes, definitely.  That should be done by default regardless of the other
improvements discussed here.

On Mon, 16 May 2011 12:45:03 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> On Mon, 16 May 2011 09:19:18 -0600 Tom Tromey <tromey@redhat.com> wrote: 
Tom> IIRC, in package.el I had to special-case some URL types, because
Tom> headers are inconsistently used.  E.g., I think the 'ftp' method won't
Tom> leave headers in the buffer.  It would be nice if this bit were handled
Tom> by the url package rather than all the callers.

SM> Sounds like an inconsistency that needs fixing, indeed.  You might want
SM> to make a bug-report for it.

I had problems using a file:// URL with package.el, which made local
testing a pain.  The problem was exactly with the missing headers, same
as the ftp:// URLs.  So I'll second the bug report if Tom opens it, or
open a new one.

On Sun, 15 May 2011 03:59:52 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> For backwards compatibility (i.e., for out-of-bzr users) a binding might
LMI> be preferable (perhaps), but you're right, an extra optional parameter
LMI> `data-only-p' would make more sense.

After looking at many examples in Emacs, I can't find or think of any
need for the headers *as text* in the code I've seen.  I would:

1) rename the function, obsolete the old one, turn headers off by
default in the new version, and hunt down all the cases where code
removes them from the content.  Yes it's a pain.

2) find any real need for the headers as *text* (I found none).  If none
is found, just do the buffer-local alist Jason suggested, and maybe
provide all the headers as a single text string in case someone needs
them.

I realize this is more invasive but backwards compatibility means lots
of unnecessary code to remove the headers will keep being written.  I
volunteer to do both tasks above.

Ted

Re: more url-utils?

[Stefan Monnier]

> After looking at many examples in Emacs, I can't find or think of any
> need for the headers *as text* in the code I've seen.  I would:

That sounds right.


        Stefan

Re: more url-utils?

[Ted Zlatanov]

On Mon, 16 May 2011 16:03:19 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> After looking at many examples in Emacs, I can't find or think of any
>> need for the headers *as text* in the code I've seen.  I would:

SM> That sounds right.

...so can I go ahead with my proposal or would you rather be
conservative and keep the old behavior, making API users turn off
headers explicitly?

Ted

Re: more url-utils?

[Lennart Borgman]

2011/5/16 Ted Zlatanov <tzz@lifelogs.com>:
> On Mon, 16 May 2011 16:03:19 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
>>> After looking at many examples in Emacs, I can't find or think of any
>>> need for the headers *as text* in the code I've seen.  I would:
>
> SM> That sounds right.
>
> ...so can I go ahead with my proposal or would you rather be
> conservative and keep the old behavior, making API users turn off
> headers explicitly?

I would go for Julien's suggestion with a function
`url-remove-headers' which also parsed the headers and returned a
alist of the headers.

Re: more url-utils?

[Ted Zlatanov]

On Mon, 16 May 2011 21:22:02 +0200 Lennart Borgman <lennart.borgman@gmail.com> wrote: 

LB> 2011/5/16 Ted Zlatanov <tzz@lifelogs.com>:
>> On Mon, 16 May 2011 16:03:19 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> 
>>>> After looking at many examples in Emacs, I can't find or think of any
>>>> need for the headers *as text* in the code I've seen.  I would:
>> 
SM> That sounds right.
>> 
>> ...so can I go ahead with my proposal or would you rather be
>> conservative and keep the old behavior, making API users turn off
>> headers explicitly?

LB> I would go for Julien's suggestion with a function
LB> `url-remove-headers' which also parsed the headers and returned a
LB> alist of the headers.

I'm trying to avoid that approach: we just established the headers are
almost never necessary in the buffer.  So we should not ask the API
users to do anything extra to remove them.

Ted

Re: more url-utils?

[Lennart Borgman]

2011/5/16 Ted Zlatanov <tzz@lifelogs.com>:
> On Mon, 16 May 2011 21:22:02 +0200 Lennart Borgman <lennart.borgman@gmail.com> wrote:
>
> LB> 2011/5/16 Ted Zlatanov <tzz@lifelogs.com>:
>>> On Mon, 16 May 2011 16:03:19 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>>
>>>>> After looking at many examples in Emacs, I can't find or think of any
>>>>> need for the headers *as text* in the code I've seen.  I would:
>>>
> SM> That sounds right.
>>>
>>> ...so can I go ahead with my proposal or would you rather be
>>> conservative and keep the old behavior, making API users turn off
>>> headers explicitly?
>
> LB> I would go for Julien's suggestion with a function
> LB> `url-remove-headers' which also parsed the headers and returned a
> LB> alist of the headers.
>
> I'm trying to avoid that approach: we just established the headers are
> almost never necessary in the buffer.  So we should not ask the API
> users to do anything extra to remove them.

It can be combined with your approach with an extra argument.

Re: more url-utils?

[Ted Zlatanov]

On Mon, 16 May 2011 21:33:55 +0200 Lennart Borgman <lennart.borgman@gmail.com> wrote: 

LB> 2011/5/16 Ted Zlatanov <tzz@lifelogs.com>:
>> On Mon, 16 May 2011 21:22:02 +0200 Lennart Borgman <lennart.borgman@gmail.com> wrote:
LB> I would go for Julien's suggestion with a function
LB> `url-remove-headers' which also parsed the headers and returned a
LB> alist of the headers.
>> 
>> I'm trying to avoid that approach: we just established the headers are
>> almost never necessary in the buffer.  So we should not ask the API
>> users to do anything extra to remove them.

LB> It can be combined with your approach with an extra argument.

The existence of `url-remove-headers' is a symptom of a bad API and
should be unnecessary, together with extra arguments.

The usage would be (remember, `url-retrieve-synchronously' is obsoleted
in my proposal in favor or `url-retrieve2-synchronously'):

(if (setq buf (url-retrieve2-synchronously url))
     (with-current-buffer buf
       (message "Header x is %s" (assq 'x url-headers-alist))
       (message "Header y is %s" (assoc "y" url-headers-alist))
       (message "The headers as a string are %s" url-headers-as-string)
       (message "data is %s" (buffer-string))
       ...)
  (...handle errors here...))

which is, I think, how everyone wants to use this function based on
browsing a whole bunch of code.  So inserting the headers would be just
(insert url-headers-as-string) and too trivial to provide as a function.

Ted

Re: more url-utils?

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

On Mon, May 16 2011, Ted Zlatanov wrote:

> I'm trying to avoid that approach: we just established the headers are
> almost never necessary in the buffer.  So we should not ask the API
> users to do anything extra to remove them.

That can be true. When you fetch an URL, you want to know if the fetch
result is a 200 OK or a 404 Not Found. That's the absolute minimum thing
you want to do if you plan to write correct code.
So just dropping the header sounds like a very bad programming practice,
and not something I'd encourage.

I wrote code with url.el. And what I always did is (search-forward
"\n\n") to move point between headers and content, and then parse the
headers to at least check the return code (expecting 200 usually).

I often also needed the Content-Type (so clearly an alist of parsed
headers would be welcome).

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: more url-utils?

[Ted Zlatanov]

On Mon, 16 May 2011 21:52:43 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Mon, May 16 2011, Ted Zlatanov wrote:
>> I'm trying to avoid that approach: we just established the headers are
>> almost never necessary in the buffer.  So we should not ask the API
>> users to do anything extra to remove them.

JD> That can be true. When you fetch an URL, you want to know if the fetch
JD> result is a 200 OK or a 404 Not Found. That's the absolute minimum thing
JD> you want to do if you plan to write correct code.
JD> So just dropping the header sounds like a very bad programming practice,
JD> and not something I'd encourage.

You won't get a HTTP status (or any headers, sometimes) from a non-HTTP
URL.  So the status is only useful in some cases and relying on it makes
the code inflexible, depending on the transport protocol's artifacts.

JD> I wrote code with url.el. And what I always did is (search-forward
JD> "\n\n") to move point between headers and content, and then parse the
JD> headers to at least check the return code (expecting 200 usually).

That's a symptom of a bad API that tied you to HTTP headers and what I
was trying to eliminate.

On Mon, 16 May 2011 16:53:37 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> After looking at many examples in Emacs, I can't find or think of any
>>>> need for the headers *as text* in the code I've seen.  I would:
SM> That sounds right.
>> ...so can I go ahead with my proposal or would you rather be
>> conservative and keep the old behavior, making API users turn off
>> headers explicitly?

SM> There are 3 parts.
SM> - removal of header text.
SM> - backward compatibility.
SM> - providing header info in a non-textual format.

Ah, OK.  So my proposal does not fit because you want backwards
compatibility.  I was hoping to eliminate the bad API altogether but
this is at least a step forward.  Let me know if you want me to
implement any parts of this.

Thanks
Ted

Re: more url-utils?

[Stefan Monnier]

> Ah, OK.  So my proposal does not fit because you want backwards
> compatibility.

I think that's right, although I'm not completely clear on what is
exactly your proposal.

> I was hoping to eliminate the bad API altogether but this is at least
> a step forward.  Let me know if you want me to implement any parts
> of this.

If you can provide a patch that give us a clean new API together with
some backward compatibility layer, that would be great.


        Stefan

Re: more url-utils?

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

On Mon, May 16 2011, Ted Zlatanov wrote:

> That's a symptom of a bad API that tied you to HTTP headers and what I
> was trying to eliminate.

Indeed Ted, you're right. I was only thinking about HTTP, stupid I am. :)

This probably means that the API should be changed (or should evolve by
addition/deprecation at least) to return a buffer with data, but let
access things like a status code (meaning something more general than
HTTP code) and possibly extra data like the HTTP headers or whatever can
be useful contextual information for the retrieved URL.

I don't know every backend url.el supports, so I can't imagine what a
good abstraction might be. :)

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: more url-utils?

[Stefan Monnier]

>>> After looking at many examples in Emacs, I can't find or think of any
>>> need for the headers *as text* in the code I've seen.  I would:
SM> That sounds right.
> ...so can I go ahead with my proposal or would you rather be
> conservative and keep the old behavior, making API users turn off
> headers explicitly?

There are 3 parts.
- removal of header text.
- backward compatibility.
- providing header info in a non-textual format.

All three are important.


        Stefan

Re: more url-utils?

[joakim]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> After looking at many examples in Emacs, I can't find or think of any
>>>> need for the headers *as text* in the code I've seen.  I would:
> SM> That sounds right.
>> ...so can I go ahead with my proposal or would you rather be
>> conservative and keep the old behavior, making API users turn off
>> headers explicitly?
>
> There are 3 parts.
> - removal of header text.
> - backward compatibility.
> - providing header info in a non-textual format.
>
> All three are important.

Teds proposal addresses all three concerns right?
- The new API does what coders expect, no information is lost
- the old api is retained but marked obsolete
- headers are accessible in alist format

Its more work but Teds volunteered to do it. Great!

>
>         Stefan

-- 
Joakim Verona

Re: more url-utils?

[Ted Zlatanov]

On Mon, 16 May 2011 23:07:49 +0200 joakim@verona.se wrote: 

j> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>>> After looking at many examples in Emacs, I can't find or think of any
>>>>> need for the headers *as text* in the code I've seen.  I would:
SM> That sounds right.
>>> ...so can I go ahead with my proposal or would you rather be
>>> conservative and keep the old behavior, making API users turn off
>>> headers explicitly?
>> 
>> There are 3 parts.
>> - removal of header text.
>> - backward compatibility.
>> - providing header info in a non-textual format.
>> 
>> All three are important.

j> Teds proposal addresses all three concerns right?
j> - The new API does what coders expect, no information is lost
j> - the old api is retained but marked obsolete
j> - headers are accessible in alist format

j> Its more work but Teds volunteered to do it. Great!

I don't think marking the old API obsolete will preserve backwards
compatibility.  But if Stefan thinks so, I'll do it according to my
proposal.  Otherwise he'll have to decide what the API will be.

I don't mind the extra work if it improves Emacs.  Right now using
url-*.el to just get some data is pretty painful compared to Perl's LWP,
for instance.

Ted

Re: more url-utils?

[Lennart Borgman]

2011/5/16 Ted Zlatanov <tzz@lifelogs.com>:
> On Mon, 16 May 2011 23:07:49 +0200 joakim@verona.se wrote:
>
> j> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>>>> After looking at many examples in Emacs, I can't find or think of any
>>>>>> need for the headers *as text* in the code I've seen.  I would:
> SM> That sounds right.
>>>> ...so can I go ahead with my proposal or would you rather be
>>>> conservative and keep the old behavior, making API users turn off
>>>> headers explicitly?
>>>
>>> There are 3 parts.
>>> - removal of header text.
>>> - backward compatibility.
>>> - providing header info in a non-textual format.
>>>
>>> All three are important.
>
> j> Teds proposal addresses all three concerns right?
> j> - The new API does what coders expect, no information is lost
> j> - the old api is retained but marked obsolete
> j> - headers are accessible in alist format
>
> j> Its more work but Teds volunteered to do it. Great!
>
> I don't think marking the old API obsolete will preserve backwards
> compatibility.  But if Stefan thinks so, I'll do it according to my
> proposal.  Otherwise he'll have to decide what the API will be.
>
> I don't mind the extra work if it improves Emacs.  Right now using
> url-*.el to just get some data is pretty painful compared to Perl's LWP,
> for instance.

Yes, it is pretty painful and hardly documented. Take for example
url-http-parse-headers. Anyone can say what it does by just reading
the doc string?

Improvements will save a lot of time!

Re: more url-utils?

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 16 May 2011 23:07:49 +0200 joakim@verona.se wrote: 
>
> j> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>>>>> After looking at many examples in Emacs, I can't find or think of any
>>>>>> need for the headers *as text* in the code I've seen.  I would:
> SM> That sounds right.
>>>> ...so can I go ahead with my proposal or would you rather be
>>>> conservative and keep the old behavior, making API users turn off
>>>> headers explicitly?
>>> 
>>> There are 3 parts.
>>> - removal of header text.
>>> - backward compatibility.
>>> - providing header info in a non-textual format.
>>> 
>>> All three are important.
>
> j> Teds proposal addresses all three concerns right?
> j> - The new API does what coders expect, no information is lost
> j> - the old api is retained but marked obsolete
> j> - headers are accessible in alist format
>
> j> Its more work but Teds volunteered to do it. Great!
>
> I don't think marking the old API obsolete will preserve backwards
> compatibility.  But if Stefan thinks so, I'll do it according to my
> proposal.  Otherwise he'll have to decide what the API will be.
>
> I don't mind the extra work if it improves Emacs.  Right now using
> url-*.el to just get some data is pretty painful compared to Perl's LWP,
> for instance.

True. We had a little inhouse "competition" comparing implementations of
reading and parsing an internal REST datasource. My Emacs implementation
was pretty lean compared with Android/Iphone implementations, but the
Clojure implementation was five lines of code. Perls would be 5
characters I presume :)

>
> Ted
>

-- 
Joakim Verona

Re: more url-utils?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> There are 3 parts.
> - removal of header text.
> - backward compatibility.
> - providing header info in a non-textual format.

Adding a new optional parameter `no-headers' (or something) to
`url-retrieve' (and -synchronous) could do the trick, but the caller
(who wants no headers) have to do some pretty deep inspection to find
out whether this version of `url-retrieve' has that parameter or not.

So how about a new function, with a signature something like this:

(defun url-fetch (url &keywords callback cbargs silent))

The `callback' is optional, and if not provided, makes `url-fetch' be
synchronous.

(Too bad Emacs doesn't really have a culture for closures, because
then we could get rid of cbargs, too.  Being able to say stuff like

(defun save-url (file)
  (url-fetch url :callback (lambda (status)
                             (write-region (point-min) (point-max) file))))

would be kinda elegant, I think, instead of

(defun save-url (file)
  (url-fetch url :callback (lambda (status file)
                             (write-region (point-min) (point-max) file))
                 :cbargs (list file)))

But that's a different discussion, perhaps.  :-)
                             
-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: more url-utils?

[Stefan Monnier]

>> There are 3 parts.
>> - removal of header text.
>> - backward compatibility.
>> - providing header info in a non-textual format.

> Adding a new optional parameter `no-headers' (or something) to
> `url-retrieve' (and -synchronous) could do the trick, but the caller
> (who wants no headers) have to do some pretty deep inspection to find
> out whether this version of `url-retrieve' has that parameter or not.

Agreed.

> So how about a new function, with a signature something like this:

> (defun url-fetch (url &keywords callback cbargs silent))

Sounds about right.

> The `callback' is optional, and if not provided, makes `url-fetch' be
> synchronous.

Yes, I used the same convention in mpc.el.

> (Too bad Emacs doesn't really have a culture for closures, because
> then we could get rid of cbargs, too.  Being able to say stuff like

This change to URL would be for Emacs-24 which does support closures
natively (and I encourage people to use this facility), so we could
definitely drop the `cbargs'.


        Stefan

Re: more url-utils?

[Ted Zlatanov]

On Sat, 14 May 2011 12:32:55 +0200 joakim@verona.se wrote: 

j> Would the defun below be suitable to include in url-util?
j> You nearly always want to work with the data part of the document.
j> The code snippet below occurs a lot in different packages.

j> A wrapper that downloads a file(like curl or wget) would be nice also.

j> (defun org-feed-skip-http-headers (buffer)
...

Yes to both, but like Lars I think the header-skipping should be an
option for the current API and not a new function.

Ted

Re: more url-utils?

[Stefan Monnier]

j> Would the defun below be suitable to include in url-util?
j> You nearly always want to work with the data part of the document.
j> The code snippet below occurs a lot in different packages.

j> A wrapper that downloads a file(like curl or wget) would be nice also.

j> (defun org-feed-skip-http-headers (buffer)
> ...

> Yes to both, but like Lars I think the header-skipping should be an
> option for the current API and not a new function.

While Emacs uses them a lot, dynamically-scoped variables used as
implicit extra parameters are bad.  In many cases, it's difficult to
avoid using them, but in this case, calling a function after the
url-retrieve call works just as well, so there's no excuse.


        Stefan

Re: more url-utils?

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> While Emacs uses them a lot, dynamically-scoped variables used as
Stefan> implicit extra parameters are bad.  In many cases, it's difficult to
Stefan> avoid using them, but in this case, calling a function after the
Stefan> url-retrieve call works just as well, so there's no excuse.

IIRC, in package.el I had to special-case some URL types, because
headers are inconsistently used.  E.g., I think the 'ftp' method won't
leave headers in the buffer.  It would be nice if this bit were handled
by the url package rather than all the callers.

Tom

Re: more url-utils?

[Stefan Monnier]

Stefan> While Emacs uses them a lot, dynamically-scoped variables used as
Stefan> implicit extra parameters are bad.  In many cases, it's difficult to
Stefan> avoid using them, but in this case, calling a function after the
Stefan> url-retrieve call works just as well, so there's no excuse.
> IIRC, in package.el I had to special-case some URL types, because
> headers are inconsistently used.  E.g., I think the 'ftp' method won't
> leave headers in the buffer.  It would be nice if this bit were handled
> by the url package rather than all the callers.

Sounds like an inconsistency that needs fixing, indeed.  You might want
to make a bug-report for it.


        Stefan


PS: Of course, that doesn't mean that the way to distinguish the "with
header" and "without header" cases should be via
a dynamically-scoped variable.

Re: more url-utils?

[joakim]

Tom Tromey <tromey@redhat.com> writes:

>>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> Stefan> While Emacs uses them a lot, dynamically-scoped variables used as
> Stefan> implicit extra parameters are bad.  In many cases, it's difficult to
> Stefan> avoid using them, but in this case, calling a function after the
> Stefan> url-retrieve call works just as well, so there's no excuse.
>
> IIRC, in package.el I had to special-case some URL types, because
> headers are inconsistently used.  E.g., I think the 'ftp' method won't
> leave headers in the buffer.  It would be nice if this bit were handled
> by the url package rather than all the callers.

Changing 
  (url-retrieve-synchronously URL)
To
  (url-retrieve-synchronously URL &optional remove-headers)

ought to sattisfy all participants in this thread?

and it could be done somewhat like this:

(defun url-retrieve-synchronously (url &optional remove-headers)
  (let ((buffer (url-retrieve-synchronously-old-implementation url))
    (if remove-headers 
    (url-remove-headers buffer
    ))))

(defun url-remove-headers (buffer)
      (with-current-buffer buffer
        (widen)
        (goto-char (point-min))
        (search-forward "\n\n")
        (delete-region (point-min) (point))
        buffer)
)
but we would also need to look out for the special cases Tom mentioned.
Good enough? 

url-remove-headers is separate so it can be re-used in a url-retrieve
callback for instance.


>
> Tom

-- 
Joakim Verona

Re: more url-utils?

[Lennart Borgman]

On Mon, May 16, 2011 at 7:04 PM,  <joakim@verona.se> wrote:
>
> Changing
>  (url-retrieve-synchronously URL)
> To
>  (url-retrieve-synchronously URL &optional remove-headers)
>
> ought to sattisfy all participants in this thread?

I did not follow, but you certainly want to check the headers many
times. Perhaps optionally return a list with buffer + headers?

Re: more url-utils?

[Stefan Monnier]

> Changing
>   (url-retrieve-synchronously URL)
> To
>   (url-retrieve-synchronously URL &optional remove-headers)
> ought to sattisfy all participants in this thread?

Let's first add the extra "remove-headers" function and fix
Tom's issue.


        Stefan

Re: more url-utils?

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 711 bytes --]

On Mon, May 16 2011, joakim@verona.se wrote:

> Changing 
>   (url-retrieve-synchronously URL)
> To
>   (url-retrieve-synchronously URL &optional remove-headers)
>
> ought to sattisfy all participants in this thread?

Not sure it's a good idea.

Having a function to remove/parse headers is more useful, and calling it
like:

(with-current-buffer (url-retrieve-synchronously URL)
  (url-remove-headers)
  (stuff))

seems enough to me.

And, actually, most of the time you do not want especially to remove
them but at least to go after them. This is a very common code to
search-forward \n\n to move point after the headers.

My 2¢,

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: more url-utils?

[Chong Yidong]

joakim@verona.se writes:

> Would the defun below be suitable to include in url-util?
> You nearly always want to work with the data part of the document.
> The code snippet below occurs a lot in different packages.
>
> A wrapper that downloads a file(like curl or wget) would be nice also.
>
> (defun org-feed-skip-http-headers (buffer)
>   "Remove HTTP headers from BUFFER, and return it.
> Assumes headers are indeed present!"
>   (with-current-buffer buffer
>     (widen)
>     (goto-char (point-min))
>     (search-forward "\n\n")
>     (delete-region (point-min) (point))
>     buffer))

Good idea, though maybe the wrapper function should incorporate
url-http-parse-response too (see the package-handle-response function in
package.el).  Please feel free to propose a patch.

Re: more url-utils?

[Ted Zlatanov]

On Tue, 17 May 2011 00:04:50 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Ah, OK.  So my proposal does not fit because you want backwards
>> compatibility.

SM> I think that's right, although I'm not completely clear on what is
SM> exactly your proposal.

This is my proposal, after thinking about it for a bit.

Current top-level API:

Dynamic binding: `url-request-data', `url-request-method' and
`url-request-extra-headers' according to the docstring; also according
to a comment: url-standalone-mode, url-gateway-unplugged,
w3-honor-stylesheets, url-confirmation-func, url-cookie-multiple-line,
url-cookie-{{,secure-}storage,confirmation}.

(defun url-retrieve (url callback &optional cbargs silent) ...)
(defun url-retrieve-synchronously (url) ...)

Both of these result in a buffer containing

"[HEADERS]

[BODY]"

Proposed API (based on code review throughout Emacs and my feeling for
what would be "natural" in ELisp):

;;; buffer-local variables: url-retrieved-ok url-headers-alist url-headers-as-string
(defmacro with-url-contents-buffer (url url-options-alist callback-closure &rest body)
...)

1) Synchronous usage with errors:

(with-url-contents-buffer "http://host" '((url-request-method "POST")) nil
  (message "this will run SYNCHRONOUSLY since the callback-closure is nil")
  (message "this could throw an error")
  (message "Header x is %s" (assq 'x url-headers-alist))
  (message "Header y is %s" (assoc "y" url-headers-alist))
  (message "The headers as a string are %s" url-headers-as-string)
  (when url-retrieved-ok (message "The retrieval went fine"))
  (message "data is %s" (buffer-string))

2) Synchronous usage without errors:

(with-url-contents-buffer "http://host" '((url-request-method "POST")) 'noerror
  (message "this will run SYNCHRONOUSLY since the callback-closure is nil")
  (message "this will not throw an error"))

3) Asynchronous usage with a callback (`url-headers-alist',
`url-headers-as-string', and `url-retrieved-ok' are still available):

(with-url-contents-buffer "http://host" '((url-request-method "POST")) callback-closure
  (message "this will run AFTER the retrieval and the callback-closure are done"))

4) Asynchronous usage with no callback (`url-headers-alist',
`url-headers-as-string', and `url-retrieved-ok' are still available):

(with-url-contents-buffer "http://host" '((url-request-method "POST")) t
  (message "this will run AFTER the retrieval is done, there is no callback"))

Implicitly we don't have an asynchronous mode that throws errors.

I also think we should have an incremental mode where each retrieved
chunk is given to the callback in the buffer as it arrives.  That would
let us handle large downloads better.  The API would not change, though,
only the `url-options-alist' would have a `url-incremental' key and the
callback would look at the buffer-local variable
`url-incremental-byte-offset'.

SM> If you can provide a patch that give us a clean new API together with
SM> some backward compatibility layer, that would be great.

After we've settled on the API, please.  I think the above may be too
different for many :)

Ted

Re: more url-utils?

[Stefan Monnier]

> 3) Asynchronous usage with a callback (`url-headers-alist',
> `url-headers-as-string', and `url-retrieved-ok' are still available):

> (with-url-contents-buffer "http://host" '((url-request-method "POST")) callback-closure
>   (message "this will run AFTER the retrieval and the callback-closure are done"))

In which sense is this asynchronous?  What's the difference between the
code run in call-back-closure from the code in `body'?


        Stefan

Re: more url-utils?

[Ted Zlatanov]

On Tue, 17 May 2011 21:11:23 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> 3) Asynchronous usage with a callback (`url-headers-alist',
>> `url-headers-as-string', and `url-retrieved-ok' are still available):

>> (with-url-contents-buffer "http://host" '((url-request-method "POST")) callback-closure
>> (message "this will run AFTER the retrieval and the callback-closure are done"))

SM> In which sense is this asynchronous?

Same as the current url.el code.

SM> What's the difference between the code run in call-back-closure from
SM> the code in `body'?

Since I'm proposing a macro, the callback-closure is funcalled while the
body is inlined.

Ted

Re: more url-utils?

[Stefan Monnier]

>>> 3) Asynchronous usage with a callback (`url-headers-alist',
>>> `url-headers-as-string', and `url-retrieved-ok' are still available):

>>> (with-url-contents-buffer "http://host" '((url-request-method "POST")) callback-closure
>>> (message "this will run AFTER the retrieval and the callback-closure are done"))

SM> In which sense is this asynchronous?
> Same as the current url.el code.

So you mean that, while the `body' is run after the retrieval is done,
the code after with-url-contents-buffer will be run earlier?

SM> What's the difference between the code run in call-back-closure from
SM> the code in `body'?
> Since I'm proposing a macro, the callback-closure is funcalled while the
> body is inlined.

I don't know what you mean by "funcalled vs inlined" (in my world,
"inlined" means it's implemented slightly differently, but the behavior
is identical).


        Stefan

Re: more url-utils?

[Ted Zlatanov]

On Wed, 18 May 2011 09:16:07 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> 3) Asynchronous usage with a callback (`url-headers-alist',
>>>> `url-headers-as-string', and `url-retrieved-ok' are still available):

>>>> (with-url-contents-buffer "http://host" '((url-request-method "POST")) callback-closure
>>>> (message "this will run AFTER the retrieval and the callback-closure are done"))

SM> In which sense is this asynchronous?
>> Same as the current url.el code.

SM> So you mean that, while the `body' is run after the retrieval is done,
SM> the code after with-url-contents-buffer will be run earlier?

Yes in the asynchronous case.  This is just one case out of several the
API supports and without a body this case behaves just like
`url-retrieve'.  If you don't think the asynchronous body is a good idea
because it's hard to read, we can eliminate it by providing a separate
function for the asynchronous case that only takes a callback-closure.
To me it seems reasonable the way I've done it.

SM> What's the difference between the code run in call-back-closure from
SM> the code in `body'?

The body is easier to write explicitly but the callback-closure offers
more flexibility and backwards compatibility.  They are more or less the
same otherwise (see below for the pseudocode), it's just different types
of syntactic sugar for different situations.  I think the majority of
uses will be in the synchronous mode, where `with-url-contents-buffer'
is very convenient.

>> Since I'm proposing a macro, the callback-closure is funcalled while the
>> body is inlined.

SM> I don't know what you mean by "funcalled vs inlined" (in my world,
SM> "inlined" means it's implemented slightly differently, but the behavior
SM> is identical).

They are different because the resulting code is different, I don't know
how to explain it any better.  Maybe if I show the pseudocode?

The body passed to the macro is inlined and the callback-closure is not,
so the macro will look like this in the synchronous case:

`(with-temp-buffer
   ... get url contents ...
   ,@(when callback-closure '(funcall callback-closure))
   ,@body)

I'll need to set up a lexical bind of the callback-closure to make this
work, but I think it's reasonable otherwise.  I can write the actual
code if you agree.  Or you can tell me how the API should look.

The asynchronous case will be a bit more complicated, wrapping the body
in a temporary closure and using that as a second callback.  But as I
said above we can disallow the body in the asynchronous mode if you
think that's too complicated.

Ted

Re: more url-utils?

[Stefan Monnier]

SM> What's the difference between the code run in call-back-closure from
SM> the code in `body'?
> The body is easier to write explicitly but the callback-closure offers
> more flexibility and backwards compatibility.  They are more or less the
> same otherwise (see below for the pseudocode), it's just different types
> of syntactic sugar for different situations.  I think the majority of
> uses will be in the synchronous mode, where `with-url-contents-buffer'
> is very convenient.

I think you're just afraid of lambda ;-0

Here's my proposal:

(defun url-fetch (url &optional silent callback)
  "Fetch URL into current buffer.
If CALLBACK is nil, do it synchronously and return a status flag."

But I also like the idea of passing url-request-method and such as
explicit arguments.  And I'm not sure what it should return if CALLBACK
is non-nil (both in the case where the request is performed
asynchronously and when it's performed synchronously).


        Stefan

Re: more url-utils?

[Ted Zlatanov]

On Wed, 18 May 2011 11:09:25 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> What's the difference between the code run in call-back-closure from
SM> the code in `body'?
>> The body is easier to write explicitly but the callback-closure offers
>> more flexibility and backwards compatibility.  They are more or less the
>> same otherwise (see below for the pseudocode), it's just different types
>> of syntactic sugar for different situations.  I think the majority of
>> uses will be in the synchronous mode, where `with-url-contents-buffer'
>> is very convenient.

SM> I think you're just afraid of lambda ;-0

...or are you afraid of macros? ;)

SM> Here's my proposal:

SM> (defun url-fetch (url &optional silent callback)
SM>   "Fetch URL into current buffer.
SM> If CALLBACK is nil, do it synchronously and return a status flag."

OK.  The status flag should be nil on failure and t on success.  Let's
not tie the `url-fetch' status flag to the HTTP status.

SM> But I also like the idea of passing url-request-method and such as
SM> explicit arguments.

(defun* url-fetch
    (url &rest spec
         &key silent callback request-data request-method
              request-extra-headers standalone-mode gateway-unplugged
              honor-stylesheets confirmation-func cookie-multiple-line
              cookie-storage cookie-confirmation cookie-secure-storage
              cookie-secure-confirmation
         &allow-other-keys)
...)

This has less compile-time parameter validation but is much more
flexible long-term.

SM> And I'm not sure what it should return if CALLBACK is non-nil (both
SM> in the case where the request is performed asynchronously and when
SM> it's performed synchronously).

By the docstring you gave, if CALLBACK is non-nil, it must be
asynchronous.  So CALLBACK can't be used synchronously.

In the asynchronous mode we could return a lambda (see that? I used a
lambda!) that can be evaluated to wait until CALLBACK completes and then
returns whatever CALLBACK returned.

Ted

Re: more url-utils?

[Stefan Monnier]

SM> I think you're just afraid of lambda ;-0
> ...or are you afraid of macros? ;)

That's partly true.  But your `body' is nothing more than the body of
the callback if passed as a lambda (and indeed within the macro you'll
have to wrap `body' with a `lambda' to use it in the process-sentinel),
so providing both `callback' and `body' arguments is not very useful.
And of course `callback' has the advantage that it takes an argument
(which will of course be `status') whereas `body' doesn't have such an
easy and obvious access to the status.

SM> But I also like the idea of passing url-request-method and such as
SM> explicit arguments.

> (defun* url-fetch
>     (url &rest spec
>          &key silent callback request-data request-method
>               request-extra-headers standalone-mode gateway-unplugged
>               honor-stylesheets confirmation-func cookie-multiple-line
>               cookie-storage cookie-confirmation cookie-secure-storage
>               cookie-secure-confirmation
>          &allow-other-keys)
> ...)

I'm not sure all those args make sense, but yes, that sounds
about right.  We should think hard about what those args should be,
within the larger context of URL rather than only http.

SM> And I'm not sure what it should return if CALLBACK is non-nil (both
SM> in the case where the request is performed asynchronously and when
SM> it's performed synchronously).
> By the docstring you gave, if CALLBACK is non-nil, it must be
> asynchronous.  So CALLBACK can't be used synchronously.

No, callback just makes it possible to do the request asynchronously,
but it can still be performed synchronously and for some URLs we may not
know how to perform them asynchronously (as the docstring of
url-retrieve already explains).

> In the asynchronous mode we could return a lambda (see that? I used a
> lambda!) that can be evaluated to wait until CALLBACK completes and then
                       ^^^^^^^^^
                        called

> returns whatever CALLBACK returned.

That's a good idea.  Kind of like a future.


        Stefan

Re: more url-utils?

[Ted Zlatanov]

On Wed, 18 May 2011 19:05:57 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> But your `body' is nothing more than the body of the callback if
SM> passed as a lambda (and indeed within the macro you'll have to wrap
SM> `body' with a `lambda' to use it in the process-sentinel), so
SM> providing both `callback' and `body' arguments is not very useful.

I thought it was more readable to cradle statements in a "with-xyz"
macro (e.g. `with-temp-file', `with-parsed-tramp-file-name') than to
craft a lambda for a callback.  It's not a big deal, I can write the
macro on top of `url-fetch' anyhow.

>> (defun* url-fetch
>> (url &rest spec
>> &key silent callback request-data request-method
>> request-extra-headers standalone-mode gateway-unplugged
>> honor-stylesheets confirmation-func cookie-multiple-line
>> cookie-storage cookie-confirmation cookie-secure-storage
>> cookie-secure-confirmation
>> &allow-other-keys)
>> ...)

SM> I'm not sure all those args make sense, but yes, that sounds
SM> about right.  We should think hard about what those args should be,
SM> within the larger context of URL rather than only http.

Yeah.  So perhaps we simply write `url-http-fetch' that specifies the
base `url-fetch' props and then adds its own list on top.

SM> No, callback just makes it possible to do the request asynchronously,
SM> but it can still be performed synchronously and for some URLs we may not
SM> know how to perform them asynchronously (as the docstring of
SM> url-retrieve already explains).

OK.

>> In the asynchronous mode we could return a lambda (see that? I used a
>> lambda!) that can be evaluated to wait until CALLBACK completes and then
SM>                        ^^^^^^^^^
SM>                         called

>> returns whatever CALLBACK returned.

SM> That's a good idea.  Kind of like a future.

All right, I'll provide that.  Are you OK with the defun* approach and
having each `url-{http,ftp,etc}-fetch' function build on the base
`url-fetch'?

Also I think the callback should get the status and then &rest plist.
The plist will depend on the URL protocol but there will be common keys
to determine the protocol, if there were headers, etc.  That's a little
bit less functional but the data will not be hidden in buffer-local
variables like it is now (although those will still be available).
There will be a lot more function parameters on the stack, though--I
don't know if that's a problem.

Ted

Re: more url-utils?

[Stefan Monnier]

> Yeah.  So perhaps we simply write `url-http-fetch' that specifies the
> base `url-fetch' props and then adds its own list on top.

Hmm... maybe that's the right approach, indeed, but I wouldn't know
(for lack of experience).

> All right, I'll provide that.  Are you OK with the defun* approach and

Sure.

> having each `url-{http,ftp,etc}-fetch' function build on the base
> `url-fetch'?

It doesn't sound too bad, tho maybe I'd have expected that url-fetch calls
one of url-<transport>-fetch, rather than the other way around.

> Also I think the callback should get the status and then &rest plist.
> The plist will depend on the URL protocol but there will be common keys
> to determine the protocol, if there were headers, etc.  That's a little
> bit less functional but the data will not be hidden in buffer-local
> variables like it is now (although those will still be available).

I think it's OK to use buffer-local vars to keep things like
url-header-alist.
[...]
wait, it's OK to use buffer-local vars in a buffer created for the
occasion (as in url-retrieve) but indeed if url-fetch stores the result
in the current buffer as I suggested, then it's ugly to modify the
buffer's local vars as a side-effect and it's better to pass it as an
argument to the callback.

> There will be a lot more function parameters on the stack, though--I
> don't know if that's a problem.

It's not a problem but I'd much prefer an "alist" than a "&rest
plist".


        Stefan

Re: more url-utils?

[Ted Zlatanov]

On Wed, 18 May 2011 23:28:41 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> All right, I'll provide that.  Are you OK with the defun* approach and

SM> Sure.

OK, it's on my TODO list.

>> Also I think the callback should get the status and then &rest plist.
>> The plist will depend on the URL protocol but there will be common keys
>> to determine the protocol, if there were headers, etc.  That's a little
>> bit less functional but the data will not be hidden in buffer-local
>> variables like it is now (although those will still be available).

SM> I think it's OK to use buffer-local vars to keep things like
SM> url-header-alist.
SM> [...]
SM> wait, it's OK to use buffer-local vars in a buffer created for the
SM> occasion (as in url-retrieve) but indeed if url-fetch stores the result
SM> in the current buffer as I suggested, then it's ugly to modify the
SM> buffer's local vars as a side-effect and it's better to pass it as an
SM> argument to the callback.

Right, `url-fetch' no longer implies `with-temp-buffer'.

>> There will be a lot more function parameters on the stack, though--I
>> don't know if that's a problem.

SM> It's not a problem but I'd much prefer an "alist" than a "&rest
SM> plist".

OK.  But if the status is nil we couldn't get any data so there are no
URL properties.  So we can have just one callback parameter "url-info"
which is nil if the request failed and an alist if it worked.

How would the callback find out what actually failed?  Does it even need
to?  I think it's enough to know the request failed, the user can look
in *Messages* to find out.

Ted

Re: more url-utils?

[Stefan Monnier]

> OK.  But if the status is nil we couldn't get any data so there are no
> URL properties.  So we can have just one callback parameter "url-info"
> which is nil if the request failed and an alist if it worked.

I think that a boolean status is not sufficient.  E.g. in case of
failure, we want to have some information about why (e.g. an error message).

> How would the callback find out what actually failed?  Does it even need
> to?  I think it's enough to know the request failed, the user can look
> in *Messages* to find out.

I don't think that's good enough.


        Stefan

Re: more url-utils?

[Ted Zlatanov]

On Thu, 19 May 2011 08:53:22 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> OK.  But if the status is nil we couldn't get any data so there are no
>> URL properties.  So we can have just one callback parameter "url-info"
>> which is nil if the request failed and an alist if it worked.

SM> I think that a boolean status is not sufficient.  E.g. in case of
SM> failure, we want to have some information about why (e.g. an error
SM> message).

..but we don't want to throw an error.  Hrm.  How about "url-info" is
three possible types:

1) nil = failed, no information available (should happen only if there's
no handler for the URL's protocol)

2) atom (string or symbol) = failed, information is the string or the
symbol's properties

3) alist = success

Ted

Re: more url-utils?

[Stefan Monnier]

>>> OK.  But if the status is nil we couldn't get any data so there are no
>>> URL properties.  So we can have just one callback parameter "url-info"
>>> which is nil if the request failed and an alist if it worked.

SM> I think that a boolean status is not sufficient.  E.g. in case of
SM> failure, we want to have some information about why (e.g. an error
SM> message).

> ..but we don't want to throw an error.  Hrm.  How about "url-info" is
> three possible types:

> 1) nil = failed, no information available (should happen only if there's
> no handler for the URL's protocol)

> 2) atom (string or symbol) = failed, information is the string or the
> symbol's properties

How 'bout: the result of url-fetch (either returned in passed to the
callback), is list of the form (STATUS . INFO): if STATUS is nil, it
means the call was a success and INFO is an alist of auxiliary info such
as headers and stuff, and if STATUS is non-nil, it should be an error
symbol and INFO is some failure information such that one can do
(signal (car result) (cdr result)).


        Stefan

Re: more url-utils?

[Ted Zlatanov]

On Thu, 19 May 2011 10:28:05 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> OK.  But if the status is nil we couldn't get any data so there are no
>>>> URL properties.  So we can have just one callback parameter "url-info"
>>>> which is nil if the request failed and an alist if it worked.

SM> I think that a boolean status is not sufficient.  E.g. in case of
SM> failure, we want to have some information about why (e.g. an error
SM> message).

>> ..but we don't want to throw an error.  Hrm.  How about "url-info" is
>> three possible types:

>> 1) nil = failed, no information available (should happen only if there's
>> no handler for the URL's protocol)

>> 2) atom (string or symbol) = failed, information is the string or the
>> symbol's properties

SM> How 'bout: the result of url-fetch (either returned in passed to the
SM> callback), is list of the form (STATUS . INFO): if STATUS is nil, it
SM> means the call was a success and INFO is an alist of auxiliary info such
SM> as headers and stuff, and if STATUS is non-nil, it should be an error
SM> symbol and INFO is some failure information such that one can do
SM> (signal (car result) (cdr result)).

Sure, that's a nice solution.  I'll use that.

Ted

Re: more url-utils?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> How 'bout: the result of url-fetch (either returned in passed to the
> callback), is list of the form (STATUS . INFO): if STATUS is nil, it
> means the call was a success and INFO is an alist of auxiliary info such
> as headers and stuff, and if STATUS is non-nil, it should be an error
> symbol and INFO is some failure information such that one can do
> (signal (car result) (cdr result)).

Say you have a function that's supposed to do something with a web page,
`parse-page'.  The it would be very nice, I think, if the signature of
that function could be the same no matter whether we're fetching the
page synchronously or not.

That is, it would be nice if it were possible to say either

(apply #'parse-page (url-fetch "http://google.com"))

to do it synchronously or say 

(url-fetch "http://google.com" 'parse-page)

to do it asynchronously.

This would imply that the signature used for callbacks should be
something like

(defun parse-page (buffer &optional status headers)
  ...)

Does that make sense?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: more url-utils?

[Stefan Monnier]

> That is, it would be nice if it were possible to say either

> (apply #'parse-page (url-fetch "http://google.com"))

> to do it synchronously or say 

> (url-fetch "http://google.com" 'parse-page)

I was thinking of using

> (funcall #'parse-page (url-fetch "http://google.com"))
[...]
> (url-fetch "http://google.com" 'parse-page)

I.e. the callback takes a single argument.


        Stefan

Re: more url-utils?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (funcall #'parse-page (url-fetch "http://google.com"))
> [...]
>> (url-fetch "http://google.com" 'parse-page)
>
> I.e. the callback takes a single argument.

Which would be a list of buffer/info/headers?

Presently the callback is called with the current buffer set to the
return buffer.  If the async is to have the same signature as the sync,
then that's not really practical.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: more url-utils?

[Stefan Monnier]

>>> (funcall #'parse-page (url-fetch "http://google.com"))
>> [...]
>>> (url-fetch "http://google.com" 'parse-page)
>> I.e. the callback takes a single argument.
> Which would be a list of buffer/info/headers?

For example, yes.

> Presently the callback is called with the current buffer set to the
> return buffer.  If the async is to have the same signature as the sync,
> then that's not really practical.

The sync case can set-buffer as well, so both choices are possible
without breaking consistency.  I don't have a strong preference
either way.


        Stefan

Re: more url-utils?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The sync case can set-buffer as well, so both choices are possible
> without breaking consistency.  I don't have a strong preference
> either way.

So calling (url-fetch "http://google.com") would change the current
buffer?  I think that sounds like very surprising behaviour... 

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: more url-utils?

[Stefan Monnier]

>> The sync case can set-buffer as well, so both choices are possible
>> without breaking consistency.  I don't have a strong preference
>> either way.

> So calling (url-fetch "http://google.com") would change the current
> buffer?  I think that sounds like very surprising behaviour... 

Oh, wait, no, because I wanted url-fetch to insert the result in the
current-buffer, rather than creating a brand new buffer.


        Stefan

Re: more url-utils?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Oh, wait, no, because I wanted url-fetch to insert the result in the
> current-buffer, rather than creating a brand new buffer.

Yeah, that makes sense, perhaps.  In the async case too?

Hm.  Are there any charset issues?  I mean, if you're fetching an image,
it's a binary thing, and if you want to allow already having stuff in
the buffer, there might be issues...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: more url-utils?

[Stefan Monnier]

>> Oh, wait, no, because I wanted url-fetch to insert the result in the
>> current-buffer, rather than creating a brand new buffer.
> Yeah, that makes sense, perhaps.  In the async case too?

Yes.

> Hm.  Are there any charset issues?

I don't know what you're referring to.

> I mean, if you're fetching an image, it's a binary thing, and if you
> want to allow already having stuff in the buffer, there might be
> issues...

url-fetch could be defined to replace the current content of the buffer,
if that's deemed necessary (i.e. mostly if it needs to use
set-buffer-multibyte, which should ideally only ever be used on empty
buffers).


        Stefan

Re: more url-utils?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> url-fetch could be defined to replace the current content of the buffer,
> if that's deemed necessary (i.e. mostly if it needs to use
> set-buffer-multibyte, which should ideally only ever be used on empty
> buffers).

Right...  so it wouldn't be like `insert-file-contents', where you'd get
more stuff into a buffer already containing stuff.

The normal synchronous calling pattern would be

(with-temp-buffer
  (let ((info (url-fetch "http://google.com")))
    ... do stuff))

while the async case would be

(url-fetch "http://google.com" (lambda (info) ... do stuff))

Yeah, that looks nice to me.
    
-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: more url-utils?

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 19 May 2011 08:53:22 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>
>>> OK.  But if the status is nil we couldn't get any data so there are no
>>> URL properties.  So we can have just one callback parameter "url-info"
>>> which is nil if the request failed and an alist if it worked.
>
> SM> I think that a boolean status is not sufficient.  E.g. in case of
> SM> failure, we want to have some information about why (e.g. an error
> SM> message).
>
> ..but we don't want to throw an error.  Hrm.  How about "url-info" is
> three possible types:
>
> 1) nil = failed, no information available (should happen only if there's
> no handler for the URL's protocol)
>
> 2) atom (string or symbol) = failed, information is the string or the
> symbol's properties
>
> 3) alist = success

This seems ok from my user-viewpoint. OTOH I wouldnt mind handling
protocol errors with exceptions. Its what one does in many other
languages anyway.

BTW thanks everyone for driving this forward from my initial inadequate
proposal!

>
> Ted
>

-- 
Joakim Verona

Re: more url-utils?

[Stefan Monnier]

> This seems ok from my user-viewpoint. OTOH I wouldnt mind handling
> protocol errors with exceptions. Its what one does in many other
> languages anyway.

For synchronous calls, signalling exceptions would be a better
solution indeed.  But that doesn't work nearly as well for the
asynchronous case.


        Stefan

ELisp futures and continuations/coroutines (was: more url-utils?)

[Ted Zlatanov]

On Wed, 18 May 2011 19:05:57 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> That's a good idea.  Kind of like a future.

Speaking of futures, can we have a general facility for those?

1) a real "future" type

2) a way to construct it and check if the value is available

3) process sentinel tie-in

I'm sure much of this facility is either possible or already available
in ELisp, I just don't know the pieces and I think it would be very
convenient to provide a "future" type.  WDYT?

Similarly it would be very nice to provide continuations and coroutines
if those are not already possible.  They don't need a top-level type
though, a lexical closure is sufficient I think.  Please tell me how I'm
wrong :)

Ted

Re: ELisp futures and continuations/coroutines

[Stefan Monnier]

SM> That's a good idea.  Kind of like a future.
> Speaking of futures, can we have a general facility for those?
> 1) a real "future" type
> 2) a way to construct it and check if the value is available
> 3) process sentinel tie-in

Patches welcome (preferably Elisp-only for a start).

> I'm sure much of this facility is either possible or already available
> in ELisp, I just don't know the pieces and I think it would be very
> convenient to provide a "future" type.  WDYT?

I don't think it's already available, nor do I think similar things are
commonly used currently.

> Similarly it would be very nice to provide continuations and coroutines
> if those are not already possible.  They don't need a top-level type
> though, a lexical closure is sufficient I think.  Please tell me how I'm
> wrong :)

Given the way we implement the stack call/cc is difficult to implement.
But the concurrency branch should make it easy to have coroutines.


        Stefan

Re: ELisp futures and continuations/coroutines

[joakim]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 18 May 2011 19:05:57 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
>
> SM> That's a good idea.  Kind of like a future.
>
> Speaking of futures, can we have a general facility for those?
>
> 1) a real "future" type
>
> 2) a way to construct it and check if the value is available
>
> 3) process sentinel tie-in
>
> I'm sure much of this facility is either possible or already available
> in ELisp, I just don't know the pieces and I think it would be very
> convenient to provide a "future" type.  WDYT?
>
> Similarly it would be very nice to provide continuations and coroutines
> if those are not already possible.  They don't need a top-level type
> though, a lexical closure is sufficient I think.  Please tell me how I'm
> wrong :)

Continuations makes for very convenient web frameworks. Yum!

>
> Ted
>

-- 
Joakim Verona

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Thu, 19 May 2011 16:16:30 +0200 joakim@verona.se wrote: 

j> Continuations makes for very convenient web frameworks. Yum!

That's probably not useful for Emacs, but generally network interactions
and iteration-oriented code are much easier when you can keep a
continuation state.

Ted

Re: ELisp futures and continuations/coroutines

[Thien-Thi Nguyen]

() Ted Zlatanov <tzz@lifelogs.com>
() Thu, 19 May 2011 08:46:21 -0500

   3) process sentinel tie-in

What do you think of fsm.el for this?
Hmm, i see it's not (yet?) a part of Emacs.

-- 
a sig, not big, i fig, you dig?

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Thu, 19 May 2011 17:09:25 +0200 Thien-Thi Nguyen <ttn@gnuvola.org> wrote: 

TN> () Ted Zlatanov <tzz@lifelogs.com>
TN> () Thu, 19 May 2011 08:46:21 -0500

TN>    3) process sentinel tie-in

TN> What do you think of fsm.el for this?
TN> Hmm, i see it's not (yet?) a part of Emacs.

You mean fsm-0.1ttn5.el from gnu.emacs.sources?
(http://lists.gnu.org/archive/html/gnu-emacs-sources/2007-02/msg00006.html)

Do you want to provide an example of how it would work to tie a
hypothetical future type with process sentinels through fsm.el?  It
seems reasonable but I don't know enough to say.

Ted

Re: ELisp futures and continuations/coroutines

[Thien-Thi Nguyen]

() Ted Zlatanov <tzz@lifelogs.com>
() Thu, 19 May 2011 10:31:21 -0500

   You mean fsm-0.1ttn5.el from gnu.emacs.sources?
   (http://lists.gnu.org/archive/html/gnu-emacs-sources/2007-02/msg00006.html)

Yes, or the original (also to gnu-emacs-sources).  Same idea.

   Do you want to provide an example of how it would work to tie a
   hypothetical future type with process sentinels through fsm.el?  It
   seems reasonable but I don't know enough to say.

No, because i don't know what such a future type would do.  I just know
that i have successfully (and more pleasantly than initially imagined)
used fsm to implement a multicast proxy server (1 in, N out), which had
as its core requirement, the ability to chunk computation into disparate
pieces to be evaluated later.  From this congruent experience, i would
guess that a "future type" would be some subset of the fsm internals.

Analogous to the "keymap type", which is really just a list with symbol
‘keymap’ in its CAR, i am guessing one could poke around fsm internals
and derive a "future type", without having to go to C.

Which is to say, i also don't know enough to say, but i see a similarity.

-- 
a sig, not big, i fig, you dig?

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Thu, 19 May 2011 18:40:47 +0200 Thien-Thi Nguyen <ttn@gnuvola.org> wrote: 

TN> i don't know what such a future type would do.
...
TN> i would guess that a "future type" would be some subset of the fsm
TN> internals.

Why tie futures to fsm.el or deferred.el specifically?  That seems like
an implementation detail.

I'd rather establish a futures API:

- create a future
- is it done?
- get result
- inspect thrown error, if any

which is simple, obviously, but it's essential so all the
implementations that use it speak the same language.

TN> Analogous to the "keymap type", which is really just a list with symbol
TN> ‘keymap’ in its CAR, i am guessing one could poke around fsm internals
TN> and derive a "future type", without having to go to C.

TN> Which is to say, i also don't know enough to say, but i see a similarity.

Yes, it could be that way, or it could be a defstruct as in the
deferred.el Masashi-san posted.  Ideally no C code would be needed.  I
like the defstruct because it's simple yet has structure and accessor
functions.  I would copy the deferred defstruct without the `next' or
`cancel' fields and without any callback/errorback defaults.  Here's a
first cut, pretty trivial code:

#+begin_src lisp
(defstruct future callback errorback status value)

(defun future-done-p (future)
  (future-status future))

(defun future-errored-p (future)
  (eq (future-status future) 'error))

(defun future-finish (future &optional status)
  (unless (future-done-p future)
    (setf (future-status future) (or status t))
    (when (future-callback future)
      (funcall (future-callback future) future))))

(defun future-errored (future error)
  (unless (future-done-p future)
    (setf (future-status future) 'error)
    (when (future-errorback future)
      (funcall (future-errorback future) future error))))

(defun future-call (future)
  (unless (future-done-p future)
    (let ((ff (future-value future)))
      (when (functionp ff)
        ;; TODO: needs error handling
        (setf (future-value future)
              (funcall ff))))
    (future-finish future)))
#+end_src

`future-errored-p' breaks if the return value is 'error, though the
correct callback will be invoked even then.

I like the name "future" because it's fairly standard nowadays, it's a
noun unlike "deferred," there are no global functions or variables with
that string in Emacs except some org-mode variables, and it's very clear.

If the above looks OK I can finish the implementation (error handling in
`future-call' and around the callback/errorback invocations).  Then any
library can use ":include future" to add its own details on top of the
`future' defstruct.

Ted

Re: ELisp futures and continuations/coroutines

[Thien-Thi Nguyen]

() Ted Zlatanov <tzz@lifelogs.com>
() Thu, 19 May 2011 18:48:49 -0500

   Why tie futures to fsm.el or deferred.el specifically?  That seems like
   an implementation detail.

Well, one important detail, whether you consider it implementation
or design, is who kicks the evaluation (and when).  In the case of
fsm.el, this is done by either a timer or from process input.

A "future type" w/o this detail well-defined is no better than
a simple closure (IMHO) because it has no concept of time / timing.

Since invocation must be well-defined, so must the reception and
stashing of the returned value(s) be done in a coordinated way.

I think the "hello world" (demonstrating minimal sufficiency)
of such a system is countdown timer.

   I'd rather establish a futures API:

   - create a future
   - is it done?
   - get result
   - inspect thrown error, if any

   which is simple, obviously, but it's essential so all the
   implementations that use it speak the same language.

The answer to "is it done" is "never", if it never is kicked, tickled,
invoked, prodded, probed, or otherwise moved to action.

   I like the defstruct because it's simple

Yes, it's much cleaner than fsm's plist munging, but (as i tried to express
above), that is just the shape of the sail and not the motion of the boat.

   (defstruct future callback errorback status value)
   [...]

Keep going!

   If the above looks OK I can finish the implementation (error handling in
   `future-call' and around the callback/errorback invocations).  Then any
   library can use ":include future" to add its own details on top of the
   `future' defstruct.

Minimalism is nice, but i suppose i like a more full-featured "future type",
one that can self-terminate, for example.  Conniving computers, grumble...
(i'll have to dig up and post some old fsm toys -- get a dozen of them going
at once and you'll see what i mean :-D).

-- 
a sig, not big, i fig, you dig?

Re: ELisp futures and continuations/coroutines

[Mohsen BANAN]

>>>>> On Fri, 20 May 2011 03:23:57 +0200, Thien-Thi Nguyen <ttn@gnuvola.org> said:

  Thien-Thi> A "future type" w/o this detail well-defined is no better than
  Thien-Thi> a simple closure (IMHO) because it has no concept of time / timing.

If the point is that:
  
   This is not a case of either or, but both.

then, I agree.

In which case if there is meaningful room for
layering, then great! But, I don't suspect much.

In any case, here is a pointer to a C fsm
interface, documentation and examples that I built
and used (in RFC-2188, RFC-2524, ...) 15 years ago.

   http://www.neda.com/PLPC/110301#x1-1520006.19

I am in no way suggesting its inclusion.

But the documentation is relatively good and may
be worth reusing.

...Mohsen

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Thu, 19 May 2011 21:18:50 -0700 Mohsen BANAN <list-general@mohsen.1.banan.byname.net> wrote: 

MB> In any case, here is a pointer to a C fsm
MB> interface, documentation and examples that I built
MB> and used (in RFC-2188, RFC-2524, ...) 15 years ago.

MB>    http://www.neda.com/PLPC/110301#x1-1520006.19

That brings back memories of using ObjectTime (a FSM programming IDE)
back in college :)

Ted

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Fri, 20 May 2011 03:23:57 +0200 Thien-Thi Nguyen <ttn@gnuvola.org> wrote: 

TN> () Ted Zlatanov <tzz@lifelogs.com>
TN> () Thu, 19 May 2011 18:48:49 -0500

TN>    Why tie futures to fsm.el or deferred.el specifically?  That seems like
TN>    an implementation detail.

TN> Well, one important detail, whether you consider it implementation
TN> or design, is who kicks the evaluation (and when).  In the case of
TN> fsm.el, this is done by either a timer or from process input.

Yes, but deferred.el does it with a timer-triggered queue drain, storing
the next deferred item in the current one, and the concurrency branch
might do it with a separate thread.  These implementation details
obscure the purpose of the "future" type: to hold a potential value.

TN> A "future type" w/o this detail well-defined is no better than
TN> a simple closure (IMHO) because it has no concept of time / timing.
...
TN> Minimalism is nice, but i suppose i like a more full-featured "future type",
TN> one that can self-terminate, for example.  Conniving computers, grumble...
TN> (i'll have to dig up and post some old fsm toys -- get a dozen of them going
TN> at once and you'll see what i mean :-D).

Why should a future type encapsulate any time knowledge?  The whole
point is that you don't know when and if it will be available.

You could derive a expected-future which will have some knowledge of
when it might have the data.  A self-managed-future could run its own
functions.  And so on.  But I really want a clean, simple API that can
be used by any library.

Let's get this done, simple though it is, and then see how it will
integrate with fsm.el and deferred.el.  I really think it will benefit
everyone to start simple.

TN> Yes, [defstruct is] much cleaner than fsm's plist munging, but (as i
TN> tried to express above), that is just the shape of the sail and not
TN> the motion of the boat.

Heh heh.  I guess we're close to land then.

TN> Keep going!

The below adds a way to cancel a future.  I think error handling is all
that's left unsaid.

Ted

#+begin_src lisp
(defstruct future callback errorback status value)

(defun future-done-p (future)
  (future-status future))

(defun future-errored-p (future)
  (eq (future-status future) 'error))

(defun future-cancelled-p (future)
  (eq (future-status future) 'cancel))

(defun future-finish (future &optional status)
  (unless (future-done-p future)
    (setf (future-status future) (or status t))
    (when (future-callback future)
      (funcall (future-callback future) future))))

(defun future-errored (future error)
  (unless (future-done-p future)
    (setf (future-status future) 'error)
    (when (future-errorback future)
      (funcall (future-errorback future) future error))))

(defun future-call (future)
  (unless (future-done-p future)
    (let ((ff (future-value future)))
      (when (functionp ff)
        ;; TODO: needs error handling
        (setf (future-value future)
              (funcall ff))))
    (future-finish future)))

(defun future-cancel (future)
  (unless (future-done-p future)
    (future-finish future 'cancel)))

#+end_src

Re: ELisp futures and continuations/coroutines

[Thien-Thi Nguyen]

() Ted Zlatanov <tzz@lifelogs.com>
() Fri, 20 May 2011 10:29:39 -0500

   [knowledge of time is design, not implementation, thus out of scope
    of the proposed "future type"]

Ok, i can respect that.

   Let's get this done, simple though it is, and then see how it will
   integrate with fsm.el and deferred.el.  I really think it will benefit
   everyone to start simple.

   #+begin_src lisp
   (defstruct future callback errorback status value)

   (defun future-done-p (future)
     (future-status future))

   (defun future-errored-p (future)
     (eq (future-status future) 'error))

   (defun future-cancelled-p (future)
     (eq (future-status future) 'cancel))

   (defun future-finish (future &optional status)
     (unless (future-done-p future)
       (setf (future-status future) (or status t))
       (when (future-callback future)
         (funcall (future-callback future) future))))

   (defun future-errored (future error)
     (unless (future-done-p future)
       (setf (future-status future) 'error)
       (when (future-errorback future)
         (funcall (future-errorback future) future error))))

   (defun future-call (future)
     (unless (future-done-p future)
       (let ((ff (future-value future)))
         (when (functionp ff)
           ;; TODO: needs error handling
           (setf (future-value future)
                 (funcall ff))))
       (future-finish future)))

   (defun future-cancel (future)
     (unless (future-done-p future)
       (future-finish future 'cancel)))

   #+end_src

The next step (speaking from an armchair designer's pov) would be
to rebase fsm.el and deferred.el onto this data structure.

before:   fsm:                   deferred:
          API                    API
          internals              internals

after:    fsm:                   deferred:
          API                    API
          fsm glue               deferred glue
          struct ‘future’        struct ‘future’

I guess there will be similarities between the fsm and deferred glue that
we'd be wise to "move" into ‘struct future’, but that remains to be seen.

-- 
a sig, not big, i fig, you dig?

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

On Sun, 22 May 2011 15:17:32 +0200 Thien-Thi Nguyen <ttn@gnuvola.org> wrote: 

TN> () Ted Zlatanov <tzz@lifelogs.com>
TN> () Fri, 20 May 2011 10:29:39 -0500

TZZ>    Let's get this done, simple though it is, and then see how it will
TZZ>    integrate with fsm.el and deferred.el.  I really think it will benefit
TZZ>    everyone to start simple.
...
TN> The next step (speaking from an armchair designer's pov) would be
TN> to rebase fsm.el and deferred.el onto this data structure.

Stefan, Chong, anyone else, please let me know if you have objections to
the attached future.el package.  I will add error handling and add it to
Emacs tonight if there are no objections.  It's pretty simple code.

TN> I guess there will be similarities between the fsm and deferred glue that
TN> we'd be wise to "move" into ‘struct future’, but that remains to be seen.

I'll gladly modify it if we find such similarities once we start
reworking fsm.el and deferred.el to use it.

Ted


[-- Attachment #2: future.el --]
[-- Type: application/emacs-lisp, Size: 2084 bytes --]

Re: ELisp futures and continuations/coroutines

[Stefan Monnier]

> Stefan, Chong, anyone else, please let me know if you have objections to
> the attached future.el package.  I will add error handling and add it to
> Emacs tonight if there are no objections.  It's pretty simple code.

IIUC this will only be used by URL, at least at first.  Is there any
other package which is likely to use it in the foreseeable "future"?
If not, then I'd recommend to name it "url-future" for now.


        Stefan


PS: I'd expect future-finish and future-errored to signal an error is
the future is already done-p.  And future-call should return the
future-value.

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

On Mon, 23 May 2011 11:59:02 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Stefan, Chong, anyone else, please let me know if you have objections to
>> the attached future.el package.  I will add error handling and add it to
>> Emacs tonight if there are no objections.  It's pretty simple code.

SM> IIUC this will only be used by URL, at least at first.  Is there any
SM> other package which is likely to use it in the foreseeable "future"?
SM> If not, then I'd recommend to name it "url-future" for now.

The idea from the beginning was that this is a true data type, not an
auxiliary library.  At least deferred.el and fsm.el will use it, and I
am in favor of adding deferred.el to Emacs as I mentioned.

I would rather not isolate its scope to just URL interactions because
rewriting url-future-* to future-* later will be annoying and because it
makes the package look like it has narrow utility.

SM> PS: I'd expect future-finish and future-errored to signal an error is
SM> the future is already done-p.  And future-call should return the
SM> future-value.

Thanks, I made those changes, see attached.

Ted


[-- Attachment #2: future.el --]
[-- Type: application/emacs-lisp, Size: 2294 bytes --]

Re: ELisp futures and continuations/coroutines

[SAKURAI Masashi]

At Mon, 23 May 2011 10:10:44 -0500,
Ted Zlatanov wrote:
> :
> The idea from the beginning was that this is a true data type, not an
> auxiliary library.  At least deferred.el and fsm.el will use it, and I
> am in favor of adding deferred.el to Emacs as I mentioned.

What can I do to add deferred.el to Emacs ?


--
SAKURAI, Masashi (family, given)

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Tue, 24 May 2011 00:42:58 +0900 SAKURAI Masashi <m.sakurai@kiwanami.net> wrote: 

SM> At Mon, 23 May 2011 10:10:44 -0500,
SM> Ted Zlatanov wrote:
>> :
>> The idea from the beginning was that this is a true data type, not an
>> auxiliary library.  At least deferred.el and fsm.el will use it, and I
>> am in favor of adding deferred.el to Emacs as I mentioned.

SM> What can I do to add deferred.el to Emacs ?

You need to sign the assignment papers and then I can add it to the GNU
ELPA, which is the Emacs 24 package repository.  It may get moved to the
Emacs core later, depending on how widely it's used, but in the GNU ELPA
it will be very easy to find and install.

Also please see the discussion in this thread about url-future.el.  If
you can consider augmenting that defstruct instead of using the one in
deferred.el, it would be nice so we're all speaking the same data
language.  I asked the same of Thien-Thi Nguyen for fsm.el.  But it's
not a requirement for inclusion in the GNU ELPA and you don't have to do
it.

Ted

Re: ELisp futures and continuations/coroutines

[SAKURAI Masashi]

At Tue, 24 May 2011 21:07:50 -0500,
Ted Zlatanov wrote:
> :
> SM> What can I do to add deferred.el to Emacs ?
> 
> You need to sign the assignment papers and then I can add it to the GNU
> ELPA, which is the Emacs 24 package repository.  It may get moved to the
> Emacs core later, depending on how widely it's used, but in the GNU ELPA
> it will be very easy to find and install.

Thank you for your advice.
I sent my signing request form to gnu.
I have to learn the GNU ELPA.

> Also please see the discussion in this thread about url-future.el.  If
> you can consider augmenting that defstruct instead of using the one in
> deferred.el, it would be nice so we're all speaking the same data
> language.  I asked the same of Thien-Thi Nguyen for fsm.el.  But it's
> not a requirement for inclusion in the GNU ELPA and you don't have to do
> it.

I am reading the thread to catch up the discussion.
I will comment as soon as possible.

--
SAKURAI, Masashi (family, given)

Re: ELisp futures and continuations/coroutines

[SAKURAI Masashi]

At Tue, 24 May 2011 21:07:50 -0500,
Ted Zlatanov wrote:
>  :
> Also please see the discussion in this thread about url-future.el.  If
> you can consider augmenting that defstruct instead of using the one in
> deferred.el, it would be nice so we're all speaking the same data
> language.  I asked the same of Thien-Thi Nguyen for fsm.el.  But it's
> not a requirement for inclusion in the GNU ELPA and you don't have to do
> it.

I have read all discussion messages in this thread (about more
url-utils and future in elisp) carefully at last. 
Those discussions are so interesting. 
I wrote my comment as follows. Sorry for long comment.


About the "future" type for the just url-fetch function, I think it is
good enough to transfer values asynchronously to the waiting function.
However, I think it is somewhat primitive to construct general
asynchronous programs.

I have met so many difficulties during developing some Emacs
applications[1] which has rich UIs and communication with external
resources, such as handling many callbacks and waiting for them to
finish. For example, getting images by wget -> making thumbnails ->
displaying the images in the buffer, during then, the application
notifies the user of progress of tasks and may display results or
error messages.

OTOH, in JavaScript, there are async supports in many major libs:
jQuery.Deferred, Mochikit.Async, dojo.Deferred and so on. We can
easily write async codes right now. Now, even .NET4[2] provides an
async lib like JSDeferred. So, I'm sure elisp should have async
support too.

While I'm not a researcher for the concurrent programing,
investigating such other async libraries, I found that asynchronous
programing needs at least following two functions:

1) sequential connecting
2) waiting for async tasks done (all of them or the earliest one).

One can define many other aspects of async programing.
Ex. composition of tasks, error handling, canceling, propagating
values, adding tasks after executed, restarting tasks, notifying
progress and so on[3].

Recently, I found an utility program url-queue.el (Fetching web pages
in parallel) was added to lisp/url in trunk. In the single-thread
model like the Emacs, the task queue requires the callback
interface. So, I think some library which supports concurrent
programing model will be needed in near future, like deferred.el 
and concurrent.el.

Furthermore, url-queue.el seems to implement the semaphore which is
one of concurrent patterns. I made the semaphore in concurrent.el
because I needed. I have also seen the similar code in
emacs-w3m[4]. The code of w3m-proc.el manages number of sub-processes
and abstraction of async and sync calling.  So, I think those
concurrent patterns have been needed already.

Though concurrent.el has some patterns those were implemented for my
applications, of course, it doesn't cover all patterns. According to
other languages and some books, STM, Agent, Actor, Reentrant Lock and
Read-Write Lock are argued as concurrent programing.


Last, if my experience of development of deferred.el and concurrent.el
would help the Emacs's advance, I would be happy. I don't mind if
the libraries will be added to GNU ELPA or Emacs, even the maintainers
write a subset code from scratch.

Foot notes:
[1] My deferred.el applications
  * A network multiplayer game which I mentioned the previous post.
  https://github.com/kiwanami/emacs-3d-demo
  (movie) http://www.youtube.com/watch?v=YS5CUj9oUhs
  * In-line image manager with cacoo.com, the on-line diagramming tool
  https://github.com/kiwanami/emacs-cacoo
  (movie) http://www.youtube.com/watch?v=pwZvT5HFQjY
  * PDF file manager
  https://github.com/kiwanami/emacs-anything-books
  (movie) http://www.youtube.com/watch?v=jfEHmgCIc08
  * Inertial scrolling (see the movie of cacoo.el)
  https://github.com/kiwanami/emacs-inertial-scroll

[2] .NET
  * Async Tasks - Simplify Asynchronous Programming with Tasks
  http://msdn.microsoft.com/en-us/magazine/ff959203.aspx
  Moreover, Task and Rx enables to collect async resources by LINQ
  * Reactive Extensions
  http://msdn.microsoft.com/en-us/data/gg577609

[3] Aspects of async libs
  Sorry for only Japanese article I found.
  * (in Japanese)
  http://digitechlog.com/2010/03/12/7-asynchronous-processing-open-source-actionscript-library.html 
  * (machine translation)
  http://translate.google.co.jp/translate?hl=ja&sl=ja&tl=en&u=http://digitechlog.com/2010/03/12/7-asynchronous-processing-open-source-actionscript-library.html
  Following article may be helpful for designing async lib.
  * Virtual Panel: How to Survive Asynchronous Programming in JavaScript (InfoQ)
  http://www.infoq.com/articles/surviving-asynchronous-programming-in-javascript

[4] Emacs-w3m
  * web site: http://emacs-w3m.namazu.org/index-en.html
  * w3m-proc.el: http://cvs.namazu.org/emacs-w3m/w3m-proc.el?view=markup

--
SAKURAI, Masashi (family, given)

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Fri, 03 Jun 2011 22:59:48 +0900 SAKURAI Masashi <m.sakurai@kiwanami.net> wrote: 

SM> At Tue, 24 May 2011 21:07:50 -0500,
SM> Ted Zlatanov wrote:
>> :
>> Also please see the discussion in this thread about url-future.el.  If
>> you can consider augmenting that defstruct instead of using the one in
>> deferred.el, it would be nice so we're all speaking the same data
>> language.  I asked the same of Thien-Thi Nguyen for fsm.el.  But it's
>> not a requirement for inclusion in the GNU ELPA and you don't have to do
>> it.

SM> About the "future" type for the just url-fetch function, I think it is
SM> good enough to transfer values asynchronously to the waiting function.
SM> However, I think it is somewhat primitive to construct general
SM> asynchronous programs.

The future type is intentionally a simple pure data type.  It can be
extended with `defstruct'.  Is the data type missing something you need?
Can you convert concurrent.el to derive its data type from the
url-future struct and use the url-future-* functions, or is there
something blocking that conversion?  When I looked at concurrent.el, the
data type it uses was very similar to the url-future defstruct.

SM> While I'm not a researcher for the concurrent programing,
SM> investigating such other async libraries, I found that asynchronous
SM> programing needs at least following two functions:

SM> 1) sequential connecting
SM> 2) waiting for async tasks done (all of them or the earliest one).

SM> One can define many other aspects of async programing.
SM> Ex. composition of tasks, error handling, canceling, propagating
SM> values, adding tasks after executed, restarting tasks, notifying
SM> progress and so on[3].

Sure, but all of these are functional requirements.  url-future.el only
addressed the "futures" protocol, so we can speak a common data
language.  It doesn't provide any actual async support.  That's the job
for concurrent.el and deferred.el among others.

SM> Though concurrent.el has some patterns those were implemented for my
SM> applications, of course, it doesn't cover all patterns. According to
SM> other languages and some books, STM, Agent, Actor, Reentrant Lock and
SM> Read-Write Lock are argued as concurrent programing.

I'm not sure how much of that is needed in Emacs.  Perhaps these needs
will become more apparent when the concurrency branch is merged, though
I have no idea when that will hapen.

SM> Last, if my experience of development of deferred.el and concurrent.el
SM> would help the Emacs's advance, I would be happy. I don't mind if
SM> the libraries will be added to GNU ELPA or Emacs, even the maintainers
SM> write a subset code from scratch.

You need to sign the assignment papers.  One of the Emacs maintainers
can help you with that.  I think assign@gnu.org is the general contact
point for any copyright assignment questions.

Ted

Re: ELisp futures and continuations/coroutines

[SAKURAI Masashi]

At Fri, 03 Jun 2011 11:06:22 -0500,
Ted Zlatanov wrote:
> :
> Sure, but all of these are functional requirements.  url-future.el only
> addressed the "futures" protocol, so we can speak a common data
> language.  It doesn't provide any actual async support.  That's the job
> for concurrent.el and deferred.el among others.

I have understood "speaking a common data language" you mentioned.
OK. I see. I would try to modify deferred.el to fit the data type,
after I check the code of the url-fetch.

> SM> Though concurrent.el has some patterns those were implemented for my
> SM> applications, of course, it doesn't cover all patterns. According to
> SM> other languages and some books, STM, Agent, Actor, Reentrant Lock and
> SM> Read-Write Lock are argued as concurrent programing.
> 
> I'm not sure how much of that is needed in Emacs.  Perhaps these needs
> will become more apparent when the concurrency branch is merged, though
> I have no idea when that will hapen.

I see.

Recently I'm interesting in the Emacs as an application runtime environment.
So, I'm also interesting in the thread 'Emacs as a desktop environment'.

> SM> Last, if my experience of development of deferred.el and concurrent.el
> SM> would help the Emacs's advance, I would be happy. I don't mind if
> SM> the libraries will be added to GNU ELPA or Emacs, even the maintainers
> SM> write a subset code from scratch.
> 
> You need to sign the assignment papers.  One of the Emacs maintainers
> can help you with that.  I think assign@gnu.org is the general contact
> point for any copyright assignment questions.

I sent my request to assign@gnu.org in last month, and
I'm waiting for assignment papers from gnu.org. 

--
SAKURAI, Masashi (family, given)

Re: ELisp futures and continuations/coroutines

[SAKURAI Masashi]

> SM> Last, if my experience of development of deferred.el and concurrent.el
> SM> would help the Emacs's advance, I would be happy. I don't mind if
> SM> the libraries will be added to GNU ELPA or Emacs, even the maintainers
> SM> write a subset code from scratch.
> 
> You need to sign the assignment papers.  One of the Emacs maintainers
> can help you with that.  I think assign@gnu.org is the general contact
> point for any copyright assignment questions.

I have signed and received the assignment paper copy.  I would learn
the GNU ELPA. I'm going to check the emacs-devel threads about the
ELPA. Which documents should I read?

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Wed, 29 Jun 2011 14:31:32 +0900 SAKURAI Masashi <m.sakurai@kiwanami.net> wrote: 

SM> Last, if my experience of development of deferred.el and concurrent.el
SM> would help the Emacs's advance, I would be happy. I don't mind if
SM> the libraries will be added to GNU ELPA or Emacs, even the maintainers
SM> write a subset code from scratch.
>> 
>> You need to sign the assignment papers.  One of the Emacs maintainers
>> can help you with that.  I think assign@gnu.org is the general contact
>> point for any copyright assignment questions.

SM> I have signed and received the assignment paper copy.  I would learn
SM> the GNU ELPA. I'm going to check the emacs-devel threads about the
SM> ELPA. Which documents should I read?

If you could check out the http://bzr.savannah.gnu.org/r/emacs/elpa/
branch (or the bzr:// equivalent), there's a README in there.  You can
also prepare a tarball and Chong and I can help you convert it to a
ELPA-style package, if you prefer.  It should be easy :)

Ted

Re: ELisp futures and continuations/coroutines

[SAKURAI Masashi]

At Wed, 29 Jun 2011 07:01:39 -0500,
Ted Zlatanov wrote:
> :
> SM> I have signed and received the assignment paper copy.  I would learn
> SM> the GNU ELPA. I'm going to check the emacs-devel threads about the
> SM> ELPA. Which documents should I read?
> 
> If you could check out the http://bzr.savannah.gnu.org/r/emacs/elpa/
> branch (or the bzr:// equivalent), there's a README in there.  You can
> also prepare a tarball and Chong and I can help you convert it to a
> ELPA-style package, if you prefer.  It should be easy :)

Thank you for your advice. I will check it.

Re: ELisp futures and continuations/coroutines

[Chong Yidong]

SAKURAI Masashi <m.sakurai@kiwanami.net> writes:

> I would learn the GNU ELPA. I'm going to check the emacs-devel threads
> about the ELPA. Which documents should I read?

See the Emacs Lisp Reference Manual's node on Packaging in trunk.

Re: ELisp futures and continuations/coroutines

[Stefan Monnier]

> The idea from the beginning was that this is a true data type, not an
> auxiliary library.  At least deferred.el and fsm.el will use it, and I
> am in favor of adding deferred.el to Emacs as I mentioned.

I don't know of any package using deferred either, so it doesn't seem
urgent to add to Emacs's core.  I'd rather keep such things on ELPA for
now until they prove their necessity.

> I would rather not isolate its scope to just URL interactions because
> rewriting url-future-* to future-* later will be annoying

Maybe "annoying" but trivial to do.

> and because it makes the package look like it has narrow utility.

As long as no other package uses anything similar, it has
narrow utility.


        Stefan

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

On Mon, 23 May 2011 12:45:51 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> The idea from the beginning was that this is a true data type, not an
>> auxiliary library.  At least deferred.el and fsm.el will use it, and I
>> am in favor of adding deferred.el to Emacs as I mentioned.

SM> I don't know of any package using deferred either, so it doesn't seem
SM> urgent to add to Emacs's core.  I'd rather keep such things on ELPA for
SM> now until they prove their necessity.

OK.

>> I would rather not isolate its scope to just URL interactions because
>> rewriting url-future-* to future-* later will be annoying

SM> Maybe "annoying" but trivial to do.

OK, I did as you asked.  See attached, which has commentary, error
handling, and ERT tests.  I rewrote some trivial functions as defmacros.

Thanks
Ted


[-- Attachment #2: url-future.el --]
[-- Type: application/emacs-lisp, Size: 4850 bytes --]

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

On Tue, 24 May 2011 21:02:38 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> OK, I did as you asked.  See attached, which has commentary, error
TZ> handling, and ERT tests.  I rewrote some trivial functions as defmacros.

I'm trying to push and keep getting an error (break-lock didn't work):

Using saved push location: /home/tzz/source/emacs/trunk/
Unable to obtain lock  held by tzz                                                                                                                                         
at vcs-noshell [process #20860], acquired 1 second ago.
See "bzr help break-lock" for more.
bzr: ERROR: Could not acquire lock "(remote lock)": bzr+ssh://tzz@bzr.savannah.gnu.org/emacs/                                                                              

I have to run and can't figure out how to fix it; can someone please
commit the attached patch that adds url-future.el?

Thanks
Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: url-future.patch --]
[-- Type: text/x-diff, Size: 5419 bytes --]

=== modified file 'lisp/url/ChangeLog'
--- lisp/url/ChangeLog	2011-05-29 02:45:03 +0000
+++ lisp/url/ChangeLog	2011-05-31 10:46:44 +0000
@@ -1,3 +1,7 @@
+2011-05-31  Teodor Zlatanov  <tzz@lifelogs.com>
+
+	* url-future.el: Add general futures facility.
+
 2011-05-29  Leo Liu  <sdl.web@gmail.com>
 
 	* url-cookie.el (url-cookie): Add option :named so that

=== added file 'lisp/url/url-future.el'
--- lisp/url/url-future.el	1970-01-01 00:00:00 +0000
+++ lisp/url/url-future.el	2011-05-31 10:45:57 +0000
@@ -0,0 +1,126 @@
+;;; url-future.el --- general futures facility for url.el
+
+;; Copyright (C) 2011  Free Software Foundation, Inc.
+
+;; Author: Teodor Zlatanov <tzz@lifelogs.com>
+;; Keywords: data
+
+;; This file is part of GNU Emacs.
+;;
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Make a url-future (basically a defstruct):
+;; (make-url-future :value (lambda () (calculation goes here))
+;;   :callback (lambda (future) (use future on success))
+;;   :errorback (lambda (future &rest error) (error handler)))
+
+;; Then either call it with `url-future-call' or cancel it with
+;; `url-future-cancel'.  Generally the functions will return the
+;; future itself, not the value it holds.  Also the functions will
+;; throw a url-future-already-done error if you try to call or cancel
+;; a future more than once.
+
+;; So, to get the value:
+;; (when (url-future-completed-p future) (url-future-value future))
+
+;; See the ERT tests and the code for futher details.
+
+;;; Code:
+
+(eval-when-compile (require 'cl))
+(eval-when-compile (require 'ert))
+
+(defstruct url-future callback errorback status value)
+
+(defmacro url-future-done-p (url-future)
+  `(url-future-status ,url-future))
+
+(defmacro url-future-completed-p (url-future)
+  `(eq (url-future-status ,url-future) t))
+
+(defmacro url-future-errored-p (url-future)
+  `(eq (url-future-status ,url-future) 'error))
+
+(defmacro url-future-cancelled-p (url-future)
+  `(eq (url-future-status ,url-future) 'cancel))
+
+(defun url-future-finish (url-future &optional status)
+  (if (url-future-done-p url-future)
+      (signal 'error 'url-future-already-done)
+    (setf (url-future-status url-future) (or status t))
+    ;; the status must be such that the future was completed
+    ;; to run the callback
+    (when (url-future-completed-p url-future)
+      (funcall (or (url-future-callback url-future) 'ignore)
+               url-future))
+    url-future))
+
+(defun url-future-errored (url-future errorcons)
+  (if (url-future-done-p url-future)
+      (signal 'error 'url-future-already-done)
+    (setf (url-future-status url-future) 'error)
+    (setf (url-future-value url-future) errorcons)
+    (funcall (or (url-future-errorback url-future) 'ignore)
+             url-future errorcons)))
+
+(defun url-future-call (url-future)
+  (if (url-future-done-p url-future)
+      (signal 'error 'url-future-already-done)
+    (let ((ff (url-future-value url-future)))
+      (when (functionp ff)
+        (condition-case catcher
+            (setf (url-future-value url-future)
+                  (funcall ff))
+          (error (url-future-errored url-future catcher)))
+        (url-future-value url-future)))
+    (if (url-future-errored-p url-future)
+        url-future
+      (url-future-finish url-future))))
+
+(defun url-future-cancel (url-future)
+  (if (url-future-done-p url-future)
+      (signal 'error 'url-future-already-done)
+    (url-future-finish url-future 'cancel)))
+
+(ert-deftest url-future-test ()
+  (let* ((text "running future")
+         (good (make-url-future :value (lambda () (format text))
+                                :callback (lambda (f) (set 'saver f))))
+         (bad (make-url-future :value (lambda () (/ 1 0))
+                               :errorback (lambda (&rest d) (set 'saver d))))
+         (tocancel (make-url-future :value (lambda () (/ 1 0))
+                                    :callback (lambda (f) (set 'saver f))
+                                    :errorback (lambda (&rest d)
+                                                 (set 'saver d))))
+         saver)
+    (should (equal good (url-future-call good)))
+    (should (equal good saver))
+    (should (equal text (url-future-value good)))
+    (should (url-future-completed-p good))
+    (should-error (url-future-call good))
+    (setq saver nil)
+    (should (equal bad (url-future-call bad)))
+    (should-error (url-future-call bad))
+    (should (equal saver (list bad '(arith-error))))
+    (should (url-future-errored-p bad))
+    (setq saver nil)
+    (should (equal (url-future-cancel tocancel) tocancel))
+    (should-error (url-future-call tocancel))
+    (should (null saver))
+    (should (url-future-cancelled-p tocancel))))
+
+(provide 'url-future)
+;;; url-future.el ends here

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Tue, 31 May 2011 05:53:17 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Tue, 24 May 2011 21:02:38 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 
TZ> OK, I did as you asked.  See attached, which has commentary, error
TZ> handling, and ERT tests.  I rewrote some trivial functions as defmacros.

TZ> I'm trying to push and keep getting an error (break-lock didn't work):

TZ> Using saved push location: /home/tzz/source/emacs/trunk/
TZ> Unable to obtain lock  held by tzz                                                                                                                                         
TZ> at vcs-noshell [process #20860], acquired 1 second ago.
TZ> See "bzr help break-lock" for more.
TZ> bzr: ERROR: Could not acquire lock "(remote lock)": bzr+ssh://tzz@bzr.savannah.gnu.org/emacs/                                                                              

TZ> I have to run and can't figure out how to fix it; can someone please
TZ> commit the attached patch that adds url-future.el?

Apparently the push worked anyhow, though it looked to me like it
didn't.  I'm mystified.

Ted

Re: ELisp futures and continuations/coroutines (was: more url-utils?)

[SAKURAI Masashi]

Hi, Ted

I have just made an asynchronous library, deferred.el and concurrent.el.

deferred.el   : https://github.com/kiwanami/emacs-deferred/blob/master/README.markdown
concurrent.el : https://github.com/kiwanami/emacs-deferred/blob/master/README-concurrent.markdown

deferred.el is inspired by JSDeferred that comes from Mochikit.Async(JS) and Twisted(Python).

The chain of asynchronous callbacks likes the continuation.  Indeed,
concurrent.el that is build on deferred.el, has the pseud-thread, the generator and
the dataflow variable which may be similar to a "future" type.

A sample code of concurrent.el
https://github.com/kiwanami/emacs-deferred/blob/master/concurrent-sample.el

Here is a demo for the networking and UI thread. (Sorry for machine translation...)
http://translate.google.co.jp/translate?js=n&prev=_t&hl=ja&ie=UTF-8&layout=2&eotf=1&sl=ja&tl=en&u=http://d.hatena.ne.jp/kiwanami/20110413/1302709536

I have written some applications built on deferred.el and I found it works well.

I think handling of asynchronous callbacks is getting important for
the Emacs in the future. Because the Emacs interpreter is built on
the single thread model (Node.js also employs the single thread
model), the Emacs Lisp should have a way to write complicated
asynchronous codes more easily.

I would like to include deferred.el in the Emacs package.


--
SAKURAI, Masashi (family, given)


At Thu, 19 May 2011 08:46:21 -0500,
Ted Zlatanov wrote:
> 
> On Wed, 18 May 2011 19:05:57 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 
> 
> SM> That's a good idea.  Kind of like a future.
> 
> Speaking of futures, can we have a general facility for those?
> 
> 1) a real "future" type
> 
> 2) a way to construct it and check if the value is available
> 
> 3) process sentinel tie-in
> 
> I'm sure much of this facility is either possible or already available
> in ELisp, I just don't know the pieces and I think it would be very
> convenient to provide a "future" type.  WDYT?
> 
> Similarly it would be very nice to provide continuations and coroutines
> if those are not already possible.  They don't need a top-level type
> though, a lexical closure is sufficient I think.  Please tell me how I'm
> wrong :)
> 
> Ted
> 
> 

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Fri, 20 May 2011 02:03:54 +0900 SAKURAI Masashi <m.sakurai@kiwanami.net> wrote: 

SM> I have just made an asynchronous library, deferred.el and concurrent.el.

SM> deferred.el   : https://github.com/kiwanami/emacs-deferred/blob/master/README.markdown
SM> concurrent.el : https://github.com/kiwanami/emacs-deferred/blob/master/README-concurrent.markdown

SM> deferred.el is inspired by JSDeferred that comes from Mochikit.Async(JS) and Twisted(Python).

I looked at the code and it's pretty nice.  It needs some small cleanup
to be part of Emacs.

SM> I have written some applications built on deferred.el and I found it works well.

It's a good library, but maybe too complicated for general use.  I don't
know how the maintainers feel about anaphoric macros and chained
futures.  The resulting code, while neat, is IMO hard to understand if
you don't understand the underlying macros.

Also, the library does a lot more than futures, and I think it would be
nice to separate just the futures (the `deferred' defstruct).  But the
way they are implemented currently, the queue implementation is implicit
in the future definition itself.

SM> I think handling of asynchronous callbacks is getting important for
SM> the Emacs in the future. Because the Emacs interpreter is built on
SM> the single thread model (Node.js also employs the single thread
SM> model), the Emacs Lisp should have a way to write complicated
SM> asynchronous codes more easily.

I think with lexical closures Emacs is getting closer to true
concurrency, which is better than simulating it in a single-threaded
way.  I like your approach, it's just (like fsm.el) that it's a very
specific way of doing things that doesn't fit with the rest of ELisp
code, so I don't think it should be the *standard* way to do futures or
continuations.

This is just my opinion; the Emacs maintainers should say what they
think.

SM> I would like to include deferred.el in the Emacs package.

I'm in favor, it's a good library and more useful in the core than in
the GNU ELPA.

Ted

Re: ELisp futures and continuations/coroutines

[SAKURAI Masashi]

At Thu, 19 May 2011 17:51:20 -0500,
Ted Zlatanov wrote:
> :
> It's a good library, but maybe too complicated for general use.  I don't
> know how the maintainers feel about anaphoric macros and chained
> futures.  The resulting code, while neat, is IMO hard to understand if
> you don't understand the underlying macros.

I employed the anaphoric macros to simulate the method chains in
conventional OOP languages or the "do" syntax in Haskell. I feel
that it is not so good too. I would learn the better ideas.

* JavaScript (JSDeferred):
    http.get("http://example.com/").
    next(function(result) {
        alert(result);
    });

* Haskell:
    main = do x <- getContents
              putStr x

* elisp (deferred.el):
    (deferred:$
      (deferred:url-retrieve "http://www.gnu.org")
      (deferred:nextc it
        (lambda (buf)
          (insert (with-current-buffer buf (buffer-string)))
          (kill-buffer buf))))

> Also, the library does a lot more than futures, and I think it would be
> nice to separate just the futures (the `deferred' defstruct).  But the
> way they are implemented currently, the queue implementation is implicit
> in the future definition itself.

Yes. The library deferred.el focuses on just supporting asynchronous
callback programing. So, it is different from the "future" type that
you mentions in some aspects. Although on the higher abstract layer,
those ideas and implementations may be identical, the users like 
practical ones.

It seems that many JavaScript developers like the word "deferred", and
that the Java, C++ and other functional languages developers like
"future".


> I think with lexical closures Emacs is getting closer to true
> concurrency, which is better than simulating it in a single-threaded
> way.  I like your approach, it's just (like fsm.el) that it's a very
> specific way of doing things that doesn't fit with the rest of ELisp
> code, so I don't think it should be the *standard* way to do futures or
> continuations.

I see. I think the developers who are familiar to JavaScript's
deferred like deferred.el.


> SM> I would like to include deferred.el in the Emacs package.
> 
> I'm in favor, it's a good library and more useful in the core than in
> the GNU ELPA.

Thank you. I'm encouraged.


--
SAKURAI, Masashi (family, given)

Re: ELisp futures and continuations/coroutines

[Ted Zlatanov]

On Sat, 21 May 2011 00:49:09 +0900 SAKURAI Masashi <m.sakurai@kiwanami.net> wrote: 

SM> I employed the anaphoric macros to simulate the method chains in
SM> conventional OOP languages or the "do" syntax in Haskell. I feel
SM> that it is not so good too. I would learn the better ideas.

Perhaps you can use macros to simplify things, so instead of 

 (deferred:$
   (deferred:url-retrieve "http://www.gnu.org")
   (deferred:nextc it
     (lambda (buf)
       (insert (with-current-buffer buf (buffer-string)))
       (kill-buffer buf))))

the user could say (when we're specifically working with URL retrieval)

(deferred:do-url
  "http://www.gnu.org" ;; string implies url-retrieve
  ;; lambda implies "nextc it" so the following two are equivalent
  (lambda (buf) ...)
  (nextc it (lambda (buf) ...))

but this is just an idea.  I don't mean to tell you how it should work
and IMO it's usable as is.  Other, more experienced Lisp programmers
should tell you what they think because I'm just a beginner.

Ted