Re: async 1.0

[John Wiegley <jwiegley@gmail.com>]

>>>>> Christopher Schmidt <christopher@ch.ristopher.com> writes:

>>> Other than that, GNU Emacs has excellent asynchronous process and network
>>> facilities already.  Why not use them?
>> 
>> And these would be?  I'm not aware of any core service that lets me
>> asynchronously call lambdas.

> I do not like that you snipped context out of my quotes.

I really am very sorry if I dropped your context.  I didn't meant to
antagonize, just to reduce the length of my reply.

> Why do you want complex forms to be called asynchronously out of the context
> of the original Emacs process?

My core argument for async.el is that certain Emacs primitives, such as
`copy-file' and `process-send-region', cannot benefit from the tricks employed
by packages like deferred.el.  I think deferred.el is an excellent approach to
deferred computation: when things don't need to happen right away, they can be
interleaved with the user's commands as the interpreter becomes available.

But there are cases when Emacs *must* block the main process for minutes at a
time (such as copying a 10G file on my local machine).  It is these cases that
async.el is designed to address.  Otherwise, if deferred.el can solve the
problem, I think it's preferrable to use that over full asynchronicity.  It's
easier to debug with it, behavior is less surprising, and backtraces are more
revealing.

So we actually have many tools in our toolbox.  Async.el would just be one
more, with its own caveats and gotchas.  So far we have:

    1. `start-process'.  Works great, if you have an external executable to
       call.  `copy-file' cannot be made asynchronous using this primitive
       alone, as you cannot depend on the existence of "cp" or "copy.exe" on
       all platforms.

    2. Timers.  This is how deferred.el and a few other packages give the
       semblance of asynchronicity: by returning to the caller immediately,
       and queueing up code to be executed as the interpreter becomes
       available.

    3. Self-hosting (aka async.el), where a child Emacs is spawned to do work
       on behalf of the parent Emacs.

    4. The "concurrency" branch in Emacs Bzr.  I'd really like to know more
       about what's in here, and what problems it creates and solves.

Each of the first three has been used many times over, by many packages.
Eshell uses #1 to make use of external programs asynchronously; el-get (thanks
to Dave Abrahams) used to use #2 to make updates and installs asynchronous;
deferred.el is based on #2; #3 is used by elnode and helm, who rolled their
own implementations of what async.el does.

So async.el does not invent anything new.  The same approach (`start-process'
of a child Emacs) is currently "out there" and being used.  I just wanted to
wrap a simple and generalized interface around this practice, to aid in its
further use.

> The point I was trying to make is that async.el can make development and
> actual end-user usage both simpler and harder.  Some issues could be avoided
> by using non-forking asynchronous I/O in the first place.  "A nice
> short-term solution with severe long-term consequences."  (Now that I looked
> up 'severe' in a dictionary, I think it is the wrong term.  I wanted to say
> that async.el can cause unexpected surprises.)

Yes, this is the bane of asynchronicity in general.  And perhaps the best
argument for why it should always be "opt-in".

> Implementing asynchronous smtp on top of smtpmail within 19 LOC is
> impressive.  No doubt whatsoever.  This is a great opt-in feature.  You did
> not mention opt-in-ness before and to me it was not obvious from the
> context, especially because AFAICT dired-async.el unconditionally redefines
> dired internals to use async.el.

Sorry if I failed to make my position more clear.  I want async.el to be
available to everyone, but by no means do I want to inflict it on everyone
without their consent.  I take a very conservative approach to core Emacs
changes, since this is a platform people use to get real work done.  I _live_
in Emacs; it's core of my professional life.  I wouldn't want it to be
disrupted any more than I would want to do the same to you.

> I am sorry for offending you if that is the case.  I have lots of respect
> for your work.[1]

None taken, Christopher.  I'm just a parent defending his little child. :)

John