Re: async 1.0

Re: async 1.0

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

>     async.el uses `start-process' to spawn a child Emacs process for the
>     purpose of invoking the lambda that is passed as the first argument to
>     `async-start'.

> I see it would work, but wouldn't it be horribly slow to start up?

You know, Emacs is suprisingly fast to execute if you use "emacs -batch -Q".
For example:

    vulcan ~ $ average -n 100 emacs -Q -batch --eval "(+ 10 20)"
    0.0393513631821

It's only 39.4ms overhead per invocation to use Emacs this way -- plus the
time required load modules used by the async process, and injection of the
variable environment.  For example:

    vulcan ~ $ average -n 100 emacs -Q -batch \
                    --eval "(progn (require 'smtpmail) (+ 10 20))"
    0.143285052776

Now we're up to 143ms.  What makes this OK is that the job I'm asking
`async-start' to do takes many seconds, such that the overhead of starting a
new Emacs interpreter is completely lost in the time required to do the work
-- time otherwise spent with me staring at a wait cursor.

As a further example: In Gnus, hitting M-g on the emacs-devel group takes
around 60s (I have a lot of back articles).  If async.el were integrated, this
operation would return instantly, with some kind of fontification to show the
group being updated in the background, and blocking behavior if I hit RET to
actually visit the group before it's done.

That's kind of how dired-async.el operates now: When you copy a bunch of large
files, control returns to you immediately, and each file is colorized with a
yellow background that gets removed once that copy operation is completed.

John

Re: async 1.0

[Teemu Likonen]

John Wiegley [2012-06-21 16:23:26 -0500] wrote:

> You know, Emacs is suprisingly fast to execute if you use "emacs -batch -Q".
> For example:
>
>     vulcan ~ $ average -n 100 emacs -Q -batch --eval "(+ 10 20)"
>     0.0393513631821

It loads quickly from cache:

    $ time emacs -Q --batch --eval t

    real    0m0.044s
    user    0m0.024s
    sys     0m0.020s


But:

    $ sync; sudo sh -c 'echo 3 >/proc/sys/vm/drop_caches'
    $ time emacs -Q --batch --eval t

    real    0m2.288s
    user    0m0.036s
    sys     0m0.020s

Re: async 1.0

[John Wiegley]

>>>>> Teemu Likonen <tlikonen@iki.fi> writes:

> It loads quickly from cache:
>     real    0m0.044s

> But:
>     real    0m2.288s

Understood, but still miniscule compared to the 30 seconds I have to wait to
send an e-mail with a multi-megabyte attachment.

Now, a smart `fork' primitive could make this all moot, if it were able to
fork the "base" state of Emacs only....

John

Re: async 1.0

[Michael Albinus]

John Wiegley <johnw@newartisans.com> writes:

>>>>>> Teemu Likonen <tlikonen@iki.fi> writes:
>
>> It loads quickly from cache:
>>     real    0m0.044s
>
>> But:
>>     real    0m2.288s
>
> Understood, but still miniscule compared to the 30 seconds I have to wait to
> send an e-mail with a multi-megabyte attachment.
>
> Now, a smart `fork' primitive could make this all moot, if it were able to
> fork the "base" state of Emacs only....

Or you keep the Emacs daemons running in the background, waiting for new
tasks. Communication could be via D-Bus (queued services). But I have no
idea, how much it would improve reactiveness.

And it wouldn't run on all platforms Emacs supports.

> John

Best regards, Michael.

Re: async 1.0

[John Wiegley]

>>>>> Michael Albinus <michael.albinus@gmx.de> writes:

> Or you keep the Emacs daemons running in the background, waiting for new
> tasks. Communication could be via D-Bus (queued services). But I have no
> idea, how much it would improve reactiveness.
> 
> And it wouldn't run on all platforms Emacs supports.

And it introduces state bugs, which would be insane to debug, since presently
there is no a way to debug the child Emacs.  I'm having a hard time even
getting stack traces from the point of failure (a condition-case handler gives
you the backtrace for the handler, not the error's origin.  This may need new
functionality in Emacs to pass the trace as part of the error data).

I've been talking with someone else who is interested in creating a Swank
backend for Elisp that would make such debugging possible, but even then it's
not something I'd want to rely on after only the 20th `async-start' call
failed, though all the prior ones had succeeded...

The startup cost for async Emacs is currently dwarfed by the size of the tasks
I'm using it for.  If your async job is only running for a few hundred
milliseconds, I would say don't use async.el for that job, rather than trying
to optimize that case.

John

Re: async 1.0

[Samuel Bronson]

John Wiegley <jwiegley@gmail.com> wrote:

> >>>>> Michael Albinus <michael.albinus@gmx.de> writes:
> 
> > Or you keep the Emacs daemons running in the background, waiting for new
> > tasks. Communication could be via D-Bus (queued services). But I have no
> > idea, how much it would improve reactiveness.
> > 
> > And it wouldn't run on all platforms Emacs supports.
> 
> And it introduces state bugs, which would be insane to debug, since presently
> there is no a way to debug the child Emacs.  I'm having a hard time even
> getting stack traces from the point of failure (a condition-case handler gives
> you the backtrace for the handler, not the error's origin.  This may need new
> functionality in Emacs to pass the trace as part of the error data).

I've long thought Emacs should keep the backtrace along with the 
error/condition/exception (whichever you want to call it), like Python 
does: the way things are now is fairly painful even for single-process 
debugging, in all but the simplest cases.  Captured backtraces would 
allow the user to manually examine only those errors that they are 
actually interested in, and should permit handlers to augment them with 
additional information before re-throwing.

Re: async 1.0

[John Wiegley]

>>>>> Samuel Bronson <naesten@gmail.com> writes:

> I've long thought Emacs should keep the backtrace along with the
> error/condition/exception (whichever you want to call it), like Python does:
> the way things are now is fairly painful even for single-process debugging,
> in all but the simplest cases.  Captured backtraces would allow the user to
> manually examine only those errors that they are actually interested in, and
> should permit handlers to augment them with additional information before
> re-throwing.

I agree.  Does anybody with experience in this area know how difficult this
would be to add?

John

Re: async 1.0

[Samuel Bronson]

On Jul 5, 2012, at 12:09 AM, John Wiegley wrote:

>>>>>> Samuel Bronson <naesten@gmail.com> writes:
>
>> I've long thought Emacs should keep the backtrace along with the
>> error/condition/exception (whichever you want to call it), like  
>> Python does:
>> the way things are now is fairly painful even for single-process  
>> debugging,
>> in all but the simplest cases.  Captured backtraces would allow the  
>> user to
>> manually examine only those errors that they are actually  
>> interested in, and
>> should permit handlers to augment them with additional information  
>> before
>> re-throwing.
>
> I agree.  Does anybody with experience in this area know how  
> difficult this
> would be to add?

I haven't got any experience to speak of, but I did notice that the  
current API for error/condition handling involves giving handlers a  
single value of the form (ERROR-SYMBOL . DATA), where ERROR-SYMBOL and  
DATA are the two things that were passed to `signal'.  There is no  
obvious place to cram an extra BACKTRACE value, and making a new type  
that acted like a cons but had an extra field would be rather evil...

I noticed this while trying to figure out how to get ert.el to report  
*both* a backtrace *and* the accompanying error message when an error  
occurs during testing.  Right now, it's easy enough to get a backtrace  
(ert.el already does this using a call to `backtrace' in an `unwind- 
protect' cleanup) or an error message (just quote/comment out the  
cleanup)...

Oh. I can just remove the "(kill-emacs 2)", I guess?  Of course, that  
changes the exit code for this case to 1, which is meant to indicate  
test failures, not testing errors.  And the backtrace is absurdly  
long, and comes before the actual error message.

I guess there's always the `debugger' variable?

Re: async 1.0

[John Wiegley]

>>>>> Sivaram Neelakantan <nsivaram.net@gmail.com> writes:

> As a lowly user of Emacs, is this going to get into the main branch as I
> seem to be doing a lot of stuff in Emacs that makes the wait icon a regular
> screen feature?

Getting async.el into the main branch can happen next week, if Stefan and RMS
approve.  But modifying Emacs packages to take advantage of asnynchronicity
would require a lot more time and testing.

One thing that would make async.el _incredibly_ useful would be a `call/cc'
function in Emacs.  Then I could write the following:

    (defun some-complicated-command ()
      (interactive)
      (async-start/cc (lambda ()
                          ... do hard processing in a sub-emacs ...)

                        ;; Instead of being called with just the return value, the
                        ;; finish function is now also called with the result of
                        ;; (call/cc) after invoking the sub-Emacs

                        (lambda (ret cont)
                          (funcall cont)))

      (message "This processing happens *after* the callback is called."))

The difference here is that right now, `async-start' returns to the *caller*
immediately, with finalization code in the callback.  If I had `call/cc',
`async-start/cc' would return to the *user* immediately.  The callback's only
job would be to call the continuation, which would resume execution of the
stack that has spawned the asynchronous process.

The issue this addresses is that right now, if you call `async-start' when
call stacks are deep (Gnus, Eshell, etc.), the code has to be made aware that
finalization has been moved to a callback function.  It essentially has to be
"short-circuited", to know that asynchronicity is involved.

With continuations, we can push the current runtime stack into a variable, pop
back to the Emacs event loop immediately, and then resume that runtime stack
once the asynchronous function has finished executing.

Stefan/Chong: how hard would this be to add?  I'm aware we'd have to move away
from reliance on the C stack; but since they were able to make Python
stackless, I wonder how hard it would be to make Emacs Lisp the same?

John

Re: async 1.0

[Stefan Monnier]

> Stefan/Chong: how hard would this be to add?  I'm aware we'd have to
> move away from reliance on the C stack; but since they were able to
> make Python stackless, I wonder how hard it would be to make Emacs
> Lisp the same?

It's pretty much the same problem as the one that the concurrency branch
tackled: having several stacks at the same time.

In the concurrency branch, they made switching from one branch to another
fairly fast because it seemed to be an important property.  In your
case, that property might be less important, so you could use
a different approach where you unwind&rewind the specpdl stack.

Not using the C stack at all would require significant changes to the
C code, tho.  But you don't need that as long as you enforce that the
continuations are not called more than once.  Then you just need several
C stacks (pthreads provides that, for example).


        Stefan

Re: async 1.0

[Stefan Monnier]

>> As a lowly user of Emacs, is this going to get into the main branch as I
>> seem to be doing a lot of stuff in Emacs that makes the wait icon a regular
>> screen feature?
> Getting async.el into the main branch can happen next week, if Stefan and RMS
> approve.

I'm not completely sure about including it at this point.  But you can
put it up on GNU ELPA, of course.


        Stefan

Re: async 1.0

[Lennart Borgman]

On Sat, Jun 23, 2012 at 5:00 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>>> As a lowly user of Emacs, is this going to get into the main branch as I
>>> seem to be doing a lot of stuff in Emacs that makes the wait icon a regular
>>> screen feature?
>> Getting async.el into the main branch can happen next week, if Stefan and RMS
>> approve.
>
> I'm not completely sure about including it at this point.  But you can
> put it up on GNU ELPA, of course.

Here is a somewhat related wish for RPC calls between two Emacs:

http://sourceforge.net/mailarchive/forum.php?thread_name=4FE5DA0C.8000203%40siege-engine.com&forum_name=cedet-devel

Re: async 1.0

[John Wiegley]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I'm not completely sure about including it at this point.  But you can put
> it up on GNU ELPA, of course.

I want this to be a core service any Emacs author can use, not something that
requires users to install a package from ELPA.

Is there a problem with my including it?  I am fully signed up with the FSF.

John

Re: async 1.0

[Jeremiah Dodds]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> I'm not completely sure about including it at this point.  But you can put
>> it up on GNU ELPA, of course.
>
> I want this to be a core service any Emacs author can use, not something that
> requires users to install a package from ELPA.
>
> Is there a problem with my including it?  I am fully signed up with the FSF.
>
> John

It strikes me as the type of thing that could benefit by releasing
through ELPA first, and then including in core, a sort of "beta" release
style that should help catch any ... things that need caught. Just my
two cents.

Re: async 1.0

[Thierry Volpiatto]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> I'm not completely sure about including it at this point.  But you can put
>> it up on GNU ELPA, of course.
>
> I want this to be a core service any Emacs author can use, not something that
> requires users to install a package from ELPA.
Agree, I vote for inclusion in Emacs, this package works really well now.

> Is there a problem with my including it?  I am fully signed up with the FSF.
>
> John
>
>

-- 
  Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: async 1.0

[Christopher Schmidt]

John Wiegley <jwiegley@gmail.com> writes:

> I want this to be a core service any Emacs author can use, not
                      ^^^^^^^^^^^^
> something that requires users to install a package from ELPA.
>
> Is there a problem with my including it?  I am fully signed up with
> the FSF.

Whilst this is an incredibly interesting package, I do not think that
async.el should be a "core service" that may be used by other core
packages.

This approach to asynchronousy breaks things.  I have customizations all
over the place.  I add functionality to tons of hooks and I have lots of
defadvice's around core functions.  `async-inject-variables' does not
help very much in this regard as it just handles the tip of the iceberg.
It does not remotely cover all customisation points that might need to
be translated to the child Emacs instance.

smtpmail-async would break my setup.  It would break my setup in a way
very hard to fix.

Other than that, GNU Emacs has excellent asynchronous process and
network facilities already.  Why not use them?  In this case one does
not loose customisation, progress, error, debugging and bidirectional
communication functionality.  Of course this would require some
development effort, but in the end there are only very few core packages
(tramp, dired, mail sending, anything else?) that benefit from a true
asynchronous dispatch-and-forget implementation anyway.

There are tons of real issues with async.el.  To make a long story
short, I think true asynchronous I/O in Emacs is a hard, and forking is
not a solution.

Do not get me wrong.  This is an interesting package.  When it comes to
side-effect free CPU work that does not require user feedback this is
the way to go.  (I have no actual example for this use-case, though.)
When it comes to I/O, and this is what all posts in this thread are
about, async.el IMO totally undermines everything Emacs stands for.  It
seems to be like a forbidden fruit.  A nice short-term solution with
severe long-term consequences.

If this package makes it into GNU Emacs, please make every single usage
opt-in by default.  Document on a per use-case basis which functions and
hooks are run in the child Emacs instance.  If async.el somehow finds
its way on my machine, the very first form in my init.el will be
responsible for redefining all `async-'-functions to unconditionally
signal an error.

Slightly off topic...  I do not agree with GNU ELPA not being applicable
for core services.  I think every single package that is not absolutely
necessary for running a bare Emacs should be moved to GNU ELPA.  In
particular I am thinking of packages like CEDET, Calc, Calendar, ECB,
ERC, Gnus, Org-mode, all prog-modes, eshell, mh-e, rmail, shell, term
etc.  On top of that, IMO every single core package should have a copy
on GNU ELPA so one can to overwrite the native GNU Emacs one with the
one from GNU ELPA.  This would decouple all packages from the Emacs
release cycle and allow bug fixes to be distributed instantly.  On top
of that, this would make actual core emacs development, test, release
and bug management a lot easier.[1]

Just my two cents.

        Christopher

[1] I realise this would require substantial modifications to pretty
much every single aspect of GNU Emacs and GNU ELPA.  Yet I think this is
an approach worth discussing.

Re: async 1.0

[Bastien]

Hi Christopher,

Christopher Schmidt <christopher@ch.ristopher.com> writes:

> I think every single package that is not absolutely
> necessary for running a bare Emacs should be moved to GNU ELPA.  In
> particular I am thinking of packages like CEDET, Calc, Calendar, ECB,
> ERC, Gnus, Org-mode, all prog-modes, eshell, mh-e, rmail, shell, term
> etc.  

I disagree.

As Eli said, this is just some additional MO in GNU Emacs.
Moreover, there is no useless burden when launching Emacs.
If there is, that can be fixed without removing stuff.

I'm fine with the current policy for GNU Emacs and GNU ELPA,
as they complete each other in a useful way: a light-weight
policy for GNU ELPA ("from FSF-copyrighted authors") and a
strong policy for Emacs ("be accepted by Emacs maintainers.")

By moving stuff to GNU ELPA, you will not only decentralize
maintainance (it is already decentralized for many packages),
you will also remove the pride of "being part of GNU Emacs".  
If you want people to be pride of "being part of GNU ELPA", 
then you will have to define a stronger policy for GNU ELPA.  
And you will end up with more work, as you'll now have two
things to maintain.

(Also, I think it's good for a package to have two categories
of users: regular users that use the package from GNU Emacs,
and power users that want to test the latest versions.  Telling 
people to use the constantly updated version from GNU ELPA will 
make your userbase perhaps too homogeneous.)

> On top of that, IMO every single core package should have a copy
> on GNU ELPA so one can to overwrite the native GNU Emacs one with the
> one from GNU ELPA.  

At least for Org-mode, this is already the case.

> This would decouple all packages from the Emacs
> release cycle and allow bug fixes to be distributed instantly.  On top
> of that, this would make actual core emacs development, test, release
> and bug management a lot easier.[1]

There is Elisp code everywhere on the web, emacswiki, etc.  It *is* 
pretty chaotic and creative.  Having many useful packages included
in GNU Emacs helps users to find some order in this chaos, and my
guess is that it's part of the motivations of Elisp hackers.

Best,

-- 
 Bastien

Re: async 1.0

[John Wiegley]

>>>>> Christopher Schmidt <christopher@ch.ristopher.com> writes:

> Whilst this is an incredibly interesting package, I do not think that
> async.el should be a "core service" that may be used by other core packages.

I think that it should definitely be a core service, but _of course_, "opt-in"
until it is discovered that each particular use of async.el cannot break in
any way (and I'm open to the thought that this may never be true, and it will
be opt-in forever).

For example, I have changes to dired to make it asynchronous.  They work great
here, and I'm not sure what all your doom and gloom is about.  However, I
wouldn't imagine enabling such a change by default; I would add a
`dired-use-async' variable.

*However*, such a variable loses a lot of value if dired is part of Emacs but
async isn't.  How can I modify dired (and it does require core modifications
to support async.el) if the changes rely on macros contained in a package from
ELPA?  Dired gets byte-compiled as part of the Emacs bootstrap, yet those
macros are not present until the user installs async.el from ELPA.  Do they
have to re-compile dired to establish the proper definitions in byte-code?

Further, even though enabling `dired-use-async' purports to make dired async,
it doesn't really.  The user has to additionally install async through ELPA
for that variable to become meaningful.  This is a needless burden on the
user, not all of whom are familiar and comfortable with installing packages
through ELPA.  Async should be something people can just turn on, not
something they have to jump through hoops to get.

So, I think async.el should be a core services so that other core services can
rely on its definitions at compile time, as Emacs is being built.  Otherwise,
changing dired, eshell, smtpmail, etc., to use async.el ends up being way too
hacky to be worth doing.

I believe requiring distribution through ELPA would kill chance at real
adoption for async.el.

> smtpmail-async would break my setup.  It would break my setup in a way very
> hard to fix.

Can you explain that a bit more?  Otherwise, it sounds like FUD.

> Other than that, GNU Emacs has excellent asynchronous process and network
> facilities already.  Why not use them?

And these would be?  I'm not aware of any core service that lets me
asynchronously call lambdas.

> If this package makes it into GNU Emacs, please make every single usage
> opt-in by default.

I wouldn't dream of enabling this by default for unsuspecting users.  It needs
years of battle testing before I'd even begin to have that kind of confidence.

> If async.el somehow finds its way on my machine, the very first form in my
> init.el will be responsible for redefining all `async-'-functions to
> unconditionally signal an error.

Again, a bit FUD-y.

> Slightly off topic...  I do not agree with GNU ELPA not being applicable for
> core services.  I think every single package that is not absolutely
> necessary for running a bare Emacs should be moved to GNU ELPA.  In
> particular I am thinking of packages like CEDET, Calc, Calendar, ECB, ERC,
> Gnus, Org-mode, all prog-modes, eshell, mh-e, rmail, shell, term etc.  On
> top of that, IMO every single core package should have a copy on GNU ELPA so
> one can to overwrite the native GNU Emacs one with the one from GNU ELPA.
> This would decouple all packages from the Emacs release cycle and allow bug
> fixes to be distributed instantly.  On top of that, this would make actual
> core emacs development, test, release and bug management a lot easier.[1]

This is a discussion for another thread, please.

John

Re: async 1.0

[Christopher Schmidt]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Christopher Schmidt <christopher@ch.ristopher.com> writes:
[...]
> I think that it should definitely be a core service, but _of course_,
> "opt-in" until it is discovered that each particular use of async.el
> cannot break in any way (and I'm open to the thought that this may
> never be true, and it will be opt-in forever).

You did not mention that before.

[...]
> Further, even though enabling `dired-use-async' purports to make dired
> async, it doesn't really.  The user has to additionally install async
> through ELPA for that variable to become meaningful.  This is a
> needless burden on the user, not all of whom are familiar and
> comfortable with installing packages through ELPA.  Async should be
> something people can just turn on, not something they have to jump
> through hoops to get.

I was not arguing in favour of making async.el a GNU ELPA package rather
than a core GNU Emacs package.  I did not question asynchronous beef-ups
of dired or smtpmail per se.

[...]
>> smtpmail-async would break my setup.  It would break my setup in a
>> way very hard to fix.
>
> Can you explain that a bit more?  Otherwise, it sounds like FUD.

This is off-topic.  Executive summary: I rewrote the network stack of
Emacs in order to perform certificate pinning and OCSP verification.
This is integrated into a setup where user-visible mutable state (this
is fuzzy!) is stored persistently on a per-frame basis - frames bound to
distinct modes and instances - with these instances keeping track off
all file and network activity.  This stuff is integrated into my OS
environment, with Emacs modifying firewall rules whenever necessary.

>> Other than that, GNU Emacs has excellent asynchronous process and
>> network facilities already.  Why not use them?
>
> And these would be?  I'm not aware of any core service that lets me
> asynchronously call lambdas.

I do not like that you snipped context out of my quotes.

    Other than that, GNU Emacs has excellent asynchronous process and
    network facilities already.  Why not use them?  In this case one
    does not loose customisation, progress, error, debugging and
    bidirectional communication functionality.  Of course this would
    require some development effort, but in the end there are only very
    few core packages (tramp, dired, mail sending, anything else?) that
    benefit from a true asynchronous dispatch-and-forget implementation
    anyway.
    [...]
    When it comes to side-effect free CPU work that does not require
    user feedback this [async.el] is the way to go.  (I have no actual
    example for this use-case, though.)  When it comes to I/O, and this
    is what all posts in this thread are about, async.el IMO totally
    undermines everything Emacs stands for.

Why do you want complex forms to be called asynchronously out of the
context of the original Emacs process?

Does the advantage of an easy and little intrusive implementation in
comparison to Masashi's deferred.el or the use of manual async
primitives outweigh the loss of "customisation, progress, error,
debugging and bidirectional communication functionality"?  Does it
outweigh that a minority of end-users would have to fiddle with async to
translate their customizations into child emacsen?  Is there any non-I/O
(non-tramp) related use-case for async.el?

The point I was trying to make is that async.el can make development and
actual end-user usage both simpler and harder.  Some issues could be
avoided by using non-forking asynchronous I/O in the first place.  "A
nice short-term solution with severe long-term consequences."  (Now that
I looked up 'severe' in a dictionary, I think it is the wrong term.  I
wanted to say that async.el can cause unexpected surprises.)

[...]
>> If async.el somehow finds its way on my machine, the very first form
>> in my init.el will be responsible for redefining all
>> `async-'-functions to unconditionally signal an error.
>
> Again, a bit FUD-y.

I am sorry.  That comment was not appropriate, it does not matter.


Implementing asynchronous smtp on top of smtpmail within 19 LOC is
impressive.  No doubt whatsoever.  This is a great opt-in feature.  You
did not mention opt-in-ness before and to me it was not obvious from the
context, especially because AFAICT dired-async.el unconditionally
redefines dired internals to use async.el.

John, I have no intention to spread FUD.  I am sorry for offending you
if that is the case.  I have lots of respect for your work.[1]  I am not
a GNU Emacs developer and I do not want to be one.  I was just
brainstorming, raising a few points that went through my head, from an
Emacs user point of view, when I read the messages regarding async.el.
I was under the impression that you would use async.el to revamp the
core packages of Emacs, one after another, wreaking havoc upon my
precious customizations.

Greetings,

        Christopher

[1] I borrowed some stuff from your dotemacs.  Eshell is featured in a
Hollywood movie and your work on git modularization should make Boost
fun again.  I loved your OpenGenera blog post ;)  This is great stuff!

Re: async 1.0

[John Wiegley]

>>>>> Christopher Schmidt <christopher@ch.ristopher.com> writes:

>>> Other than that, GNU Emacs has excellent asynchronous process and network
>>> facilities already.  Why not use them?
>> 
>> And these would be?  I'm not aware of any core service that lets me
>> asynchronously call lambdas.

> I do not like that you snipped context out of my quotes.

I really am very sorry if I dropped your context.  I didn't meant to
antagonize, just to reduce the length of my reply.

> Why do you want complex forms to be called asynchronously out of the context
> of the original Emacs process?

My core argument for async.el is that certain Emacs primitives, such as
`copy-file' and `process-send-region', cannot benefit from the tricks employed
by packages like deferred.el.  I think deferred.el is an excellent approach to
deferred computation: when things don't need to happen right away, they can be
interleaved with the user's commands as the interpreter becomes available.

But there are cases when Emacs *must* block the main process for minutes at a
time (such as copying a 10G file on my local machine).  It is these cases that
async.el is designed to address.  Otherwise, if deferred.el can solve the
problem, I think it's preferrable to use that over full asynchronicity.  It's
easier to debug with it, behavior is less surprising, and backtraces are more
revealing.

So we actually have many tools in our toolbox.  Async.el would just be one
more, with its own caveats and gotchas.  So far we have:

    1. `start-process'.  Works great, if you have an external executable to
       call.  `copy-file' cannot be made asynchronous using this primitive
       alone, as you cannot depend on the existence of "cp" or "copy.exe" on
       all platforms.

    2. Timers.  This is how deferred.el and a few other packages give the
       semblance of asynchronicity: by returning to the caller immediately,
       and queueing up code to be executed as the interpreter becomes
       available.

    3. Self-hosting (aka async.el), where a child Emacs is spawned to do work
       on behalf of the parent Emacs.

    4. The "concurrency" branch in Emacs Bzr.  I'd really like to know more
       about what's in here, and what problems it creates and solves.

Each of the first three has been used many times over, by many packages.
Eshell uses #1 to make use of external programs asynchronously; el-get (thanks
to Dave Abrahams) used to use #2 to make updates and installs asynchronous;
deferred.el is based on #2; #3 is used by elnode and helm, who rolled their
own implementations of what async.el does.

So async.el does not invent anything new.  The same approach (`start-process'
of a child Emacs) is currently "out there" and being used.  I just wanted to
wrap a simple and generalized interface around this practice, to aid in its
further use.

> The point I was trying to make is that async.el can make development and
> actual end-user usage both simpler and harder.  Some issues could be avoided
> by using non-forking asynchronous I/O in the first place.  "A nice
> short-term solution with severe long-term consequences."  (Now that I looked
> up 'severe' in a dictionary, I think it is the wrong term.  I wanted to say
> that async.el can cause unexpected surprises.)

Yes, this is the bane of asynchronicity in general.  And perhaps the best
argument for why it should always be "opt-in".

> Implementing asynchronous smtp on top of smtpmail within 19 LOC is
> impressive.  No doubt whatsoever.  This is a great opt-in feature.  You did
> not mention opt-in-ness before and to me it was not obvious from the
> context, especially because AFAICT dired-async.el unconditionally redefines
> dired internals to use async.el.

Sorry if I failed to make my position more clear.  I want async.el to be
available to everyone, but by no means do I want to inflict it on everyone
without their consent.  I take a very conservative approach to core Emacs
changes, since this is a platform people use to get real work done.  I _live_
in Emacs; it's core of my professional life.  I wouldn't want it to be
disrupted any more than I would want to do the same to you.

> I am sorry for offending you if that is the case.  I have lots of respect
> for your work.[1]

None taken, Christopher.  I'm just a parent defending his little child. :)

John

ELPA and core services

[John Wiegley]

>>>>> Christopher Schmidt <christopher@ch.ristopher.com> writes:

> I do not agree with GNU ELPA not being applicable for core services.  I
> think every single package that is not absolutely necessary for running a
> bare Emacs should be moved to GNU ELPA.

Please, please no.  I use lots of Emacsen on lots of machines, and only a few
have my configuration copied over.  Please don't turn unconfigured Emacsen
into brain-dead zombies of little value.  I rely on the fact that Eshell
exists on every Windows machine that has Emacs.  And not all of these have a
network connection for me to even use ELPA, if that company's web proxy would
even allow it.

I think you may be considering this issue only from the
single-power-user-with-net-access point of view.

> On top of that, IMO every single core package should have a copy on GNU ELPA
> so one can to overwrite the native GNU Emacs one with the one from GNU ELPA.
> This would decouple all packages from the Emacs release cycle and allow bug
> fixes to be distributed instantly.

Now, this I agree with completely.  ELPA overrides is a great idea, and I
think it could accelerate development -- as long as inter-package version
dependencies are managed.  If the newest Gnus suddenly depends on the the
newest something-else, that something-else should be installed/updated along
with it automatically.

John

Re: ELPA and core services

[Lennart Borgman]

On Sun, Jun 24, 2012 at 11:12 PM, John Wiegley <jwiegley@gmail.com> wrote:
>>>>>> Christopher Schmidt <christopher@ch.ristopher.com> writes:
>
>> I do not agree with GNU ELPA not being applicable for core services.  I
>> think every single package that is not absolutely necessary for running a
>> bare Emacs should be moved to GNU ELPA.
>
> Please, please no.  I use lots of Emacsen on lots of machines, and only a few
> have my configuration copied over.  Please don't turn unconfigured Emacsen
> into brain-dead zombies of little value.  I rely on the fact that Eshell
> exists on every Windows machine that has Emacs.  And not all of these have a
> network connection for me to even use ELPA, if that company's web proxy would
> even allow it.
>
> I think you may be considering this issue only from the
> single-power-user-with-net-access point of view.
>
>> On top of that, IMO every single core package should have a copy on GNU ELPA
>> so one can to overwrite the native GNU Emacs one with the one from GNU ELPA.
>> This would decouple all packages from the Emacs release cycle and allow bug
>> fixes to be distributed instantly.
>
> Now, this I agree with completely.  ELPA overrides is a great idea, and I
> think it could accelerate development -- as long as inter-package version
> dependencies are managed.  If the newest Gnus suddenly depends on the the
> newest something-else, that something-else should be installed/updated along
> with it automatically.

I agree with John on both points here.

Re: ELPA and core services

[Stefan Monnier]

> have my configuration copied over.  Please don't turn unconfigured Emacsen
> into brain-dead zombies of little value.  I rely on the fact that Eshell

There's no such plan, of course.  The only plan I have in this area is
to bundle some ELPA packages in the Emacs release tarball and maybe
after that move some large externally maintained bundled packages (think
Org or Gnus) to ELPA (while keeping them bundled in the release).

But I have many other things to do, and it's not very high on my list
of priorities.


        Stefan

ELPA and core services

[Stephen J. Turnbull]

John Wiegley writes:

 > Please, please no.  I use lots of Emacsen on lots of machines, and only a few
 > have my configuration copied over.  Please don't turn unconfigured Emacsen
 > into brain-dead zombies of little value.

That won't happen.  XEmacs made the mistake of distributing core
XEmacs without the SUMO package, but Emacs doesn't need to make the
same mistake, and even if it did it would be easy to rectify once you
have a package system.  (It's also not that big a mistake, as the
continued, if attenuated, existence of XEmacs shows.)

 > Now, this I agree with completely.  ELPA overrides is a great idea, and I
 > think it could accelerate development -- as long as inter-package version
 > dependencies are managed.

I doubt it will accelerate development by much.  Managing the real
dependencies between separately maintained code bases (even if they're
in the same tree!) is what slows down development, not distributing
changes.

Re: ELPA and core services

[Achim Gratz]

John Wiegley writes:
>> On top of that, IMO every single core package should have a copy on GNU ELPA
>> so one can to overwrite the native GNU Emacs one with the one from GNU ELPA.
>> This would decouple all packages from the Emacs release cycle and allow bug
>> fixes to be distributed instantly.
>
> Now, this I agree with completely.  ELPA overrides is a great idea, and I
> think it could accelerate development -- as long as inter-package version
> dependencies are managed.  If the newest Gnus suddenly depends on the the
> newest something-else, that something-else should be installed/updated along
> with it automatically.

It may be a great idea, but even ignoring inter-package dependencies it
currently doesn't really work due to the way autoloads are generated and
used in both Emacs and ELPA.  Putting two different fixes for this issue
into Emacs and ELPA to me seems a bad idea, so that suggests something
more general would be needed and extending ELPA to allow for "layered"
installations by default (site-elpa, elpa-24.x, user-elpa, ...) and
delivering core packages via a "built-in" ELPA might becmome an option.

Distribution of bug fixes is actually a seperate issue that could be
solved in other ways, IMHO.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: ELPA and core services

[Tom Tromey]

>>>>> "Achim" == Achim Gratz <Stromeko@nexgo.de> writes:

Achim> It may be a great idea, but even ignoring inter-package dependencies it
Achim> currently doesn't really work due to the way autoloads are generated and
Achim> used in both Emacs and ELPA.

Emacs should use the package.el activation code for cases where a
package is going to be distributed both in Emacs and separately.
That was my original intent; just nobody has implemented this on the
Emacs side.

Tom

Re: ELPA and core services

[Achim Gratz]

Tom Tromey writes:
> Emacs should use the package.el activation code for cases where a
> package is going to be distributed both in Emacs and separately.
> That was my original intent; just nobody has implemented this on the
> Emacs side.

Sorry for digging up this old thread again, this is what I currently
have in package-directory-list from emacs-24 pretest:

("/usr/local/share/emacs/24.2.92/site-lisp/elpa"
 "/usr/local/share/emacs/site-lisp/elpa"
 "/usr/share/emacs/site-lisp/elpa")

This seems to collect the directories in reverse from load-path and
other such variables (I don't know if the precedence is also in that
order, but from a cursory look not) and it is missing a package
directory for the Emacs installation, maybe it should be :
"/usr/local/share/emacs/24.2.92/lisp/packages".

Any Emacs core package that is also in ELPA would then be moved into
…/packages and would be pre-activated unless the same package was
present somewhere up the package directory list or the user explicitly
deactivated that package.  Is that what you had in mind?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

stdlib for Emacs? [was: async 1.0]

[Stephen J. Turnbull]

Christopher Schmidt writes:

 > Slightly off topic...  I do not agree with GNU ELPA not being applicable
 > for core services.  I think every single package that is not absolutely
 > necessary for running a bare Emacs should be moved to GNU ELPA.

If you want to know what that would look like, look at XEmacs.  It's a
win for XEmacs, but I have to suspect that win has a lot to do with
getting free maintenance for the core of many packages that are core
parts of Emacs, plus a different development culture (see below).  The
XEmacs packagization side either has a volunteer maintainer --
sometimes the core maintainer but we prefer not to burden them if
another volunteer is available -- or it's done by the "XEmacs Dev
Team".  Note that packages maintained by the dev team typically have
much longer response times, etc, even when there is an upstream.

It's not obvious to me that it would be a win for Emacs to *move*
those packages to ELPA, even if it would reduce duplication of effort
and cooperation costs.  Ask the Gnus team.

 > On top of that, IMO every single core package should have a copy on
 > GNU ELPA so one can to overwrite the native GNU Emacs one with the
 > one from GNU ELPA.  This would decouple all packages from the Emacs
 > release cycle and allow bug fixes to be distributed instantly.

Bug fixes are already mostly distributed instantly, by bzr.

So AFAICS what you are claiming is that this would speed up
distribution of bug fixes to users who don't care enough about Emacs
to build it themselves.  This is very unlikely based on XEmacs
experience.  AFAICT, such users mostly get their fixes when they are
picked up by the OS distros, but guess what?  The distros don't
distribute upstream binary packages, they build their own from their
own repos which are (near) clones of Emacs's or upstream.

 > On top of that, this would make actual core emacs development,
 > test, release and bug management a lot easier.[1]

No.  It only helps if package distribution is completely decoupled
from Emacs distribution, but this would be very costly to the Emacs "I
can hack any part of Emacs" mindset because it would require codifying
a lot of APIs that are currently more or less internal.  (The
fungibility of "internal-ish" APIs causes me a certain amount of
hair-tearing every time we do a major sync to Emacs, since XEmacs is
by necessity and by preference more friendly to fixed APIs than
Emacs.)  It would also probably result in more integration bugs
getting through to end users.

 > [1] I realise this would require substantial modifications to pretty
 > much every single aspect of GNU Emacs and GNU ELPA.  Yet I think this is
 > an approach worth discussing.

As a gross estimate, it took Steve Baur and a few others about one
man-year of effort to break out the packages that were available in
XEmacs 20.4 for the XEmacs 20.5 release in 1996.  There are a lot more
packages in Emacs now, and it would probably be more effort in total
to do the same for Emacs.

Even in hindsight, I don't think the package split-off in XEmacs was a
mistake.  It didn't turn out as well as hoped, but it could have in
theory, even in the theory I have now.  But Emacs is in a different
situation.  GNU ELPA is successful, let it grow for a while and see
what happens before turning Emacs upside down. ;-)

Re: async 1.0

[Stephen J. Turnbull]

John Wiegley writes:

 > One thing that would make async.el _incredibly_ useful would be a `call/cc'
 > function in Emacs.

XEmacs efs (or maybe it's dired -- not to be confused with any dired
in Emacs IIRC, you need the XEmacs versions) has an implementation of
call/cc.  I don't know how close to "true" call/cc it is.  For correct
info, ask Mike Sperber <mike@xemacs.org>.

Re: async 1.0

[John Wiegley]

>>>>> Stephen J Turnbull <stephen@xemacs.org> writes:

> XEmacs efs (or maybe it's dired -- not to be confused with any dired in
> Emacs IIRC, you need the XEmacs versions) has an implementation of call/cc.
> I don't know how close to "true" call/cc it is.  For correct info, ask Mike
> Sperber <mike@xemacs.org>.

I'll have to check that out.

Eshell has a poor man's implementation of call/cc, called `eshell-do-eval'.
It throws `eshell-defer' as the equivalent of calling `call/cc', returning an
object you can `eval' later to resume execution.

The way it works is by taking the form you want to `eval' and transforming it
as it evaluates, performing substitutions so that (let ((a (+ 10 20))) ...)
becomes (let ((a 30)) ...), etc., on down to where you throw `eshell-defer'
(and of course removing the throw).  The next time you evaluate, no additional
evaluations are done until you reach the point where you deferred.

The downsides are: (a) it's slow to walk and eval in Lisp, (b) it doesn't work
with byte-compiled forms, and (c) it's limited to a subset of the Emacs Lisp
language, since each single special form needs to be treated specially, and I
only implemented transforms for the ones that Eshell actually used.  Oh, and
(a) it doesn't peer past function call boundaries.

That said, it could be extended to fully support Emacs Lisp; it would simply
require that nothing be byte-compiled.  Unless bytecode stream instrumentation
was added to the mix...

On the other hand, how hard would it be to make Emacs stackless and provide a
native call/cc that works not only with byte-compiled code, but with
interposing native function calls?  For example:

    (let* ((a (+ 10 20))
           (c '(1 2 3)))
           (d (mapcar #'(lambda (b) (+ b a) (call/cc)) c)))

I cannot think of any way to fake continuations here, other than having
intimate of every native function, and transforming the `let' into this:

    (let* ((a 30)
           (b '(2 3))
           (d (cons 1 (mapcar #'(lambda (b) (+ b a)) c))))

The problem gets pretty hairy fairly quickly.

John

Re: async 1.0

[Michael Sperber]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> John Wiegley writes:
>
>  > One thing that would make async.el _incredibly_ useful would be a `call/cc'
>  > function in Emacs.
>
> XEmacs efs (or maybe it's dired -- not to be confused with any dired
> in Emacs IIRC, you need the XEmacs versions) has an implementation of
> call/cc.  I don't know how close to "true" call/cc it is.  For correct
> info, ask Mike Sperber <mike@xemacs.org>.

It doesn't have call/cc, but large parts of it are effectively written
in CPS.  EFS also rolls its own version of closures.

-- 
Cheers =8-} Mike
Friede, Völkerverständigung und überhaupt blabla

Re: async 1.0

[Richard Stallman]

I didn't realize it could be so fast.  On my machine,
emacs -Q -batch --eval "(+ 10 20)" takes about 1/4 second.

--
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: async 1.0

[John Wiegley]

>>>>> Richard Stallman <rms@gnu.org> writes:

> I didn't realize it could be so fast.  On my machine, emacs -Q -batch --eval
> "(+ 10 20)" takes about 1/4 second.

What kind of hardware are you using?  The Emacs that I'm running in off an SSD
with a lot of cache RAM available.  If I do it on my laptop, which also has an
SSD, I get the same times: 0.03713s.

I've also found another use for async.el: sandboxing.  The following allows me
execute a lambda synchronously, but without affecting the host Emacs
environment:

    (async-get (async-start FUNC))

I see this idiom being useful enough that I've reduced it to one function,
(async-sandbox FUNC).

Here's an example of using sandboxing to byte-compile a file, without letting
`eval-when-compile' forms affect the host environment:

    (let ((proc
           (async-start
            `(lambda ()
               (require 'bytecomp)
               ,(async-inject-variables "\\`load-path\\'")
               (let ((default-directory ,(file-name-directory file)))
                 (add-to-list 'load-path default-directory)
                 (ignore-errors
                   (load ,file))
                 ;; returns nil if there were any errors
                 (prog1
                     (byte-compile-file ,file)
                   (load ,file)))))))

      (unless (condition-case err
                  (async-get proc)
                (error
                 (ignore (message "Error: %s" err))))
        (ignore (message "Recompiling %s...FAILED" file))))

John

Re: async 1.0

[Richard Stallman]

I am using an Yeeloong with an SSD that is somewhat slow.

--
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call

Re: async 1.0

[zwz]

This may be a little off the topic,
but I just came across a report about your bag getting stolen in Argentina.

Did you lost your Yeeloong? 


Richard Stallman <rms@gnu.org> writes:

> I am using an Yeeloong with an SSD that is somewhat slow.
>
> --
> Dr Richard Stallman
> President, Free Software Foundation
> 51 Franklin St
> Boston MA 02110
> USA
> www.fsf.org  www.gnu.org
> Skype: No way! That's nonfree (freedom-denying) software.
>   Use Ekiga or an ordinary phone call

Re: async 1.0

[Richard Stallman]

My Yeeloong was stolen.  Unfortunately we did not have a recent
backup for the software configuration, and I still have some problems.

--
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call