Async package.el

Async package.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 2562 bytes --]

This is a heads up that I've pushed some commits that implement
asynchronicity in package.el. I'll explain implementation details below,
but I want to list the surface changes first.

- New variable `package-menu-async'. If this is non-nil the package menu
uses asynchronous operations when refreshing the archives and when
installing packages. For the purpose of testing on the devlist, I've set
the default to t. We can discuss whether we prefer this on or off when
Emacs gets released.

- For the sake of simplicity, when you hit `x' in the package-menu you will
no longer get two separate prompts for install/delete. You'll be prompted a
single time for everything.

- If you don't use the package-menu, nothing has changed on the surface.
That is, if you invoke `M-x package-install' or call this function with a
single argument everything will be synchronous. But the implementation is
there (determined by the function's 3rd and 4th args), so we could use the
prefix arg to specify async here.

Cheers to all,
Artur


-----
Implementation (for those who care)
-----

- Most functions which involve downloading data now take an extra optional
ASYNC argument (some of them also take an additional CALLBACK argument).
This defaults to nil. If it is t, any downloads involved are performed
asynchronously (and, if appropriate, CALLBACK is called afterwards).

- The only really async function (actually a macro) is
`package--with-work-buffer-async'. All other functions simply propagate
their ASYNC argument down to this one. If ASYNC nil, it just defers the
work back to `package--with-work-buffer'.

- Archives are refreshed all at the same time (which leads to a signficant
speed boost if you have several archives configured), so there's a new
variable `package--downloads-in-progress' to keep track of which archive
downloads are still ongoing. The function
`package--update-downloads-in-progress' takes care of controling this
variable and calling a relevant hook when it's all done.

- Package downloads are performed in sequence, because it is generally not
safe to do them in parallel (and deciding when/how it is safe is
complicated). So there's no variable to keep track of them.
`package-menu--perform-transaction' takes care to call `package-install' on
one package at a time (and possibly calls `package-delete' afterwards). In
turn, `package-install' takes care to install everything in the order
specified by `package-compute-transaction' (which ensures dependencies come
before the package itself). This is all done with recursive callbacks.

[-- Attachment #2: Type: text/html, Size: 2860 bytes --]

Re: Async package.el

[Dmitry Gutov]

Hi Artur,

On 04/06/2015 01:46 PM, Artur Malabarba wrote:
> This is a heads up that I've pushed some commits that implement
> asynchronicity in package.el. I'll explain implementation details below,
> but I want to list the surface changes first.

One change I've noticed recently, is that the last few times I ran `M-x 
list-packages', it didn't display the "new" packages at the top.

Might that be related to the recent changes?

> - For the sake of simplicity, when you hit `x' in the package-menu you
> will no longer get two separate prompts for install/delete. You'll be
> prompted a single time for everything.

I think "INSTALL (xxx-y.z) and DELETE (xxx-z.t)" is needlessly verbose. 
It should probably be "Upgrade (xxx to z.t)", with maybe "and delete the 
currently installed versions".

> - Most functions which involve downloading data now take an extra
> optional ASYNC argument (some of them also take an additional CALLBACK
> argument). This defaults to nil. If it is t, any downloads involved are
> performed asynchronously (and, if appropriate, CALLBACK is called
> afterwards).

Shouldn't there be some new tests?

> - The only really async function (actually a macro) is
> `package--with-work-buffer-async'. All other functions simply propagate
> their ASYNC argument down to this one. If ASYNC nil, it just defers the
> work back to `package--with-work-buffer'.

I have to say that package upgrading being only partially asynchronous 
looks perplexing. You say yes - it starts downloading stuff in the 
background - I can move around in the buffer, then suddenly the 
background process takes over in a synchronous fashion - I can't move 
cursor anymore and just watch some messages in the echo area. The 
Compile-Log only pops up at the end.

While parallel downloading sounds good, maybe the overall interface 
should still be presented in a synchronous fashion.

> - Archives are refreshed all at the same time (which leads to a
> signficant speed boost if you have several archives configured), so
> there's a new variable `package--downloads-in-progress' to keep track of
> which archive downloads are still ongoing. The function
> `package--update-downloads-in-progress' takes care of controling this
> variable and calling a relevant hook when it's all done.

I think this is the main upside of the latest changes, currently.

> - Package downloads are performed in sequence, because it is generally
> not safe to do them in parallel

Why not? Simply downloading in parallel should be safe. Maybe not all at 
once, but 2-4 at a time, especially if they come from different 
archives, would be beneficial.

Re: Async package.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 2677 bytes --]

Thanks for all the feedback Dmitry,

> One change I've noticed recently, is that the last few times I ran `M-x
> list-packages', it didn't display the "new" packages at the top.
>
> Might that be related to the recent changes?
>
Yes. I'm fixing it now.

>
>>    - For the sake of simplicity, when you hit `x' in the package-menu
>>    you
>>    will no longer get two separate prompts for install/delete. You'll be
>>    prompted a single time for everything.
>>
>>  I think "INSTALL (xxx-y.z) and DELETE (xxx-z.t)" is needlessly verbose.
>> It should probably be "Upgrade (xxx to z.t)", with maybe "and delete the
>> currently installed versions".
>>
>  I agree.

>
>>    - Most functions which involve downloading data now take an extra
>>    optional ASYNC argument (some of them also take an additional
>>    CALLBACK
>>    argument). This defaults to nil. If it is t, any downloads involved
>>    are
>>    performed asynchronously (and, if appropriate, CALLBACK is called
>>    afterwards).
>>
>>  Shouldn't there be some new tests?
>
> Yes. I'm still trying to figure out how to do async tests. Currently all
package.el tests use a local directory as the repository, but the async
functions only kick-in for remote repositories.
Should I use GNU Elpa for the tests? Might that cause hydra to unexpectedly
complain due to random connection issues?

> I have to say that package upgrading being only partially asynchronous
> looks perplexing. You say yes - it starts downloading stuff in the
> background - I can move around in the buffer, then suddenly the background
> process takes over in a synchronous fashion - I can't move cursor anymore
> and just watch some messages in the echo area. The Compile-Log only pops up
> at the end.
>
> While parallel downloading sounds good, maybe the overall interface should
> still be presented in a synchronous fashion.
>
I agree that's annoying and I have some similar plans. We can change the
package-installation logic to first download all files simultaneously, and
then unpack/compile all of them in the proper order. So large upgrades
would have a significant speed improvement.

>
>>    - Package downloads are performed in sequence, because it is
>>    generally
>>    not safe to do them in parallel
>>
>>  Why not? Simply downloading in parallel should be safe. Maybe not all
> at once, but 2-4 at a time, especially if they come from different
> archives, would be beneficial.
>
> I mis-expressed myself. Package downloads are safe in parallel, what makes
it non-trivial is that the subsequent unpacking/compiling needs to be done
in the right order (curently a package is installed as soon as its download
finishes).

[-- Attachment #2: Type: text/html, Size: 4151 bytes --]

Re: Async package.el

[Dmitry Gutov]

On 04/06/2015 05:32 PM, Artur Malabarba wrote:

> Yes. I'm fixing it now.

Thank you.

> I agree.

I'd also prefer to revert to use the sentence case (non-caps).

b6610d55470c7e835472a581977ab6fad537c8b6 did that change, despite having 
the word "refactor" in its message. :P

> Yes. I'm still trying to figure out how to do async tests. Currently all
> package.el tests use a local directory as the repository, but the async
> functions only kick-in for remote repositories.
> Should I use GNU Elpa for the tests? Might that cause hydra to
> unexpectedly complain due to random connection issues?

I think ideally you'd run a small package archive server locally, 
launched with a tiny Python script.

> I agree that's annoying and I have some similar plans. We can change the
> package-installation logic to first download all files simultaneously,
> and then unpack/compile all of them in the proper order. So large
> upgrades would have a significant speed improvement.

Right, that should be optimal, but the requires changes are probably 
non-trivial.

To take this further (and forgive the negativity), maybe we should 
back-pedal on adding the asynchronous interface to downloading package 
archives, too. While doing it in parallel is great, refreshing the table 
of packages right under the user's nose is bound to create problems. 
Here's one:

M-x list-packages, press `i' in the displayed list, wait for the refresh 
to be done, see the `I' disappear.

Can you imagine something like Ubuntu Software Center doing this and 
updating the packages list while the user's interacting with the 
interface? Aside from it being confusing, the more interface elements 
there are, the higher are the odds of this kind of update breaking 
something.

While `list-packages' is far from complexity of analogous tools, maybe 
we should walk in that direction without having to support the 
complexity of asynchronously updating the packages list.

The slowness of `list-packages' launch could be tackled in different 
ways. We want to have up-to-date information, but we don't have to:

- Update it each time `M-x list-packages' is run. Doing it only once a 
day, by default, should be sufficient for most uses. GNU ELPA is updated 
less often than that anyway.

- Wait until the user invokes `M-x list-packages' to update it. We could 
do it while Emacs is idle. Of course, we user might end up not doing 
anything the packages in the current session anyway, so it'll require 
some experimenting. Maybe check that `M-x list-packages' was called at 
least once since the idle update, and, conversely, still perform the 
synchronous update at the beginning of `list-packages' if the archives 
haven't been updated for a while.

Both approaches should also lower the packages archives' server load, at 
the cost of being slightly less up-to-date.

Re: Async package.el

[Stefan Monnier]

> To take this further (and forgive the negativity), maybe we should
> back-pedal on adding the asynchronous interface to downloading package
> archives, too.

I strongly disagree.

> While doing it in parallel is great, refreshing the table of
> packages right under the user's nose is bound to create problems.

PCL-CVS and VC-Dir have been doing the same for years with
great success.  It shouldn't cause problems.  But indeed, to make it
work well requires extra care.

> - Wait until the user invokes `M-x list-packages' to update it. We could do
> it while Emacs is idle.

I suspect you mean to add a "Don't" at the very beginning.  It'd be
acceptable to add such a (mis)feature, but it should very clearly be OFF
by default, since we don't want Emacs to "call home" in such a way
by default.


        Stefan

Re: Async package.el

[Dmitry Gutov]

On 04/08/2015 12:46 AM, Stefan Monnier wrote:

> I strongly disagree.

I'm not sure you actually do. By "interface", I mean how the user works 
with it, not whether we use asynchronous calls under the hood (which, 
with care, usually can improve the user experience).

>> While doing it in parallel is great, refreshing the table of
>> packages right under the user's nose is bound to create problems.
>
> PCL-CVS and VC-Dir have been doing the same for years with
> great success.  It shouldn't cause problems.  But indeed, to make it
> work well requires extra care.

I haven't used PCL-CVS, but VC-Dir is a good counter-example: while it's 
refreshed asynchronously, we don't really expect the user to do much 
while that happens, other than maybe look at the output. It's just nicer 
to have a responsive Emacs during this operation (which could be long in 
certain cases).

And when vc-dir buffer is initially displayed, we don't have any "stale" 
data about the repository. First the data is fetched (though 
asynchronously), then the user can interact with it.

>> - Wait until the user invokes `M-x list-packages' to update it. We could do
>> it while Emacs is idle.
>
> I suspect you mean to add a "Don't" at the very beginning.  It'd be

I kinda did, implicitly (the last sentence before the list ended with 
"we don't have to"). Admittedly, it's awkward wording.

> acceptable to add such a (mis)feature, but it should very clearly be OFF
> by default, since we don't want Emacs to "call home" in such a way
> by default.

Fair enough. I'm also more partial to the option 1 (only refresh once a 
day), because it's more predictable, but the above would be closer to 
how other systems and applications look for updates (in the background, 
without bothering the user), so it should be something to consider.

Re: Async package.el

[Stefan Monnier]

> I haven't used PCL-CVS, but VC-Dir is a good counter-example: while it's
> refreshed asynchronously, we don't really expect the user to do much while
> that happens, other than maybe look at the output. It's just nicer to
> have a responsive Emacs during this operation (which could be long in
> certain cases).

I think the issue is that currently, when the async download ends, we
just completely rebuild the package-list, losing all previous info.
This needs to be fixed.  In PCL-CVS and in VC-Dir, when the async
operation ends, we merge the result with the current buffer's content.
So the user can mark/unmark elements which the async operation is in
process and these things aren't lost afterwards (and neither is the
position of point).

> And when vc-dir buffer is initially displayed, we don't have any "stale"
> data about the repository.

In PCL-CVS we did.

>> acceptable to add such a (mis)feature, but it should very clearly be OFF
>> by default, since we don't want Emacs to "call home" in such a way
>> by default.
> Fair enough. I'm also more partial to the option 1 (only refresh
> once a day), because it's more predictable, but the above would be closer to
> how other systems and applications look for updates (in the background,
> without bothering the user), so it should be something to consider.

I'm perfectly OK with changing the "auto-update" so that it's not done
at every call to list-packages (e.g. once a day is perfectly fine,
indeed.  I'd even agree with "never" and just let the user request an
update manually).  What I object to, is to do it even before list-packages
is invoked.


        Stefan

Re: Async package.el

[Dmitry Gutov]

On 04/08/2015 04:32 PM, Stefan Monnier wrote:

> This needs to be fixed.  In PCL-CVS and in VC-Dir, when the async
> operation ends, we merge the result with the current buffer's content.

I think packages are a bit more complex than entries in PCL-CVS and 
VC-Dir, and the effort spend on it might have gone to something else. 
But of course, people only work on what's interesting to them anyway.

> So the user can mark/unmark elements which the async operation is in
> process and these things aren't lost afterwards (and neither is the
> position of point).

We also have `describe-package' buffers, some of which might be open and 
get out of date after one such update (which is probably not that big a 
deal).

Further, we have keyword filtering. So the update would have to keep 
that on too. Any similar feature would also require a change here.

>> And when vc-dir buffer is initially displayed, we don't have any "stale"
>> data about the repository.
>
> In PCL-CVS we did.

Ok, true.

> I'm perfectly OK with changing the "auto-update" so that it's not done
> at every call to list-packages (e.g. once a day is perfectly fine,
> indeed.  I'd even agree with "never" and just let the user request an
> update manually).  What I object to, is to do it even before list-packages
> is invoked.

We could only start those idle updates after list-packages had been 
invoked at least once.

Re: Async package.el

[raman]

Also, attempting to make package async shows up a problem that is lying
in wait as Emacs gets multithreading capabilities -- most elisp packages
are written assuming a sync/blocking world -- especially with respect to
user-feedback. So even as package downloads and installs things
asynchronously, the underlying elisp modules that download, unpack and
compile output messages continuously in the echo area -- wonder how
distracting that is to someone who can see. I can say that it well-nigh
makes things impossible with Emacspeak at present --  I need to find a
way of telling #'message   not to talk when called from an async
task  -- or more generally, if called from anything except the main UI
thread. Any ideas?

Re: Async package.el

[Rasmus]

Hi,

raman <tv.raman.tv@gmail.com> writes:

> I need to find a
> way of telling #'message   not to talk when called from an async
> task  -- or more generally, if called from anything except the main UI
> thread. Any ideas?

Is something like this too hackish?

(cl-letf (((symbol-function 'message) 'ignore))
         whatever)

In general I agree that it would be good if there existed a way to mute
Emacs while running something.  This is also useful for scripts,
e.g. exporting (org) documents from the bash prompt.

—Rasmus

-- 
One thing that is clear: it's all down hill from here 

Re: Async package.el

[Artur Malabarba]

>> I need to find a
>> way of telling #'message   not to talk when called from an async
>> task  -- or more generally, if called from anything except the main UI
>> thread. Any ideas?
>
> Is something like this too hackish?
>
> (cl-letf (((symbol-function 'message) 'ignore))
>          whatever)
>

This is generally what I do too. IIRC, the only drawback is that this
uses unwind-protect, which is not triggered by the max-lisp-eval-depth
error. So if you hit that error, message's function definition won't
be restored.
You can either just live with that, or you can wrap the code inside
the `cl-letf' in a `condition-case' statement and to catch that error.

Re: Async package.el

[Artur Malabarba]

There are some C-level functions that send messages without using
`message', but these are very few.
There's also an emacs package dedicated to muting messages, I think
it's called `shut-up' (or something like that).

2015-04-09 15:50 GMT+01:00 Artur Malabarba <bruce.connor.am@gmail.com>:
>>> I need to find a
>>> way of telling #'message   not to talk when called from an async
>>> task  -- or more generally, if called from anything except the main UI
>>> thread. Any ideas?
>>
>> Is something like this too hackish?
>>
>> (cl-letf (((symbol-function 'message) 'ignore))
>>          whatever)
>>
>
> This is generally what I do too. IIRC, the only drawback is that this
> uses unwind-protect, which is not triggered by the max-lisp-eval-depth
> error. So if you hit that error, message's function definition won't
> be restored.
> You can either just live with that, or you can wrap the code inside
> the `cl-letf' in a `condition-case' statement and to catch that error.

async message

[Ivan Shmakov]

>>>>> Rasmus  <rasmus@gmx.us> writes:
>>>>> Hi, raman <tv.raman.tv@gmail.com> writes:

 >> I need to find a way of telling #'message not to talk when called
 >> from an async task — or more generally, if called from anything
 >> except the main UI thread.  Any ideas?

 > Is something like this too hackish?

 > (cl-letf (((symbol-function 'message) 'ignore)) whatever)

	While I tend to agree there’s a problem with the echo area, I’d
	rather have ‘message’ /still/ add to the *Messages* buffer even
	when otherwise “muted”.

[…]

-- 
FSF associate member #7257  http://boycottsystemd.org/  … 3013 B6A0 230E 334A

Re: async message

[raman]

Adding to the message buffer while muted would definitely be a good
thing.

I dont believe any of the calls to #'message come from the native C
layer in the case of package.el

Re: Async package.el

[raman]

Unclear how well that hack would work -- though cl-let is a win to an extent.

Re: Async package.el

[Stefan Monnier]

> Also, attempting to make package async shows up a problem that is lying
> in wait as Emacs gets multithreading capabilities -- most elisp packages
> are written assuming a sync/blocking world -- especially with respect to
> user-feedback. So even as package downloads and installs things
> asynchronously, the underlying elisp modules that download, unpack and
> compile output messages continuously in the echo area

AFAIK those messages are under the control of package.el, so we
can/should simply fix those.  E.g. compilation shouldn't emit any
messages, since warnings and such should go to a buffer instead.

Basically when installing a (set of) packages, there should only be
3 messages, only the last 2 being "asynchronous":

    downloading...
    installing...
    installing...done


-- Stefan

Re: Async package.el

[raman]

Sounds reasonable. I was afraid the "Extracting ..." messages  came from
Emacs utilities that package uses to unpack an archive, if not, then
this is easier to solve.

Re: Async package.el

[Artur Malabarba]

> AFAIK those messages are under the control of package.el, so we
> can/should simply fix those.  E.g. compilation shouldn't emit any
> messages, since warnings and such should go to a buffer instead.

But it does. `byte-recompile-directory' emits 3 messages per file
being compiled ("Checking", "Compiling", and "Wrote"), and 1 message
per file not to be compiled ("Checking").
These are not under the control of package.el, and I'd appreciate if
someone could do something about them.

Maybe only print to the compilation buffer, or only message when the
command was called interactively.

Re: Async package.el

[Stefan Monnier]

> Maybe only print to the compilation buffer, or only message when the
> command was called interactively.

Changing the behavior based on whether the call is interactive
sounds like a good solution.


        Stefan

Re: Async package.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 2017 bytes --]

Somehow I hadn't seen this message before.

On Apr 7, 2015 2:19 AM, "Dmitry Gutov" <dgutov@yandex.ru> wrote:
> I'd also prefer to revert to use the sentence case (non-caps).

I changed it to caps because, if there's a long list of packages to be
installed, it's a little easy to overlook the fact that you're also being
asked about a deletion.
Maybe just the "Delete" word should be all-caps, and only if there are
other operations in the same message. Or maybe "Delete" should just come
first.

> I think ideally you'd run a small package archive server locally,
launched with a tiny Python script.

I'd appreciate any help with that. The only server I know how to run
nowadays is with Jekyll.

> Right, that should be optimal, but the requires changes are probably
non-trivial.

Indeed.

> To take this further (and forgive the negativity), maybe we should
back-pedal on adding the asynchronous interface to downloading package
archives, too. While doing it in parallel is great, refreshing the table of
packages right under the user's nose is bound to create problems. Here's
one:
>
> M-x list-packages, press `i' in the displayed list, wait for the refresh
to be done, see the `I' disappear.

I agree this needs addressing, but it will take a lot more than that for me
to backpedal on the archive refreshing part. I'm actually very happy with
the resulting UX. (Unlike the package-installation part, which I'm still
not thrilled about).

One way to address this is to simply not regenerate the buffer if anything
has been marked. In this situation, the "refresh finished" message can be
accompanied by a "hit g to revert buffer" message.
This would be easily scalable. Whenever a new feature is added which
involves some semi persistent information, we'd just extend the definition
of "anything has been marked".

A second, more sophisticated way would be to not revert the buffer at all.
Instead, we carefully update the information currently displayed in the
buffer. Though this is more troublesome, of course.

[-- Attachment #2: Type: text/html, Size: 2416 bytes --]

Re: Async package.el

[Dmitry Gutov]

On 04/08/2015 02:26 AM, Artur Malabarba wrote:
> Somehow I hadn't seen this message before.

Ok. I figured you were just taking your time composing the reply.

> I changed it to caps because, if there's a long list of packages to be
> installed, it's a little easy to overlook the fact that you're also
> being asked about a deletion.

Well, I don't know. Usually it's just the one word (UPGRADE), and by 
itself it looks grating.

> Maybe just the "Delete" word should be all-caps, and only if there are
> other operations in the same message.

Use bold face, but keep it sentence-case?

> Or maybe "Delete" should just come first.

Yeah, this would probably be the best approach.

> I'd appreciate any help with that. The only server I know how to run
> nowadays is with Jekyll.

The first example runs a server that will serve the current directory: 
http://www.linuxjournal.com/content/tech-tip-really-simple-http-server-python

We don't need much more than that, right?

> I agree this needs addressing, but it will take a lot more than that for
> me to backpedal on the archive refreshing part. I'm actually very happy
> with the resulting UX. (Unlike the package-installation part, which I'm
> still not thrilled about).

Personally, I'm a little annoyed with the blinking. And the small 
stuttering when the table is being regenerated. And the fact that I can 
miss the message when the archives get refreshed just by interacting 
with Emacs (maybe flooring C-n right after the buffer is displayed), and 
then remain vaguely unsure the refresh has happened. Yeah, most of this 
could just be the aversion to change. :)

Here's a more subtle scenario, by the way (but easier to fix): M-x 
list-packages, then PageDown. Wait for the refresh -> see the current 
line get scrolled to the middle of the window.

> One way to address this is to simply not regenerate the buffer if
> anything has been marked. In this situation, the "refresh finished"
> message can be accompanied by a "hit g to revert buffer" message.
> This would be easily scalable. Whenever a new feature is added which
> involves some semi persistent information, we'd just extend the
> definition of "anything has been marked".

I guess that would work.

> A second, more sophisticated way would be to not revert the buffer at
> all. Instead, we carefully update the information currently displayed in
> the buffer. Though this is more troublesome, of course.

And more dangerous, at least theoretically. Here's a made up scenario: I 
select a package, the archive updates, it contains a new version of the 
same package, which has unresolved dependencies (a situation which we 
mark as "incompatible"). Whether we transfer the installation mark or 
not, the user could install the package they wanted (at least if the 
archive server still contains the file); now they can't. It might have 
even jumped away to the bottom.

Re: Async package.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 3142 bytes --]

> The first example runs a server that will serve the current directory:
http://www.linuxjournal.com/content/tech-tip-really-simple-http-server-python
>
> We don't need much more than that, right?

I think that should be enough, thanks.

>> I agree this needs addressing, but it will take a lot more than that for
>> me to backpedal on the archive refreshing part. I'm actually very happy
>> with the resulting UX. (Unlike the package-installation part, which I'm
>> still not thrilled about).
>
>
> Personally, I'm a little annoyed with the blinking. And the small
stuttering when the table is being regenerated.

Next in my list of TODOs is to change the behaviour when package-menu-async
is nil. Right now, it just reverts to the old behaviour (sequential
downloads + hang interface), but I plan to make it do "parallel downloads +
hang the interface", which might be more to your liking.

> And the fact that I can miss the message when the archives get refreshed
just by interacting with Emacs (maybe flooring C-n right after the buffer
is displayed), and then remain vaguely unsure the refresh has happened.

Also next on the list is a visual cue to indicate there's an ongoing
operation. Something like this:



So there should be no doubt about when has the update finished (this
spinner will be local to the packages buffer, so it won't distract you if
you're trying to work). I'm not sure how many will find this "useful
information" and how many will think it "annoying distraction from hell".

> Here's a more subtle scenario, by the way (but easier to fix): M-x
list-packages, then PageDown. Wait for the refresh -> see the current line
get scrolled to the middle of the window.

>
>
>> One way to address this is to simply not regenerate the buffer if
>> anything has been marked. In this situation, the "refresh finished"
>> message can be accompanied by a "hit g to revert buffer" message.
>> This would be easily scalable. Whenever a new feature is added which
>> involves some semi persistent information, we'd just extend the
>> definition of "anything has been marked".
>
>
> I guess that would work.
>
>
>> A second, more sophisticated way would be to not revert the buffer at
>> all. Instead, we carefully update the information currently displayed in
>> the buffer. Though this is more troublesome, of course.
>
>
> And more dangerous, at least theoretically. Here's a made up scenario: I
select a package, the archive updates, it contains a new version of the
same package, which has unresolved dependencies (a situation which we mark
as "incompatible"). Whether we transfer the installation mark or not, the
user could install the package they wanted (at least if the archive server
still contains the file); now they can't. It might have even jumped away to
the bottom.

If I understand the way archive servers work, they all just offer the
latest package version. So, in this scenario, trying to install the old
(compatible) version would lead to 404 error anyway. If that's the case,
then, arguably, removing the user's install mark would be the correct
choice (although an explanatory message might be in order).

[-- Attachment #2: Type: text/html, Size: 3946 bytes --]

Re: Async package.el

[Dmitry Gutov]

On 04/08/2015 12:43 PM, Artur Malabarba wrote:

> Next in my list of TODOs is to change the behaviour when
> package-menu-async is nil. Right now, it just reverts to the old
> behaviour (sequential downloads + hang interface), but I plan to make it
> do "parallel downloads + hang the interface", which might be more to
> your liking.

I think if you intend to keep the asynchronous interface, first we 
should try to make it work for most people. It's not that I absolutely 
hate it anyway.

Maybe try the "merge individual items a la PCL-CVS" strategy, as 
suggested by Stefan.

> Also next on the list is a visual cue to indicate there's an ongoing
> operation. Something like this:

Spinners are cute, but maybe we can start with a [waiting...] mode-line 
entry, a la vc-dir.

> So there should be no doubt about when has the update finished (this
> spinner will be local to the packages buffer, so it won't distract you
> if you're trying to work). I'm not sure how many will find this "useful
> information" and how many will think it "annoying distraction from hell".

Thanks. I think it's useful, and it has precedent, mentioned above.

>  > Here's a more subtle scenario, by the way (but easier to fix): M-x
> list-packages, then PageDown. Wait for the refresh -> see the current
> line get scrolled to the middle of the window.

^ You haven't commented on this one.

>  >> One way to address this is to simply not regenerate the buffer if
>  >> anything has been marked. In this situation, the "refresh finished"
>  >> message can be accompanied by a "hit g to revert buffer" message.
>  > I guess that would work.

Let's think about this: suppose the user has selected a few items with 
'i', then she's going to press 'x'.

- If she hasn't managed to hit it yet, are we comfortable with 
overtaking the interface just before that, with the "would you like to 
reload" question? If we do, we both interrupt the user's train of 
thought and risk confusing her with "press y on n" if she presses 'x' by 
inertia right after the new prompt.

- If she's pressed 'x' already, will we be able to avoid interrupting 
the subsequent 'y or n' prompt? Then, when she presses 'y', will the 
"would you like to revert" prompt appear immediately?

If she chooses to revert, could that lead to problems with installation, 
by introducing inconsistency in the package information?

> If I understand the way archive servers work, they all just offer the
> latest package version. So, in this scenario, trying to install the old
> (compatible) version would lead to 404 error anyway. If that's the case,
> then, arguably, removing the user's install mark would be the correct
> choice (although an explanatory message might be in order).

GNU ELPA does keep the previous versions, see the "old versions" list: 
http://elpa.gnu.org/packages/let-alist.html

Even if other package archives don't, keeping older versions is a valid 
direction for improving package.el. We don't want to make it harder.

Re: Async package.el

[Artur Malabarba]

>> Next in my list of TODOs is to change the behaviour when
>> package-menu-async is nil. Right now, it just reverts to the old
>> behaviour (sequential downloads + hang interface), but I plan to make it
>> do "parallel downloads + hang the interface", which might be more to
>> your liking.
>
> I think if you intend to keep the asynchronous interface, first we should
> try to make it work for most people. It's not that I absolutely hate it
> anyway.

It's also almost trivial to just hang the interface while a variable
is non-nil. So it's no sacrifice to implement it.

> Maybe try the "merge individual items a la PCL-CVS" strategy, as suggested
> by Stefan.

Yes, I do prefer this strategy too. For now I'm letting it bounce
around in my head and mature a bit while I focus on some work stuff,
but, as long as it doesn't turn out to be a monumental effort, I'll
end up writing it at some point (unless someone else beats me to it).

>> Also next on the list is a visual cue to indicate there's an ongoing
>> operation. Something like this:
>
> Spinners are cute, but maybe we can start with a [waiting...] mode-line
> entry, a la vc-dir.

Done. We can discuss if we want to improve it.

>>  > Here's a more subtle scenario, by the way (but easier to fix): M-x
>> list-packages, then PageDown. Wait for the refresh -> see the current
>> line get scrolled to the middle of the window.
>
>
> ^ You haven't commented on this one.

Sorry. I figured this will get automatically fixed by any of the
options already being discussed. For instance, if we do the "merge
individual items", then nothing in the algorithm would recenter the
window and this inconvenience should disapear.
AFAICT, the only case where scrolling can happen in this scenario is
if N new packages are added to the top while the cursor was less than
N lines from the bottom of the window, and we should be able to avoid
a complete recentering by let-binding `scroll-step' to a small value.
(but I may be going too deep into the technicalities of something
that's not even impemented).

>>  >> One way to address this is to simply not regenerate the buffer if
>>  >> anything has been marked. In this situation, the "refresh finished"
>>  >> message can be accompanied by a "hit g to revert buffer" message.
>>  > I guess that would work.
>
>
> Let's think about this: suppose the user has selected a few items with 'i',
> then she's going to press 'x'.
>
> - If she hasn't managed to hit it yet, are we comfortable with overtaking
> the interface just before that, with the "would you like to reload"
> question? If we do, we both interrupt the user's train of thought and risk
> confusing her with "press y on n" if she presses 'x' by inertia right after
> the new prompt.

The suggestion was not to use a question prompt, but to simply message
"Package refresh done, type g to revert buffer" instead of the ususal
"Package refresh done".

> - If she's pressed 'x' already, will we be able to avoid interrupting the
> subsequent 'y or n' prompt? Then, when she presses 'y', will the "would you
> like to revert" prompt appear immediately?

I thought we could use  `(minibuffer-window-active-p
(selected-window))' to check whether the minibuffer is active before
sending the message. Unfortunately, although this works with
`read-string' it doesn't work with `read-key' (which is what
`y-or-n-p' uses).

> If she chooses to revert, could that lead to problems with installation, by
> introducing inconsistency in the package information?

No. If she reverts, any "i" marks will be forgotten. This is a flaw of
this approach, but one fortunate consequence is that there won't be
any such inconsistency problems.
> GNU ELPA does keep the previous versions, see the "old versions" list:
> http://elpa.gnu.org/packages/let-alist.html
>
> Even if other package archives don't, keeping older versions is a valid
> direction for improving package.el. We don't want to make it harder.

Well, we're not really making it harder. This behaviour has always
been the case. As soon as you `list-packages', old-compatible packages
get forgotten in exchange for new-incompatible ones. The only
difference is that now the user has a brief chance to see the old
package before it is brutally taken beyond her reach.
I think I can fix this if we want, but I don't think it's a flaw of
the async interface.

Re: Async package.el

[Dmitry Gutov]

On 04/08/2015 09:39 PM, Artur Malabarba wrote:

> Done. We can discuss if we want to improve it.

Looks fine to me.

Here's another weird thing I see now, which again could be unrelated:

I have two archives configured: GELPA and MELPA. After Emacs was 
launched, I can `M-x list-packages', and the list starts with 2048-game, 
@ and abc-mode. 4clojure (and many other packages) is listed as "incompat".

But after loading ends, 4clojure gets inserted at the second place. If I 
quit it and call `M-x list-packages' in the same session, 4clojure is there.

But if I restart Emacs, 4clojure is again missing at the beginning until 
the loading is over.

> Sorry. I figured this will get automatically fixed by any of the
> options already being discussed. For instance, if we do the "merge
> individual items", then nothing in the algorithm would recenter the
> window and this inconvenience should disapear.

Of course, I just wanted to make sure you haven't entirely missed it. 
"Yup" would've been okay. :)

> The suggestion was not to use a question prompt, but to simply message
> "Package refresh done, type g to revert buffer" instead of the ususal
> "Package refresh done".

If it's a message, it'll disappear after the next command. Then, I can 
be only sure the list is refreshed if I don't select any items.

And this information won't be present in the interface, aside from the 
transient message. Kinda suboptimal.

> No. If she reverts, any "i" marks will be forgotten. This is a flaw of
> this approach, but one fortunate consequence is that there won't be
> any such inconsistency problems.

No, I mean if she agrees to revert after the installation has started 
(or maybe just the downloading part started, and hasn't finished yet).

> The only
> difference is that now the user has a brief chance to see the old
> package before it is brutally taken beyond her reach.

One could say it's worse than never seeing it, though. Wasted effort.

Re: Async package.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 2178 bytes --]

By the way, thanks for taking the time to try it out and think of all
possible issues.l

> I have two archives configured: GELPA and MELPA. After Emacs was
launched, I can `M-x list-packages', and the list starts with 2048-game, @
and abc-mode. 4clojure (and many other packages) is listed as "incompat".
>
> But after loading ends, 4clojure gets inserted at the second place. If I
quit it and call `M-x list-packages' in the same session, 4clojure is there.

I don't get that bevahiour from emacs -Q, but there was a problem with the
incompatible table a few days ago. Maybe your elc file is outdated?

>> The suggestion was not to use a question prompt, but to simply message

>> "Package refresh done, type g to revert buffer" instead of the ususal
>> "Package refresh done".
>
> If it's a message, it'll disappear after the next command. Then, I can be
only sure the list is refreshed if I don't select any items.
>
> And this information won't be present in the interface, aside from the
transient message. Kinda suboptimal.

Just like any other keybind. That's more of a general issue with the
package menu or Emacs itself. We can add "g-redisplay" to the quick-help
menu under `h' (which, btw, could use some improving.

>> No. If she reverts, any "i" marks will be forgotten. This is a flaw of
>> this approach, but one fortunate consequence is that there won't be
>> any such inconsistency problems.
>
>
> No, I mean if she agrees to revert after the installation has started (or
maybe just the downloading part started, and hasn't finished yet).

Ah, I see. Then we're fine. All information needed for a transaction is
gathered the moment you hit `x' (before you even answer `y'). Whatever
happens to the buffer after that is irrelevant.

>> The only
>> difference is that now the user has a brief chance to see the old
>> package before it is brutally taken beyond her reach.
>
>
> One could say it's worse than never seeing it, though. Wasted effort.

Yes, but you're wasting a few seconds of effort on a situation that will
happen very rarely. If that were a frequent occasion it might even be
infuriating, but how often do compatible packages become incompatible?

[-- Attachment #2: Type: text/html, Size: 2616 bytes --]

Re: Async package.el

[Dmitry Gutov]

On 04/09/2015 01:14 PM, Artur Malabarba wrote:
> By the way, thanks for taking the time to try it out and think of all
> possible issues.l

No problem, being critical is easy.

> I don't get that bevahiour from emacs -Q, but there was a problem with
> the incompatible table a few days ago. Maybe your elc file is outdated?

'emacs -Q' won't let you reproduce this, because (I think) it won't load 
~/.emacs.d/elpa/archives/melpa/archive-contents during `package-initilize'.

I've pulled the latest changes, re-bootstrapped Emacs, removed 
~/.emacs.d/elpa/archives, but after restarting Emacs, the problem persists.

> Just like any other keybind. That's more of a general issue with the
> package menu or Emacs itself. We can add "g-redisplay" to the quick-help
> menu under `h' (which, btw, could use some improving.

Certainly. But `g' is a common keybinding.

My point was different: in VC-Dir, when [waiting...] disappears from the 
mode-line, I can be sure that the list is not outdated.

Not so with your proposal. "The list is outdated" - this information 
won't be apparent from the interface.

Anyway, merging the items on the fly would solve that.

> Ah, I see. Then we're fine. All information needed for a transaction is
> gathered the moment you hit `x' (before you even answer `y'). Whatever
> happens to the buffer after that is irrelevant.

Good.

> Yes, but you're wasting a few seconds of effort on a situation that will
> happen very rarely. If that were a frequent occasion it might even be
> infuriating, but how often do compatible packages become incompatible?

Indeed, that should be rare. I don't have a scarier scenario to offer 
right now, so maybe we can ignore this downside (but preferably handle 
the possibility with care; and not move the "to install" mark to the 
now-incompatible package).

Re: Async package.el

[Artur Malabarba]

2015-04-09 13:34 GMT+01:00 Dmitry Gutov >> I don't get that bevahiour
from emacs -Q, but there was a problem with
>> the incompatible table a few days ago. Maybe your elc file is outdated?
>
>
> 'emacs -Q' won't let you reproduce this, because (I think) it won't load
> ~/.emacs.d/elpa/archives/melpa/archive-contents during `package-initilize'.
>
> I've pulled the latest changes, re-bootstrapped Emacs, removed
> ~/.emacs.d/elpa/archives, but after restarting Emacs, the problem persists.

Found it. The compatibility table was being initialize before the list
of built-in packages. Should be fixed now.

> Indeed, that should be rare. I don't have a scarier scenario to offer right
> now, so maybe we can ignore this downside (but preferably handle the
> possibility with care; and not move the "to install" mark to the
> now-incompatible package).

Yup

Re: Async package.el

[Daiki Ueno]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> I agree that's annoying and I have some similar plans. We can change
> the package-installation logic to first download all files
> simultaneously, and then unpack/compile all of them in the proper
> order. So large upgrades would have a significant speed improvement.

Though maybe it might be off topic, I was wondering if it is possible to
do the compilation (and perhaps downloading / unpacking) phase in a
separate Emacs process, so that the running Emacs session is not
affected by loading the compile-time requirements of a package.

Given that Emacs is rather a lightweight program these days, I guess it
isn't too impractical perhaps?  To implement that, maybe one would add a
few batch functions in package.el and a comint interface to communicate
with the inferior Emacs.

Regards,
-- 
Daiki Ueno

Re: Async package.el

[Dmitry Gutov]

On 04/07/2015 08:31 AM, Daiki Ueno wrote:

> Though maybe it might be off topic, I was wondering if it is possible to
> do the compilation (and perhaps downloading / unpacking) phase in a
> separate Emacs process, so that the running Emacs session is not
> affected by loading the compile-time requirements of a package.

https://github.com/jwiegley/emacs-async/blob/master/async-bytecomp.el 
implements something like that.

Re: Async package.el

[Artur Malabarba]

2015-04-07 10:13 GMT+01:00 Dmitry Gutov <dgutov@yandex.ru>:
> On 04/07/2015 08:31 AM, Daiki Ueno wrote:
>
>> Though maybe it might be off topic, I was wondering if it is possible to
>> do the compilation (and perhaps downloading / unpacking) phase in a
>> separate Emacs process, so that the running Emacs session is not
>> affected by loading the compile-time requirements of a package.
>
>
> https://github.com/jwiegley/emacs-async/blob/master/async-bytecomp.el
> implements something like that.

This is exactly what I've done in Paradox as well. There were a few
bugs at first, but I think they're all ironed out now so it should be
very doable.

Still, I'd need to hear how other people on the dev list feels about
using a second Emacs instance for something. Last time I mentioned it
to Stefan he said he'd prefer for the installation to be done by the
running Emacs.

Re: Async package.el

[Robert Pluim]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> 2015-04-07 10:13 GMT+01:00 Dmitry Gutov <dgutov@yandex.ru>:
>
> Still, I'd need to hear how other people on the dev list feels about
> using a second Emacs instance for something. Last time I mentioned it
> to Stefan he said he'd prefer for the installation to be done by the
> running Emacs.

Using a second Emacs runs the risk that it's not the same version as the
running Emacs. How likely is that to happen? For instance, the only Emacs in
my path is not the one I'm typing this in.

Regards

Robert

Re: Async package.el

[Artur Malabarba]

> Using a second Emacs runs the risk that it's not the same version as the
> running Emacs. How likely is that to happen? For instance, the only Emacs in
> my path is not the one I'm typing this in.

Your system path is irrelevant in this scenario. Emacs can tell the
full filename of the executable that's running it with
`(expand-file-name invocation-name invocation-directory)', and that is
what we would use to run the second instance.

The only situation where they can diverge is if the Emacs executable
was recompiled while the current one is running, and then we could add
some form of version check to protect against it.

Re: Async package.el

[Robert Pluim]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> Using a second Emacs runs the risk that it's not the same version as the
>> running Emacs. How likely is that to happen? For instance, the only Emacs in
>> my path is not the one I'm typing this in.
>
> Your system path is irrelevant in this scenario. Emacs can tell the
> full filename of the executable that's running it with
> `(expand-file-name invocation-name invocation-directory)', and that is
> what we would use to run the second instance.

OK.

> The only situation where they can diverge is if the Emacs executable
> was recompiled while the current one is running, and then we could add
> some form of version check to protect against it.

That happens to me sometimes as well, but in that case I've pointed the
gun at my own foot and pulled the trigger :-) I'd consider it rare
enough that you don't need to expend a lot of effort to prevent it.

Re: Async package.el

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

> > The only situation where they can diverge is if the Emacs executable
> > was recompiled while the current one is running, and then we could add
> > some form of version check to protect against it.
>
> That happens to me sometimes as well,

Me too. And I suspect many others on this list as well.

> but in that case I've pointed the
> gun at my own foot and pulled the trigger :-) I'd consider it rare
> enough that you don't need to expend a lot of effort to prevent it.

Not only that, but even when it does happens it's unlikely to cause
problems. Because the background process does very little interaction with
the running Emacs, even different versions should be mostly safe.

[-- Attachment #2: Type: text/html, Size: 831 bytes --]

Re: Async package.el

[Stefan Monnier]

> Last time I mentioned it to Stefan he said he'd prefer for the
> installation to be done by the running Emacs.

Indeed.  This said, I'm not dead set against running a sub-process,
especially not if it's configurable (so I can keep the current
behavior).

IOW, I think my preference is "as a user" rather than "as a maintainer".


        Stefan

Re: Async package.el

[Daiki Ueno]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Still, I'd need to hear how other people on the dev list feels about
> using a second Emacs instance for something. Last time I mentioned it
> to Stefan he said he'd prefer for the installation to be done by the
> running Emacs.

By the way, I saw several ELPA packages come with tests.  If the process
separation idea is feasible, perhaps it might also be worth considering
to run the bundled tests using it, since tests could consume resources
and tend to have side-effects.

Regards,
-- 
Daiki Ueno

Re: Async package.el

[Achim Gratz]

Artur Malabarba writes:
> Still, I'd need to hear how other people on the dev list feels about
> using a second Emacs instance for something. Last time I mentioned it
> to Stefan he said he'd prefer for the installation to be done by the
> running Emacs.

Doing it in the running Emacs is currently impossible to do correctly
for packages that are already loaded (in an earlier version) and change
certain implementation details.  So either Emacs grows the necessary
infrastructure to not fail in these instances or we need to reliably
start a new Emacs instance for the compilation.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

Re: Async package.el

[Stefan Monnier]

> Doing it in the running Emacs is currently impossible to do correctly
> for packages that are already loaded (in an earlier version) and change
> certain implementation details.

IIRC, the "currently" above is not fully true any more.
At least, Emacs-25 tries to handle the known problematic cases.
It's far from 100% bulletproof, admittedly.


        Stefan

Re: Async package.el

[Stefan Monnier]

> This is a heads up that I've pushed some commits that implement
> asynchronicity in package.el.

Thanks, Artur.

> - New variable `package-menu-async'. If this is non-nil the package menu
> uses asynchronous operations when refreshing the archives and when
> installing packages. For the purpose of testing on the devlist, I've set
> the default to t.

Good.

> - Archives are refreshed all at the same time (which leads to a signficant
> speed boost if you have several archives configured), so there's a new

Good.

> - Package downloads are performed in sequence, because it is generally not

Good as well (we could refine this to allow a maximum of
N outstanding downloads, but this is good enough for now).


        Stefan