bug#39187: Fwd: "Package refresh is already in progress, please wait..."

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Jean-Christophe Helary]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

Thanks in advance !

> Begin forwarded message:
> 
> From: Stefan Kangas <stefan@marxist.se>
> Subject: Re: "Package refresh is already in progress, please wait..."
> Date: January 19, 2020 1:20:42 JST
> To: Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
> Cc: Help Gnu Emacs mailing list <help-gnu-emacs@gnu.org>
> 
> Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org> writes:
>> 
>> It looks like emacs does not properly handle package refreshes when the internet connection has been killed in the middle of an refresh.
>> 
>> I often get this message in that case and I can't call package until I actually kill emacs and restart it.
> 
> That's a bug which was almost certainly introduced in a6d87ea045 by
> yours truly.  Could you please file a new bug report by running M-x
> report-emacs-bug or emailing bug-gnu-emacs@gnu.org?  Please include a
> reference to this thread.
> 
> Thanks in advance.
> 
> Best regards,
> Stefan Kangas

[-- Attachment #2: Type: text/html, Size: 3697 bytes --]

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Stefan Kangas]

Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
writes:

> From: Stefan Kangas <stefan@marxist.se>
> Subject: Re: "Package refresh is already in progress, please wait..."
> Date: January 19, 2020 1:20:42 JST
> To: Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
> Cc: Help Gnu Emacs mailing list <help-gnu-emacs@gnu.org>
>
> Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org> writes:
>
> > It looks like emacs does not properly handle package refreshes when
> > the internet connection has been killed in the middle of an
> > refresh.
> > 
> > I often get this message in that case and I can't call package
> > until I actually kill emacs and restart it.

This is indeed a pretty bad bug.  Thank you for bringing it to our
attention.

For reference, here is the discussion about this on help-gnu-emacs:

    https://lists.gnu.org/archive/html/help-gnu-emacs/2020-01/msg00119.html

> That's a bug which was almost certainly introduced in a6d87ea045 by
> yours truly.  Could you please file a new bug report by running M-x
> report-emacs-bug or emailing bug-gnu-emacs@gnu.org?  Please include a
> reference to this thread.

I think the fix here is simply to make sure that these processes times
out.

I will look into this, but that should not stop anyone else from
looking into it if they feel so inclined.  Please also if anyone has a
better idea for how to fix it, feel free to share it.

Best regards,
Stefan Kangas


PS.  Is there a way to mark this as blocking Emacs 27.1?  I don't
think we should release a pre-test with this issue.

PS2.  If we can't find a fix in time before the pre-release, it is
better to back out commit a6d87ea045 on the release branch.  (We could
then leave it on master and fix it there, I think.)  But that that
will hopefully not be necessary.

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Michael Albinus]

Stefan Kangas <stefan@marxist.se> writes:

> PS.  Is there a way to mark this as blocking Emacs 27.1?  I don't
> think we should release a pre-test with this issue.

For previous Emacs releases, there was a bug collecting blocking bugs
for that release. See debbugs-gnu-emacs-blocking-reports in
debbugs-gnu.el.

Unfortunately, this isn't used anymore. I found it convenient.

Best regards, Michael.

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Sun, 19 Jan 2020 10:09:41 +0100
> Cc: Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>,
>  39187@debbugs.gnu.org
> 
> Stefan Kangas <stefan@marxist.se> writes:
> 
> > PS.  Is there a way to mark this as blocking Emacs 27.1?  I don't
> > think we should release a pre-test with this issue.
> 
> For previous Emacs releases, there was a bug collecting blocking bugs
> for that release. See debbugs-gnu-emacs-blocking-reports in
> debbugs-gnu.el.
> 
> Unfortunately, this isn't used anymore. I found it convenient.

Feel free to create such a report.  But I don't promise I will agree
with every bug added to that list, I might have questions or I might
disagree (and remove the bug from the list).

Thanks.

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> For previous Emacs releases, there was a bug collecting blocking bugs
>> for that release. See debbugs-gnu-emacs-blocking-reports in
>> debbugs-gnu.el.
>>
>> Unfortunately, this isn't used anymore. I found it convenient.
>
> Feel free to create such a report.  But I don't promise I will agree
> with every bug added to that list, I might have questions or I might
> disagree (and remove the bug from the list).

Of course.

I've created bug#39200 for Emacs 27.1, and bug#39202 for Emacs
28.1. I've also marked bug#39187 as blocking bug#39200.

However, this will only work if this list of blocking bugs is maintained
by people.

> Thanks.

Best regards, Michael.

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Stefan Kangas]

Michael Albinus <michael.albinus@gmx.de> writes:

> I've created bug#39200 for Emacs 27.1, and bug#39202 for Emacs
> 28.1. I've also marked bug#39187 as blocking bug#39200.
>
> However, this will only work if this list of blocking bugs is maintained
> by people.

Thanks.  I suggest to let emacs-devel know about this.

Best regards,
Stefan Kangas

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Michael Albinus]

Stefan Kangas <stefan@marxist.se> writes:

>> I've created bug#39200 for Emacs 27.1, and bug#39202 for Emacs
>> 28.1. I've also marked bug#39187 as blocking bug#39200.
>>
>> However, this will only work if this list of blocking bugs is maintained
>> by people.
>
> Thanks.  I suggest to let emacs-devel know about this.

Will do. Just now I'm working on debbugs-gnu.el, in order to let it know
these special bugs. A new debbugs release shall appear soon.

OTOH, I don't know whether Eli has a policy when to declare a bug
blocking for a Emacs given release. We should avoid that everybody
declares her own pet bug as release-critical.

admin/release-process speaks only about the "how" of creating / handling
blocking bugs, not about "when".

> Best regards,
> Stefan Kangas

Best regards, Michael.

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Stefan Kangas]

>> > It looks like emacs does not properly handle package refreshes when
>> > the internet connection has been killed in the middle of an
>> > refresh.
>> > 
>> > I often get this message in that case and I can't call package
>> > until I actually kill emacs and restart it.
[...]
> I think the fix here is simply to make sure that these processes times
> out.

Unless I'm missing something, it turns out that this is not that easy
to do.  At least I couldn't figure out how to do it.  I found the
following probably relevant bug:

 bug#34607: url-retrieve-synchronously: TIMEOUT option value takes no effect

For now at least, I'm giving up on finding a solution where url does
the right thing and times out. 

-----

I see two ways forward:

1. Remove the warning and accept that we might launch several
   processes doing the same update one after the other.  I would like
   to avoid that on master, but it might be the least intrusive change
   for emacs-27.
  
2. When we run a refresh, save the current time.  If the user tries to
   run another refresh, check if more than X seconds has elapsed since
   the old one was started.  If it has been more than X seconds,
   prompt the user to kill the old refresh and re-launch it.  It won't
   be perfect, but it will be better than what we had before at least.

Does anyone else have any ideas or preferences here?  Otherwise, I'll
revert the change that exposed this bug on the emacs-27 branch, and
implement the second change on master.

Best regards,
Stefan Kangas

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

> I see two ways forward:
>
> 1. Remove the warning and accept that we might launch several
>    processes doing the same update one after the other.  I would like
>    to avoid that on master, but it might be the least intrusive change
>    for emacs-27.

This part is now done on the emacs-27 branch.  Please test it and see
that it now works as before.

> 2. When we run a refresh, save the current time.  If the user tries to
>    run another refresh, check if more than X seconds has elapsed since
>    the old one was started.  If it has been more than X seconds,
>    prompt the user to kill the old refresh and re-launch it.  It won't
>    be perfect, but it will be better than what we had before at least.

I'm working on a fix for the master branch as outlined above.

Note that we probably also have to change the condition for the menu
entry, see Bug#39436.

Best regards,
Stefan Kangas

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Jean-Christophe Helary]

> On Feb 10, 2020, at 5:33, Stefan Kangas <stefan@marxist.se> wrote:
> 
> Stefan Kangas <stefan@marxist.se> writes:
> 
>> I see two ways forward:
>> 
>> 1. Remove the warning and accept that we might launch several
>>   processes doing the same update one after the other.  I would like
>>   to avoid that on master, but it might be the least intrusive change
>>   for emacs-27.
> 
> This part is now done on the emacs-27 branch.  Please test it and see
> that it now works as before.

Can you specify how that is expected to work so that I am sure I make the right test ?

Jean-Christophe 

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Stefan Kangas]

Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
writes:

>> This part is now done on the emacs-27 branch.  Please test it and see
>> that it now works as before.
>
> Can you specify how that is expected to work so that I am sure I make the right test ?

1. Run revert-buffer (bound to 'g' by default).

2. Disable your internet connection temporarily and then enable it
   again.

3. Run revert-buffer again.

It should no longer display an error message and instead start a new
package archive refresh.

Best regards,
Stefan Kangas

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Jean-Christophe Helary]

> On Feb 11, 2020, at 0:30, Stefan Kangas <stefan@marxist.se> wrote:
> 
> Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
> writes:
> 
>>> This part is now done on the emacs-27 branch.  Please test it and see
>>> that it now works as before.
>> 
>> Can you specify how that is expected to work so that I am sure I make the right test ?
> 
> 1. Run revert-buffer (bound to 'g' by default).
> 
> 2. Disable your internet connection temporarily and then enable it
>   again.
> 
> 3. Run revert-buffer again.
> 
> It should no longer display an error message and instead start a new
> package archive refresh.

Thank you Stefan,

I'm rebuilding emacs right now and will let you know how that worked.


Jean-Christophe Helary
-----------------------------------------------
http://mac4translators.blogspot.com @brandelune

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Jean-Christophe Helary]

> On Feb 11, 2020, at 0:30, Stefan Kangas <stefan@marxist.se> wrote:
> 
> Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
> writes:
> 
>>> This part is now done on the emacs-27 branch.  Please test it and see
>>> that it now works as before.
>> 
>> Can you specify how that is expected to work so that I am sure I make the right test ?
> 
> 1. Run revert-buffer (bound to 'g' by default).
> 
> 2. Disable your internet connection temporarily and then enable it
>   again.
> 
> 3. Run revert-buffer again.
> 
> It should no longer display an error message and instead start a new
> package archive refresh.

Stephan

Excellent. Thank you so much.

When the connection is off, there is a message that the packages can't be downloaded.

Then, when I reconnect to the internet, the refresh works as expected.

Would it be possible to also have that on master ?

Jean-Christophe 

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Stefan Kangas]

Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
writes:

>> It should no longer display an error message and instead start a new
>> package archive refresh.
>
> Stephan
>
> Excellent. Thank you so much.
>
> When the connection is off, there is a message that the packages can't be downloaded.
>
> Then, when I reconnect to the internet, the refresh works as expected.

Thank you very much for testing and describing the behaviour you see
in detail.  That confirms that this bug should no longer block the
emacs-27 release.

> Would it be possible to also have that on master ?

I'm working on a different fix for the master branch, which will try
to avoid running more than one refresh at the same time.  It's on the
top of my things to do in Emacs, but I unfortunately haven't found the
time to finish it yet.

Best regards,
Stefan Kangas

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Jean-Christophe Helary]

> On Feb 11, 2020, at 15:25, Stefan Kangas <stefan@marxist.se> wrote:
> 
> Jean-Christophe Helary <jean.christophe.helary@traduction-libre.org>
> writes:
> 
>>> It should no longer display an error message and instead start a new
>>> package archive refresh.
>> 
>> Stephan
>> 
>> Excellent. Thank you so much.
>> 
>> When the connection is off, there is a message that the packages can't be downloaded.
>> 
>> Then, when I reconnect to the internet, the refresh works as expected.
> 
> Thank you very much for testing and describing the behaviour you see
> in detail.

Well, you wanted a test :)

>  That confirms that this bug should no longer block the
> emacs-27 release.
> 
>> Would it be possible to also have that on master ?
> 
> I'm working on a different fix for the master branch, which will try
> to avoid running more than one refresh at the same time.  It's on the
> top of my things to do in Emacs, but I unfortunately haven't found the
> time to finish it yet.

Too bad, because I usually work on a recent local build of master and that would be really nice to have a temporary solution there as well.

Jean-Christophe 

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Stefan Kangas]

Stefan Kangas <stefan@marxist.se> writes:

>>> > It looks like emacs does not properly handle package refreshes when
>>> > the internet connection has been killed in the middle of an
>>> > refresh.
>>> > 
>>> > I often get this message in that case and I can't call package
>>> > until I actually kill emacs and restart it.
> [...]
>> I think the fix here is simply to make sure that these processes times
>> out.
>
> Unless I'm missing something, it turns out that this is not that easy
> to do.  At least I couldn't figure out how to do it.  I found the
> following probably relevant bug:
>
>  bug#34607: url-retrieve-synchronously: TIMEOUT option value takes no effect
>
> For now at least, I'm giving up on finding a solution where url does
> the right thing and times out. 
>
> -----
>
> I see two ways forward:
>
> 1. Remove the warning and accept that we might launch several
>    processes doing the same update one after the other.  I would like
>    to avoid that on master, but it might be the least intrusive change
>    for emacs-27.
>   
> 2. When we run a refresh, save the current time.  If the user tries to
>    run another refresh, check if more than X seconds has elapsed since
>    the old one was started.  If it has been more than X seconds,
>    prompt the user to kill the old refresh and re-launch it.  It won't
>    be perfect, but it will be better than what we had before at least.
>
> Does anyone else have any ideas or preferences here?  Otherwise, I'll
> revert the change that exposed this bug on the emacs-27 branch, and
> implement the second change on master.

The more I've been looking at and thinking about this, the less
convinced I've been that it's a good idea to do something like (2).
It would not be a proper solution, but a hack, really.

The correct fix would be to make sure the connection times out.  But
I'm not sure that it's worth going deep into the guts of url.el right
now.

Lars has been working on a replacement library (with-url) which AFAIU
he hopes to merge in the coming months.  Without having looked at it
in great detail, it seems to me that we should try to make package.el
use the with-url library once it's merged.  Hopefully, that means that
timeouts will work properly.  (Otherwise, we can fix it.)

Once timeouts work, it would then be easy to reintroduce the warning
about duplicate package refreshes.

I have consequently done (1) instead also on master: the commit is now
reverted and this warning removed.  Please test that it now works as
expected, and thank you for your patience.

Best regards,
Stefan Kangas

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Stefan Kangas]

close 39187 27.1
thanks

Stefan Kangas <stefan@marxist.se> writes:

> I have consequently done (1) instead also on master: the commit is now
> reverted and this warning removed.  Please test that it now works as
> expected, and thank you for your patience.

No further comments within 2 weeks, so I'm closing this bug now.

Best regards,
Stefan Kangas

bug#39187: Fwd: "Package refresh is already in progress, please wait..."

[Jean-Christophe Helary]

> On Apr 8, 2020, at 10:22, Stefan Kangas <stefan@marxist.se> wrote:
> 
> close 39187 27.1
> thanks
> 
> Stefan Kangas <stefan@marxist.se> writes:
> 
>> I have consequently done (1) instead also on master: the commit is now
>> reverted and this warning removed.  Please test that it now works as
>> expected, and thank you for your patience.
> 
> No further comments within 2 weeks, so I'm closing this bug now.

Indeed, the workaround you created seems to work. Thank you.

Jean-Christophe