bug#24849: Is Emacs put in idle mode when window is not focused?

bug#24849: Is Emacs put in idle mode when window is not focused?

[Johan Andersson]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

Hello,

I have written an Emacs package called Prodigy (
https://github.com/rejeep/prodigy.el/). The package is used to manage
external services from within Emacs. A service is some kind of running
process, usually a web server or database.

There has always been an issue that I have not investigated further. It
seems that after a while, if Emacs is not focused, it ends up in some idle
mode. The effect is that requests are very slow, almost hangs for a few
seconds. If I focus the Emacs window, the request goes through immediately.

The reason I'm asking about it now is that this has gotten even worse in
Emacs 25.

My question is if anyone knows what this is about and if it's possible to
solve?

Thanks!

-- 
Johan Andersson
System Developer, Burt
www.burtcorp.com
Cell: +46 761 041607
https:// <http://twitter.com/rejeep>github.com/rejeep |
http://twitter.com/rejeep | http://twitter.com/burtcorp

[-- Attachment #2: Type: text/html, Size: 1639 bytes --]

bug#24849: Is Emacs put in idle mode when window is not focused?

[Eli Zaretskii]

> From: Johan Andersson <johan.andersson@burtcorp.com>
> Date: Tue, 1 Nov 2016 09:44:20 +0100
> 
> I have written an Emacs package called Prodigy (https://github.com/rejeep/prodigy.el/). The package is used
> to manage external services from within Emacs. A service is some kind of running process, usually a web
> server or database.
> 
> There has always been an issue that I have not investigated further. It seems that after a while, if Emacs is
> not focused, it ends up in some idle mode. The effect is that requests are very slow, almost hangs for a few
> seconds. If I focus the Emacs window, the request goes through immediately.
> 
> The reason I'm asking about it now is that this has gotten even worse in Emacs 25.
> 
> My question is if anyone knows what this is about and if it's possible to solve?

Please describe in more detail what you package does and how it is
related to "non-idle" Emacs.  IOW, why does your package care whether
Emacs is idle or not.

My suspicion is that your package somehow assumes that the Emacs main
loop is always running at the same fast speed, which is not true.  In
particular, when the frame doesn't have the focus, Emacs 25 stops the
blinking cursor timer, and if that is the only high-frequency activity
in Emacs (i.e. there's no other timers or external events that drive
the event loop), then yes, Emacs will only crank the event loop at
very low frequency.

But I'm not sure this is related to your problem, because I don't know
what does your package expect from Emacs.

bug#24849: Is Emacs put in idle mode when window is not focused?

[Johan Andersson]

[-- Attachment #1: Type: text/plain, Size: 2799 bytes --]

Sorry for not being clear enough.

With Prodigy, you define a service. Usually that is a path, command and
arguments to the command. You can then via the Prodigy GUI manage
(start/stop/restart/etc...) these services.

To put it very simple, what happens in Prodigy when a service is started is
basically this:

(let* ((default-directory "/tmp")
       (process (start-process "server" nil "python" "-m"
"SimpleHTTPServer" "8000")))
  (set-process-filter
   process
   (lambda (_ output)
     ;; ...
     )))

What happens in practice is that, when I get to work, I select the services
I need and start them (unless Emacs was killed, they are already started).
Sometimes I use Emacs quite frequently and then this is not so much of an
issue because Emacs does not have time to idle. But when I don't use Emacs
for a while, it will hang waiting for the response from the service
(because Emacs is idle).

Hope you understand my issue better!

On Tue, Nov 1, 2016 at 5:22 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Johan Andersson <johan.andersson@burtcorp.com>
> > Date: Tue, 1 Nov 2016 09:44:20 +0100
> >
> > I have written an Emacs package called Prodigy (
> https://github.com/rejeep/prodigy.el/). The package is used
> > to manage external services from within Emacs. A service is some kind of
> running process, usually a web
> > server or database.
> >
> > There has always been an issue that I have not investigated further. It
> seems that after a while, if Emacs is
> > not focused, it ends up in some idle mode. The effect is that requests
> are very slow, almost hangs for a few
> > seconds. If I focus the Emacs window, the request goes through
> immediately.
> >
> > The reason I'm asking about it now is that this has gotten even worse in
> Emacs 25.
> >
> > My question is if anyone knows what this is about and if it's possible
> to solve?
>
> Please describe in more detail what you package does and how it is
> related to "non-idle" Emacs.  IOW, why does your package care whether
> Emacs is idle or not.
>
> My suspicion is that your package somehow assumes that the Emacs main
> loop is always running at the same fast speed, which is not true.  In
> particular, when the frame doesn't have the focus, Emacs 25 stops the
> blinking cursor timer, and if that is the only high-frequency activity
> in Emacs (i.e. there's no other timers or external events that drive
> the event loop), then yes, Emacs will only crank the event loop at
> very low frequency.
>
> But I'm not sure this is related to your problem, because I don't know
> what does your package expect from Emacs.
>



-- 
Johan Andersson
System Developer, Burt
www.burtcorp.com
Cell: +46 761 041607
https:// <http://twitter.com/rejeep>github.com/rejeep |
http://twitter.com/rejeep | http://twitter.com/burtcorp

[-- Attachment #2: Type: text/html, Size: 4507 bytes --]

bug#24849: Is Emacs put in idle mode when window is not focused?

[Eli Zaretskii]

> From: Johan Andersson <johan.andersson@burtcorp.com>
> Date: Tue, 1 Nov 2016 20:48:59 +0100
> Cc: 24849@debbugs.gnu.org
> 
> (let* ((default-directory "/tmp")
> (process (start-process "server" nil "python" "-m" "SimpleHTTPServer" "8000")))
> (set-process-filter
> process
> (lambda (_ output)
> ;; ...
> )))
> 
> What happens in practice is that, when I get to work, I select the services I need and start them (unless
> Emacs was killed, they are already started). Sometimes I use Emacs quite frequently and then this is not so
> much of an issue because Emacs does not have time to idle. But when I don't use Emacs for a while, it will
> hang waiting for the response from the service (because Emacs is idle).

So you are saying that the service, which is a sub-process of Emacs,
produces some output, but Emacs doesn't read that output timely enough
because it's idle?  That's not possible, I think: when Emacs is idle,
it is most of the time stuck inside a call to 'pselect', which should
return immediately when some input arrived from a sub-process.

So I guess I still don't understand something in your setup.  But
what?

bug#24849: Is Emacs put in idle mode when window is not focused?

[Johan Andersson]

[-- Attachment #1: Type: text/plain, Size: 1745 bytes --]

I added the set-process-filter in case that would make a difference, but I
don't think that is the issue here. The issue is that the services (sub
processes of Emacs as you say) takes a long time to respond when Emacs is
idle. If I run the same command from a terminal, that doesn't happen.

On Tue, Nov 1, 2016 at 9:13 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Johan Andersson <johan.andersson@burtcorp.com>
> > Date: Tue, 1 Nov 2016 20:48:59 +0100
> > Cc: 24849@debbugs.gnu.org
> >
> > (let* ((default-directory "/tmp")
> > (process (start-process "server" nil "python" "-m" "SimpleHTTPServer"
> "8000")))
> > (set-process-filter
> > process
> > (lambda (_ output)
> > ;; ...
> > )))
> >
> > What happens in practice is that, when I get to work, I select the
> services I need and start them (unless
> > Emacs was killed, they are already started). Sometimes I use Emacs quite
> frequently and then this is not so
> > much of an issue because Emacs does not have time to idle. But when I
> don't use Emacs for a while, it will
> > hang waiting for the response from the service (because Emacs is idle).
>
> So you are saying that the service, which is a sub-process of Emacs,
> produces some output, but Emacs doesn't read that output timely enough
> because it's idle?  That's not possible, I think: when Emacs is idle,
> it is most of the time stuck inside a call to 'pselect', which should
> return immediately when some input arrived from a sub-process.
>
> So I guess I still don't understand something in your setup.  But
> what?
>



-- 
Johan Andersson
System Developer, Burt
www.burtcorp.com
Cell: +46 761 041607
https:// <http://twitter.com/rejeep>github.com/rejeep |
http://twitter.com/rejeep | http://twitter.com/burtcorp

[-- Attachment #2: Type: text/html, Size: 2929 bytes --]

bug#24849: Is Emacs put in idle mode when window is not focused?

[Daniel Colascione]

BTW: please don't top-post

On 11/01/2016 01:18 PM, Johan Andersson wrote:
> I added the set-process-filter in case that would make a difference, but
> I don't think that is the issue here. The issue is that the services
> (sub processes of Emacs as you say) takes a long time to respond when
> Emacs is idle. If I run the same command from a terminal, that doesn't
> happen.

Are you sure that your subprocess isn't buffering its output?

Can you catch Emacs in the act?

(Assuming GNU/Linux here.) Run emacs with "strace -s256 -o trace -ff -tt 
emacs" and look at the trace.* files produced. You should have one for 
each thread in each process. You should be able to identify the process 
involved from the system calls in each. (Look for execve.) Now, look for 
the write(2) system call from your subprocess and see how much time 
passes between that system call and Emacs waking up from pselect.

If Emacs wakes up very soon after that write(2), the problem lies in 
your code. If there is a big delay between that write(2) and Emacs 
returning from pselect, the problem is likely in Emacs.

>
> On Tue, Nov 1, 2016 at 9:13 PM, Eli Zaretskii <eliz@gnu.org
> <mailto:eliz@gnu.org>> wrote:
>
>     > From: Johan Andersson <johan.andersson@burtcorp.com
>     <mailto:johan.andersson@burtcorp.com>>
>     > Date: Tue, 1 Nov 2016 20:48:59 +0100
>     > Cc: 24849@debbugs.gnu.org <mailto:24849@debbugs.gnu.org>
>     >
>     > (let* ((default-directory "/tmp")
>     > (process (start-process "server" nil "python" "-m" "SimpleHTTPServer" "8000")))
>     > (set-process-filter
>     > process
>     > (lambda (_ output)
>     > ;; ...
>     > )))
>     >
>     > What happens in practice is that, when I get to work, I select the services I need and start them (unless
>     > Emacs was killed, they are already started). Sometimes I use Emacs quite frequently and then this is not so
>     > much of an issue because Emacs does not have time to idle. But when I don't use Emacs for a while, it will
>     > hang waiting for the response from the service (because Emacs is idle).
>
>     So you are saying that the service, which is a sub-process of Emacs,
>     produces some output, but Emacs doesn't read that output timely enough
>     because it's idle?  That's not possible, I think: when Emacs is idle,
>     it is most of the time stuck inside a call to 'pselect', which should
>     return immediately when some input arrived from a sub-process.
>
>     So I guess I still don't understand something in your setup.  But
>     what?
>
>
>
>
> --
> Johan Andersson
> System Developer, Burt
> www.burtcorp.com <http://www.burtcorp.com>
> Cell: +46 761 041607
> https:// <http://twitter.com/rejeep>github.com/rejeep
> <http://github.com/rejeep> | http://twitter.com/rejeep |
> http://twitter.com/burtcorp

bug#24849: Is Emacs put in idle mode when window is not focused?

[Johan Andersson]

[-- Attachment #1: Type: text/plain, Size: 3481 bytes --]

> BTW: please don't top-post

Sorry, what?

> Are you sure that your subprocess isn't buffering its output?

No. Does that make a difference?

> Can you catch Emacs in the act?

What do you mean?

> Assuming GNU/Linux here.

On a Mac here, but I might be able to find a GNU/Linux machine at work tomo.

On Tue, Nov 1, 2016 at 9:28 PM, Daniel Colascione <dancol@dancol.org> wrote:

> BTW: please don't top-post
>
> On 11/01/2016 01:18 PM, Johan Andersson wrote:
>
>> I added the set-process-filter in case that would make a difference, but
>> I don't think that is the issue here. The issue is that the services
>> (sub processes of Emacs as you say) takes a long time to respond when
>> Emacs is idle. If I run the same command from a terminal, that doesn't
>> happen.
>>
>
> Are you sure that your subprocess isn't buffering its output?
>
> Can you catch Emacs in the act?
>
> (Assuming GNU/Linux here.) Run emacs with "strace -s256 -o trace -ff -tt
> emacs" and look at the trace.* files produced. You should have one for each
> thread in each process. You should be able to identify the process involved
> from the system calls in each. (Look for execve.) Now, look for the
> write(2) system call from your subprocess and see how much time passes
> between that system call and Emacs waking up from pselect.
>
> If Emacs wakes up very soon after that write(2), the problem lies in your
> code. If there is a big delay between that write(2) and Emacs returning
> from pselect, the problem is likely in Emacs.
>
>
>> On Tue, Nov 1, 2016 at 9:13 PM, Eli Zaretskii <eliz@gnu.org
>> <mailto:eliz@gnu.org>> wrote:
>>
>>     > From: Johan Andersson <johan.andersson@burtcorp.com
>>     <mailto:johan.andersson@burtcorp.com>>
>>     > Date: Tue, 1 Nov 2016 20:48:59 +0100
>>     > Cc: 24849@debbugs.gnu.org <mailto:24849@debbugs.gnu.org>
>>     >
>>     > (let* ((default-directory "/tmp")
>>     > (process (start-process "server" nil "python" "-m"
>> "SimpleHTTPServer" "8000")))
>>     > (set-process-filter
>>     > process
>>     > (lambda (_ output)
>>     > ;; ...
>>     > )))
>>     >
>>     > What happens in practice is that, when I get to work, I select the
>> services I need and start them (unless
>>     > Emacs was killed, they are already started). Sometimes I use Emacs
>> quite frequently and then this is not so
>>     > much of an issue because Emacs does not have time to idle. But when
>> I don't use Emacs for a while, it will
>>     > hang waiting for the response from the service (because Emacs is
>> idle).
>>
>>     So you are saying that the service, which is a sub-process of Emacs,
>>     produces some output, but Emacs doesn't read that output timely enough
>>     because it's idle?  That's not possible, I think: when Emacs is idle,
>>     it is most of the time stuck inside a call to 'pselect', which should
>>     return immediately when some input arrived from a sub-process.
>>
>>     So I guess I still don't understand something in your setup.  But
>>     what?
>>
>>
>>
>>
>> --
>> Johan Andersson
>> System Developer, Burt
>> www.burtcorp.com <http://www.burtcorp.com>
>> Cell: +46 761 041607
>> https:// <http://twitter.com/rejeep>github.com/rejeep
>> <http://github.com/rejeep> | http://twitter.com/rejeep |
>> http://twitter.com/burtcorp
>>
>


-- 
Johan Andersson
System Developer, Burt
www.burtcorp.com
Cell: +46 761 041607
https:// <http://twitter.com/rejeep>github.com/rejeep |
http://twitter.com/rejeep | http://twitter.com/burtcorp

[-- Attachment #2: Type: text/html, Size: 6779 bytes --]

bug#24849: Is Emacs put in idle mode when window is not focused?

[Daniel Colascione]

On 11/01/2016 01:33 PM, Johan Andersson wrote:
>> BTW: please don't top-post
>
> Sorry, what?

Please start your reply below the text to which you're replying.

     A: Because we read from top to bottom, left to right.
     Q: Why should I start my reply below the quoted text?

     A: Because it messes up the order in which people normally read text.
     Q: Why is top-posting such a bad thing?

     A: The lost context.
     Q: What makes top-posted replies harder to read than bottom-posted?

     A: Yes.
     Q: Should I trim down the quoted part of an email to which I'm 
replying?


>
>> Are you sure that your subprocess isn't buffering its output?
>
> No. Does that make a difference?

Emacs can't respond to what it can't see. If your subprocess isn't 
pushing bytes to the kernel and is instead buffering internally, Emacs 
can't see those bytes. Internal buffering is a common cause of 
complaints about parent processes not seeing output from children.

>
>> Can you catch Emacs in the act?
>
> What do you mean?
>
>> Assuming GNU/Linux here.
>
> On a Mac here, but I might be able to find a GNU/Linux machine at work tomo.

dtrace, which OS X supports, has similar system-call-tracing 
capabilities, although I don't know the correct incantation off the top 
of my head.

>

bug#24849: Is Emacs put in idle mode when window is not focused?

[Johan Andersson]

[-- Attachment #1: Type: text/plain, Size: 2044 bytes --]

On Tue, Nov 1, 2016 at 9:37 PM, Daniel Colascione <dancol@dancol.org> wrote:

> On 11/01/2016 01:33 PM, Johan Andersson wrote:
>
>> BTW: please don't top-post
>>>
>>
>> Sorry, what?
>>
>
> Please start your reply below the text to which you're replying.
>
>     A: Because we read from top to bottom, left to right.
>     Q: Why should I start my reply below the quoted text?
>
>     A: Because it messes up the order in which people normally read text.
>     Q: Why is top-posting such a bad thing?
>
>     A: The lost context.
>     Q: What makes top-posted replies harder to read than bottom-posted?
>
>     A: Yes.
>     Q: Should I trim down the quoted part of an email to which I'm
> replying?
>
>
>
>> Are you sure that your subprocess isn't buffering its output?
>>>
>>
>> No. Does that make a difference?
>>
>
> Emacs can't respond to what it can't see. If your subprocess isn't pushing
> bytes to the kernel and is instead buffering internally, Emacs can't see
> those bytes. Internal buffering is a common cause of complaints about
> parent processes not seeing output from children.
>
>
>> Can you catch Emacs in the act?
>>>
>>
>> What do you mean?
>>
>> Assuming GNU/Linux here.
>>>
>>
>> On a Mac here, but I might be able to find a GNU/Linux machine at work
>> tomo.
>>
>
> dtrace, which OS X supports, has similar system-call-tracing capabilities,
> although I don't know the correct incantation off the top of my head.
>
>
>>
> Please start your reply below the text to which you're replying.

Ok, then I guess that's default in Gmail. This better?

> ... Internal buffering is a common cause of complaints about parent
processes not seeing output from children.

I understand that, but that's not my issue here. The sub process might not
have any output at all, but I would still expect that sub process to
respond to my requests.

-- 
Johan Andersson
System Developer, Burt
www.burtcorp.com
Cell: +46 761 041607
https:// <http://twitter.com/rejeep>github.com/rejeep |
http://twitter.com/rejeep | http://twitter.com/burtcorp

[-- Attachment #2: Type: text/html, Size: 4451 bytes --]

bug#24849: Is Emacs put in idle mode when window is not focused?

[Daniel Colascione]

On 11/01/2016 01:41 PM, Johan Andersson wrote:
  >> Please start your reply below the text to which you're replying.
>
> Ok, then I guess that's default in Gmail. This better?

Yep. But please also trim your replies, as I have done.


>> ... Internal buffering is a common cause of complaints about parent
> processes not seeing output from children.
>
> I understand that, but that's not my issue here. The sub process might
> not have any output at all, but I would still expect that sub process to
> respond to my requests.

Ah, I see. I think I've misunderstood. The problem isn't that Emacs 
isn't responding to the subprocess saying something to Emacs, but that 
the subprocess _itself_ isn't responding quickly to requests it's 
getting over the network, and that this stalling occurs only when your 
subprocess is run under Emacs? We shouldn't be seeing this issue either: 
Emacs subprocesses do not run in some weird environment.

I still think a system call trace will help clear up this problem.

bug#24849: Is Emacs put in idle mode when window is not focused?

[Eli Zaretskii]

> From: Johan Andersson <johan.andersson@burtcorp.com>
> Date: Tue, 1 Nov 2016 21:18:28 +0100
> Cc: 24849@debbugs.gnu.org
> 
> I added the set-process-filter in case that would make a difference, but I don't think that is the issue here. The
> issue is that the services (sub processes of Emacs as you say) takes a long time to respond when Emacs is
> idle. If I run the same command from a terminal, that doesn't happen.

"To respond"?  That is, Emacs sends something to the service, and the
time between that and the response is too long?

Are you sure it's the time between the input to the service and its
response that's long?  Could it be that Emacs delays sending the input
instead?

Maybe you could describe in more detail how the communications between
Emacs and the service work.  E.g., what triggers sending input from
Emacs to the service?

bug#24849: Is Emacs put in idle mode when window is not focused?

[Johan Andersson]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]

Yes, now you understand the issue correctly!

> I still think a system call trace will help clear up this problem.

I will see if I can get a GNU/Linux machine.

If you want to try it out on your machine, it's easy to do:

1. Install Prodigy: https://github.com/rejeep/prodigy.el/ (package-install
prodigy)
2. Add a simple service:
(prodigy-define-service
  :name "test"
  :cwd "/tmp"
  :command "python"
  :args '("-m" "SimpleHTTPServer" "8000"))
3. Start prodigy (M-x prodigy)
4. Start the service (press s)

[-- Attachment #2: Type: text/html, Size: 2580 bytes --]

bug#24849: Is Emacs put in idle mode when window is not focused?

[Johan Andersson]

[-- Attachment #1: Type: text/plain, Size: 289 bytes --]

> Emacs sends something to the service, and the time between that and the
response is too long?

No, Emacs starts the service, but the communication to that service never
goes via Emacs. For example for the web server example above, all
communication to the service will go via a browser.

[-- Attachment #2: Type: text/html, Size: 619 bytes --]

bug#24849: Is Emacs put in idle mode when window is not focused?

[Alan Third]

On Tue, Nov 01, 2016 at 09:33:59PM +0100, Johan Andersson wrote:
> > Assuming GNU/Linux here.
> 
> On a Mac here, but I might be able to find a GNU/Linux machine at work tomo.

Hi Johan,

Your problem might be related to bug 22993:

https://lists.gnu.org/archive/html/bug-gnu-emacs/2016-03/msg00590.html

Somewhere down the thread there are a number of work‐arounds
available that you can try.
-- 
Alan Third

bug#24849: Is Emacs put in idle mode when window is not focused?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The package is used to manage
  > external services from within Emacs. A service is some kind of running
  > process, usually a web server or database.

Could you explain more concretely the scenario you have in mind here?
Does "database" mean a program running on your own machine?
Does "a web server" mean you're running Emacs on the web server machine
and using Emacs to control it?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#24849: Is Emacs put in idle mode when window is not focused?

[Johan Andersson]

[-- Attachment #1: Type: text/plain, Size: 237 bytes --]

That might actually be it. Tried it out now and it seems to work. Will try
it out today and see if it works.

For reference, to disable nap mode for Emacs, run:

  defaults write org.gnu.Emacs NSAppSleepDisabled -bool YES

Thanks a lot!

[-- Attachment #2: Type: text/html, Size: 496 bytes --]

bug#24849: Is Emacs put in idle mode when window is not focused?

[Johan Andersson]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

Seems like Alan had the solution above, but for reference I will answer
your question anyways.

> Does "database" mean a program running on your own machine?

Yes.

> Does "a web server" mean you're running Emacs on the web server machine
and using Emacs to control it?

No, both the database, web server and Emacs are running on my local machine.


Here is a scenario:

I develop a web application locally. To do that I need a web server,
database and a background worker. I have defined services for all of these.
When I work on this app, I start all of these services in Prodigy. Then I
visit for example http://localhost:3000 in my browser. The app also
connects to the database.

[-- Attachment #2: Type: text/html, Size: 1518 bytes --]