sit-for

sit-for

[David Kastrup]

Since we have the new sit-for implementation, I have a lot of times
when Emacs just pauses in busy waiting for input.  This happens
spontaneously.  One situation where it happens frequently is when
reading news with gnus.  If I type C-g at such a time with
debug-on-quit set to t, I just get


Debugger entered--Lisp error: (quit)

in the *Backtrace* buffer.

In contrast, when Emacs is actually waiting in a non-busy wait for
input, the backtrace is

Debugger entered--Lisp error: (quit)
  signal(quit nil)
  keyboard-quit()
  call-interactively(keyboard-quit)

and this is what I would expect without those random lockups which
tend to last for seconds and gobble up CPU power.

This is really a nuisance.  The change to sit-for is a fundamental
change to some core mechanism of Emacs, and it is currently seemingly
breaking quite a few things, apart from causing strange effects.

It does not look like the implications of this code are obvious to
anybody.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: sit-for

[Chong Yidong]

David Kastrup <dak@gnu.org> writes:

> Since we have the new sit-for implementation, I have a lot of times
> when Emacs just pauses in busy waiting for input.  This happens
> spontaneously.  One situation where it happens frequently is when
> reading news with gnus.

I haven't seen this problem in my usage of gnus.  It would be helpful
if you could be more specific: does the problem happen with the
2006-07-26 change to `read-event'?  If so, did it start to happen at
that time, or did it already happen after the Lisp-level `sit-for' was
implemented?

If it only started after the `read-event' change, one possibility is
that read_char is getting stuck somewhere before we get to the point
where we check if the timeout has elapsed.

> This is really a nuisance.  The change to sit-for is a fundamental
> change to some core mechanism of Emacs, and it is currently seemingly
> breaking quite a few things, apart from causing strange effects.

That's the story of the Emacs 22 feature freeze.

Re: sit-for

[David Kastrup]

Chong Yidong <cyd@stupidchicken.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Since we have the new sit-for implementation, I have a lot of times
>> when Emacs just pauses in busy waiting for input.  This happens
>> spontaneously.  One situation where it happens frequently is when
>> reading news with gnus.
>
> I haven't seen this problem in my usage of gnus.

It is not just gnus.  And it is not easy to see: Emacs gets somewhat
sluggish, sometimes the cursor blinks quite less than expected, and
CPU usage goes up, up, up, which is of course mostly noticeable when
you have power management, and the fans start waking up.

> It would be helpful if you could be more specific: does the problem
> happen with the 2006-07-26 change to `read-event'?

Already before.  I have now recompiled Emacs and will see whether this
changes something.

> If so, did it start to happen at that time, or did it already happen
> after the Lisp-level `sit-for' was implemented? 
>
> If it only started after the `read-event' change, one possibility is
> that read_char is getting stuck somewhere before we get to the point
> where we check if the timeout has elapsed.
>
>> This is really a nuisance.  The change to sit-for is a fundamental
>> change to some core mechanism of Emacs, and it is currently
>> seemingly breaking quite a few things, apart from causing strange
>> effects.
>
> That's the story of the Emacs 22 feature freeze.


-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: sit-for

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Chong Yidong <cyd@stupidchicken.com> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> Since we have the new sit-for implementation, I have a lot of times
>>> when Emacs just pauses in busy waiting for input.  This happens
>>> spontaneously.  One situation where it happens frequently is when
>>> reading news with gnus.
>>
>> I haven't seen this problem in my usage of gnus.
>
> It is not just gnus.  And it is not easy to see: Emacs gets somewhat
> sluggish, sometimes the cursor blinks quite less than expected, and
> CPU usage goes up, up, up, which is of course mostly noticeable when
> you have power management, and the fans start waking up.
>
>> It would be helpful if you could be more specific: does the problem
>> happen with the 2006-07-26 change to `read-event'?
>
> Already before.  I have now recompiled Emacs and will see whether this
> changes something.

Update: it still happens.  I am working for a while in a web browser,
suddenly the fans engage, the Emacs frame (not even mapped when this
happens) shows a very slow and erratically blinking cursor, and `top'
shows that Emacs is consuming close to 100% of CPU power.

So yes, even an off-screen Emacs sitting idle in some frame suddenly
decides to suck up all CPU power.

>> If so, did it start to happen at that time, or did it already
>> happen after the Lisp-level `sit-for' was implemented?
>>
>> If it only started after the `read-event' change, one possibility
>> is that read_char is getting stuck somewhere before we get to the
>> point where we check if the timeout has elapsed.
>>
>>> This is really a nuisance.  The change to sit-for is a fundamental
>>> change to some core mechanism of Emacs, and it is currently
>>> seemingly breaking quite a few things, apart from causing strange
>>> effects.
>>
>> That's the story of the Emacs 22 feature freeze.

I can't remember any change as invasive as that without a
correspondingly bad bug it was trying to fix.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: sit-for

[Chong Yidong]

> Update: it still happens.  I am working for a while in a web browser,
> suddenly the fans engage, the Emacs frame (not even mapped when this
> happens) shows a very slow and erratically blinking cursor, and `top'
> shows that Emacs is consuming close to 100% of CPU power.
>
> So yes, even an off-screen Emacs sitting idle in some frame suddenly
> decides to suck up all CPU power.

My guess is that it's a `sit-for' call in the jit-lock stealth timer
causing the problem.

There are two places in read_char that could conceivably carry out
some expensive operations while waiting for input.  I doubt these are
the culprits, but to rule them out, could you apply the following
patch and see if the problem still occurs?

Apart from that, there are two other possibilities I can think of that
may cause this kind of trouble.  The first is that some invalid "input
event" is continuously firing, causing wait_reading_process_output to
return instantly.  Since kbd_buffer_get_event loops calling
wait_reading_process_output until it gets a valid event, this could
cause high CPU consumption.  If this is indeed the case, we'll
probably have to somehow fix wait_read_process_output itself to ignore
those fake events (which makes the entire strategy of the Lisp-level
sit-for redundant).

The other possibility is some kind of bad timer interaction, in which
a timer is called during sit-for, but that timer itself contains a
sit-for, so each sit-for never returns.  But that kind of bug would
probably have affected the old sit-for implementation too, so this
seems unlikely.

*** emacs/src/keyboard.c.~1.861.~	2006-07-28 09:21:33.000000000 -0400
--- emacs/src/keyboard.c	2006-07-29 10:06:29.000000000 -0400
***************
*** 2556,2561 ****
--- 2556,2570 ----
  	    /* Normal case: no input arrived during redisplay.  */
  	    break;
  
+ 	  /* Don't keep looping if the timeout expires.  */
+ 	  if (end_time)
+ 	    {
+ 	      EMACS_TIME now;
+ 	      EMACS_GET_TIME (now);
+ 	      if (EMACS_TIME_GE (now, *end_time))
+ 		goto exit;
+ 	    }
+ 
  	  /* Input arrived and pre-empted redisplay.
  	     Process any events which are not user-visible.  */
  	  swallow_events (0);
***************
*** 2749,2755 ****
  
    /* Maybe autosave and/or garbage collect due to idleness.  */
  
!   if (INTERACTIVE && NILP (c))
      {
        int delay_level, buffer_size;
  
--- 2758,2764 ----
  
    /* Maybe autosave and/or garbage collect due to idleness.  */
  
!   if (INTERACTIVE && NILP (c) && !end_time)
      {
        int delay_level, buffer_size;

Re: sit-for

[Richard Stallman]

    > That's the story of the Emacs 22 feature freeze.

This change is a bug fix.  sit-for was waking up when no input
was available.

Re: sit-for

[Richard Stallman]

    Update: it still happens.  I am working for a while in a web browser,
    suddenly the fans engage, the Emacs frame (not even mapped when this
    happens) shows a very slow and erratically blinking cursor, and `top'
    shows that Emacs is consuming close to 100% of CPU power.

Please run Emacs under GDB, so you can suspend it several times while
this is happening.  Make a backtrace (C and Lisp) each time, and by
and by you'll see a pattern which shows what part of Emacs is doing
this.

Re: sit-for

[Kim F. Storm]

Chong Yidong <cyd@stupidchicken.com> writes:

> The other possibility is some kind of bad timer interaction, in which
> a timer is called during sit-for, but that timer itself contains a
> sit-for, so each sit-for never returns.  But that kind of bug would
> probably have affected the old sit-for implementation too, so this
> seems unlikely.

I don't know if it is related, but from time to time, the cursor
disappears completely (with blinking cursor).  I have not been able to
reproduce this in any reliable way.  This is an old bug, discussed on
the list 1-2 years ago. It still happens with the new sit-for ...


Since wait_read_process_output now has an end_time rather than a
duration, it seems possible to make a check on reentry to
wait_read_process_output whether an "outer" call has an end_time
BEFORE the end_time of the current call ... and do something sensible,
e.g.  modify the current end_time to the same as the outer end_time.

In that way, nested sit-for calls would never make emacs wait longer
than "allowed" by an outer sit-for call.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: sit-for

[Richard Stallman]

    Since wait_read_process_output now has an end_time rather than a
    duration, it seems possible to make a check on reentry to
    wait_read_process_output whether an "outer" call has an end_time
    BEFORE the end_time of the current call ... and do something sensible,
    e.g.  modify the current end_time to the same as the outer end_time.

This seems like a good idea.

Re: sit-for

[Chong Yidong]

> Since we have the new sit-for implementation, I have a lot of times
> when Emacs just pauses in busy waiting for input.  This happens
> spontaneously.  One situation where it happens frequently is when
> reading news with gnus.

Another possibility just occurred to me.  Unlike the old sit-for, the
new sit-for is not interrupted by input coming from processes (as
opposed to user input).  If gnus (or some other package) relies on
this behavior, a bug will arise.

The question is: is this a bug of the new sit-for?  If sit-for is
changed to interrupt due to processes, we have the same problem as
before: input coming from miscellaneous async processes will interfere
with towers of hanoi (or other animation code).

One possibility is to bring back the old sit-for with its warts
(interruptable even if no new input events are available) and change
the animation code to use `read-event' with its new timeout argument.

Another possibility is to leave sit-for as it is, try to find code
that relies on sit-for returning due to process output, and fix it.

Re: sit-for

[Kim F. Storm]

Chong Yidong <cyd@stupidchicken.com> writes:

>> Since we have the new sit-for implementation, I have a lot of times
>> when Emacs just pauses in busy waiting for input.  This happens
>> spontaneously.  One situation where it happens frequently is when
>> reading news with gnus.
>
> Another possibility just occurred to me.  Unlike the old sit-for, the
> new sit-for is not interrupted by input coming from processes (as
> opposed to user input).  If gnus (or some other package) relies on
> this behavior, a bug will arise.

IMO, sit-for should never be interrupted by input coming from a
subprocess (that is what accept-process-output is for), and code
which relies on that behaviour is wrong.

Process output is _not_ input in the normal sense.  AFAICS, process
output is still read during sit-for and passed to the proper filters
or buffers--so the new sit-for is doing TRT.

>
> The question is: is this a bug of the new sit-for?  If sit-for is
> changed to interrupt due to processes, we have the same problem as
> before: input coming from miscellaneous async processes will interfere
> with towers of hanoi (or other animation code).
>
> One possibility is to bring back the old sit-for with its warts
> (interruptable even if no new input events are available) and change
> the animation code to use `read-event' with its new timeout argument.

That would be a big step backwards!

>
> Another possibility is to leave sit-for as it is, try to find code
> that relies on sit-for returning due to process output, and fix it.

Richard specifically asked [someone] to check all sit-for calls to see
if the new behaviour would break them.  So [we] should already have
done this (but I don't think anybody actually did that).

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: sit-for

[Chong Yidong]

>> Another possibility is to leave sit-for as it is, try to find code
>> that relies on sit-for returning due to process output, and fix it.
>
> Richard specifically asked [someone] to check all sit-for calls to see
> if the new behaviour would break them.  So [we] should already have
> done this (but I don't think anybody actually did that).

I did check, but maybe I missed something non-obvious.  The strange
behavior of Gnus reported by David Kastrup seems to indicate that some
code somewhere is relying on some undocumented feature of sit-for.

Re: sit-for

[Chong Yidong]

David Kastrup <dak@gnu.org> writes:

>>> Since we have the new sit-for implementation, I have a lot of times
>>> when Emacs just pauses in busy waiting for input.  This happens
>>> spontaneously.  One situation where it happens frequently is when
>>> reading news with gnus.
>>
>> I haven't seen this problem in my usage of gnus.
>
> It is not just gnus.  And it is not easy to see: Emacs gets somewhat
> sluggish, sometimes the cursor blinks quite less than expected, and
> CPU usage goes up, up, up, which is of course mostly noticeable when
> you have power management, and the fans start waking up.

Does setting jit-lock-stealth-time to nil change anything?

Re: sit-for

[David Kastrup]

storm@cua.dk (Kim F. Storm) writes:

> Chong Yidong <cyd@stupidchicken.com> writes:
>
>>> Since we have the new sit-for implementation, I have a lot of times
>>> when Emacs just pauses in busy waiting for input.  This happens
>>> spontaneously.  One situation where it happens frequently is when
>>> reading news with gnus.
>>
>> Another possibility just occurred to me.  Unlike the old sit-for, the
>> new sit-for is not interrupted by input coming from processes (as
>> opposed to user input).  If gnus (or some other package) relies on
>> this behavior, a bug will arise.
>
> IMO, sit-for should never be interrupted by input coming from a
> subprocess (that is what accept-process-output is for), and code
> which relies on that behaviour is wrong.

accept-process-output does not redraw the screen, so it can't do that.

> Process output is _not_ input in the normal sense.  AFAICS, process
> output is still read during sit-for and passed to the proper filters
> or buffers--so the new sit-for is doing TRT.

_If_ it indeed processes the input.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: sit-for

[David Kastrup]

Chong Yidong <cyd@stupidchicken.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>>>> Since we have the new sit-for implementation, I have a lot of times
>>>> when Emacs just pauses in busy waiting for input.  This happens
>>>> spontaneously.  One situation where it happens frequently is when
>>>> reading news with gnus.
>>>
>>> I haven't seen this problem in my usage of gnus.
>>
>> It is not just gnus.  And it is not easy to see: Emacs gets somewhat
>> sluggish, sometimes the cursor blinks quite less than expected, and
>> CPU usage goes up, up, up, which is of course mostly noticeable when
>> you have power management, and the fans start waking up.
>
> Does setting jit-lock-stealth-time to nil change anything?

Unfortunately, the behavior occured just once after I last updated
Emacs to the new timer-based stuff, and I did not debug it then since
it just made me shrug and say "same bug as before".  So the frequency
of that occurence seemingly has dropped sharply, at least.  I am
rather sure, however, that I encountered it this one time after
recompiling Emacs.  If I manage to have it happen again, I'll sic the
debugger on it, something I should have done the first time.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: sit-for

[Richard Stallman]

    > Another possibility just occurred to me.  Unlike the old sit-for, the
    > new sit-for is not interrupted by input coming from processes (as
    > opposed to user input).  If gnus (or some other package) relies on
    > this behavior, a bug will arise.

    IMO, sit-for should never be interrupted by input coming from a
    subprocess (that is what accept-process-output is for), and code
    which relies on that behaviour is wrong.

Did the old sit-for really wake up when process input came in?
It was never supposed to -- that would have been a bug.  
But I don't recall seeing any reports of such a bug.
The bug was that input events that did not correspond to any
real input could wake it up.  But process output does not
work by generating input events.