High CPU load

High CPU load

[David Reitter]

Recent Emacs 22 branch builds (Carbon) from CVS appear to occasionally  
cause high CPU loads (100-140% on my dual core). This has been  
reported by various users and I'm seeing it myself as well.

Consistently, this happens when AUCTeX (latex-mode) is used (and  
perhaps other things like RefTeX and ispell).
This has been occurring since the switch to Leopard (10.5).

Thread Viewer shows 8 active threads in the Emacs process, two of them  
working.

The stack trace of the first one shows select$DARWIN_EXTSN (right  
after _pthread_start).
The second one is in mach_msg_trap, CFRunLoopRunSpecific, CFLoopRun,  
TSystemNotificationTask::SystemNotificationTaskProc(void*),  
PrivateMPEntryPoint.


More details from within the running Emacs (which runs nicely while  
the load problem occurs):

(process-list) returns (#<process> ispell)

flyspell-mode is not enabled, but aspell (via 'flyspell-buffer') was  
used at some point.

(process-sentinel (car (process-list)))  returns nil
(process-status (car (process-list))) returns `run'
(kill-process (car (process-list))) works as intended. CPU load  
continues.

killed latex buffer. CPU load continues.

(get-internal-run-time) reflects the high CPU load.

timer-list is nil.
timer-idle-qlist is:

([t 0 0 125000 t show-paren-function nil t]
  [nil 0 0 500000 0.5 blink-cursor-start nil t]
  [nil 0 0 500000 t jit-lock-context-fontify nil t]
  [nil 0 1 199999 t reftex-view-crossref-when-idle nil t]
  [nil 0 5 0 t mac-cleanup-expired-apple-events nil t])

`cancel-function-timers' for any of these doesn't help the situation  
(I tried all except blink-cursor-start).
timer-event-last is correct.


I have no guarantees that this would also occur when started with -Q -  
since the problem only occurs sporadically, I can't test that as  
easily. But I can't see how Lisp-level changes or the patches we use  
could cause this. Can I investigate this further?


In GNU Emacs 22.1.50.1 (i386-apple-darwin8.11.1, Carbon Version 1.6.0)
  of 2007-11-26 on plume.sr.unh.edu - Aquamacs Distribution 1.2
Windowing system distributor `Apple Inc.', version 10.5.1
configured using `configure  '--without-x' '--prefix=/usr/local''

Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: nil
   value of $LC_CTYPE: nil
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: nil
   value of $LC_NUMERIC: nil
   value of $LC_TIME: nil
   value of $LANG: nil
   locale-coding-system: iso-8859-1
   default-enable-multibyte-characters: t

Major mode: Text

Minor modes in effect:
   smart-frame-positioning-mode: t
   aquamacs-styles-mode: t
   shell-dirtrack-mode: t
   recentf-mode: t
   encoded-kbd-mode: t
   osx-key-mode: t
   show-paren-mode: t
   delete-selection-mode: t
   pc-selection-mode: t
   cua-mode: t
   tooltip-mode: t
   mac-input-method-mode: t
   tool-bar-mode: 0
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   unify-8859-on-encoding-mode: t
   utf-translate-cjk-mode: t
   auto-compression-mode: t
   column-number-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent input:
<menu-bar> <help-menu> <send-emacs-bug-report> 1 0
0 <backspace> <backspace> <backspace> <backspace> H
i g h SPC C P U SPC l o a d SPC o n SPC C a r b n SPC
<backspace> <backspace> o n SPC p o r t SPC w i t h
SPC <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> SPC ( w i t h SPC A U C T e
X ) <return> <switch-frame> <menu-bar> <help-menu>
<report-emacs-bug>

Recent messages:
Defining fontset: lucida_typewriter14
Loading /Users/dr/Library/Preferences/Aquamacs Emacs/frame- 
positions.el (source)...done
Aquamacs is based on GNU Emacs 22, a part of the GNU/Linux system.
It is Free Software: you can improve and redistribute it under the GNU  
General Public License, version 3
or later. Copyright (C) 2007 Free Software Foundation, Inc. (C) 2007  
D. Reitter. No Warranty.
Aquamacs is based on GNU Emacs 22, a part of the GNU/Linux system.
It is Free Software: you can improve and redistribute it under the GNU  
General Public License, version
3 or later. Copyright (C) 2007 Free Software Foundation, Inc. (C) 2007  
D. Reitter. No Warranty.
Loading emacsbug...done
call-interactively: Text is read-only

Re: High CPU load

[YAMAMOTO Mitsuharu]

>>>>> On Tue, 27 Nov 2007 10:38:00 +0000, David Reitter <david.reitter@gmail.com> said:

> Recent Emacs 22 branch builds (Carbon) from CVS appear to
> occasionally cause high CPU loads (100-140% on my dual core). This
> has been reported by various users and I'm seeing it myself as well.

Does this happen also on unmodified Carbon Emacs, or Aquamacs Emacs
only?

> Consistently, this happens when AUCTeX (latex-mode) is used (and
> perhaps other things like RefTeX and ispell).  This has been
> occurring since the switch to Leopard (10.5).

I have no idea.  Could you try replacing mac.c with the one in the
trunk (Rev. 1.84) to see if this changes the situation?

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: High CPU load

[David Reitter]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

On 27 Nov 2007, at 13:06, YAMAMOTO Mitsuharu wrote:

>> Recent Emacs 22 branch builds (Carbon) from CVS appear to
>> occasionally cause high CPU loads (100-140% on my dual core). This
>> has been reported by various users and I'm seeing it myself as well.
>
> Does this happen also on unmodified Carbon Emacs, or Aquamacs Emacs
> only?

I don't know. Seiji, did you get any reports?

>> Consistently, this happens when AUCTeX (latex-mode) is used (and
>> perhaps other things like RefTeX and ispell).  This has been
>> occurring since the switch to Leopard (10.5).
>
> I have no idea.  Could you try replacing mac.c with the one in the
> trunk (Rev. 1.84) to see if this changes the situation?

OK, done that, and I will distribute this variant to testers. I reckon  
it will be a while before we can tell.



[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2193 bytes --]

Re: High CPU load

[David Reitter]

On 27 Nov 2007, at 13:06, YAMAMOTO Mitsuharu wrote:
> I have no idea.  Could you try replacing mac.c with the one in the
> trunk (Rev. 1.84) to see if this changes the situation?

OK, having done that, I've got a report from a user who still  
experiences the problem:



Begin forwarded message:

> From: Ugur Ozdemir <uozdemir@artsci.wustl.edu>
> Date: 27 November 2007 17:51:16 GMT
> To: Development of Aquamacs Emacs <aquamacs-devel@aquamacs.org>
> Subject: Re: [Aquamacs-devel] Aquamacs / CPU Load Problem
> Reply-To: Development of Aquamacs Emacs <aquamacs-devel@aquamacs.org>
>
> David,
>
> I had the same problem with this latest nightly build. I do not know
> these things very much here are the threads that go crazy in the
> thread viewer:
>
> thr 05807	select$DARWIN_EXTSN
> 			-pthread_start
> 			thread_start
>
> thr 05c07	mach-msg-trap
> 			CFRunLoopRunSpecific
> 			CFRunLoopRun
> 			TSystemNotificationTask::SystemNotificationTaskProc(void*)
> 			PrivateMPEntryPoint
> 			-pthread_start
> 			thread_start
>
> Best,
>
> Ugur
>
> On Nov 27, 2007, at 10:33 AM, David Reitter wrote:
>
>> I'm writing to ask your help in diagnosing the CPU hogging problem
>> with recent Aquamacs versions in Leopard.
>>
>> We have an experimental fix to address the problem (we don't know
>> what it is - that's why it's purely experimental) and we'd like to
>> know whether this works. Can you download, install and use the
>> latest nightly Intel build from http://aquamacs.org/nightlies.shtml?
>> If you're on a PPC Mac, please do let me know.
>>
>>

Re: High CPU load

[Seiji Zenitani]

On 2007/11/27, at 11:29, David Reitter wrote:
> On 27 Nov 2007, at 13:06, YAMAMOTO Mitsuharu wrote:
>
>>> Recent Emacs 22 branch builds (Carbon) from CVS appear to
>>> occasionally cause high CPU loads (100-140% on my dual core). This
>>> has been reported by various users and I'm seeing it myself as well.
>>
>> Does this happen also on unmodified Carbon Emacs, or Aquamacs Emacs
>> only?
>
> I don't know. Seiji, did you get any reports?
>
I've got no reports.

Re: High CPU load

[YAMAMOTO Mitsuharu]

>>>>> On Tue, 27 Nov 2007 16:29:34 +0000, David Reitter <david.reitter@gmail.com> said:

>> Does this happen also on unmodified Carbon Emacs, or Aquamacs Emacs
>> only?

> I don't know. Seiji, did you get any reports?

Ah, I meant bare CVS version.  Carbon Emacs Package is also a modified
Carbon Emacs.

>>>>> On Tue, 27 Nov 2007 17:57:54 +0000, David Reitter <david.reitter@gmail.com> said:

> On 27 Nov 2007, at 13:06, YAMAMOTO Mitsuharu wrote:
>> I have no idea.  Could you try replacing mac.c with the one in the
>> trunk (Rev. 1.84) to see if this changes the situation?

> OK, having done that, I've got a report from a user who still  
> experiences the problem:

> Begin forwarded message:

>> From: Ugur Ozdemir <uozdemir@artsci.wustl.edu>
>> Date: 27 November 2007 17:51:16 GMT
>> To: Development of Aquamacs Emacs <aquamacs-devel@aquamacs.org>
>> Subject: Re: [Aquamacs-devel] Aquamacs / CPU Load Problem
>> Reply-To: Development of Aquamacs Emacs <aquamacs-devel@aquamacs.org>
>> 
>> David,
>> 
>> I had the same problem with this latest nightly build. I do not know
>> these things very much here are the threads that go crazy in the
>> thread viewer:
>> 
>> thr 05807	select$DARWIN_EXTSN
>> -pthread_start
>> thread_start
>> 
>> thr 05c07	mach-msg-trap
>> CFRunLoopRunSpecific
>> CFRunLoopRun
>> TSystemNotificationTask::SystemNotificationTaskProc(void*)
>> PrivateMPEntryPoint
>> -pthread_start
>> thread_start

Thanks for testing.

The second thread seems to be spawned when one uses the file dialog on
Leopard.  Carbon's Navigation Services (file dialogs) have heavily
changed on Leopard so it uses Cocoa's NSSavePanel/NSOpenPanel (you can
see the difference by typing `/' or `~').  So I suspect some bugs have
been introduced there.  Maybe you can avoid this situation by setting
use-dialog-box to nil in the meanwhile.

I also guess high CPU usage has something to do with filesystems over
network, because SystemNotificationTaskProc above seems to monitor
file system volume and the socket monitor thread (the first thread
above) is also busy.  Maybe you can monitor network packets while high
CPU load is observed.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: High CPU load

[YAMAMOTO Mitsuharu]

>>>>> On Wed, 28 Nov 2007 10:24:00 +0900, YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp> said:

> Maybe you can avoid this situation by setting use-dialog-box to nil
> in the meanwhile.

Actually, setting use-file-dialog to nil was sufficient.

I could reproduce the high CPU load with unmodified Carbon Emacs on
Leopard using the following steps.

  1. Create 7 instances of shell buffers using C-u M-x shell repeatedly.
  2. Kill all the shell buffers.
  3. Open a file dialog and cancel it.

> I also guess high CPU usage has something to do with filesystems
> over network, because SystemNotificationTaskProc above seems to
> monitor file system volume and the socket monitor thread (the first
> thread above) is also busy.  Maybe you can monitor network packets
> while high CPU load is observed.

This guess was not right.  Normally, SystemNotificationTaskProc waits
for a Unix domain socket /var/run/mDNSResponder to become readable.
But if the file descriptor is the one that was used in the Emacs main
thread previously, the socket is waited to become either readable or
writable because of the specification of CFSocketCreateWithNative.
Since /var/run/mDNSResponder is always writable, that leads to a busy
loop.

Please try the following patch.  It invalidates CFSocket objects every
time to avoid the above situation.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Index: src/mac.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/mac.c,v
retrieving revision 1.77.2.3
diff -c -p -r1.77.2.3 mac.c
*** src/mac.c	1 Aug 2007 22:20:41 -0000	1.77.2.3
--- src/mac.c	29 Nov 2007 00:18:55 -0000
*************** static ComponentInstance as_scripting_co
*** 79,84 ****
--- 79,85 ----
  /* The single script context used for all script executions.  */
  static OSAID as_script_context;
  
+ #ifndef MAC_OS_X
  #if TARGET_API_MAC_CARBON
  static int wakeup_from_rne_enabled_p = 0;
  #define ENABLE_WAKEUP_FROM_RNE (wakeup_from_rne_enabled_p = 1)
*************** static int wakeup_from_rne_enabled_p = 0
*** 87,92 ****
--- 88,94 ----
  #define ENABLE_WAKEUP_FROM_RNE 0
  #define DISABLE_WAKEUP_FROM_RNE 0
  #endif
+ #endif
  
  #ifndef MAC_OSX
  static OSErr posix_pathname_to_fsspec P_ ((const char *, FSSpec *));
*************** Lisp_Object
*** 1127,1144 ****
  cfdate_to_lisp (date)
       CFDateRef date;
  {
!   static const CFGregorianDate epoch_gdate = {1970, 1, 1, 0, 0, 0.0};
!   static CFAbsoluteTime epoch = 0.0, sec;
!   int high, low;
! 
!   if (epoch == 0.0)
!     epoch = CFGregorianDateGetAbsoluteTime (epoch_gdate, NULL);
  
!   sec = CFDateGetAbsoluteTime (date) - epoch;
    high = sec / 65536.0;
    low = sec - high * 65536.0;
  
!   return list3 (make_number (high), make_number (low), make_number (0));
  }
  
  
--- 1129,1143 ----
  cfdate_to_lisp (date)
       CFDateRef date;
  {
!   CFTimeInterval sec;
!   int high, low, microsec;
  
!   sec = CFDateGetAbsoluteTime (date) + kCFAbsoluteTimeIntervalSince1970;
    high = sec / 65536.0;
    low = sec - high * 65536.0;
+   microsec = (sec - floor (sec)) * 1000000.0;
  
!   return list3 (make_number (high), make_number (low), make_number (microsec));
  }
  
  
*************** xrm_get_preference_database (application
*** 1826,1833 ****
  
    GCPRO3 (database, quarks, value);
  
-   BLOCK_INPUT;
- 
    app_id = kCFPreferencesCurrentApplication;
    if (application)
      {
--- 1825,1830 ----
*************** xrm_get_preference_database (application
*** 1879,1886 ****
      CFRelease (key_set);
    CFRelease (app_id);
  
-   UNBLOCK_INPUT;
- 
    UNGCPRO;
  
    return database;
--- 1876,1881 ----
*************** extern int noninteractive;
*** 4994,5001 ****
        SELECT_TIMEOUT_THRESHOLD_RUNLOOP seconds).
        -> Create CFSocket for each socket and add it into the current
           event RunLoop so that the current event loop gets quit when
!          the socket becomes ready.  Then ReceiveNextEvent can wait for
!          both kinds of inputs.
     4. Otherwise.
        -> Periodically poll the window input channel while repeatedly
           executing `select' with a short timeout
--- 4989,4996 ----
        SELECT_TIMEOUT_THRESHOLD_RUNLOOP seconds).
        -> Create CFSocket for each socket and add it into the current
           event RunLoop so that the current event loop gets quit when
!          the socket becomes ready.  Then CFRunLoopRunInMode can wait
!          for both kinds of inputs.
     4. Otherwise.
        -> Periodically poll the window input channel while repeatedly
           executing `select' with a short timeout
*************** socket_callback (s, type, address, data,
*** 5017,5028 ****
       const void *data;
       void *info;
  {
-   int fd = CFSocketGetNative (s);
-   SELECT_TYPE *ofds = (SELECT_TYPE *)info;
- 
-   if ((type == kCFSocketReadCallBack && FD_ISSET (fd, &ofds[0]))
-       || (type == kCFSocketConnectCallBack && FD_ISSET (fd, &ofds[1])))
-     QuitEventLoop (GetCurrentEventLoop ());
  }
  #endif	/* SELECT_USE_CFSOCKET */
  
--- 5012,5017 ----
*************** select_and_poll_event (nfds, rfds, wfds,
*** 5032,5073 ****
       SELECT_TYPE *rfds, *wfds, *efds;
       EMACS_TIME *timeout;
  {
!   OSStatus err = noErr;
    int r = 0;
  
!   /* Try detect_input_pending before ReceiveNextEvent in the same
       BLOCK_INPUT block, in case that some input has already been read
       asynchronously.  */
    BLOCK_INPUT;
!   ENABLE_WAKEUP_FROM_RNE;
!   if (!detect_input_pending ())
      {
!       EMACS_TIME select_timeout;
!       EventTimeout timeoutval =
! 	(timeout
! 	 ? (EMACS_SECS (*timeout) * kEventDurationSecond
! 	    + EMACS_USECS (*timeout) * kEventDurationMicrosecond)
! 	 : kEventDurationForever);
  
        EMACS_SET_SECS_USECS (select_timeout, 0, 0);
        r = select (nfds, rfds, wfds, efds, &select_timeout);
        if (timeoutval == 0.0)
! 	err = eventLoopTimedOutErr;
!       else if (r == 0)
  	{
  #if USE_CG_DRAWING
  	  mac_prepare_for_quickdraw (NULL);
  #endif
! 	  err = ReceiveNextEvent (0, NULL, timeoutval,
! 				  kEventLeaveInQueue, NULL);
  	}
      }
-   DISABLE_WAKEUP_FROM_RNE;
    UNBLOCK_INPUT;
  
    if (r != 0)
      return r;
!   else if (err == noErr)
      {
        /* Pretend that `select' is interrupted by a signal.  */
        detect_input_pending ();
--- 5021,5084 ----
       SELECT_TYPE *rfds, *wfds, *efds;
       EMACS_TIME *timeout;
  {
!   int timedout_p = 0;
    int r = 0;
+   EMACS_TIME select_timeout;
+   EventTimeout timeoutval =
+     (timeout
+      ? (EMACS_SECS (*timeout) * kEventDurationSecond
+ 	+ EMACS_USECS (*timeout) * kEventDurationMicrosecond)
+      : kEventDurationForever);
+   SELECT_TYPE orfds, owfds, oefds;
  
!   if (timeout == NULL)
!     {
!       if (rfds) orfds = *rfds;
!       if (wfds) owfds = *wfds;
!       if (efds) oefds = *efds;
!     }
! 
!   /* Try detect_input_pending before CFRunLoopRunInMode in the same
       BLOCK_INPUT block, in case that some input has already been read
       asynchronously.  */
    BLOCK_INPUT;
!   while (1)
      {
!       if (detect_input_pending ())
! 	break;
  
        EMACS_SET_SECS_USECS (select_timeout, 0, 0);
        r = select (nfds, rfds, wfds, efds, &select_timeout);
+       if (r != 0)
+ 	break;
+ 
        if (timeoutval == 0.0)
! 	timedout_p = 1;
!       else
  	{
  #if USE_CG_DRAWING
  	  mac_prepare_for_quickdraw (NULL);
  #endif
! 	  if (CFRunLoopRunInMode (kCFRunLoopDefaultMode,
! 				  timeoutval >= 0 ? timeoutval : 100000, true)
! 	      == kCFRunLoopRunTimedOut)
! 	    timedout_p = 1;
  	}
+ 
+       if (timeout == NULL && timedout_p)
+ 	{
+ 	  if (rfds) *rfds = orfds;
+ 	  if (wfds) *wfds = owfds;
+ 	  if (efds) *efds = oefds;
+ 	}
+       else
+ 	break;
      }
    UNBLOCK_INPUT;
  
    if (r != 0)
      return r;
!   else if (!timedout_p)
      {
        /* Pretend that `select' is interrupted by a signal.  */
        detect_input_pending ();
*************** sys_select (nfds, rfds, wfds, efds, time
*** 5084,5108 ****
       SELECT_TYPE *rfds, *wfds, *efds;
       EMACS_TIME *timeout;
  {
!   OSStatus err = noErr;
    int r;
    EMACS_TIME select_timeout;
!   static SELECT_TYPE ofds[3];
  
    if (inhibit_window_system || noninteractive
        || nfds < 1 || rfds == NULL || !FD_ISSET (0, rfds))
      return select (nfds, rfds, wfds, efds, timeout);
  
    FD_CLR (0, rfds);
!   ofds[0] = *rfds;
  
    if (wfds)
!     ofds[1] = *wfds;
    else
!     FD_ZERO (&ofds[1]);
  
    if (efds)
!     ofds[2] = *efds;
    else
      {
        EventTimeout timeoutval =
--- 5095,5119 ----
       SELECT_TYPE *rfds, *wfds, *efds;
       EMACS_TIME *timeout;
  {
!   int timedout_p = 0;
    int r;
    EMACS_TIME select_timeout;
!   SELECT_TYPE orfds, owfds, oefds;
  
    if (inhibit_window_system || noninteractive
        || nfds < 1 || rfds == NULL || !FD_ISSET (0, rfds))
      return select (nfds, rfds, wfds, efds, timeout);
  
    FD_CLR (0, rfds);
!   orfds = *rfds;
  
    if (wfds)
!     owfds = *wfds;
    else
!     FD_ZERO (&owfds);
  
    if (efds)
!     oefds = *efds;
    else
      {
        EventTimeout timeoutval =
*************** sys_select (nfds, rfds, wfds, efds, time
*** 5130,5212 ****
        if (r != 0 || timeoutval == 0.0)
  	return r;
  
!       *rfds = ofds[0];
        if (wfds)
! 	*wfds = ofds[1];
  
  #if SELECT_USE_CFSOCKET
        if (timeoutval > 0 && timeoutval <= SELECT_TIMEOUT_THRESHOLD_RUNLOOP)
  	goto poll_periodically;
  
!       /* Try detect_input_pending before ReceiveNextEvent in the same
! 	 BLOCK_INPUT block, in case that some input has already been
! 	 read asynchronously.  */
        BLOCK_INPUT;
-       ENABLE_WAKEUP_FROM_RNE;
        if (!detect_input_pending ())
  	{
! 	  int minfd, fd;
  	  CFRunLoopRef runloop =
  	    (CFRunLoopRef) GetCFRunLoopFromEventLoop (GetCurrentEventLoop ());
! 	  static const CFSocketContext context = {0, ofds, NULL, NULL, NULL};
! 	  static CFMutableDictionaryRef sources;
! 
! 	  if (sources == NULL)
! 	    sources =
! 	      CFDictionaryCreateMutable (NULL, 0, NULL,
! 					 &kCFTypeDictionaryValueCallBacks);
  
  	  for (minfd = 1; ; minfd++) /* nfds-1 works as a sentinel.  */
  	    if (FD_ISSET (minfd, rfds) || (wfds && FD_ISSET (minfd, wfds)))
  	      break;
  
  	  for (fd = minfd; fd < nfds; fd++)
  	    if (FD_ISSET (fd, rfds) || (wfds && FD_ISSET (fd, wfds)))
  	      {
! 		void *key = (void *) fd;
! 		CFRunLoopSourceRef source =
! 		  (CFRunLoopSourceRef) CFDictionaryGetValue (sources, key);
  
  		if (source == NULL)
  		  {
- 		    CFSocketRef socket =
- 		      CFSocketCreateWithNative (NULL, fd,
- 						(kCFSocketReadCallBack
- 						 | kCFSocketConnectCallBack),
- 						socket_callback, &context);
- 
- 		    if (socket == NULL)
- 		      continue;
- 		    source = CFSocketCreateRunLoopSource (NULL, socket, 0);
  		    CFRelease (socket);
! 		    if (source == NULL)
! 		      continue;
! 		    CFDictionaryAddValue (sources, key, source);
! 		    CFRelease (source);
  		  }
  		CFRunLoopAddSource (runloop, source, kCFRunLoopDefaultMode);
  	      }
  
  #if USE_CG_DRAWING
  	  mac_prepare_for_quickdraw (NULL);
  #endif
! 	  err = ReceiveNextEvent (0, NULL, timeoutval,
! 				  kEventLeaveInQueue, NULL);
! 
! 	  for (fd = minfd; fd < nfds; fd++)
! 	    if (FD_ISSET (fd, rfds) || (wfds && FD_ISSET (fd, wfds)))
! 	      {
! 		void *key = (void *) fd;
! 		CFRunLoopSourceRef source =
! 		  (CFRunLoopSourceRef) CFDictionaryGetValue (sources, key);
  
! 		CFRunLoopRemoveSource (runloop, source, kCFRunLoopDefaultMode);
! 	      }
  	}
-       DISABLE_WAKEUP_FROM_RNE;
        UNBLOCK_INPUT;
  
!       if (err == noErr || err == eventLoopQuitErr)
  	{
  	  EMACS_SET_SECS_USECS (select_timeout, 0, 0);
  	  return select_and_poll_event (nfds, rfds, wfds, efds,
--- 5141,5244 ----
        if (r != 0 || timeoutval == 0.0)
  	return r;
  
!       *rfds = orfds;
        if (wfds)
! 	*wfds = owfds;
  
  #if SELECT_USE_CFSOCKET
        if (timeoutval > 0 && timeoutval <= SELECT_TIMEOUT_THRESHOLD_RUNLOOP)
  	goto poll_periodically;
  
!       /* Try detect_input_pending before CFRunLoopRunInMode in the
! 	 same BLOCK_INPUT block, in case that some input has already
! 	 been read asynchronously.  */
        BLOCK_INPUT;
        if (!detect_input_pending ())
  	{
! 	  int minfd, fd, nsocks;
  	  CFRunLoopRef runloop =
  	    (CFRunLoopRef) GetCFRunLoopFromEventLoop (GetCurrentEventLoop ());
! #if MAC_OS_X_VERSION_MAX_ALLOWED >= 1020
! 	  CFSocketRef *shead, *s;
! #else
! 	  CFRunLoopSourceRef *shead, *s;
! #endif
  
  	  for (minfd = 1; ; minfd++) /* nfds-1 works as a sentinel.  */
  	    if (FD_ISSET (minfd, rfds) || (wfds && FD_ISSET (minfd, wfds)))
  	      break;
  
+ 	  nsocks = 1;
+ 	  for (fd = minfd + 1; fd < nfds; fd++)
+ 	    if (FD_ISSET (fd, rfds) || (wfds && FD_ISSET (fd, wfds)))
+ 	      nsocks++;
+ 
+ #if MAC_OS_X_VERSION_MAX_ALLOWED >= 1020
+ 	  shead = alloca (sizeof (CFSocketRef) * nsocks);
+ #else
+ 	  shead = alloca (sizeof (CFRunLoopSourceRef) * nsocks);
+ #endif
+ 	  s = shead;
  	  for (fd = minfd; fd < nfds; fd++)
  	    if (FD_ISSET (fd, rfds) || (wfds && FD_ISSET (fd, wfds)))
  	      {
! 		CFSocketRef socket =
! 		  CFSocketCreateWithNative (NULL, fd,
! 					    (kCFSocketReadCallBack
! 					     | kCFSocketConnectCallBack),
! 					    socket_callback, NULL);
! 		CFRunLoopSourceRef source;
! 
! 		if (socket == NULL)
! 		  continue;
! #if MAC_OS_X_VERSION_MAX_ALLOWED >= 1020
! 		{
! 		  CFOptionFlags flags = CFSocketGetSocketFlags (socket);
  
+ 		  CFSocketSetSocketFlags (socket,
+ 					  flags & ~kCFSocketCloseOnInvalidate);
+ 		}
+ #endif
+ 		source = CFSocketCreateRunLoopSource (NULL, socket, 0);
  		if (source == NULL)
  		  {
  		    CFRelease (socket);
! 		    continue;
  		  }
  		CFRunLoopAddSource (runloop, source, kCFRunLoopDefaultMode);
+ #if MAC_OS_X_VERSION_MAX_ALLOWED >= 1020
+ 		CFRelease (source);
+ 		*s = socket;
+ #else
+ 		CFRelease (socket);
+ 		*s = source;
+ #endif
+ 		s++;
  	      }
  
  #if USE_CG_DRAWING
  	  mac_prepare_for_quickdraw (NULL);
  #endif
! 	  if (CFRunLoopRunInMode (kCFRunLoopDefaultMode,
! 				  timeoutval >= 0 ? timeoutval : 100000, true)
! 	      == kCFRunLoopRunTimedOut)
! 	    timedout_p = 1;
  
! 	  do
! 	    {
! 	      --s;
! #if MAC_OS_X_VERSION_MAX_ALLOWED >= 1020
! 	      CFSocketInvalidate (*s);
! #else
! 	      CFRunLoopRemoveSource (runloop, *s, kCFRunLoopDefaultMode);
! #endif
! 	      CFRelease (*s);
! 	    }
! 	  while (s != shead);
  	}
        UNBLOCK_INPUT;
  
!       if (!timedout_p)
  	{
  	  EMACS_SET_SECS_USECS (select_timeout, 0, 0);
  	  return select_and_poll_event (nfds, rfds, wfds, efds,
*************** sys_select (nfds, rfds, wfds, efds, time
*** 5242,5252 ****
  	if (r != 0)
  	  return r;
  
! 	*rfds = ofds[0];
  	if (wfds)
! 	  *wfds = ofds[1];
  	if (efds)
! 	  *efds = ofds[2];
  
  	if (timeout)
  	  {
--- 5274,5284 ----
  	if (r != 0)
  	  return r;
  
! 	*rfds = orfds;
  	if (wfds)
! 	  *wfds = owfds;
  	if (efds)
! 	  *efds = oefds;
  
  	if (timeout)
  	  {
*************** init_mac_osx_environment ()
*** 5409,5418 ****
--- 5441,5452 ----
  void
  mac_wakeup_from_rne ()
  {
+ #ifndef MAC_OSX
    if (wakeup_from_rne_enabled_p)
      /* Post a harmless event so as to wake up from
         ReceiveNextEvent.  */
      mac_post_mouse_moved_event ();
+ #endif
  }
  #endif
  

Re: High CPU load

[David Reitter]

On 29 Nov 2007, at 00:33, YAMAMOTO Mitsuharu wrote:
>
>  1. Create 7 instances of shell buffers using C-u M-x shell  
> repeatedly.
>  2. Kill all the shell buffers.
>  3. Open a file dialog and cancel it.

This does not produce the problem on Aquamacs.

>
> Please try the following patch.  It invalidates CFSocket objects every
> time to avoid the above situation.

Okay, but your patch doesn't apply to version 1.77.2.3 of mac.c  
despite some nudging. (I don't understand why, though.) Could you send  
another one, please?

patch -l -F 15 -c  <~/mac.p
patching file mac.c
Hunk #1 succeeded at 79 with fuzz 3.
Hunk #2 succeeded at 88 with fuzz 3.
Hunk #3 succeeded at 1129 with fuzz 3.
Hunk #7 succeeded at 5012 with fuzz 3.
Hunk #8 FAILED at 5021.
Hunk #9 succeeded at 5095 with fuzz 3.
Hunk #10 FAILED at 5141.
Hunk #11 FAILED at 5274.
Hunk #12 succeeded at 5441 with fuzz 3.
3 out of 12 hunks FAILED -- saving rejects to file mac.c.rej

Re: High CPU load

[YAMAMOTO Mitsuharu]

>>>>> On Thu, 29 Nov 2007 01:09:20 +0000, David Reitter <david.reitter@gmail.com> said:

> On 29 Nov 2007, at 00:33, YAMAMOTO Mitsuharu wrote:
>> 
>> 1. Create 7 instances of shell buffers using C-u M-x shell  
>> repeatedly.
>> 2. Kill all the shell buffers.
>> 3. Open a file dialog and cancel it.

> This does not produce the problem on Aquamacs.

You may need to adjust the number of shell buffers to create/kill
using the output of `lsof' command so the reuse of a file descriptor
can happen between Emacs and SystemNotification threads.

>> Please try the following patch.  It invalidates CFSocket objects every
>> time to avoid the above situation.

> Okay, but your patch doesn't apply to version 1.77.2.3 of mac.c  
> despite some nudging. (I don't understand why, though.) Could you send  
> another one, please?

Invalidation of CFSocket in every sys_select call turned out to be
problematic in another situation, anyway.  Here is another patch that
the invalidation is done via emacs_close call.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Index: src/mac.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/mac.c,v
retrieving revision 1.77.2.3
diff -c -p -r1.77.2.3 mac.c
*** src/mac.c	1 Aug 2007 22:20:41 -0000	1.77.2.3
--- src/mac.c	29 Nov 2007 02:11:40 -0000
*************** extern int noninteractive;
*** 5009,5014 ****
--- 5009,5016 ----
  #if SELECT_USE_CFSOCKET
  #define SELECT_TIMEOUT_THRESHOLD_RUNLOOP 0.2
  
+ static CFMutableDictionaryRef mac_cf_sockets;
+ 
  static void
  socket_callback (s, type, address, data, info)
       CFSocketRef s;
*************** select_and_poll_event (nfds, rfds, wfds,
*** 5079,5084 ****
--- 5081,5110 ----
  }
  
  int
+ mac_try_close_socket (fd)
+      int fd;
+ {
+ #if SELECT_USE_CFSOCKET
+   if (mac_cf_sockets)
+     {
+       void *key = (void *) fd;
+       CFSocketRef socket =
+ 	(CFSocketRef) CFDictionaryGetValue (mac_cf_sockets, key);
+ 
+       if (socket)
+ 	{
+ 	  CFSocketInvalidate (socket);
+ 	  CFDictionaryRemoveValue (mac_cf_sockets, key);
+ 
+ 	  return 1;
+ 	}
+     }
+ #endif
+ 
+   return 0;
+ }
+ 
+ int
  sys_select (nfds, rfds, wfds, efds, timeout)
       int nfds;
       SELECT_TYPE *rfds, *wfds, *efds;
*************** sys_select (nfds, rfds, wfds, efds, time
*** 5156,5161 ****
--- 5182,5192 ----
  	      CFDictionaryCreateMutable (NULL, 0, NULL,
  					 &kCFTypeDictionaryValueCallBacks);
  
+ 	  if (mac_cf_sockets == NULL)
+ 	    mac_cf_sockets =
+ 	      CFDictionaryCreateMutable (NULL, 0, NULL,
+ 					 &kCFTypeDictionaryValueCallBacks);
+ 
  	  for (minfd = 1; ; minfd++) /* nfds-1 works as a sentinel.  */
  	    if (FD_ISSET (minfd, rfds) || (wfds && FD_ISSET (minfd, wfds)))
  	      break;
*************** sys_select (nfds, rfds, wfds, efds, time
*** 5167,5173 ****
  		CFRunLoopSourceRef source =
  		  (CFRunLoopSourceRef) CFDictionaryGetValue (sources, key);
  
! 		if (source == NULL)
  		  {
  		    CFSocketRef socket =
  		      CFSocketCreateWithNative (NULL, fd,
--- 5198,5204 ----
  		CFRunLoopSourceRef source =
  		  (CFRunLoopSourceRef) CFDictionaryGetValue (sources, key);
  
! 		if (source == NULL || !CFRunLoopSourceIsValid (source))
  		  {
  		    CFSocketRef socket =
  		      CFSocketCreateWithNative (NULL, fd,
*************** sys_select (nfds, rfds, wfds, efds, time
*** 5177,5182 ****
--- 5208,5214 ----
  
  		    if (socket == NULL)
  		      continue;
+ 		    CFDictionaryAddValue (mac_cf_sockets, key, socket);
  		    source = CFSocketCreateRunLoopSource (NULL, socket, 0);
  		    CFRelease (socket);
  		    if (source == NULL)
Index: src/sysdep.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/sysdep.c,v
retrieving revision 1.279.2.1
diff -c -p -r1.279.2.1 sysdep.c
*** src/sysdep.c	25 Jul 2007 05:15:36 -0000	1.279.2.1
--- src/sysdep.c	29 Nov 2007 02:11:40 -0000
*************** emacs_close (fd)
*** 3320,3325 ****
--- 3320,3334 ----
    int did_retry = 0;
    register int rtnval;
  
+ #if defined (MAC_OSX) && defined (HAVE_CARBON)
+   {
+     extern int mac_try_close_socket P_ ((int));
+ 
+     if (mac_try_close_socket (fd))
+       return 0;
+   }
+ #endif
+ 
    while ((rtnval = close (fd)) == -1
  	 && (errno == EINTR))
      did_retry = 1;

Re: High CPU load

[David Reitter]

On 29 Nov 2007, at 02:21, YAMAMOTO Mitsuharu wrote:

> Invalidation of CFSocket in every sys_select call turned out to be
> problematic in another situation, anyway.  Here is another patch that
> the invalidation is done via emacs_close call.

People who have experienced the problem frequently have been working  
with your patch over the last days. I haven't received reports of any  
problems.

Thanks for fixing this so quickly.