bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

[Steve Revilak]

[-- Attachment #1.1: Type: text/plain, Size: 8470 bytes --]

This bug report will be sent to the Bug-GNU-Emacs mailing list
and the GNU bug tracker at debbugs.gnu.org.  Please check that
the From: line contains a valid email address.  After a delay of up
to one day, you should receive an acknowledgement at that address.

Please write in English if possible, as the Emacs maintainers
usually do not have translators for other languages.

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from `emacs -Q':



I'd like to report an org-mode regression issue.  When working with
large .org files, Emacs 24.0.90 becomes sluggish, and consumes large
amounts of CPU.

I've attached an .org file (sample.org.gz) that causes emacs to exhibit
this behavior.  I apologize for the large attachment, but I have not
been able to reproduce this issue with small .org files.

Steps to reproduce:

(1) gunzip sample.org.gz

(2) run "top" in a terminal window.  We'll use top to monitor emacs CPU
utilization.

(3) Start emacs as "emacs -Q sample.org".  Emacs should open the file
with org-mode as the major mode.

(4) Press M-> to position point at the end of the buffer.

(5) Press RET and start typing.  You don't need to type anything
meaningful: nonsense like "fdfsds fdsfds fdsfdsf fdsfdfsds" will do.
Continue typing for 10-20 seconds.

(6) While typing notice the delay between pressing keys on the keyboard,
and the characters being displayed in the buffer.  To me, this feels
similar to typing characters into an ssh session, over a slow
communications link.

(7) While typing, observe emacs CPU utilization in top.  While
performing step (5), I observed emacs using 100% CPU.  (My machine has a
quad-core 2.66 GHz Intel Xeon processor; emacs was fully utilizing one
of the CPU cores).

(8) Repeat steps (3)--(7) with emacs 23.3.  With emacs 23.3, the delays
in (6) are not present.  As for step (7), I could not get emacs 23.3 to
use more than 2--4% CPU (as reported via top).

Uname for my system is
Linux sunny 2.6.37.6-0.7-desktop #1 SMP PREEMPT 2011-07-21 02:17:24 +0200 x86_64 x86_64 x86_64 GNU/Linux

Another thing you can try: put point in line 1, column zero.  Press and
hold the keyboard down arrow until point reaches the end of buffer;
then, press and hold the keyboard up arrow until point reaches the first
line of the buffer.  Repeat this a few times.  With emacs 24, the point
movement is `jerky', and emacs uses ~ 47% CPU (as reported by top).
With emacs 23.3, the point movement is `smooth', and emacs 23.3 uses
2--4% CPU (as reported by top).


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
     `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/local/emacs-24.0.90.1/share/emacs/24.0.90/etc/DEBUG.


In GNU Emacs 24.0.90.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.22.1)
  of 2011-09-26 on sunny
Windowing system distributor `The X.Org Foundation', version 11.0.10903000
configured using `configure  '--prefix=/usr/local/emacs-24.0.90.1/''

Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: C
   value of $LC_CTYPE: nil
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: nil
   value of $LC_NUMERIC: nil
   value of $LC_TIME: nil
   value of $LANG: en_US.UTF-8
   value of $XMODIFIERS: @im=local
   locale-coding-system: utf-8-unix
   default enable-multibyte-characters: t

Major mode: Perl

Minor modes in effect:
   diff-auto-refine-mode: t
   shell-dirtrack-mode: t
   display-time-mode: t
   tooltip-mode: t
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   column-number-mode: t
   line-number-mode: t

Recent input:
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <down> 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v C-v 
C-v C-v C-v C-v C-v C-v C-v <down-mouse-1> <mouse-1> 
C-x k <return> <down-mouse-1> <mouse-1> M-x r e p o 
r t - e m a <tab> <return>

Recent messages:
Saving file /home/srevilak/sample.org...
Wrote /home/srevilak/sample.org
Mark set
Saving file /home/srevilak/sample.org...
Wrote /home/srevilak/sample.org
FOLDED
CHILDREN
SUBTREE
byte-code: Beginning of buffer [3 times]
scroll-up-command: End of buffer [5 times]
scroll-up-command: End of buffer

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr message sendmail rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mailabbrev mail-utils gmm-utils mailheader
emacsbug perl-mode tabify pcmpl-unix vc-bzr vc-sccs vc-svn vc-dir
org-clock dired-aux log-view vc-rcs two-column term disp-table ehelp
electric browse-url mule-util cal-move org-table dired pcvs pcvs-parse
pcvs-info pcvs-defs ewoc smerge-mode executable pcmpl-cvs ansi-color
ispell diff-mode help-mode view thingatpt reftex-parse newcomment
multi-isearch parse-time vc-cvs log-edit pcvs-util add-log vc ediff-merg
ediff-diff ediff-wind ediff-help ediff-util ediff-mult ediff-init ediff
vc-dispatcher skeleton reftex-vcr reftex-dcr reftex reftex-vars tex-mode
compile shell latexenc diary-lib diary-loaddefs cal-iso vc-git org-wl
org-w3m org-vm org-rmail org-mhe org-mew org-irc org-jsinfo org-infojs
org-html format-spec org-exp ob-exp org-exp-blocks org-info org-gnus
org-docview org-bibtex bibtex org-bbdb org-agenda org byte-opt warnings
bytecomp byte-compile cconv macroexp advice help-fns advice-preload
ob-emacs-lisp ob-tangle ob-ref ob-lob ob-table org-footnote org-src
ob-comint comint ring ob-keys ob ob-eval org-pcomplete pcomplete
org-list org-faces org-compat org-entities org-macs noutline outline
easy-mmode regexp-opt cal-menu easymenu calendar cal-loaddefs edmacro
kmacro paren time time-date tooltip ediff-hook vc-hooks lisp-float-type
mwheel x-win x-dnd tool-bar dnd fontset image fringe lisp-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer loaddefs
button faces cus-face files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)

[-- Attachment #1.2: sample.org.gz --]
[-- Type: application/x-gzip, Size: 40022 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

[Eli Zaretskii]

> Date: Mon, 26 Sep 2011 22:50:25 -0400
> From: Steve Revilak <steve@srevilak.net>
> 
> I'd like to report an org-mode regression issue.  When working with
> large .org files, Emacs 24.0.90 becomes sluggish, and consumes large
> amounts of CPU.

If you type this:

  M-x set-variable RET bidi-paragraph-direction RET left-to-right RET

does the problem go away?

Org mode should do that by default, but the Org mode maintainer
(CC'ed) didn't yet respond to my repeated mails about this issue.

Bastien, please tell me whether you want me to make this change in the
Emacs repository, or wait for you to do it in the Org mode repo and
then merge to Emacs.

bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

[Bastien]

Eli Zaretskii <eliz@gnu.org> writes:

> Bastien, please tell me whether you want me to make this change in the
> Emacs repository, or wait for you to do it in the Org mode repo and
> then merge to Emacs.

Please make this change in the Emacs repository.

Thanks,

-- 
 Bastien

bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

[Steve Revilak]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

>> I'd like to report an org-mode regression issue.  When working with
>> large .org files, Emacs 24.0.90 becomes sluggish, and consumes large
>> amounts of CPU.

>If you type this:
>
>  M-x set-variable RET bidi-paragraph-direction RET left-to-right RET
>
>does the problem go away?

My bug report contained two scenarios to produce sluggish response and
high CPU utilization.  These scenarios were

  (1) move point to the end of buffer, and start typing

  (2) Put point in line 1, column zero; press and hold the down (then up)
  arrows to move point down (then up) the org-mode buffer.

Setting bidi-paragraph-direction: left-to-right eliminates the problem
with scenario (1).  However, it has no effect on scenario (2).

Steve

[-- Attachment #2: Type: application/pgp-signature, Size: 313 bytes --]

bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

[Lawrence Mitchell]

Steve Revilak wrote:

>>> I'd like to report an org-mode regression issue.  When working with
>>> large .org files, Emacs 24.0.90 becomes sluggish, and consumes large
>>> amounts of CPU.

>> If you type this:

>>  M-x set-variable RET bidi-paragraph-direction RET left-to-right RET

>> does the problem go away?

> My bug report contained two scenarios to produce sluggish response and
> high CPU utilization.  These scenarios were

>  (1) move point to the end of buffer, and start typing

>  (2) Put point in line 1, column zero; press and hold the down (then up)
>  arrows to move point down (then up) the org-mode buffer.

> Setting bidi-paragraph-direction: left-to-right eliminates the problem
> with scenario (1).  However, it has no effect on scenario (2).

Setting bidi-display-reordering to nil in the buffer has the
effect that moving point is no longer jerky for me.

Lawrence

bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

[Eli Zaretskii]

> From: Bastien <bzg@altern.org>
> Cc: Steve Revilak <steve@srevilak.net>,  9610@debbugs.gnu.org
> Date: Tue, 27 Sep 2011 08:02:47 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Bastien, please tell me whether you want me to make this change in the
> > Emacs repository, or wait for you to do it in the Org mode repo and
> > then merge to Emacs.
> 
> Please make this change in the Emacs repository.

Done in revision 105940.

bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

[Eli Zaretskii]

> Date: Tue, 27 Sep 2011 09:47:45 -0400
> From: Steve Revilak <steve@srevilak.net>
> Cc: 9610@debbugs.gnu.org, Bastien Guerry <bzg@gnu.org>, steve@srevilak.net
> 
> My bug report contained two scenarios to produce sluggish response and
> high CPU utilization.  These scenarios were
> 
>   (1) move point to the end of buffer, and start typing
> 
>   (2) Put point in line 1, column zero; press and hold the down (then up)
>   arrows to move point down (then up) the org-mode buffer.

Sorry, I missed the second one.

> Setting bidi-paragraph-direction: left-to-right eliminates the problem
> with scenario (1).  However, it has no effect on scenario (2).

Yes, there was another reason for the slowdown.  I found an
optimization of the new display that solves the problem for me with
your sample Org file.  The fix is committed as revision 105941 on the
trunk.  If you cannot or don't want to build the trunk, the patch is
reproduced below; please try it.  On my machine, the cursor motion is
now as fast as in Emacs 23 with the same file.

I'm closing the bug.  Feel free to reopen if there are any leftovers.

Thanks.

=== modified file 'src/ChangeLog'
--- src/ChangeLog	2011-09-27 08:37:07 +0000
+++ src/ChangeLog	2011-09-27 17:18:31 +0000
@@ -1,3 +1,11 @@
+2011-09-27  Eli Zaretskii  <eliz@gnu.org>
+
+	* xdisp.c (handle_invisible_prop): If invisible text ends on a
+	newline, reseat the iterator instead of bidi-iterating there one
+	character at a time.  (Bug#9610)
+	(BUFFER_POS_REACHED_P, move_it_in_display_line_to): Bail when past
+	TO_CHARPOS if the bidi iterator is at base embedding level.
+
 2011-09-27  Andreas Schwab  <schwab@linux-m68k.org>
 
 	* lread.c (readevalloop): Use correct code for NBSP.

=== modified file 'src/xdisp.c'
--- src/xdisp.c	2011-09-24 16:28:25 +0000
+++ src/xdisp.c	2011-09-27 17:18:31 +0000
@@ -4056,40 +4056,67 @@ handle_invisible_prop (struct it *it)
 	  /* The position newpos is now either ZV or on visible text.  */
 	  if (it->bidi_p && newpos < ZV)
 	    {
-	      /* With bidi iteration, the region of invisible text
-		 could start and/or end in the middle of a non-base
-		 embedding level.  Therefore, we need to skip
-		 invisible text using the bidi iterator, starting at
-		 IT's current position, until we find ourselves
-		 outside the invisible text.  Skipping invisible text
-		 _after_ bidi iteration avoids affecting the visual
-		 order of the displayed text when invisible properties
-		 are added or removed.  */
-	      if (it->bidi_it.first_elt && it->bidi_it.charpos < ZV)
-		{
-		  /* If we were `reseat'ed to a new paragraph,
-		     determine the paragraph base direction.  We need
-		     to do it now because next_element_from_buffer may
-		     not have a chance to do it, if we are going to
-		     skip any text at the beginning, which resets the
-		     FIRST_ELT flag.  */
-		  bidi_paragraph_init (it->paragraph_embedding,
-				       &it->bidi_it, 1);
-		}
-	      do
+	      EMACS_INT bpos = CHAR_TO_BYTE (newpos);
+
+	      if (FETCH_BYTE (bpos) == '\n'
+		  || (newpos > BEGV && FETCH_BYTE (bpos - 1) == '\n'))
 		{
-		  bidi_move_to_visually_next (&it->bidi_it);
+		  /* If the invisible text ends on a newline or the
+		     character after a newline, we can avoid the
+		     costly, character by character, bidi iteration to
+		     newpos, and instead simply reseat the iterator
+		     there.  That's because all bidi reordering
+		     information is tossed at the newline.  This is a
+		     big win for modes that hide complete lines, like
+		     Outline, Org, etc.  (Implementation note: the
+		     call to reseat_1 is necessary, because it signals
+		     to the bidi iterator that it needs to reinit its
+		     internal information when the next element for
+		     display is requested.  */
+		  struct text_pos tpos;
+
+		  SET_TEXT_POS (tpos, newpos, bpos);
+		  reseat_1 (it, tpos, 0);
+		}
+	      else	/* Must use the slow method.  */
+		{
+		  /* With bidi iteration, the region of invisible text
+		     could start and/or end in the middle of a
+		     non-base embedding level.  Therefore, we need to
+		     skip invisible text using the bidi iterator,
+		     starting at IT's current position, until we find
+		     ourselves outside the invisible text.  Skipping
+		     invisible text _after_ bidi iteration avoids
+		     affecting the visual order of the displayed text
+		     when invisible properties are added or
+		     removed.  */
+		  if (it->bidi_it.first_elt && it->bidi_it.charpos < ZV)
+		    {
+		      /* If we were `reseat'ed to a new paragraph,
+			 determine the paragraph base direction.  We
+			 need to do it now because
+			 next_element_from_buffer may not have a
+			 chance to do it, if we are going to skip any
+			 text at the beginning, which resets the
+			 FIRST_ELT flag.  */
+		      bidi_paragraph_init (it->paragraph_embedding,
+					   &it->bidi_it, 1);
+		    }
+		  do
+		    {
+		      bidi_move_to_visually_next (&it->bidi_it);
+		    }
+		  while (it->stop_charpos <= it->bidi_it.charpos
+			 && it->bidi_it.charpos < newpos);
+		  IT_CHARPOS (*it) = it->bidi_it.charpos;
+		  IT_BYTEPOS (*it) = it->bidi_it.bytepos;
+		  /* If we overstepped NEWPOS, record its position in
+		     the iterator, so that we skip invisible text if
+		     later the bidi iteration lands us in the
+		     invisible region again. */
+		  if (IT_CHARPOS (*it) >= newpos)
+		    it->prev_stop = newpos;
 		}
-	      while (it->stop_charpos <= it->bidi_it.charpos
-		     && it->bidi_it.charpos < newpos);
-	      IT_CHARPOS (*it) = it->bidi_it.charpos;
-	      IT_BYTEPOS (*it) = it->bidi_it.bytepos;
-	      /* If we overstepped NEWPOS, record its position in the
-		 iterator, so that we skip invisible text if later the
-		 bidi iteration lands us in the invisible region
-		 again. */
-	      if (IT_CHARPOS (*it) >= newpos)
-		it->prev_stop = newpos;
 	    }
 	  else
 	    {
@@ -7880,7 +7907,9 @@ move_it_in_display_line_to (struct it *i
   ((op & MOVE_TO_POS) != 0					\
    && BUFFERP (it->object)					\
    && (IT_CHARPOS (*it) == to_charpos				\
-       || (!it->bidi_p && IT_CHARPOS (*it) > to_charpos)	\
+       || ((!it->bidi_p						\
+	    || BIDI_AT_BASE_LEVEL (it->bidi_it))		\
+	   && IT_CHARPOS (*it) > to_charpos)			\
        || (it->what == IT_COMPOSITION				\
 	   && ((IT_CHARPOS (*it) > to_charpos			\
 		&& to_charpos >= it->cmp_it.charpos)		\
@@ -7912,7 +7941,13 @@ move_it_in_display_line_to (struct it *i
       if ((op & MOVE_TO_POS) != 0
 	  && BUFFERP (it->object)
 	  && it->method == GET_FROM_BUFFER
-	  && ((!it->bidi_p && IT_CHARPOS (*it) > to_charpos)
+	  && (((!it->bidi_p
+		/* When the iterator is at base embedding level, we
+		   are guaranteed that characters are delivered for
+		   display in strictly increasing order of their
+		   buffer positions.  */
+		|| BIDI_AT_BASE_LEVEL (it->bidi_it))
+	       && IT_CHARPOS (*it) > to_charpos)
 	      || (it->bidi_p
 		  && (prev_method == GET_FROM_IMAGE
 		      || prev_method == GET_FROM_STRETCH

bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

[Eli Zaretskii]

> From: Lawrence Mitchell <wence@gmx.li>
> Date: Tue, 27 Sep 2011 16:10:25 +0100
> 
> Setting bidi-display-reordering to nil in the buffer has the
> effect that moving point is no longer jerky for me.

I think with the latest trunk, you won't need to fiddle with this
variable anymore.

bug#9610: 24.0.90; org-mode: sluggish response and high CPU utilization with large .org files

[Bastien]

Eli Zaretskii <eliz@gnu.org> writes:

>> Please make this change in the Emacs repository.
>
> Done in revision 105940.

Thanks!  I added your change to Org's git repo:

http://orgmode.org/w/?p=org-mode.git;a=commit;h=1a97f29c342d85960a65c0bd992fee7c87850da5

Best,

-- 
 Bastien