Memory leak due to bidi?

Memory leak due to bidi?

[Eli Zaretskii]

This is from bug #9221

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 01 Aug 2011 22:23:18 -0400
> Subject: bug#9221: Memory leak
> 
> I'm seeing some weird behavior linked to insane memory consumption.
> E.g. my Gnus session tends to grow to more than 2GB and then become
> unusage and unkillable (or close enough: in D state, "kill -9" isn't
> enough to make it stop use CPU, tho I guess that CPU use is really due
> to something like the OS being in the process of dumping the core file,
> tho I don't see any left over core files).

Does anyone else using the trunk version see similar growth of the
memory footprint?  If so, please add your observations to the bug
report.

Re: Memory leak due to bidi?

[Stefan Monnier]

>> I'm seeing some weird behavior linked to insane memory consumption.
>> E.g. my Gnus session tends to grow to more than 2GB and then become
>> unusage and unkillable (or close enough: in D state, "kill -9" isn't
>> enough to make it stop use CPU, tho I guess that CPU use is really due
>> to something like the OS being in the process of dumping the core file,
>> tho I don't see any left over core files).

> Does anyone else using the trunk version see similar growth of the
> memory footprint?  If so, please add your observations to the bug
> report.

BTW, this problem is new: I've been running with bidi-display-reordering
set to t for more than a year now and the problem only appeared in the
last couple of weeks.


        Stefan

Re: Memory leak due to bidi?

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> BTW, this problem is new: I've been running with bidi-display-reordering
> set to t for more than a year now and the problem only appeared in the
> last couple of weeks.

Has anybody written an, er, memory usage statistics thing?

That is, something that could summarise memory usage by bytes used in
buffers, in strings, in conses and "rest", which would be pure C-level
allocations.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Memory leak due to bidi?

[Andreas Schwab]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> That is, something that could summarise memory usage by bytes used in
> buffers, in strings, in conses and "rest", which would be pure C-level
> allocations.

Like what (garbage-collect) returns?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Memory leak due to bidi?

[Lars Magne Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Like what (garbage-collect) returns?

I used to know that.  :-)

It might be even more helpful if it displayed the data in a nice buffer,
and with everything using a "byte" unit, if possible.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Memory leak due to bidi?

[joakim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> BTW, this problem is new: I've been running with bidi-display-reordering
>> set to t for more than a year now and the problem only appeared in the
>> last couple of weeks.
>
> Has anybody written an, er, memory usage statistics thing?
>
> That is, something that could summarise memory usage by bytes used in
> buffers, in strings, in conses and "rest", which would be pure C-level
> allocations.

(garbage-collect)

memory-usage.el makes it human readable.

I used to have lots of memory issues. At the time I made the following
notes:

- create a memory meter in the modeline, then it would at least be
  possible to see when memory is consumed, and maybe get a clue.

- modify the garbage collector, to make some form of dump to a file,
  when the mark stage occurs. (this is like the "jhat" java tool)

- modify the garb so as to flag an object for pretend delete. then a
  pretend garb would be done, and a measure of how much memory would be
  reclaimed if that object was to be deleted would be returned.


  
-- 
Joakim Verona

Re: Memory leak due to bidi?

[Lars Magne Ingebrigtsen]

joakim@verona.se writes:

> memory-usage.el makes it human readable.

Is that something that could be put into GNU ELPA?

> I used to have lots of memory issues. At the time I made the following
> notes:
>
> - create a memory meter in the modeline, then it would at least be
>   possible to see when memory is consumed, and maybe get a clue.

That sounds useful.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Memory leak due to bidi?

[joakim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> joakim@verona.se writes:
>
>> memory-usage.el makes it human readable.
>
> Is that something that could be put into GNU ELPA?
>
>> I used to have lots of memory issues. At the time I made the following
>> notes:
>>
>> - create a memory meter in the modeline, then it would at least be
>>   possible to see when memory is consumed, and maybe get a clue.
>
> That sounds useful.


gnu elpa would be a good place:

;;; memory-usage.el --- Analyze the memory usage of Emacs in various ways

;; Copyright (C) 2002, 2004  Free Software Foundation, Inc.

;; Author: Stefan Monnier <address@bogus.example.com>
;; Keywords: maint

;;  minor mods by joakim verona

;; This file is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation; either version 2, or (at your option)
;; any later version.

;; This file is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to
;; the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
;; Boston, MA 02111-1307, USA.

;;; Commentary:

;; 

;;; Code:

(defun buffer-size-bytes (b)
  "Return total number of bytes in the buffer contents."
  (with-current-buffer b
    (save-restriction
      (widen)
      (- (position-bytes (point-max)) (position-bytes (point-min))))))

(defun buffer-gap-bytes (b)
  "Return total number of bytes in the buffer gap."
  (with-current-buffer b
    (gap-size)))

(defun buffer-total-bytes (b)
  "Return total number of ralloc bytes used by buffer."
  (with-current-buffer b
    (save-restriction
      (widen) 
      (+ (position-bytes (point-max))
         (- (position-bytes (point-min)))
         (gap-size)))))

;;;###autoload
(defun memory-usage ()
  "List all buffers and their memory usage."
  (interactive)
  (pop-to-buffer (get-buffer-create "*Buffer Details*"))
  (erase-buffer)
  (let* ((bufs (buffer-list))
         (num (length bufs))
         (gc-stats (garbage-collect))
         (conses (nth 0 gc-stats))
         (symbols (nth 1 gc-stats))
         (markers (nth 2 gc-stats))
         (strings (nth 3 gc-stats))
         (vectors (nth 4 gc-stats))
         (floats (nth 5 gc-stats))
         (intervals (nth 6 gc-stats))
         (lisptotal (+ (* 8 (+ (car conses) (cdr conses)))
                       (* 24 (+ (car symbols) (cdr symbols)))
                       (* 20 (+ (car markers) (cdr markers)))
                       strings
                       vectors
                       (* 12 (+ (car floats) (cdr floats)))
                       (* 28 (+ (car intervals) (cdr intervals)))))
         (buffertotalbytes (apply #'+ (mapcar #'buffer-total-bytes bufs)))
         )
    
    (insert (format "Garbage collection stats:\n%s\n\n =>" gc-stats))
    (insert (format "\t%d bytes in cons cells\n" (* 8 (+ (car conses) (cdr 
conses)))))
    (insert (format "\t%d bytes in symbols\n" (* 24 (+ (car symbols) (cdr 
symbols)))))
    (insert (format "\t%d bytes in markers\n" (* 20 (+ (car markers) (cdr 
markers)))))
    (insert (format "\t%d bytes of string chars\n" strings))
    (insert (format "\t%d bytes of vector slots\n" (* 4 vectors)))
    (insert (format "\t%d bytes in floats\n" (* 12 (+ (car floats) (cdr 
floats)))))
    (insert (format "\t%d bytes in intervals\n" (* 28 (+ (car intervals) (cdr 
intervals)))))

    (insert (format "\nTotal bytes in lisp objects (not counting string and 
vector headers): %d\n\n"
                    lisptotal))

    (insert (format "Buffer ralloc memory usage:\n%d buffers\n%d bytes total (%d in gaps)\n"
                    num
                    buffertotalbytes
                    (apply #'+ (mapcar #'buffer-gap-bytes bufs))))
    (insert (format "%10s\t%s\t%s\n\n" "Size" "Gap" "Name"))
    (insert (mapconcat
            (lambda (b)
               (format "%10d\t%s\t%s"
                       (buffer-size-bytes b)
                       (buffer-gap-bytes b)
                       (buffer-name b)))
             (sort bufs (lambda (b1 b2)
                          (> (buffer-size-bytes b1) (buffer-size-bytes b2))))
             "\n"))
    (insert "\n")
    (insert (format "total:%s" (+ lisptotal buffertotalbytes)))

    (insert "\n"))
  (goto-char (point-min)))

(provide 'memory-usage)
;; arch-tag: 04e012f0-3c59-4319-8d1a-e86204671ec5
;;; memory-usage.el ends here

-- 
Joakim Verona

Re: Memory leak due to bidi?

[Andreas Röhler]

Am 02.08.2011 21:12, schrieb Stefan Monnier:
>>> I'm seeing some weird behavior linked to insane memory consumption.
>>> E.g. my Gnus session tends to grow to more than 2GB and then become
>>> unusage and unkillable (or close enough: in D state, "kill -9" isn't
>>> enough to make it stop use CPU, tho I guess that CPU use is really due
>>> to something like the OS being in the process of dumping the core file,
>>> tho I don't see any left over core files).
>
>> Does anyone else using the trunk version see similar growth of the
>> memory footprint?  If so, please add your observations to the bug
>> report.
>
> BTW, this problem is new: I've been running with bidi-display-reordering
> set to t for more than a year now and the problem only appeared in the
> last couple of weeks.
>
>
>          Stefan
>
>

it might help knowing which is fine:

GNU Emacs 24.0.50.1 (i686-suse-linux-gnu, X toolkit, Xaw3d scroll bars) 
of 2011-05-02

Andreas

Re: Memory leak due to bidi?

[James Cloos]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> BTW, this problem is new: I've been running with bidi-display-
SM> reordering set to t for more than a year now and the problem only
SM> appeared in the last couple of weeks.

I can confirm that.

My build from 2011/06/07 did not have this bug.

My build from 2011/07/23 does.

The first was from the git mirror, commit 32278679adfe, which is
dated June 1st.

The latter is from an rsync of the bzr, revno 105313.

A few bidi commits were made on 07/23 before I synced and built.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Memory leak due to bidi?

[Stefan Monnier]

SM> BTW, this problem is new: I've been running with bidi-display-
SM> reordering set to t for more than a year now and the problem only
SM> appeared in the last couple of weeks.

> I can confirm that.

> My build from 2011/06/07 did not have this bug.

> My build from 2011/07/23 does.

> The first was from the git mirror, commit 32278679adfe, which is
> dated June 1st.

> The latter is from an rsync of the bzr, revno 105313.

> A few bidi commits were made on 07/23 before I synced and built.

Let's not forget that it may have nothing to do with bidi.  I just
happened to see bidi_shelve_cache as a "frequent" caller of mmap, but
according to Eli this is not necessarily abnormal.


        Stefan

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org,  Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 02 Aug 2011 21:10:14 -0400
> 
> Let's not forget that it may have nothing to do with bidi.  I just
> happened to see bidi_shelve_cache as a "frequent" caller of mmap, but
> according to Eli this is not necessarily abnormal.

I would be the first to be happy if it turns out cache shelving and
unshelving is not the culprit.  However, as I still have my doubts, I
instrumented the relevant code as shown in the patch below.  With this
patch, you can:

 . Put a breakpoint on some rarely-used interactive function (I use
   Fredraw_display), invoke it whenever you feel like it, and print
   the value of the variable bidi_cache_total_alloc -- it should be
   strictly zero, i.e. each allocation should be followed by a
   corresponding deallocation before Emacs gets back to its command
   loop.  If you ever see a non-zero value, let alone a value that
   goes up, please proceed with the next item below.

 . Put a hardware watchpoint on bidi_cache_total_alloc, with the
   following commands:

     print bidi_cache_total_alloc
     bt 4
     continue

   Then tell GDB to log all its output:

     set logging on

   and run Emacs until the value of bidi_cache_total_alloc again
   increases.  Then send me the logfile (gdb.txt in the directory
   where you run GDB), or look into it to find some code that
   allocates a buffer, but never frees it.

I am running with this patch for almost 3 hours, and the value of
bidi_cache_total_alloc is strictly zero every time I look at it in the
debugger.

TIA

Here's the patch:

=== modified file 'src/bidi.c'
--- src/bidi.c	2011-08-02 19:16:32 +0000
+++ src/bidi.c	2011-08-03 10:24:32 +0000
@@ -620,12 +620,15 @@ bidi_pop_it (struct bidi_it *bidi_it)
   bidi_cache_last_idx = -1;
 }
 
+ptrdiff_t bidi_cache_total_alloc;
+
 /* Stash away a copy of the cache and its control variables.  */
 void *
 bidi_shelve_cache (void)
 {
   unsigned char *databuf;
 
+  /* Empty cache.  */
   if (bidi_cache_idx == 0)
     return NULL;
 
@@ -634,6 +637,12 @@ bidi_shelve_cache (void)
 		     + sizeof (bidi_cache_start_stack)
 		     + sizeof (bidi_cache_sp) + sizeof (bidi_cache_start)
 		     + sizeof (bidi_cache_last_idx));
+  bidi_cache_total_alloc +=
+    sizeof (bidi_cache_idx) + bidi_cache_idx * sizeof (struct bidi_it)
+    + sizeof (bidi_cache_start_stack)
+    + sizeof (bidi_cache_sp) + sizeof (bidi_cache_start)
+    + sizeof (bidi_cache_last_idx);
+
   memcpy (databuf, &bidi_cache_idx, sizeof (bidi_cache_idx));
   memcpy (databuf + sizeof (bidi_cache_idx),
 	  bidi_cache, bidi_cache_idx * sizeof (struct bidi_it));
@@ -659,7 +668,7 @@ bidi_shelve_cache (void)
 
 /* Restore the cache state from a copy stashed away by bidi_shelve_cache.  */
 void
-bidi_unshelve_cache (void *databuf)
+bidi_unshelve_cache (void *databuf, int just_free)
 {
   unsigned char *p = databuf;
 
@@ -672,30 +681,47 @@ bidi_unshelve_cache (void *databuf)
     }
   else
     {
-      memcpy (&bidi_cache_idx, p, sizeof (bidi_cache_idx));
-      bidi_cache_ensure_space (bidi_cache_idx);
-      memcpy (bidi_cache, p + sizeof (bidi_cache_idx),
-	      bidi_cache_idx * sizeof (struct bidi_it));
-      memcpy (bidi_cache_start_stack,
-	      p + sizeof (bidi_cache_idx)
-	      + bidi_cache_idx * sizeof (struct bidi_it),
-	      sizeof (bidi_cache_start_stack));
-      memcpy (&bidi_cache_sp,
-	      p + sizeof (bidi_cache_idx)
-	      + bidi_cache_idx * sizeof (struct bidi_it)
-	      + sizeof (bidi_cache_start_stack),
-	      sizeof (bidi_cache_sp));
-      memcpy (&bidi_cache_start,
-	      p + sizeof (bidi_cache_idx)
-	      + bidi_cache_idx * sizeof (struct bidi_it)
-	      + sizeof (bidi_cache_start_stack) + sizeof (bidi_cache_sp),
-	      sizeof (bidi_cache_start));
-      memcpy (&bidi_cache_last_idx,
-	      p + sizeof (bidi_cache_idx)
-	      + bidi_cache_idx * sizeof (struct bidi_it)
-	      + sizeof (bidi_cache_start_stack) + sizeof (bidi_cache_sp)
-	      + sizeof (bidi_cache_start),
-	      sizeof (bidi_cache_last_idx));
+      if (just_free)
+	{
+	  ptrdiff_t idx;
+
+	  memcpy (&idx, p, sizeof (bidi_cache_idx));
+	  bidi_cache_total_alloc -=
+	    sizeof (bidi_cache_idx) + idx * sizeof (struct bidi_it)
+	    + sizeof (bidi_cache_start_stack) + sizeof (bidi_cache_sp)
+	    + sizeof (bidi_cache_start) + sizeof (bidi_cache_last_idx);
+	}
+      else
+	{
+	  memcpy (&bidi_cache_idx, p, sizeof (bidi_cache_idx));
+	  bidi_cache_ensure_space (bidi_cache_idx);
+	  memcpy (bidi_cache, p + sizeof (bidi_cache_idx),
+		  bidi_cache_idx * sizeof (struct bidi_it));
+	  memcpy (bidi_cache_start_stack,
+		  p + sizeof (bidi_cache_idx)
+		  + bidi_cache_idx * sizeof (struct bidi_it),
+		  sizeof (bidi_cache_start_stack));
+	  memcpy (&bidi_cache_sp,
+		  p + sizeof (bidi_cache_idx)
+		  + bidi_cache_idx * sizeof (struct bidi_it)
+		  + sizeof (bidi_cache_start_stack),
+		  sizeof (bidi_cache_sp));
+	  memcpy (&bidi_cache_start,
+		  p + sizeof (bidi_cache_idx)
+		  + bidi_cache_idx * sizeof (struct bidi_it)
+		  + sizeof (bidi_cache_start_stack) + sizeof (bidi_cache_sp),
+		  sizeof (bidi_cache_start));
+	  memcpy (&bidi_cache_last_idx,
+		  p + sizeof (bidi_cache_idx)
+		  + bidi_cache_idx * sizeof (struct bidi_it)
+		  + sizeof (bidi_cache_start_stack) + sizeof (bidi_cache_sp)
+		  + sizeof (bidi_cache_start),
+		  sizeof (bidi_cache_last_idx));
+	  bidi_cache_total_alloc -=
+	    sizeof (bidi_cache_idx) + bidi_cache_idx * sizeof (struct bidi_it)
+	    + sizeof (bidi_cache_start_stack) + sizeof (bidi_cache_sp)
+	    + sizeof (bidi_cache_start) + sizeof (bidi_cache_last_idx);
+	}
 
       xfree (p);
     }

=== modified file 'src/dispextern.h'
--- src/dispextern.h	2011-08-02 19:16:32 +0000
+++ src/dispextern.h	2011-08-03 10:37:42 +0000
@@ -2978,7 +2978,7 @@ extern int  bidi_mirror_char (int);
 extern void bidi_push_it (struct bidi_it *);
 extern void bidi_pop_it (struct bidi_it *);
 extern void *bidi_shelve_cache (void);
-extern void bidi_unshelve_cache (void *);
+extern void bidi_unshelve_cache (void *, int);
 
 /* Defined in xdisp.c */
 

=== modified file 'src/dispnew.c'
--- src/dispnew.c	2011-07-14 20:40:35 +0000
+++ src/dispnew.c	2011-08-03 10:32:00 +0000
@@ -5282,7 +5282,7 @@ buffer_posn_from_coords (struct window *
      argument is ZV to prevent move_it_in_display_line from matching
      based on buffer positions.  */
   move_it_in_display_line (&it, ZV, to_x, MOVE_TO_X);
-  bidi_unshelve_cache (itdata);
+  bidi_unshelve_cache (itdata, 0);
 
   Fset_buffer (old_current_buffer);
 

=== modified file 'src/indent.c'
--- src/indent.c	2011-07-14 21:35:23 +0000
+++ src/indent.c	2011-08-03 10:32:13 +0000
@@ -2135,7 +2135,7 @@ whether or not it is currently displayed
 	}
 
       SET_PT_BOTH (IT_CHARPOS (it), IT_BYTEPOS (it));
-      bidi_unshelve_cache (itdata);
+      bidi_unshelve_cache (itdata, 0);
     }
 
   if (BUFFERP (old_buffer))

=== modified file 'src/window.c'
--- src/window.c	2011-07-14 17:28:42 +0000
+++ src/window.c	2011-08-03 10:35:05 +0000
@@ -1379,7 +1379,7 @@ if it isn't already recorded.  */)
       if (it.current_y < it.last_visible_y)
 	move_it_past_eol (&it);
       value = make_number (IT_CHARPOS (it));
-      bidi_unshelve_cache (itdata);
+      bidi_unshelve_cache (itdata, 0);
 
       if (old_buffer)
 	set_buffer_internal (old_buffer);
@@ -4273,7 +4273,7 @@ window_scroll_pixel_based (Lisp_Object w
 	}
 
       start = it.current.pos;
-      bidi_unshelve_cache (itdata);
+      bidi_unshelve_cache (itdata, 0);
     }
   else if (auto_window_vscroll_p)
     {
@@ -4417,7 +4417,7 @@ window_scroll_pixel_based (Lisp_Object w
 	    }
 	  else
 	    {
-	      bidi_unshelve_cache (itdata);
+	      bidi_unshelve_cache (itdata, 0);
 	      if (noerror)
 		return;
 	      else if (n < 0)	/* could happen with empty buffers */
@@ -4434,7 +4434,7 @@ window_scroll_pixel_based (Lisp_Object w
 	    w->vscroll = 0;
 	  else
 	    {
-	      bidi_unshelve_cache (itdata);
+	      bidi_unshelve_cache (itdata, 0);
 	      if (noerror)
 		return;
 	      else
@@ -4583,7 +4583,7 @@ window_scroll_pixel_based (Lisp_Object w
 	    SET_PT_BOTH (charpos, bytepos);
 	}
     }
-  bidi_unshelve_cache (itdata);
+  bidi_unshelve_cache (itdata, 0);
 }
 
 
@@ -5010,7 +5010,7 @@ displayed_window_lines (struct window *w
   start_display (&it, w, start);
   move_it_vertically (&it, height);
   bottom_y = line_bottom_y (&it);
-  bidi_unshelve_cache (itdata);
+  bidi_unshelve_cache (itdata, 0);
 
   /* rms: On a non-window display,
      the value of it.vpos at the bottom of the screen
@@ -5116,7 +5116,7 @@ and redisplay normally--don't erase and 
 	  move_it_vertically_backward (&it, window_box_height (w) / 2);
 	  charpos = IT_CHARPOS (it);
 	  bytepos = IT_BYTEPOS (it);
-	  bidi_unshelve_cache (itdata);
+	  bidi_unshelve_cache (itdata, 0);
 	}
       else if (iarg < 0)
 	{
@@ -5164,7 +5164,7 @@ and redisplay normally--don't erase and 
 	    }
 	  if (h <= 0)
 	    {
-	      bidi_unshelve_cache (itdata);
+	      bidi_unshelve_cache (itdata, 0);
 	      return Qnil;
 	    }
 
@@ -5187,7 +5187,7 @@ and redisplay normally--don't erase and 
 	  charpos = IT_CHARPOS (it);
 	  bytepos = IT_BYTEPOS (it);
 
-	  bidi_unshelve_cache (itdata);
+	  bidi_unshelve_cache (itdata, 0);
 	}
       else
 	{

=== modified file 'src/xdisp.c'
--- src/xdisp.c	2011-08-03 05:24:30 +0000
+++ src/xdisp.c	2011-08-03 10:36:28 +0000
@@ -604,7 +604,7 @@ int current_mode_line_height, current_he
 #define SAVE_IT(ITCOPY,ITORIG,CACHE)		\
   do {						\
     if (CACHE)					\
-      xfree (CACHE);				\
+      bidi_unshelve_cache (CACHE, 1);		\
     ITCOPY = ITORIG;				\
     CACHE = bidi_shelve_cache();		\
   } while (0)
@@ -613,7 +613,7 @@ int current_mode_line_height, current_he
   do {						\
     if (pITORIG != pITCOPY)			\
       *(pITORIG) = *(pITCOPY);			\
-    bidi_unshelve_cache (CACHE);		\
+    bidi_unshelve_cache (CACHE, 0);		\
     CACHE = NULL;				\
   } while (0)
 
@@ -1341,9 +1341,9 @@ pos_visible_p (struct window *w, EMACS_I
 	  *vpos = it2.vpos;
 	}
       else
-	xfree (it2data);
+	bidi_unshelve_cache (it2data, 1);
     }
-  bidi_unshelve_cache (itdata);
+  bidi_unshelve_cache (itdata, 0);
 
   if (old_buffer)
     set_buffer_internal_1 (old_buffer);
@@ -2627,7 +2627,7 @@ init_iterator (struct it *it, struct win
 	    it->paragraph_embedding = R2L;
 	  else
 	    it->paragraph_embedding = NEUTRAL_DIR;
-	  bidi_unshelve_cache (NULL);
+	  bidi_unshelve_cache (NULL, 0);
 	  bidi_init_it (charpos, IT_BYTEPOS (*it), FRAME_WINDOW_P (it->f),
 			&it->bidi_it);
 	}
@@ -5618,7 +5618,7 @@ back_to_previous_visible_line_start (str
 	pos = --IT_CHARPOS (it2);
 	--IT_BYTEPOS (it2);
 	it2.sp = 0;
-	bidi_unshelve_cache (NULL);
+	bidi_unshelve_cache (NULL, 0);
 	it2.string_from_display_prop_p = 0;
 	it2.from_disp_prop_p = 0;
 	if (handle_display_prop (&it2) == HANDLED_RETURN
@@ -5828,7 +5828,7 @@ reseat_1 (struct it *it, struct text_pos
     {
       bidi_init_it (IT_CHARPOS (*it), IT_BYTEPOS (*it), FRAME_WINDOW_P (it->f),
 		    &it->bidi_it);
-      bidi_unshelve_cache (NULL);
+      bidi_unshelve_cache (NULL, 0);
       it->bidi_it.paragraph_dir = NEUTRAL_DIR;
       it->bidi_it.string.s = NULL;
       it->bidi_it.string.lstring = Qnil;
@@ -8096,13 +8096,13 @@ move_it_in_display_line_to (struct it *i
  done:
 
   if (atpos_data)
-    xfree (atpos_data);
+    bidi_unshelve_cache (atpos_data, 1);
   if (atx_data)
-    xfree (atx_data);
+    bidi_unshelve_cache (atx_data, 1);
   if (wrap_data)
-    xfree (wrap_data);
+    bidi_unshelve_cache (wrap_data, 1);
   if (ppos_data)
-    xfree (ppos_data);
+    bidi_unshelve_cache (ppos_data, 1);
 
   /* Restore the iterator settings altered at the beginning of this
      function.  */
@@ -8137,7 +8137,7 @@ move_it_in_display_line (struct it *it,
 	    (it, -1, prev_x, MOVE_TO_X);
 	}
       else
-	xfree (save_data);
+	bidi_unshelve_cache (save_data, 1);
     }
   else
     move_it_in_display_line_to (it, to_charpos, to_x, op);
@@ -8396,7 +8396,7 @@ move_it_to (struct it *it, EMACS_INT to_
     }
 
   if (backup_data)
-    xfree (backup_data);
+    bidi_unshelve_cache (backup_data, 1);
 
   TRACE_MOVE ((stderr, "move_it_to: reached %d\n", reached));
 }
@@ -8475,7 +8475,7 @@ move_it_vertically_backward (struct it *
       RESTORE_IT (it, it, it2data);
       if (nlines > 0)
 	move_it_by_lines (it, nlines);
-      xfree (it3data);
+      bidi_unshelve_cache (it3data, 1);
     }
   else
     {
@@ -8671,7 +8671,7 @@ move_it_by_lines (struct it *it, int dvp
 	  if (IT_CHARPOS (*it) >= start_charpos)
 	    RESTORE_IT (it, &it2, it2data);
 	  else
-	    xfree (it2data);
+	    bidi_unshelve_cache (it2data, 1);
 	}
       else
 	RESTORE_IT (it, it, it2data);

Re: Memory leak due to bidi?

[Antoine Levitt]

03/08/11 15:30, Eli Zaretskii
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org,  Eli Zaretskii <eliz@gnu.org>
>> Date: Tue, 02 Aug 2011 21:10:14 -0400
>> 
>> Let's not forget that it may have nothing to do with bidi.  I just
>> happened to see bidi_shelve_cache as a "frequent" caller of mmap, but
>> according to Eli this is not necessarily abnormal.
>
> I would be the first to be happy if it turns out cache shelving and
> unshelving is not the culprit.  However, as I still have my doubts, I
> instrumented the relevant code as shown in the patch below.  With this
> patch, you can:

Got it. It's related to another bug I reported: emacs -Q, (setq-default
word-wrap t), break, and:

(gdb) print bidi_cache_total_alloc 
$1 = 32768

Hope that helps.

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Antoine Levitt <antoine.levitt@gmail.com>
> Date: Thu, 04 Aug 2011 21:22:35 +0200
> 
> Got it. It's related to another bug I reported: emacs -Q, (setq-default
> word-wrap t), break, and:
> 
> (gdb) print bidi_cache_total_alloc 
> $1 = 32768
> 
> Hope that helps.

Yes, it helps a lot.  Believe it or not, but I was already looking at
some suspicious logic regarding this, during debugging of that other
bug.  It also explains why I don't see this in my sessions: I never
set word-wrap non-nil.

Thanks!

Re: Memory leak due to bidi?

[Kenichi Handa]

In article <m31ux3czcv.fsf@jhcloos.com>, James Cloos <cloos@jhcloos.com> writes:

>>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:
SM> BTW, this problem is new: I've been running with bidi-display-
SM> reordering set to t for more than a year now and the problem only
SM> appeared in the last couple of weeks.

> I can confirm that.

> My build from 2011/06/07 did not have this bug.

> My build from 2011/07/23 does.

Could you check the versions before and after my changes for
char-table handling?

2011-07-07  Kenichi Handa  <handa@m17n.org>

	* character.h (unicode_category_t): New enum type.

	* chartab.c (uniprop_decoder_t, uniprop_encoder_t): New types.
	(Qchar_code_property_table): New variable.
[...]

---
Kenichi Handa
handa@m17n.org

Re: Memory leak due to bidi?

[Óscar Fuentes]

Kenichi Handa <handa@m17n.org> writes:

> Could you check the versions before and after my changes for
> char-table handling?
>
> 2011-07-07  Kenichi Handa  <handa@m17n.org>
>
> 	* character.h (unicode_category_t): New enum type.
>
> 	* chartab.c (uniprop_decoder_t, uniprop_encoder_t): New types.
> 	(Qchar_code_property_table): New variable.
> [...]

The tip revision of my Emacs is precisely the revision you mention. So
far no abnormal memory consumption. bidi-display-reordering is
nil. GNU/Linux x86_64 (Kubuntu 10.10). Daily usage of Gnus, org-mode,
magit, several programming modes and an assortment of minor modes. The
current instance is running since 6 days ago. Previously it ran since it
was built (07-07) for about 3 weeks. So there was plenty of opportunity
for showing the leak.

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Wed, 03 Aug 2011 15:16:40 +0200
> Cc: Kenichi Handa <handa@m17n.org>
> 
> Kenichi Handa <handa@m17n.org> writes:
> 
> > Could you check the versions before and after my changes for
> > char-table handling?
> >
> > 2011-07-07  Kenichi Handa  <handa@m17n.org>
> >
> > 	* character.h (unicode_category_t): New enum type.
> >
> > 	* chartab.c (uniprop_decoder_t, uniprop_encoder_t): New types.
> > 	(Qchar_code_property_table): New variable.
> > [...]
> 
> The tip revision of my Emacs is precisely the revision you mention. So
> far no abnormal memory consumption. bidi-display-reordering is
> nil. GNU/Linux x86_64 (Kubuntu 10.10). Daily usage of Gnus, org-mode,
> magit, several programming modes and an assortment of minor modes. The
> current instance is running since 6 days ago. Previously it ran since it
> was built (07-07) for about 3 weeks. So there was plenty of opportunity
> for showing the leak.

Thanks for the info.

FWIW, I'm running the version that shows "the leak" on others'
machines for the last 10 days without exiting Emacs, and last time I
looked it had a 195MB memory footprint.  So I suspect the leak is
triggered by some specific usage patterns.  Or maybe I'm missing
something important.

Re: Memory leak due to bidi?

[Antoine Levitt]

03/08/11 15:47, Eli Zaretskii
>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Wed, 03 Aug 2011 15:16:40 +0200
>> Cc: Kenichi Handa <handa@m17n.org>
>> 
>> Kenichi Handa <handa@m17n.org> writes:
>> 
>> > Could you check the versions before and after my changes for
>> > char-table handling?
>> >
>> > 2011-07-07  Kenichi Handa  <handa@m17n.org>
>> >
>> > 	* character.h (unicode_category_t): New enum type.
>> >
>> > 	* chartab.c (uniprop_decoder_t, uniprop_encoder_t): New types.
>> > 	(Qchar_code_property_table): New variable.
>> > [...]
>> 
>> The tip revision of my Emacs is precisely the revision you mention. So
>> far no abnormal memory consumption. bidi-display-reordering is
>> nil. GNU/Linux x86_64 (Kubuntu 10.10). Daily usage of Gnus, org-mode,
>> magit, several programming modes and an assortment of minor modes. The
>> current instance is running since 6 days ago. Previously it ran since it
>> was built (07-07) for about 3 weeks. So there was plenty of opportunity
>> for showing the leak.
>
> Thanks for the info.
>
> FWIW, I'm running the version that shows "the leak" on others'
> machines for the last 10 days without exiting Emacs, and last time I
> looked it had a 195MB memory footprint.  So I suspect the leak is
> triggered by some specific usage patterns.  Or maybe I'm missing
> something important.

I couldn't find if someone tried it, so just in case, another data point
: it just happened to me with (setq-default bidi-display-reordering nil)

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Antoine Levitt <antoine.levitt@gmail.com>
> Date: Thu, 04 Aug 2011 18:00:04 +0200
> 
> I couldn't find if someone tried it, so just in case, another data point
> : it just happened to me with (setq-default bidi-display-reordering nil)

Thanks.  In what revision did this happen?

Re: Memory leak due to bidi?

[Antoine Levitt]

04/08/11 18:30, Eli Zaretskii
>> From: Antoine Levitt <antoine.levitt@gmail.com>
>> Date: Thu, 04 Aug 2011 18:00:04 +0200
>> 
>> I couldn't find if someone tried it, so just in case, another data point
>> : it just happened to me with (setq-default bidi-display-reordering nil)
>
> Thanks.  In what revision did this happen?

Bzr pulled from yesterday, can't remember the exact rev.

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Antoine Levitt <antoine.levitt@gmail.com>
> Date: Thu, 04 Aug 2011 19:04:12 +0200
> 
> 04/08/11 18:30, Eli Zaretskii
> >> From: Antoine Levitt <antoine.levitt@gmail.com>
> >> Date: Thu, 04 Aug 2011 18:00:04 +0200
> >> 
> >> I couldn't find if someone tried it, so just in case, another data point
> >> : it just happened to me with (setq-default bidi-display-reordering nil)
> >
> > Thanks.  In what revision did this happen?
> 
> Bzr pulled from yesterday, can't remember the exact rev.

Okay, thanks.

The code that calls bidi_shelve_cache and bidi_unshelve_cache runs
even when bidi-display-reordering is nil, so at least that code is not
off the hook yet.  Although I've been running Emacs with that code
instrumented (see the patch I posted yesterday) on 2 different
machines for many hours, and find the allocation balance to be zero
all the time and the memory footprint reasonably low (e.g., the
session where I'm typing this runs for 25 hours, and its memory usage
is 116MB).

Re: Memory leak due to bidi?

[Stephen J. Turnbull]

Antoine Levitt writes:

 > Bzr pulled from yesterday, can't remember the exact rev.

If you haven't pulled again or committed, "bzr revno" is a quick way
to get that.

Re: Memory leak due to bidi?

[Kenichi Handa]

In article <87y5zatz2f.fsf@wanadoo.es>, Óscar Fuentes <ofv@wanadoo.es> writes:

> Kenichi Handa <handa@m17n.org> writes:
> > Could you check the versions before and after my changes for
> > char-table handling?
> >
> > 2011-07-07  Kenichi Handa  <handa@m17n.org>
> >
> > 	* character.h (unicode_category_t): New enum type.
> >
> > 	* chartab.c (uniprop_decoder_t, uniprop_encoder_t): New types.
> > 	(Qchar_code_property_table): New variable.
> > [...]

> The tip revision of my Emacs is precisely the revision you mention. So
> far no abnormal memory consumption. bidi-display-reordering is
> nil. GNU/Linux x86_64 (Kubuntu 10.10). Daily usage of Gnus, org-mode,
> magit, several programming modes and an assortment of minor modes. The
> current instance is running since 6 days ago. Previously it ran since it
> was built (07-07) for about 3 weeks. So there was plenty of opportunity
> for showing the leak.

Thank you for checking.  So, my change above is not likely
the cause of the leak.

---
Kenichi Handa
handa@m17n.org

Re: Memory leak due to bidi?

[Andy Moreton]

On Wed 03 Aug 2011, Kenichi Handa wrote:

> In article <m31ux3czcv.fsf@jhcloos.com>, James Cloos <cloos@jhcloos.com> writes:
>
>>>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:
> SM> BTW, this problem is new: I've been running with bidi-display-
> SM> reordering set to t for more than a year now and the problem only
> SM> appeared in the last couple of weeks.
>
>> I can confirm that.
>
>> My build from 2011/06/07 did not have this bug.
>
>> My build from 2011/07/23 does.
>
> Could you check the versions before and after my changes for
> char-table handling?
>
> 2011-07-07  Kenichi Handa  <handa@m17n.org>
>
> 	* character.h (unicode_category_t): New enum type.
>
> 	* chartab.c (uniprop_decoder_t, uniprop_encoder_t): New types.
> 	(Qchar_code_property_table): New variable.
> [...]
>
> ---
> Kenichi Handa
> handa@m17n.org

Handa-san, can you please say which revision this was committed in ?

I can find the text above in src/Changelog, but bzr log seems unwilling
to reveal the revision number of the commit. I've given up on using VC
annotate in emacs after 10 minutes of bzr thrashing my machine with zero
output.

Thanks,

   AndyM

P.S. I know that the the choice of bzr is fixed in stone, but that
doesn't stop it from being a lousy tool with poor documentation and
horrifically bad space and time performance. Why Canonical's tool is
preferred over hg or git is a mystery to me...

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Wed, 03 Aug 2011 14:26:22 +0100
> 
> On Wed 03 Aug 2011, Kenichi Handa wrote:
> 
> > In article <m31ux3czcv.fsf@jhcloos.com>, James Cloos <cloos@jhcloos.com> writes:
> >
> >>>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:
> > SM> BTW, this problem is new: I've been running with bidi-display-
> > SM> reordering set to t for more than a year now and the problem only
> > SM> appeared in the last couple of weeks.
> >
> >> I can confirm that.
> >
> >> My build from 2011/06/07 did not have this bug.
> >
> >> My build from 2011/07/23 does.
> >
> > Could you check the versions before and after my changes for
> > char-table handling?
> >
> > 2011-07-07  Kenichi Handa  <handa@m17n.org>
> >
> > 	* character.h (unicode_category_t): New enum type.
> >
> > 	* chartab.c (uniprop_decoder_t, uniprop_encoder_t): New types.
> > 	(Qchar_code_property_table): New variable.
> > [...]
> >
> > ---
> > Kenichi Handa
> > handa@m17n.org
> 
> Handa-san, can you please say which revision this was committed in ?
> 
> I can find the text above in src/Changelog, but bzr log seems unwilling
> to reveal the revision number of the commit.

You can specify a revision by its date, see "-r date:YYYY-MM-DD" in
the bzr docs.

The version you want is 105007.

It is easier to find this version by this:

 bzr log --line -l500 | fgrep Handa

> I've given up on using VC annotate in emacs after 10 minutes of bzr
> thrashing my machine with zero output.

??? "time bzr annotate src/character.h" shows this for me:

  real    0m21.802s
  user    0m20.430s
  sys     0m0.650s

That's not long enough to be annoying, let alone 10 minutes of
thrashing.  What am I missing?

(Btw, "git annotate" is even slower, by design.)

It is easier to find this version by this:

 bzr log --line -l500 | fgrep Handa

Re: Memory leak due to bidi?

[Stefan Monnier]

>> I've given up on using VC annotate in emacs after 10 minutes of bzr
>> thrashing my machine with zero output.

> ??? "time bzr annotate src/character.h" shows this for me:

>   real    0m21.802s
>   user    0m20.430s
>   sys     0m0.650s

> That's not long enough to be annoying, let alone 10 minutes of
> thrashing.  What am I missing?

Thrashing slows the process tremendously, so the 10 minutes are probably
due to thrashing.  And thrashing is most likely due to Bzr using more
memory than Andy has on his machine.


        Stefan

Re: Memory leak due to bidi?

[Andy Moreton]

On Wed 03 Aug 2011, Stefan Monnier wrote:

>>> I've given up on using VC annotate in emacs after 10 minutes of bzr
>>> thrashing my machine with zero output.
>
>> ??? "time bzr annotate src/character.h" shows this for me:
>
>>   real    0m21.802s
>>   user    0m20.430s
>>   sys     0m0.650s
>
>> That's not long enough to be annoying, let alone 10 minutes of
>> thrashing.  What am I missing?
>
> Thrashing slows the process tremendously, so the 10 minutes are probably
> due to thrashing.  And thrashing is most likely due to Bzr using more
> memory than Andy has on his machine.

Sorry for being imprecise. I didn't mean thrashing in the sense of
spending all of its time paging in and out of swap. I meant that I had
one CPU pegged at 100% and emacs unresponsive while bzr contemplated
busily. I gave up at that point.

bzr may be fast enough in a POSIX world, but on Windows the native
package is slow. Mercurial is also written mostly in Python and is
blindingly fast by comparison, so it isn't Windows or python that's
causing the lack of performance.

    AndyM

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Wed, 03 Aug 2011 16:49:33 +0100
> 
> bzr may be fast enough in a POSIX world, but on Windows the native
> package is slow.

My timings were from Windows, FWIW.

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 03 Aug 2011 10:18:27 -0400
> Cc: Andy Moreton <andrewjmoreton@gmail.com>, emacs-devel@gnu.org
> 
> >> I've given up on using VC annotate in emacs after 10 minutes of bzr
> >> thrashing my machine with zero output.
> 
> > ??? "time bzr annotate src/character.h" shows this for me:
> 
> >   real    0m21.802s
> >   user    0m20.430s
> >   sys     0m0.650s
> 
> > That's not long enough to be annoying, let alone 10 minutes of
> > thrashing.  What am I missing?
> 
> Thrashing slows the process tremendously, so the 10 minutes are probably
> due to thrashing.

Yes, I know.

> And thrashing is most likely due to Bzr using more memory than Andy
> has on his machine.

My machine has only 1.5GB, but it doesn't thrash and produces the
annotations for character.h in 27 seconds.  Running XP with less than
what I have would thrash all the time, not just when bzr is working,
so I don't believe Andy has less.

But maybe that happens for Andy because Emacs has eaten up all the
memory.

Re: Memory leak due to bidi?

[Andy Moreton]

On Wed 03 Aug 2011, Eli Zaretskii wrote:
>
> You can specify a revision by its date, see "-r date:YYYY-MM-DD" in
> the bzr docs.
>
> The version you want is 105007.
>
> It is easier to find this version by this:
>
>  bzr log --line -l500 | fgrep Handa
>
Thanks for this Eli.

As rev 105207 seems ok, I doubt that 105007 is the cause of the memory
usage problems. 

I've tried bootstrapping from the head of trunk today and observe rapid
memory usage growth, so it seems the problem is not a transient issue
caused by not bootstrapping a clean build.

I'm running rev 105300 to write this, which seems to be OK. I'll
keep bisecting to narrow down where things start misbehaving.

>> I've given up on using VC annotate in emacs after 10 minutes of bzr
>> thrashing my machine with zero output.
>
> ??? "time bzr annotate src/character.h" shows this for me:
>
>   real    0m21.802s
>   user    0m20.430s
>   sys     0m0.650s
>
> That's not long enough to be annoying, let alone 10 minutes of
> thrashing.  What am I missing?

I've no idea - I get similar performance from the command line. trying
to use VC annotate on the file within emacs was taking forever - I
didn't find out how long as I was too impatient to wait for it.

    AndyM

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Wed, 03 Aug 2011 17:01:32 +0100
> 
> I'm running rev 105300 to write this, which seems to be OK.

If 105300 is OK (and you didn't mistype the revision number ;-), then
the commit I feared most (105208) is off the hook.

> I'll keep bisecting to narrow down where things start misbehaving.

Thank you.

Re: Memory leak due to bidi?

[Andy Moreton]

On Wed 03 Aug 2011, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Wed, 03 Aug 2011 17:01:32 +0100
>> 
>> I'm running rev 105300 to write this, which seems to be OK.
>
> If 105300 is OK (and you didn't mistype the revision number ;-), then
> the commit I feared most (105208) is off the hook.
>
>> I'll keep bisecting to narrow down where things start misbehaving.
>
> Thank you.

I've updated bug#9221 to mention that bisecting seems to point to
somewhere between r105368 (good) and r105375 (bad).

    AndyM

Re: Memory leak due to bidi?

[Stephen J. Turnbull]

Andy Moreton writes:

 > I've no idea - I get similar performance from the command line. trying
 > to use VC annotate on the file within emacs was taking forever - I
 > didn't find out how long as I was too impatient to wait for it.

You should report this to the bzr guys.  They do care about
peformance, specifically performance on Windows.  bzr is not going
away any time soon, so you may as well do what you can to help get
this fixed.

Re: Memory leak due to bidi?

[Stefan Monnier]

>> I've no idea - I get similar performance from the command line. trying
>> to use VC annotate on the file within emacs was taking forever - I
>> didn't find out how long as I was too impatient to wait for it.
> You should report this to the bzr guys.  They do care about
> peformance, specifically performance on Windows.  bzr is not going
> away any time soon, so you may as well do what you can to help get
> this fixed.

IIUC what Andy is saying is that the problem is in vc-annotate and not
in Bzr itself.  It is true that vc-annotate is fairly slow (and the same
holds for grep and compilation output).


        Stefan

Re: Memory leak due to bidi?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 04 Aug 2011 01:15:35 -0400
> Cc: Andy Moreton <andrewjmoreton@gmail.com>, emacs-devel@gnu.org
> 
> >> I've no idea - I get similar performance from the command line. trying
> >> to use VC annotate on the file within emacs was taking forever - I
> >> didn't find out how long as I was too impatient to wait for it.
> > You should report this to the bzr guys.  They do care about
> > peformance, specifically performance on Windows.  bzr is not going
> > away any time soon, so you may as well do what you can to help get
> > this fixed.
> 
> IIUC what Andy is saying is that the problem is in vc-annotate and not
> in Bzr itself.  It is true that vc-annotate is fairly slow (and the same
> holds for grep and compilation output).

FWIW, I tried vc-annotate as well, on Windows, and it was similarly
fast (which is what I'd expect for a small file with a relatively
short history).

Re: Memory leak due to bidi?

[Stephen J. Turnbull]

Stefan Monnier writes:

 > IIUC what Andy is saying is that the problem is in vc-annotate and not
 > in Bzr itself.  It is true that vc-annotate is fairly slow (and the same
 > holds for grep and compilation output).

Ah, OK, the English was arguably ambiguous, but your interpretation is
more likely I guess.

My bad.