bug#496: 23.0.60; ispell-word becomes increasingly slower

bug#496: 23.0.60; ispell-word becomes increasingly slower

[Markus Triska]

When I save the following form to test.el:

(let ((n 0))
  (with-temp-buffer
    (insert "test")
    (while t
      (setq n (1+ n))
      (when (= (mod n 100) 0)
        (message "n: %s -- %s" n (emacs-uptime)))
      (ispell-word nil t))))

and then do "emacs -Q --script test.el", I get:

   Starting new Ispell process [default] ...
   n: 100 -- 0 seconds
   n: 200 -- 0 seconds
   n: 300 -- 0 seconds
   n: 400 -- 0 seconds
   n: 500 -- 1 second
   n: 600 -- 1 second
   n: 700 -- 1 second
   n: 800 -- 1 second
   n: 900 -- 1 second
   n: 1000 -- 1 second
   n: 1100 -- 1 second
   n: 1200 -- 1 second
   n: 1300 -- 1 second
   n: 1400 -- 1 second
   n: 1500 -- 1 second
   n: 1600 -- 1 second
   n: 1700 -- 1 second
   n: 1800 -- 1 second
   n: 1900 -- 1 second
   n: 2000 -- 1 second
   ...
   n: 52400 -- 3 minutes, 12 seconds
   n: 52500 -- 3 minutes, 13 seconds
   n: 52600 -- 3 minutes, 14 seconds
   n: 52700 -- 3 minutes, 15 seconds
   ..
   n: 70500 -- 8 minutes, 30 seconds
   n: 70600 -- 8 minutes, 31 seconds
   n: 70700 -- 8 minutes, 33 seconds
   n: 70800 -- 8 minutes, 35 seconds
   n: 70900 -- 8 minutes, 36 seconds
   n: 71000 -- 8 minutes, 38 seconds
   ...

i.e., the program becomes increasingly slower, from many iterations
per second to several seconds per iteration. With "top", I also see
that aspell uses increasingly more memory. I can reproduce this on
Fedora Core 9 and OSX 10.4 with Aspell 0.60.5. In a simplified test to
check for memory leaks in aspell, I did:

   $ while true; do echo "test"; done | aspell -a

and in this case, aspell's memory usage seems to stay constant.


In GNU Emacs 23.0.60.1 (i386-apple-darwin8.11.1, GTK+ Version 2.12.9)
 of 2008-06-15 on mt-computer.local
Windowing system distributor `The XFree86 Project, Inc', version 11.0.40400000
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  value of $XMODIFIERS: nil
  locale-coding-system: nil
  default-enable-multibyte-characters: t

bug#496: 23.0.60; ispell-word becomes increasingly slower

[Eli Zaretskii]

> Resent-From: Markus Triska <markus.triska@gmx.at>
> Resent-To: bug-submit-list@donarmstrong.com
> Resent-CC: Emacs Bugs <bug-gnu-emacs@gnu.org>
> Resent-Sender: don@donarmstrong.com
> From: Markus Triska <markus.triska@gmx.at>
> Date: Sat, 28 Jun 2008 02:26:55 +0200 (CEST)
> Cc: 
> 
> 
> When I save the following form to test.el:
> 
> (let ((n 0))
>   (with-temp-buffer
>     (insert "test")
>     (while t
>       (setq n (1+ n))
>       (when (= (mod n 100) 0)
>         (message "n: %s -- %s" n (emacs-uptime)))
>       (ispell-word nil t))))
> 
> and then do "emacs -Q --script test.el", I get:
> 
>    Starting new Ispell process [default] ...
>    n: 100 -- 0 seconds
>    n: 200 -- 0 seconds
>    n: 300 -- 0 seconds
>    n: 400 -- 0 seconds
>    n: 500 -- 1 second
>    n: 600 -- 1 second
>    n: 700 -- 1 second
>    n: 800 -- 1 second
>    n: 900 -- 1 second
>    n: 1000 -- 1 second
>    n: 1100 -- 1 second
>    n: 1200 -- 1 second
>    n: 1300 -- 1 second
>    n: 1400 -- 1 second
>    n: 1500 -- 1 second
>    n: 1600 -- 1 second
>    n: 1700 -- 1 second
>    n: 1800 -- 1 second
>    n: 1900 -- 1 second
>    n: 2000 -- 1 second
>    ...
>    n: 52400 -- 3 minutes, 12 seconds
>    n: 52500 -- 3 minutes, 13 seconds
>    n: 52600 -- 3 minutes, 14 seconds
>    n: 52700 -- 3 minutes, 15 seconds
>    ..
>    n: 70500 -- 8 minutes, 30 seconds
>    n: 70600 -- 8 minutes, 31 seconds
>    n: 70700 -- 8 minutes, 33 seconds
>    n: 70800 -- 8 minutes, 35 seconds
>    n: 70900 -- 8 minutes, 36 seconds
>    n: 71000 -- 8 minutes, 38 seconds
>    ...
> 
> i.e., the program becomes increasingly slower, from many iterations
> per second to several seconds per iteration. With "top", I also see
> that aspell uses increasingly more memory. I can reproduce this on
> Fedora Core 9 and OSX 10.4 with Aspell 0.60.5. In a simplified test to
> check for memory leaks in aspell, I did:
> 
>    $ while true; do echo "test"; done | aspell -a
> 
> and in this case, aspell's memory usage seems to stay constant.

Garbage collection?

bug#496: 23.0.60; ispell-word becomes increasingly slower

[Markus Triska]

Eli Zaretskii <eliz@gnu.org> writes:

>> check for memory leaks in aspell, I did:
>> 
>>    $ while true; do echo "test"; done | aspell -a
>> 
>> and in this case, aspell's memory usage seems to stay constant.
>
> Garbage collection?

I've received a better test case from the Aspell maintainer: 

  ( while true; do echo '!'; echo "-"; echo "%"; echo "^test"; done ) | aspell -a

According to him, it is the constant switching of modes that is causing
a problem. He notes:

   "I have narrowed down the problem code. Technically it is not a leak
   as the code is still reachable (according to valgrind), which makes
   it harder to fix. I will let you know."

It is thus more an Aspell problem, but if it cannot be fixed in Aspell,
I will look whether the Emacs side of communication can be improved.

bug#496: 23.0.60; ispell-word becomes increasingly slower

[Eli Zaretskii]

> From: Markus Triska <markus.triska@gmx.at>
> Cc: 496@emacsbugs.donarmstrong.com, emacs-pretest-bug@gnu.org,
> 	bug-submit-list@donarmstrong.com, bug-gnu-emacs@gnu.org
> Date: Sat, 28 Jun 2008 11:25:48 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> check for memory leaks in aspell, I did:
> >> 
> >>    $ while true; do echo "test"; done | aspell -a
> >> 
> >> and in this case, aspell's memory usage seems to stay constant.
> >
> > Garbage collection?
> 
> I've received a better test case from the Aspell maintainer: 
> 
>   ( while true; do echo '!'; echo "-"; echo "%"; echo "^test"; done ) | aspell -a
> 
> According to him, it is the constant switching of modes that is causing
> a problem. He notes:
> 
>    "I have narrowed down the problem code. Technically it is not a leak
>    as the code is still reachable (according to valgrind), which makes
>    it harder to fix. I will let you know."
> 
> It is thus more an Aspell problem, but if it cannot be fixed in Aspell,
> I will look whether the Emacs side of communication can be improved.

But your test.el shows a similar problem with Ispell.  So either
Aspell and Ispell share the same problem, or Emacs still has something
to do with the slowdown.

bug#496: 23.0.60; ispell-word becomes increasingly slower

[Markus Triska]

Eli Zaretskii <eliz@gnu.org> writes:

> your test.el shows a similar problem with Ispell. So either Aspell and
> Ispell share the same problem, or Emacs still has something to do with
> the slowdown.

Aspell's maintainer Kevin Atkinson has generously sent me the following
patch for Aspell that fixes the problem for me with the latest CVS
version of Aspell. He says it will be in Aspell 0.60.7. Here's a simpler
test case for the problem in versions up to 0.60.6:

  while true; do echo "-"; done | aspell -a

Index: common/config.cpp
===================================================================
RCS file: /sources/aspell/aspell/common/config.cpp,v
retrieving revision 1.79
diff -u -r1.79 config.cpp
--- common/config.cpp	12 Oct 2007 00:33:17 -0000	1.79
+++ common/config.cpp	28 Jun 2008 10:42:10 -0000
@@ -1000,7 +1000,7 @@
       }
       
       assert(ki->def != 0); // if null this key should never have values
-      // directly added to it
+                            // directly added to it
       String value(entry->action == Reset ? get_default(ki) : entry->value);
       
       switch (ki->type) {
@@ -1027,11 +1027,13 @@
           goto error;
         }
 
+        clear_others(entry);
         NOTIFY_ALL(item_updated(ki, val));
         break;
         
       } case KeyInfoString:
         
+        clear_others(entry);
         NOTIFY_ALL(item_updated(ki, value));
         break;
         
@@ -1040,6 +1042,7 @@
         int num;
         
         if (sscanf(value.str(), "%i", &num) == 1 && num >= 0) {
+          clear_others(entry);
           NOTIFY_ALL(item_updated(ki, num));
         } else {
           pe = make_err(bad_value, entry->key, value, _("a positive integer"));
@@ -1049,6 +1052,9 @@
         break;
       }
       case KeyInfoList:
+
+        if (entry->action == ListSet || entry->action == ListClear)
+          clear_others(entry);
         
         NOTIFY_ALL(list_updated(ki));
         break;
@@ -1064,6 +1070,26 @@
       return (PosibErrBase &)pe;
   }
 
+  // Delete all previous entries with the same key
+  void Config::clear_others(Entry * entry) {
+
+    String & key = entry->key;
+    //printf("Clear Others For %s\n", key.c_str());
+    Entry * * cur = &first_;
+    while (*cur != entry) {
+      assert(*cur != *insert_point_);
+      if ((*cur)->key == key) {
+        //printf("JAP \"%s\"!\n", (*cur)->value.c_str());
+        Entry * tmp = *cur;
+        *cur = (*cur)->next;
+        delete tmp;
+      } else {
+        cur = &(*cur)->next;
+      }
+    }
+    
+  }
+
 #undef NOTIFY_ALL
 
 
Index: common/config.hpp
===================================================================
RCS file: /sources/aspell/aspell/common/config.hpp,v
retrieving revision 1.27
diff -u -r1.27 config.hpp
--- common/config.hpp	3 May 2005 05:08:19 -0000	1.27
+++ common/config.hpp	28 Jun 2008 10:42:10 -0000
@@ -132,6 +132,8 @@
 
     PosibErr<int> commit(Entry * entry, Conv * conv = 0);
 
+    void clear_others(Entry * entry);
+
     bool settings_read_in_;
 
   public: