cperl + isearch + font-lock-multiline sometimes very slow

cperl + isearch + font-lock-multiline sometimes very slow

[Klaus Zeitler]

Since approx. last week isearch slows sometimes down to crawling speed in
cperl files. Looking at top I see that emacs then needs around 30% CPU and
when I stop isearch, for the next minute emacs eats up even more CPU before
going back to normal behavior.
This behavior disappeared when I turned off font-lock. Therefore I checked
my font-lock settings and found out that setting font-lock-multiline to nil
also helped. So I'm guessing it has something to do with font-lock or
jit-lock. How should I proceed now to investigate this further?

Thanks

Klaus

-- 
 ------------------------------------------
|  Klaus Zeitler      Lucent Technologies  |
|  Email:             kzeitler@lucent.com  |
 ------------------------------------------
---
One Page Principle:  A specification that will not fit on one
page of 8.5x11 inch paper cannot be understood. -- Mark Ardis

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Richard Stallman]

Around a week ago we installed lots of changes in Cperl mode.
That is probably why things changed.

But I think the interesting question is why the text properties
make isearch slow.  I suspect that is where the bug is.

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

> Around a week ago we installed lots of changes in Cperl mode.
> That is probably why things changed.
>
> But I think the interesting question is why the text properties
> make isearch slow.  I suspect that is where the bug is.

search.c:
#define REGEXP_CACHE_SIZE 20

I don't know but doesn't font-lock alone sometimes need more
than 20 regexps?

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Richard Stallman]

    > But I think the interesting question is why the text properties
    > make isearch slow.  I suspect that is where the bug is.

    search.c:
    #define REGEXP_CACHE_SIZE 20

    I don't know but doesn't font-lock alone sometimes need more
    than 20 regexps?

Maybe it does, and maybe we should increase the cache size;
but why would this make isearch slow?

Is the slowness due to fontification when isearch shows
new parts of the buffer?  If so, repeating the same isearch
will be fast.  Is that the case?

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Chong Yidong]

Klaus Zeitler <kzeitler@lucent.com> writes:

> Since approx. last week isearch slows sometimes down to crawling speed in
> cperl files. Looking at top I see that emacs then needs around 30% CPU and
> when I stop isearch, for the next minute emacs eats up even more CPU before
> going back to normal behavior.
> This behavior disappeared when I turned off font-lock. Therefore I checked
> my font-lock settings and found out that setting font-lock-multiline to nil
> also helped. So I'm guessing it has something to do with font-lock or
> jit-lock. How should I proceed now to investigate this further?

I can't reproduce this.  Could you provide a detailed recipe and/or
test file?

Also, see if the slowdown goes away if you increase REGEXP_CACHE_SIZE
to (e.g.) 40 in search.c, as Kim suggested.

#define REGEXP_CACHE_SIZE 20

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Klaus Zeitler]

>>>>> "Chong" == Chong Yidong <cyd@stupidchicken.com> writes:
    Chong> 
    Chong> I can't reproduce this.  Could you provide a detailed recipe and/or
    Chong> test file?

I've now managed to reduce my .emacs file from 4000 lines to the following
3 lines:
--- snip ---
(setq font-lock-multiline t)
(require 'printing "printing" 'no-error)
(fset 'perl-mode 'cperl-mode)
--- snip ---

With these 3 lines I can still observe the problem.
1. Start emacs with .emacs containing the 3 lines above
2. load a Perl file (not too small), I use perl5db.pl from the Perl distribution
3. Watch emacs CPU usage

After approximaetly 25 seconds, the CPU usage climbs for nearly 3 minutes
to 55% and emacs behaves very sluggish. Afterwards emacs behaves normal.
No need for isearch, that was obviously a misinterpretation. Don't know
what the package 'printing' has to do with it. Maybe it's a problem with
menu-bar updates and stealth fontification.

    Chong> Also, see if the slowdown goes away if you increase REGEXP_CACHE_SIZE
    Chong> to (e.g.) 40 in search.c, as Kim suggested.
    Chong> 
    Chong> #define REGEXP_CACHE_SIZE 20

Changing to REGEXP_CACHE_SIZE to 40 didn't change the behavior.

BTW changing REGEXP_CACHE_SIZE to a value >40 caused a core with temacs
gcc  -L/usr/ccs/lib `./prefix-args -Xlinker -R/usr/openwin/lib -R/usr/local/gnu/lib -R/opt/exp/gnu/lib -R/opt/exp/lib -R/opt/exp/lib/xpm/lib` `{ set x ; test "$2" = "USE_MOTIF"; } || echo ' -R/usr/dt/lib -L/usr/dt/lib'` -L/usr/openwin/lib -L/usr/local/gnu/lib -L/opt/exp/gnu/lib -L/opt/exp/lib -L/opt/exp/lib/xpm/lib -o temacs  dispnew.o frame.o scroll.o xdisp.o xmenu.o window.o charset.o coding.o category.o ccl.o cm.o term.o xfaces.o xterm.o xfns.o xselect.o xrdb.o fontset.o xsmfns.o fringe.o image.o  emacs.o keyboard.o macros.o keymap.o sysdep.o buffer.o filelock.o insdel.o marker.o minibuf.o fileio.o dired.o filemode.o cmds.o casetab.o casefiddle.o indent.o search.o regex.o undo.o alloc.o data.o doc.o editfns.o callint.o eval.o floatfns.o fns.o print.o lread.o abbrev.o syntax.o unexelf.o bytecode.o process.o callproc.o region-cache.o sound.o atimer.o doprnt.o strftime.o intervals.o textprop.o composite.o md5.o    terminfo.o lastfile.o gmalloc.o ralloc.o vm-limit.o  widget.o mktime.o   ../lwlib/liblw.a -L/usr/openwin/lib -L/usr/local/gnu/lib -L/opt/exp/gnu/lib -L/opt/exp/lib -L/opt/exp/lib/xpm/lib -lXm -lgen -lXp -lXmu -lXt -lSM -lICE -lXext -ltiff -ljpeg -lpng -lz -lm -lungif -lXpm -lX11  -lsocket -lnsl -lkstat -lcurses -lkstat  -lm
./temacs --batch --load loadup bootstrap
make[2]: *** [bootstrap-emacs] Abort (core dumped)
make[2]: Leaving directory `/vol/freeware/SunOS-5.8/build/emacs-cvs/src'
make[1]: *** [bootstrap-build] Error 2

HTH

Klaus

-- 
 ------------------------------------------
|  Klaus Zeitler      Lucent Technologies  |
|  Email:             kzeitler@lucent.com  |
 ------------------------------------------
---
Psychiatrists say that one out of four people are mentally
ill.  Check three friends.  If they're OK, you're it.

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Chong Yidong]

Klaus Zeitler <kzeitler@lucent.com> writes:

>>>>>> "Chong" == Chong Yidong <cyd@stupidchicken.com> writes:
>     Chong> 
>     Chong> I can't reproduce this.  Could you provide a detailed recipe and/or
>     Chong> test file?
>
> I've now managed to reduce my .emacs file from 4000 lines to the following
> 3 lines:
> --- snip ---
> (setq font-lock-multiline t)
> (require 'printing "printing" 'no-error)
> (fset 'perl-mode 'cperl-mode)
> --- snip ---
>
> With these 3 lines I can still observe the problem.
> 1. Start emacs with .emacs containing the 3 lines above
> 2. load a Perl file (not too small), I use perl5db.pl from the Perl distribution
> 3. Watch emacs CPU usage
>
> After approximaetly 25 seconds, the CPU usage climbs for nearly 3 minutes
> to 55% and emacs behaves very sluggish. Afterwards emacs behaves normal.
> No need for isearch, that was obviously a misinterpretation. Don't know
> what the package 'printing' has to do with it. Maybe it's a problem with
> menu-bar updates and stealth fontification.

I still can't reproduce this, using the above .emacs file and
perl5db.pl.  A month or two ago, we fixed such a bug dealing with a
bad interaction between menu-bar updates and stealth fontification;
stealth fontification should no longer trigger menu-bar updates.
Could you just check that you don't have any old files (e.g. of
jit-lock) hanging around in your CVS copy?

Assuming it's not that, please see if it's due to stealth
fontification by changing jit-lock-stealth-time to nil and see if the
problem recurs.  Also, try setting jit-lock-stealth-time to 2 and see
if the problem occurs more quickly.

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Chong Yidong]

Chong Yidong <cyd@stupidchicken.com> writes:

>> (setq font-lock-multiline t)
>> (require 'printing "printing" 'no-error)
>> (fset 'perl-mode 'cperl-mode)
>>
>> With these 3 lines I can still observe the problem.
>> 1. Start emacs with .emacs containing the 3 lines above
>> 2. load a Perl file (not too small), I use perl5db.pl from the Perl distribution
>> 3. Watch emacs CPU usage
>>
>> After approximaetly 25 seconds, the CPU usage climbs for nearly 3 minutes
>> to 55% and emacs behaves very sluggish. Afterwards emacs behaves normal.
>> No need for isearch, that was obviously a misinterpretation.
>
> I still can't reproduce this, using the above .emacs file and
> perl5db.pl.

To be precise, I save the above three lines into foo.el, and do

 emacs -Q -l foo.el /usr/share/perl/5.8/perl5db.pl

After waiting several minutes and watching with `top', I observe that
Emacs' CPU usage never exceeds 4.5% (which is normal for jit stealth
fontification), and there is no sluggishness either in Emacs or other
programs.  Maybe the scheduler in Solaris (which I noticed you are
using) behaves funny.

Re: cperl + isearch + font-lock-multiline sometimes very slow

[martin rudalics]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

> With these 3 lines I can still observe the problem.
> 1. Start emacs with .emacs containing the 3 lines above
> 2. load a Perl file (not too small), I use perl5db.pl from the Perl distribution
> 3. Watch emacs CPU usage

Before digging into this any further: Can you reproduce the bug with
the attached patch?

[-- Attachment #2: cperl.patch --]
[-- Type: text/plain, Size: 2181 bytes --]

*** cperl-mode.el	Sat Oct 21 11:36:32 2006
--- cperl-mode.el	Sat Oct 21 17:53:32 2006
***************
*** 2822,2828 ****
  				    (skip-chars-backward " \t")
  				    (looking-at "[ \t]*[a-zA-Z_][a-zA-Z_0-9]*[ \t]*:")))
  			     (get-text-property (point) 'first-format-line)))
! 		   
  		   ;; Look at previous line that's at column 0
  		   ;; to determine whether we are in top-level decls
  		   ;; or function's arg decls.  Set basic-indent accordingly.
--- 2822,2828 ----
  				    (skip-chars-backward " \t")
  				    (looking-at "[ \t]*[a-zA-Z_][a-zA-Z_0-9]*[ \t]*:")))
  			     (get-text-property (point) 'first-format-line)))
! 
  		   ;; Look at previous line that's at column 0
  		   ;; to determine whether we are in top-level decls
  		   ;; or function's arg decls.  Set basic-indent accordingly.
***************
*** 5708,5714 ****

  (defun cperl-windowed-init ()
    "Initialization under windowed version."
!   (if (or (featurep 'ps-print) cperl-faces-init)
        ;; Need to init anyway:
        (or cperl-faces-init (cperl-init-faces))
      (add-hook 'font-lock-mode-hook
--- 5708,5715 ----

  (defun cperl-windowed-init ()
    "Initialization under windowed version."
!   (if (or ; (featurep 'ps-print)
!        cperl-faces-init)
        ;; Need to init anyway:
        (or cperl-faces-init (cperl-init-faces))
      (add-hook 'font-lock-mode-hook
***************
*** 5717,5726 ****
  		 (if (memq major-mode '(perl-mode cperl-mode))
  		     (progn
  		       (or cperl-faces-init (cperl-init-faces)))))))
!     (if (fboundp 'eval-after-load)
! 	(eval-after-load
! 	    "ps-print"
! 	  '(or cperl-faces-init (cperl-init-faces))))))

  (defvar cperl-font-lock-keywords-1 nil
    "Additional expressions to highlight in Perl mode.  Minimal set.")
--- 5718,5728 ----
  		 (if (memq major-mode '(perl-mode cperl-mode))
  		     (progn
  		       (or cperl-faces-init (cperl-init-faces)))))))
! ;;;     (if (fboundp 'eval-after-load)
! ;;; 	(eval-after-load
! ;;; 	    "ps-print"
! ;;; 	  '(or cperl-faces-init (cperl-init-faces))))
!     ))

  (defvar cperl-font-lock-keywords-1 nil
    "Additional expressions to highlight in Perl mode.  Minimal set.")

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Chong Yidong]

I managed to reproduce the bug, and figured out the problem.  If
ps-print is already loaded, cperl-init-faces is called when
cperl-mode.el is loaded instead of when cperl-mode is run.  However,
cperl-init-faces assumes cperl-font-lock-multiline is correctly set,
which is normally done at run-time; otherwise, it supplies the
incorrect font-lock keywords.  I've checked in a fix.

Re: cperl + isearch + font-lock-multiline sometimes very slow

[martin rudalics]

> I managed to reproduce the bug, and figured out the problem.  If
> ps-print is already loaded, cperl-init-faces is called when
> cperl-mode.el is loaded instead of when cperl-mode is run.  However,
> cperl-init-faces assumes cperl-font-lock-multiline is correctly set,
> which is normally done at run-time; otherwise, it supplies the
> incorrect font-lock keywords.  I've checked in a fix.

Good catch!  Does this handle the other two bugs I reported too?

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Chong Yidong]

martin rudalics <rudalics@gmx.at> writes:

>> I managed to reproduce the bug, and figured out the problem.  If
>> ps-print is already loaded, cperl-init-faces is called when
>> cperl-mode.el is loaded instead of when cperl-mode is run.  However,
>> cperl-init-faces assumes cperl-font-lock-multiline is correctly set,
>> which is normally done at run-time; otherwise, it supplies the
>> incorrect font-lock keywords.  I've checked in a fix.
>
> Good catch!  Does this handle the other two bugs I reported too?

I checked in fixes for those two as well.  Thanks very much.

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Klaus Zeitler]

>>>>> "Chong" == Chong Yidong <cyd@stupidchicken.com> writes:
    Chong> 
    Chong> I managed to reproduce the bug, and figured out the problem.  If
    Chong> ps-print is already loaded, cperl-init-faces is called when

Odd dependency. Just out of curiosity (if it isn't too complicated), what was
the reason?

    Chong> cperl-mode.el is loaded instead of when cperl-mode is run. However,
    Chong> cperl-init-faces assumes cperl-font-lock-multiline is correctly
    Chong> set, which is normally done at run-time; otherwise, it supplies the
    Chong> incorrect font-lock keywords.  I've checked in a fix.

Works great now, stealth fontification is now below 3% CPU. Many thanks.

Martin, I guess I don't need to try your patch? If you still want me to try
it, let me know.

Klaus

-- 
 ------------------------------------------
|  Klaus Zeitler      Lucent Technologies  |
|  Email:             kzeitler@lucent.com  |
 ------------------------------------------
---
When I get real bored, I like to drive downtown and get
a great parking spot, then sit in my car and count how
many people ask me if I'm leaving.     -- Steven Wright

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Stefan Monnier]

Chong> I managed to reproduce the bug, and figured out the problem.
Chong> If ps-print is already loaded, cperl-init-faces is called when

Indeed, the relevant code is quoted below, IIUC.

> Odd dependency.  Just out of curiosity (if it isn't too complicated), what
> was the reason?

You'd have to ask Ilya Zakharevich <ilyaz@cpan.org>, I believe.
I suspect it's due to hysterical raisins, mostly linked to the act that
face-initialization has changed significantly between Emacs-19 and now
(compounded by the usual not-so-subtle differences with XEmacs, plus
fact that XEmacs's own changes in the same time frame).
If you don't care about backward compatibility, this can surely be
considerably simplified.


        Stefan


(defun cperl-windowed-init ()
  "Initialization under windowed version."
  (if (or (featurep 'ps-print) cperl-faces-init)
      ;; Need to init anyway:
      (or cperl-faces-init (cperl-init-faces))
    (add-hook 'font-lock-mode-hook
	      (function
	       (lambda ()
		 (if (memq major-mode '(perl-mode cperl-mode))
		     (progn
		       (or cperl-faces-init (cperl-init-faces)))))))
    (if (fboundp 'eval-after-load)
	(eval-after-load
	    "ps-print"
	  '(or cperl-faces-init (cperl-init-faces))))))

[...]

(if (cperl-enable-font-lock) (cperl-windowed-init))

Re: cperl + isearch + font-lock-multiline sometimes very slow

[Stefan Monnier]

> You'd have to ask Ilya Zakharevich <ilyaz@cpan.org>, I believe.
> I suspect it's due to hysterical raisins, mostly linked to the act that
                                                                 f

> face-initialization has changed significantly between Emacs-19 and now
> (compounded by the usual not-so-subtle differences with XEmacs, plus
> fact that XEmacs's own changes in the same time frame).
  XXXXXXXXX


Sorry for that the speaking fast too which of.


        Stefan

Re: cperl + isearch + font-lock-multiline sometimes very slow

[martin rudalics]

Two further issues which, however, don't seem related to the bug:

(1) The following causes an Invalid face reference

(defcustom cperl-invalid-face		; Does not customize with '' on XEmacs
   (if cperl-singly-quote-face
       'underline ''underline) ; On older Emacsen was evaluated by `font-lock'
   (if cperl-singly-quote-face
       "*This face is used for highlighting trailing whitespace."
     "*Face for highlighting trailing whitespace.")
   :type 'face
   :version "21.1"
   :group 'cperl-faces)

Momentary remedy: Use

(defcustom cperl-invalid-face 'underline
   "*Face for highlighting trailing whitespace."
   :type 'face
   :version "21.1"
   :group 'cperl-faces)

instead.


(2) `cperl-mode' sets `font-lock-multiline' _globally_ to t with all
sorts of unpleasing side-effects.  This should be obviously done via
`font-lock-set-defaults'.

Momentary remedy: Change the setting of this variable in `cperl-mode'
to:
	(set (make-local-variable 'font-lock-multiline) t))

(I think `font-lock-multiline' asks for `make-variable-buffer-local' in
font-lock to defy similar attempts to set this.)