bug#1024: Large C++ files load slowly, regardless of font-lock-maximum-size

bug#1024: Large C++ files load slowly, regardless of font-lock-maximum-size

[John W]

To reproduce:
(1) Generate a 3M C++ file.
(2) Load file.
(3) Wait for emacs. <-- bug

I'm using jit-lock, but I tested lazy-lock, and the behavior is the same.

font-lock-maximum-size is the default.  I.e.
Its value is 256000

I used edebug to see what emacs was doing, and it gave me a stack like:

  c-literal-limits()
  c-neutralize-syntax-in-CPP(1 3527391 3527390)
  funcall(c-neutralize-syntax-in-CPP 1 3527391 3527390)
  (if nil c-before-font-lock-function (funcall c-before-font-lock-function (point-min) (point-max) (- ... ...)))
  (save-excursion (if c-get-state-before-change-function (funcall c-get-state-before-change-function ... ...)) (if nil c-before-font-lock-function (funcall c-before-font-lock-function ... ... ...)))
  (save-restriction (widen) (save-excursion (if c-get-state-before-change-function ...) (if nil c-before-font-lock-function ...)))
  c-common-init(c++-mode)
  c++-mode()
  set-auto-mode-0(c++-mode nil)
  set-auto-mode()
  normal-mode(t)
  after-find-file(nil t)

I blame c-before-font-lock-function .

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/usr/local/share/emacs/22.2/etc/DEBUG for instructions.


In GNU Emacs 22.2.1 (i686-pc-linux-gnu)
 of 2008-08-27 on augustine
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: C
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8
  default-enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
  encoded-kbd-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent input:
[ 6 ~ ESC [ 6 ~ ESC [ 6 ~ ESC [ 6 ~ ESC [ 6 ~ ESC [ 
6 ~ ESC [ 5 ~ ESC [ 5 ~ ESC [ 5 ~ ESC [ 5 ~ ESC [ 5 
~ ESC [ 5 ~ ESC [ 5 ~ ESC [ 5 ~ ESC [ 5 ~ ESC [ 5 ~ 
ESC [ 5 ~ ESC [ 5 ~ C-x k RET C-x C-e C-x k RET ESC 
[ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A 
ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC 
[ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ B ESC [ B 
ESC [ B ESC [ A ESC f ESC f ESC f ESC f ESC f ESC b 
C-k C-_ ESC [ 5 ~ ESC [ 6 ~ ESC [ 5 ~ ESC [ 6 ~ ESC 
[ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B 
ESC [ B ESC [ B ESC [ B ESC [ A C-@ ESC [ B ESC [ B 
ESC [ B C-x b C-g C-x o C-g ESC [ A ESC [ A ESC [ A 
ESC f ESC f ESC b ESC f ESC f ESC f ESC f ESC b C-k 
C-_ C-x b RET C-x b RET ESC x e m a c s - b DEL d e 
TAB DEL DEL TAB DEL DEL DEL DEL DEL DEL b u TAB DEL 
DEL g n TAB DEL DEL DEL DEL r e p o TAB r t TAB RE
T

Recent messages:
#<buffer Connect.i>
Auto-saving...done
Undo!
Mark set
Entering debugger...
Quit
Undo!
Auto-saving...done
Making completion list... [3 times]
Loading emacsbug...done



      

bug#1024: Large C++ files load slowly, regardless of font-lock-maximum-size

[martin rudalics]

 > I used edebug to see what emacs was doing, and it gave me a stack like:
 >
 >   c-literal-limits()
 >   c-neutralize-syntax-in-CPP(1 3527391 3527390)
 >   funcall(c-neutralize-syntax-in-CPP 1 3527391 3527390)
 >   (if nil c-before-font-lock-function (funcall c-before-font-lock-function (point-min) (point-max) (- ... ...)))
 >   (save-excursion (if c-get-state-before-change-function (funcall c-get-state-before-change-function ... ...)) (if nil c-before-font-lock-function (funcall c-before-font-lock-function ... ... ...)))
 >   (save-restriction (widen) (save-excursion (if c-get-state-before-change-function ...) (if nil c-before-font-lock-function ...)))
 >   c-common-init(c++-mode)
 >   c++-mode()
 >   set-auto-mode-0(c++-mode nil)
 >   set-auto-mode()
 >   normal-mode(t)
 >   after-find-file(nil t)
 >
 > I blame c-before-font-lock-function .

Could you please have a look at the description of bug#851

   http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=851

and try the recipe proposed there by Chong Yidong?

martin

bug#1024: Large C++ files load slowly, regardless of font-lock-maximum-size

[John W]

> Could you please have a look at the description of bug#851
>
> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=851
>
> and try the recipe proposed there by Chong Yidong?

Sure.  I tried this patch

+++ cc-engine.el	2008/04/01 21:41:21	1.56.2.11

but it made no difference; it is still slow, and when I examine the
stack, it is the same stack trace.  The file loads in 2 seconds in
fundamental mode, but takes about 2 minutes in C++ mode.  I should
have mentioned that this is a preprocessed C++ file, with many
compiler directies, such as #line 1 "somefile.h", and I now think the
problem is provoked by the compiler directives.

If I globally replace the #line stuff like so:

  #line 1 "some_file.h"
 -->
  s = "some_file.h"

the file loads in about 7 seconds.

- John


      

bug#1024: Large C++ files load slowly, regardless of font-lock-maximum-size

[Alan Mackenzie]

Hi, John,

On Thu, Sep 25, 2008 at 08:22:34PM -0700, John W wrote:
> > Could you please have a look at the description of bug#851

> > http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=851

> > and try the recipe proposed there by Chong Yidong?

> Sure.  I tried this patch

> +++ cc-engine.el	2008/04/01 21:41:21	1.56.2.11

This slowdown happened because of an earlier bug fix.  Part of that fix
is scanning the entire buffer, "fixing" things which could derail the
syntax and font locking, e.g. preprocessor lines like this:

#warning this isn't fixed yet!
                 ^
, where the apostrophe was being treated like a string opener.

That fix was not coded with a view to speed, unfortunately.  It was later
fixed in cc-mode.el version 1.75 (CVS trunk at savannah) and version
1.58.2.12 (in the Emacs 22 branch) on 2008-05-24.

The fix was incorporated into Emacs 22.3, released a few days ago.

Also, thanks for generating and posting the stack dump, which made it
trivially easy to track down the problem.

> - John

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#1024: Large C++ files load slowly, regardless of font-lock-maximum-size

[Alan Mackenzie]

Hi, John.

John W <jw_spambox@yahoo.com> wrote:
> To reproduce:
> (1) Generate a 3M C++ file.
> (2) Load file.
> (3) Wait for emacs. <-- bug

> I'm using jit-lock, but I tested lazy-lock, and the behavior is the same.

> font-lock-maximum-size is the default.  I.e.
> Its value is 256000

> I used edebug to see what emacs was doing, and it gave me a stack like:

> c-literal-limits()
>  c-neutralize-syntax-in-CPP(1 3527391 3527390)
>  funcall(c-neutralize-syntax-in-CPP 1 3527391 3527390)
>  (if nil c-before-font-lock-function (funcall c-before-font-lock-function (point-min) (point-max) (- ... ...)))
>  (save-excursion (if c-get-state-before-change-function (funcall c-get-state-before-change-function ... ...)) (if nil c-before-font-lock-function (funcall c-before-font-lock-function ... ... ...)))
>  (save-restriction (widen) (save-excursion (if c-get-state-before-change-function ...) (if nil c-before-font-lock-function ...)))
>  c-common-init(c++-mode)
>  c++-mode()
>  set-auto-mode-0(c++-mode nil)
>  set-auto-mode()
>  normal-mode(t)
>  after-find-file(nil t)

Thanks for taking the trouble to generate this stack dump; it identifies
the problem completely.

This bug was caused by a previous bug fix which introduced a complete
scan over the entire buffer when it's loaded.  One function,
`c-neutralize-syntax-in-CPP' had been coded very inefficiently.  It was
recoded for speed on 2008-05-24, committed in
.../emacs/lisp/progmodes/cc-mode.el version 1.58.2.12 (Emacs 22 branch)
and version 1.75 (trunk) at savannah.


To get the fixed version, either:
(i) Upgrade to Emacs 22.3; or
(ii) If you're using the Emacs 23 CVS, update your version.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).