Re: ntemacs hangs when openning the attached file

Re: ntemacs hangs when openning the attached file

[Eli Zaretskii]

> Date: Wed, 7 May 2008 12:01:38 +0800
> From: "yu jie" <yujie052@gmail.com>
> 
>     The current CVS header version hangs when openning the attached file.

No, it doesn't hang, it just takes a lot of time to visit this file.
I measured 61 seconds on a 3GHz machine.  This file has 86406 lines,
and uses some pretty non-standard formatting, such as this one:

  static int simpleNext(
    sqlite3_tokenizer_cursor *pCursor,  /* Cursor returned by simpleOpen */
    const char **ppToken,               /* OUT: *ppToken is the token text */
    int *pnBytes,                       /* OUT: Number of bytes in token */
    int *piStartOffset,                 /* OUT: Starting offset of token */
    int *piEndOffset,                   /* OUT: Ending offset of token */
    int *piPosition                     /* OUT: Position integer of token */
  ){

It also uses long comments formatted like this:

  /*
  ** Additional bit values that can be ORed with an affinity without
  ** changing the affinity.
  */

I'm guessing that this formatting coupled with the sheer size of the
file somehow triggers an inefficiency.

I didn't dig into the problem deep enough to find the actual culprit,
but if I run Emacs under GDB and interrupt it during the long wait, I
always see the following Lisp backtrace:

    "parse-partial-sexp" (0x82e9c4)
    "c-literal-limits" (0x82eb14)
    "c-neutralize-syntax-in-CPP" (0x82ec64)
    "c-common-init" (0x82edb4)
    "c-mode" (0x82eef4)
    "set-auto-mode-0" (0x82f034)
    "set-auto-mode" (0x82f110)
    "normal-mode" (0x82f454)
    "after-find-file" (0x82f5a4)
    "find-file-noselect-1" (0x82f6e4)
    "find-file-noselect" (0x82f834)
    "find-file" (0x82f984)
    "call-interactively" (0x82fc04)

This seems not to be related to fontification, but rather to something
that c-neutralize-syntax-in-CPP does during C Mode initialization.

Re: ntemacs hangs when openning the attached file

[Alan Mackenzie]

Stefan, Eli,

On Wed, May 07, 2008 at 11:48:47AM +0300, Eli Zaretskii wrote:
> > Date: Wed, 7 May 2008 12:01:38 +0800
> > From: "yu jie" <yujie052@gmail.com>

> >     The current CVS header version hangs when openning the attached file.

> No, it doesn't hang, it just takes a lot of time to visit this file.
> I measured 61 seconds on a 3GHz machine.  This file has 86406 lines,
> ....

and at 1 line per second would take just over a day to print.  This is a
big file.

> I'm guessing that this formatting coupled with the sheer size of the
> file somehow triggers an inefficiency.

[ .... ]

> This seems not to be related to fontification, but rather to something
> that c-neutralize-syntax-in-CPP does during C Mode initialization.

It's just that c-neutralize-syntax-in-CPP takes a long time in such a
large buffer.  However, it would take just as long if it were split into
several smaller buffers.

c-neutralize-syntax-in-CPP is a newish function which counteracts
problems cause by things like:

    #define RBRACE }

.  The answer would seem to be calling the function lazily or in the
background, a bit like jit-lock does for font locking.

Stefan, I think you mentioned a long time ago that jit could be used for
things other than font locking, but I can't find the thread.  Any clues?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: ntemacs hangs when openning the attached file

[Stefan Monnier]

>> This seems not to be related to fontification, but rather to something
>> that c-neutralize-syntax-in-CPP does during C Mode initialization.

> It's just that c-neutralize-syntax-in-CPP takes a long time in such a
> large buffer.  However, it would take just as long if it were split into
> several smaller buffers.

> c-neutralize-syntax-in-CPP is a newish function which counteracts
> problems cause by things like:

>     #define RBRACE }

> .  The answer would seem to be calling the function lazily or in the
> background, a bit like jit-lock does for font locking.

> Stefan, I think you mentioned a long time ago that jit could be used for
> things other than font locking, but I can't find the thread.  Any clues?

Not sure what you want to know.  You can use jit-lock-register, tho it's
not clear whether it does what you want.


        Stefan

Re: ntemacs hangs when openning the attached file

[Alan Mackenzie]

Hi, Eli and yu!

On Wed, May 07, 2008 at 11:48:47AM +0300, Eli Zaretskii wrote:
> > Date: Wed, 7 May 2008 12:01:38 +0800
> > From: "yu jie" <yujie052@gmail.com>
> > 
> >     The current CVS header version hangs when openning the attached file.

> No, it doesn't hang, it just takes a lot of time to visit this file.
> I measured 61 seconds on a 3GHz machine.  This file has 86406 lines,
> and uses some pretty non-standard formatting, such as this one:

The problem was that c-neutralize-syntax-in-CPP was inefficiently coded.
I've optimised it using essentially only Emacs primitives in the defun's
main loop.  It now runs almost 2 orders of magnitude faster.

Eli, I'd appreciate it very much indeed if you could review this new
code, please - earlier versions of it were peculiarly troublesome.
Thanks!

Here's the patch:


*** cc-mode.el~	2008-05-23 20:42:45.653994480 +0000
--- cc-mode.el	2008-05-23 20:50:43.941283760 +0000
***************
*** 837,863 ****
    ;;
    ;; This function is the C/C++/ObjC value of `c-before-font-lock-function'.
    ;;
    ;; This function might do invisible changes.
!   (c-save-buffer-state (limits mbeg+1 beg end)
!     ;; First calculate the region, possibly to be extended.
!     (setq beg (min begg c-old-BOM))
      (goto-char endd)
!     (when (c-beginning-of-macro)
!       (c-end-of-macro))
      (setq end (max (+ (- c-old-EOM old-len) (- endd begg))
  		   (point)))
      ;; Clear all old punctuation properties
      (c-clear-char-property-with-value beg end 'syntax-table '(1))
  
      (goto-char beg)
      (while (and (< (point) end)
  		(search-forward-regexp c-anchored-cpp-prefix end t))
        ;; If we've found a "#" inside a string/comment, ignore it.
!       (if (setq limits (c-literal-limits))
! 	  (goto-char (cdr limits))
  	(setq mbeg+1 (point))
  	(c-end-of-macro)	  ; Do we need to go forward 1 char here?  No!
! 	(c-neutralize-CPP-line mbeg+1 (point))))))
  
  (defun c-before-change (beg end)
    ;; Function to be put on `before-change-function'.  Primarily, this calls
--- 837,881 ----
    ;;
    ;; This function is the C/C++/ObjC value of `c-before-font-lock-function'.
    ;;
+   ;; Note: SPEED _MATTERS_ IN THIS FUNCTION!!!
+   ;; 
    ;; This function might do invisible changes.
!   (c-save-buffer-state (limits mbeg+1 beg end pps-position pps-state)
!     ;; First determine the region, (beg end), which may need "neutralizing".
!     ;; This may not start inside a string or comment, or a macro.
!     (goto-char begg)
!     (if (setq limits (c-literal-limits))
! 	(goto-char (cdr limits)))   ; go forward out of any string or comment.
!     (c-beginning-of-macro)
!     (setq beg (min (point) c-old-BOM))
! 
      (goto-char endd)
!     (if (setq limits (c-literal-limits))
! 	(goto-char (car limits)))  ; go backward out of any string or comment.
!     (if (c-beginning-of-macro)
! 	(c-end-of-macro))
      (setq end (max (+ (- c-old-EOM old-len) (- endd begg))
  		   (point)))
+ 
      ;; Clear all old punctuation properties
      (c-clear-char-property-with-value beg end 'syntax-table '(1))
  
      (goto-char beg)
+     (setq pps-position beg  pps-state nil)
      (while (and (< (point) end)
  		(search-forward-regexp c-anchored-cpp-prefix end t))
        ;; If we've found a "#" inside a string/comment, ignore it.
!       (setq pps-state
! 	    (parse-partial-sexp pps-position (point) nil nil pps-state)
! 	    pps-position (point))
!       (unless (or (nth 3 pps-state)	; in a string?
! 		  (nth 4 pps-state))	; in a comment?
  	(setq mbeg+1 (point))
  	(c-end-of-macro)	  ; Do we need to go forward 1 char here?  No!
! 	(c-neutralize-CPP-line mbeg+1 (point))
! 	(setq pps-state
! 	      (parse-partial-sexp pps-position (point) nil nil pps-state)
! 	      pps-position (point))))))
  
  (defun c-before-change (beg end)
    ;; Function to be put on `before-change-function'.  Primarily, this calls


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: ntemacs hangs when openning the attached file

[Stefan Monnier]

>> >     The current CVS header version hangs when openning the attached file.

>> No, it doesn't hang, it just takes a lot of time to visit this file.
>> I measured 61 seconds on a 3GHz machine.  This file has 86406 lines,
>> and uses some pretty non-standard formatting, such as this one:

> The problem was that c-neutralize-syntax-in-CPP was inefficiently coded.
> I've optimised it using essentially only Emacs primitives in the defun's
> main loop.  It now runs almost 2 orders of magnitude faster.

Sounds good.

> Eli, I'd appreciate it very much indeed if you could review this new
> code, please - earlier versions of it were peculiarly troublesome.

Don't know about Eli.  But here's some comments:

> +   ;; Note: SPEED _MATTERS_ IN THIS FUNCTION!!!
> +   ;; 
>     ;; This function might do invisible changes.
                             ^^
                            make

> +     (setq pps-position beg  pps-state nil)

It would be a lot more lispy to explicitly let-bind pps-position and
pps-state here, rather than declare them earlier without initializing
them and then initializing them here.

>       (while (and (< (point) end)
>   		(search-forward-regexp c-anchored-cpp-prefix end t))
>         ;; If we've found a "#" inside a string/comment, ignore it.
> !       (setq pps-state
> ! 	    (parse-partial-sexp pps-position (point) nil nil pps-state)
> ! 	    pps-position (point))
> !       (unless (or (nth 3 pps-state)	; in a string?
> ! 		  (nth 4 pps-state))	; in a comment?
>   	(setq mbeg+1 (point))
>   	(c-end-of-macro)	  ; Do we need to go forward 1 char here?  No!
> ! 	(c-neutralize-CPP-line mbeg+1 (point))
> ! 	(setq pps-state
> ! 	      (parse-partial-sexp pps-position (point) nil nil pps-state)
> ! 	      pps-position (point))))))

I have the impression that this second call to parse-partial-sexp
is unnecessary.


        Stefan

Re: ntemacs hangs when openning the attached file

[Alan Mackenzie]

Hi, Stefan!

On Fri, May 23, 2008 at 05:36:34PM -0400, Stefan Monnier wrote:

> > The problem was that c-neutralize-syntax-in-CPP was inefficiently
> > coded.  I've optimised it using essentially only Emacs primitives in
> > the defun's main loop.  It now runs almost 2 orders of magnitude
> > faster.

> Sounds good.

> > Eli, I'd appreciate it very much indeed if you could review this new
> > code, please - earlier versions of it were peculiarly troublesome.

> Don't know about Eli.

That's OK.  You'll do instead.  ;-)  (Thanks!)

> But here's some comments:

> > +   ;; Note: SPEED _MATTERS_ IN THIS FUNCTION!!!
> > +   ;; 
> >     ;; This function might do invisible changes.
>                              ^^
>                             make

I hereby resign from my role as project linguistic pedant.  ;-)  That one
(and ~100 others) have been in the source for ~5 years, put there by
Martin.  How could I miss this??

> > +     (setq pps-position beg  pps-state nil)

> It would be a lot more lispy to explicitly let-bind pps-position and
> pps-state here, rather than declare them earlier without initializing
> them and then initializing them here.

I'd thought that that would just be an unnecessary extra `let'.  However,
having tried it, it does make the code a bit clearer.  So yes, thank you
- I'll be doing this more in the future.

> >       (while (and (< (point) end)
> >   		(search-forward-regexp c-anchored-cpp-prefix end t))
> >         ;; If we've found a "#" inside a string/comment, ignore it.
> > !       (setq pps-state
> > ! 	    (parse-partial-sexp pps-position (point) nil nil pps-state)
> > ! 	    pps-position (point))
> > !       (unless (or (nth 3 pps-state)	; in a string?
> > ! 		  (nth 4 pps-state))	; in a comment?
> >   	(setq mbeg+1 (point))
> >   	(c-end-of-macro)	  ; Do we need to go forward 1 char here?  No!
> > ! 	(c-neutralize-CPP-line mbeg+1 (point))
> > ! 	(setq pps-state
> > ! 	      (parse-partial-sexp pps-position (point) nil nil pps-state)
> > ! 	      pps-position (point))))))

> I have the impression that this second call to parse-partial-sexp is
> unnecessary.

DUH!!!  Of course!  Wake up, Alan!  That's the whole point of the call to
`c-neutralize-CPP-line', just above.  Thanks!

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: ntemacs hangs when openning the attached file

[Eli Zaretskii]

> Date: Fri, 23 May 2008 21:01:23 +0000
> Cc: yu jie <yujie052@gmail.com>, Stefan Monnier <monnier@iro.umontreal.ca>,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Eli, I'd appreciate it very much indeed if you could review this new
> code, please - earlier versions of it were peculiarly troublesome.

I have nothing to add to Stefan's comments, except thank you for
working on this.

Re: ntemacs hangs when openning the attached file

[Alan Mackenzie]

Hi, Eli!

On Sat, May 24, 2008 at 11:18:20AM +0300, Eli Zaretskii wrote:
> > Date: Fri, 23 May 2008 21:01:23 +0000
> > Cc: yu jie <yujie052@gmail.com>, Stefan Monnier <monnier@iro.umontreal.ca>,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>
> > 
> > Eli, I'd appreciate it very much indeed if you could review this new
> > code, please - earlier versions of it were peculiarly troublesome.

> I have nothing to add to Stefan's comments, except thank you for
> working on this.

OK, thanks!  I think it should be OK now, I've just committed the change
to the Emacs-22 branch and the trunk.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: ntemacs hangs when openning the attached file

[Richard M Stallman]

When a test case is 4 meg, would you please offer to send it to those
who could use it, rather than mailing it directly to this list?

Thank you for contributing.