I seek QA for CPerl 5.2 (Emacs mode)

I seek QA for CPerl 5.2 (Emacs mode)

[Ilya Zakharevich]

  ilyaz.org/software/emacs

In my private classification, this release is pre-alpha.  However, I
still hope to promote it to "golden" without any change ;-).

Since it is a very extensive edit, bugs are possible.  Moreover, an
order of applying some heuristics changed (this is the principal
reason for the 3.5x..15x speedup I observed).  In the cases I tried,
this edit speeds up things; however, it might be that in some other
cases it makes things slower.

Thus I would like to hear about *ALL* changes to the worse comparing
to 5.0, both in correctness of operation, and in speed.  Your feedback
is very appreciated.

Changelog appended.  Thanks,
Ilya

;;;; After 5.0:
;;;  `cperl-add-tags-recurse-noxs-fullpath': new function (for -batch mode)

;;;; After 5.1:
;;;;;; Major edit.  Summary of most visible changes:

;;;;;; a) Multiple <<HERE per line allowed.
;;;;;; b) Handles multiline subroutine declaration headers (with comments).
;;;;;;    (The exception is `cperl-etags' - but it is not used in the rest
;;;;;;    of the mode.)
;;;;;; c) Fontifies multiline my/our declarations (even with comments,
;;;;;;    and with legacy `font-lock').
;;;;;; d) Major speedup of syntaxification, both immediate and postponed
;;;;;;    (3.5x to 15x [for different CPUs and versions of Emacs] on the
;;;;;;    huge real-life document I tested).
;;;;;; e) New bindings, edits to imenu.
;;;;;; f) "_" is made into word-char during fontification/syntaxification;
;;;;;;    some attempts to recognize non-word "_" during other operations too.
;;;;;; g) Detect bug in Emacs with `looking-at' inside `narrow' and bulk out.
;;;;;; h) autoload some more perldoc-related stuff
;;;;;; i) Some new convenience features: ISpell POD/HEREDOCs, narrow-to-HEREDOC
;;;;;; j) Attempt to incorporate XEmacs edits which reached me

;;;; Fine-grained changelog:
;;; `cperl-hook-after-change':	New configuration variable
;;; `cperl-vc-sccs-header':	Likewise
;;; `cperl-vc-sccs-header':	Likewise
;;; `cperl-vc-header-alist':	Default via two preceding variables
;;; `cperl-invalid-face':	Remove double quoting under XEmacs
;;;					(still needed under RMS 21.2)
;;; `cperl-tips':		Update URLs for resources
;;; `cperl-problems':		Likewise
;;; `cperl-praise':		Mention new features
;;; New C-c key bindings:	for `cperl-find-bad-style',
;;;	`cperl-pod-spell', `cperl-here-doc-spell', `cperl-narrow-to-here-doc',
;;;	`cperl-perdoc', `cperl-perldoc-at-point'
;;; CPerl Mode menu changes:	"Fix style by spaces", "Imenu on Perl Info"
;;;	moved, new submenu of Tools with Ispell entries and narrowing.
;;; `cperl-after-sub-regexp':	New defsubst
;;; `cperl-imenu--function-name-regexp-perl': Use `cperl-after-sub-regexp',
;;;				Allows heads up to head4
;;;				Allow "package;"
;;; `defun-prompt-regexp':	Use `cperl-after-sub-regexp',
;;; `paren-backwards-message':	??? Something for XEmacs???
;;; `cperl-mode':		Never auto-switch abbrev-mode off
;;;				Try to allow '_' be non-word char
;;;				Do not use `font-lock-unfontify-region-function' on XEmacs
;;;				Reset syntax cache on mode start
;;;				Support multiline facification (even
;;;				        on legacy `font-lock')
;;; `cperl-facemenu-add-face-function':	??? Some contributed code ???
;;; `cperl-after-change-function':	Since `font-lock' and `lazy-lock'
;;;         refuse to inform us whether the fontification is due to lazy
;;;         calling or due to edit to a buffer, install our own hook
;;;         (controlled by `cperl-hook-after-change')
;;; `cperl-electric-pod':	=cut may have been recognized as start
;;; `cperl-block-p':		Moved, updated for attributes
;;; `cperl-calculate-indent':	Try to allow '_' be non-word char
;;;				Support subs with attributes
;;; `cperl-where-am-i':		Queit (?) a warning
;;; `cperl-cached-syntax-table'	New function
;;; `cperl-forward-re':		Use `cperl-cached-syntax-table'
;;; `cperl-unwind-to-safe':	Recognize `syntax-type' property
;;;					changing in a middle of line
;;; `cperl-find-sub-attrs':	New function
;;; `cperl-find-pods-heres':	Allow many <<EOP per line
;;;				Allow subs with attributes
;;;				Major speedups (3.5x..15x on a real-life
;;;				        test file nph-proxy.pl)
;;;				Recognize "extproc " (OS/2)
;;;				        case-folded and only at start
;;;				/x on s///x with empty replacement was
;;;				        not recognized
;;;				Better comments
;;; `cperl-after-block-p':	Remarks on diff with `cperl-block-p'
;;;				Allow subs with attributes, labels
;;;				Do not confuse "else::foo" with "else"
;;;				Minor optimizations...
;;; `cperl-after-expr-p':	Try to allow '_' be non-word char
;;; `cperl-fill-paragraph':	Try to detect a major bug in Emacs
;;;         with `looking-at' inside `narrow' and bulk out if found
;;; `cperl-imenu--create-perl-index':	Updates for new
;;;         `cperl-imenu--function-name-regexp-perl'
;;; `cperl-outline-level':	Likewise
;;; `cperl-init-faces':		Allow multiline subroutine headers
;;;         and my/our declarations, and ones with comments
;;;				Allow subroutine attributes
;;; `cperl-imenu-on-info':	Better docstring.
;;; `cperl-etags'		Rudimentary support for attributes
;;;				Support for packages and "package;"
;;; `cperl-add-tags-recurse-noxs':	Better (?) docstring
;;; `cperl-add-tags-recurse-noxs-fullpath': Likewise
;;; `cperl-tags-hier-init':	Misprint for `fboundp' fixed
;;; `cperl-not-bad-style-regexp':	Try to allow '_' be non-word char
;;; `cperl-perldoc':		Add autoload
;;; `cperl-perldoc-at-point':	Likewise
;;; `cperl-here-doc-spell':	New function
;;; `cperl-pod-spell':		Likewise
;;; `cperl-map-pods-heres':	Likewise
;;; `cperl-get-here-doc-region':	Likewise
;;; `cperl-font-lock-fontify-region-function': Likewise (backward compatibility
;;;					        for legacy `font-lock')
;;; `cperl-font-lock-unfontify-region-function': Fix style
;;; `cperl-fontify-syntaxically':	Recognize and optimize away
;;;         deferred calls with no-change.  Governed by `cperl-hook-after-change'
;;; `cperl-fontify-update':	Recognize that syntaxification region
;;;         can be larger than fontification one.
;;;         XXXX we leave `cperl-postpone' property, so this is quadratic...
;;; `cperl-fontify-update-bad':	Temporary placeholder until
;;;         it is clear how to implement `cperl-fontify-update'.
;;; `cperl-time-fontification':	New function
;;; `attrib-group':		New text attribute
;;; `multiline':		New value: `syntax-type' text attribute

;;;; After 5.2:
;;; `cperl-emulate-lazy-lock':	New function
;;; `cperl-fontify-syntaxically': Would skip large regions
;;; Add `cperl-time-fontification', `cperl-emulate-lazy-lock' to menu
;;; Some globals were declared, but uninitialized

Re: I seek QA for CPerl 5.2 (Emacs mode)

[Randy W. Sims]

Ilya Zakharevich wrote:
>   ilyaz.org/software/emacs
> 
> In my private classification, this release is pre-alpha.  However, I
> still hope to promote it to "golden" without any change ;-).
> 
> Since it is a very extensive edit, bugs are possible.  Moreover, an
> order of applying some heuristics changed (this is the principal
> reason for the 3.5x..15x speedup I observed).  In the cases I tried,
> this edit speeds up things; however, it might be that in some other
> cases it makes things slower.
> 
> Thus I would like to hear about *ALL* changes to the worse comparing
> to 5.0, both in correctness of operation, and in speed.  Your feedback
> is very appreciated.

I notice an incredible speed up when loading/fontifying largish modules.

One wierd problem I find is if I load two copies of a certain file it 
becomes develishly slow to do anything, even scrolling. This doesn't 
seem to happen on all files; not sure if it's size or content related yet.

Eg, If I unpack the Module::Build distribution into two different 
directories, then load copies of Module/Build/Base.pm from each 
location, the second buffer becomes slow. The original buffer and any 
others are not affected. If I close both buffers, then open the same 
file again, the new buffer is also slow-I must exit emacs and restart 
before I can work with that particular file because of slowdown.

This is on Debian (testing) with GNU Emacs 21.4.1.

Other than that one problem, the new version is great. Like I said the 
speedup when loading/fontifying is amazing.

Regards,
Randy.

Re: I seek QA for CPerl 5.2 (Emacs mode)

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Randy W. Sims
<RandyS@thepierianspring.org>], who wrote in article <435556D7.7030303@thepierianspring.org>:

> One wierd problem I find is if I load two copies of a certain file it 
> becomes develishly slow to do anything, even scrolling. This doesn't 
> seem to happen on all files; not sure if it's size or content related yet.
> 
> Eg, If I unpack the Module::Build distribution into two different 
> directories, then load copies of Module/Build/Base.pm from each 
> location, the second buffer becomes slow. The original buffer and any 
> others are not affected. If I close both buffers, then open the same 
> file again, the new buffer is also slow-I must exit emacs and restart 
> before I can work with that particular file because of slowdown.

> This is on Debian (testing) with GNU Emacs 21.4.1.

Confirmed on 21.2.  This effect disappears if I switch to an older
version of font-lock (e.g., one on
  ilyaz.org/software/tmp/font-lock-old.zip
).  One does not need to load the same file or a large file; two files
of size about 80K show the same slowdown in
fontification-by-keywords.  (I used Build.pm and Build/Base.pm.)

Go to the ends of both buffers, then start to scroll back - it is
painfully slow in one of the buffers...

I have no idea how to debug this...  My conjecture is that newer
Font-lock somehow starts to think that it needs to fontify the full
buffer instead of doing only the visible part...

Yours,
Ilya

Bug in font-lock? Was: QA for CPerl 5.2

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Randy W. Sims
<RandyS@thepierianspring.org>], who wrote in article <435556D7.7030303@thepierianspring.org>:
> One wierd problem I find is if I load two copies of a certain file it 
> becomes develishly slow to do anything, even scrolling. This doesn't 
> seem to happen on all files; not sure if it's size or content related yet.

I suspect it is a bug in font-lock.el: in the copy I have
`font-lock-fontify-anchored-keywords' uses the variable `limit' to set
text-property 'font-lock-multiline.  I would think it should have used
not this estimate from above, but the actual value of (point) after
doing the `while' loop for this purpose.

This bug may cause a major slow-down, although I do not know why it
shows only with two buffers in CPerl mode...

It is hard to work-around this bug; one needs a way to calculate the
limit in advance, which may be tricky.  I need to think more about
this...

Thanks,
Ilya

Re: I seek QA for CPerl 5.2 (Emacs mode)

[Randy W. Sims]

Ilya Zakharevich wrote:
> [A complimentary Cc of this posting was sent to
> Randy W. Sims
> <RandyS@thepierianspring.org>], who wrote in article <435556D7.7030303@thepierianspring.org>:
> 
>> One wierd problem I find is if I load two copies of a certain file it 
>> becomes develishly slow to do anything, even scrolling. This doesn't 
>> seem to happen on all files; not sure if it's size or content related yet.
>>
>> Eg, If I unpack the Module::Build distribution into two different 
>> directories, then load copies of Module/Build/Base.pm from each 
>> location, the second buffer becomes slow. The original buffer and any 
>> others are not affected. If I close both buffers, then open the same 
>> file again, the new buffer is also slow-I must exit emacs and restart 
>> before I can work with that particular file because of slowdown.
> 
>> This is on Debian (testing) with GNU Emacs 21.4.1.
> 
> Confirmed on 21.2.  This effect disappears if I switch to an older
> version of font-lock (e.g., one on
>   ilyaz.org/software/tmp/font-lock-old.zip
> ).  One does not need to load the same file or a large file; two files
> of size about 80K show the same slowdown in
> fontification-by-keywords.  (I used Build.pm and Build/Base.pm.)
> 
> Go to the ends of both buffers, then start to scroll back - it is
> painfully slow in one of the buffers...
> 
> I have no idea how to debug this...  My conjecture is that newer
> Font-lock somehow starts to think that it needs to fontify the full
> buffer instead of doing only the visible part...

Unfortunately, I don't know much lisp, so I can't help there. I did just 
try it out on my Windows box with XEmacs 21.4.17, same emacs init file. 
I see no sign of the above problem. The font-lock.el here is:

;; Copyright (C) 1992-1995, 1997 Free Software Foundation, Inc.
;; Copyright (C) 1995 Amdahl Corporation.
;; Copyright (C) 1996, 2000, 2001 Ben Wing.

;; Author: Jamie Zawinski <jwz@jwz.org>, for the LISPM Preservation Society.
;; Minimally merged with FSF 19.34 by Barry Warsaw <bwarsaw@python.org>
;; Then (partially) synched with FSF 19.30, leading to:
;; Next Author: RMS
;; Next Author: Simon Marshall <simon@gnu.ai.mit.edu>
;; Latest XEmacs Author: Ben Wing
;; Maintainer: XEmacs Development Team
;; Keywords: languages, faces

Re: I seek QA for CPerl 5.2 (Emacs mode)

[Ilya Zakharevich]

[A complimentary Cc of this posting was sent to
Randy W. Sims
<RandyS@thepierianspring.org>], who wrote in article <435556D7.7030303@thepierianspring.org>:

> One wierd problem I find is if I load two copies of a certain file it 
> becomes develishly slow to do anything, even scrolling. This doesn't 
> seem to happen on all files; not sure if it's size or content related yet.

It looks like it is indeed a bug in font-lock.el.  At least I put a
workaround for this bug in v5.7, and the slowdown (apparently)
disappears; see

  ilyaz.org/software/emacs/

The workaround itself may slow things down, but I hope its effect will
not be noticable.

Thanks,
Ilya

Re: I seek QA for CPerl 5.2 (Emacs mode)

[Randy W. Sims]

Ilya Zakharevich wrote:
> [A complimentary Cc of this posting was sent to
> Randy W. Sims
> <RandyS@thepierianspring.org>], who wrote in article <435556D7.7030303@thepierianspring.org>:
> 
> 
>>One wierd problem I find is if I load two copies of a certain file it 
>>becomes develishly slow to do anything, even scrolling. This doesn't 
>>seem to happen on all files; not sure if it's size or content related yet.
> 
> 
> It looks like it is indeed a bug in font-lock.el.  At least I put a
> workaround for this bug in v5.7, and the slowdown (apparently)
> disappears; see
> 
>   ilyaz.org/software/emacs/
> 
> The workaround itself may slow things down, but I hope its effect will
> not be noticable.

It seems to be working fine now. I'll try to exercise the package over 
the next few days to see if anything else pops up, but so far it looks 
great.

Thanks for your work on cperl-mode. I can't imagine coding without it.

Randy.