Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Bob Rogers]

   I found three such bugs.  To reproduce:

   1.  Copy the Lisp source below the "----" line into a local file such
as test-lisp-commment-fill.lisp, or anything else that ends in ".lisp".
(Don't follow those instructions, just cut-and-paste!)

   2.  "emacs -Q test-lisp-commment-fill.lisp"

First bug:

   3.  Move point anywhere on the line that says "More testing" and type
M-q.  Observe that nothing happens, when it should join up the next
line.  Typing M-q on the next line DTRT.

Second bug:

   4.  Move to any of the paragraphs in the second #|..| comment (though
only the one numbered 4 needs filling), and type M-q.  You will see a
"Search failed" error; afterwards, *Messages* has:

	fill-comment-paragraph: Search failed: "\\(\\(^\\|[^\\\\
	]\\)\\(\\\\\\\\\\)*\\)\\(;+\\|#|\\) *"

FWIW, the failing search is on line 942 of lisp/textmodes/fill.el, but
making the obvious attempt to fix it (supplying NOERROR true) doesn't
completely solve the problem.

Third bug:

   5.  Add a ";" comment line at the end of the file; a single ";" on
the last line will do.

   6.  Move point to anywhere in the *second* line of the paragraph
numbered 4, and type M-q.  Not only is this paragraph not filled, but
nearly all of the comment text is duplicated several times!

   I can reproduce all of these quite readily in CVS Emacs updated last
night, but none of them fail in Emacs 21.3 (SuSE GNU/Linux 9.0).

   I am hoping the solution(s) will be obvious to someone more familiar
with this code than I.  If I don't hear anything, I will look into it
further.  (Clues gratefully accepted.  ;-)

   TIA,

					-- Bob Rogers
					   http://rgrjr.dyndns.org/

------------------------------------------------------------------------
;;;; Testing M-q in lisp-mode.

(in-package :common-lisp-user)

#|

   More testing.  Foo bar.
Baz quux.

|#

#||

   1.  Start lisp (via M-x slime).

   2.  Compile and load the mini-compiler (C-c C-k).

   3.  Compile and load this file.

   4.  Invoke (recompile-everything) so that *functions-compiled*
and *functions-called* reflect the runtime.

||#

Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Bob Rogers]

   Oops; I forgot to include one:

Fourth bug:

   7.  Move the point to the line before the paragraph numbered 1, and
type M-q.  Instead of filling, this line is deleted, and the following
message is produced:

	fill-delete-newlines: Invalid search bound (wrong side of point)

This, too, did not exist in 21.3.

					-- Bob

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Lennart Borgman (gmail)]

Bob Rogers wrote:
>    Oops; I forgot to include one:
> 
> Fourth bug:
> 
>    7.  Move the point to the line before the paragraph numbered 1, and
> type M-q.  Instead of filling, this line is deleted, and the following
> message is produced:
> 
> 	fill-delete-newlines: Invalid search bound (wrong side of point)
> 
> This, too, did not exist in 21.3.

Hi Bob,

I think it would be helpful to those trying to fix the bug if you used 
the bug reporting command from the menus:

   Help - Send Bug Report...

That bug report will include date of your build etc.

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Chong Yidong]

Bob Rogers <rogers-emacs@rgrjr.dyndns.org> writes:

> First bug:
>
>    3.  Move point anywhere on the line that says "More testing" and type
> M-q.  Observe that nothing happens, when it should join up the next
> line.  Typing M-q on the next line DTRT.

I cannot reproduce this.  Typing M-q on the indicated line joins the
lines, as it should.  This is with latest CVS, GNU Emacs 22.0.96.29
(i686-pc-linux-gnu, GTK+ Version 2.10.6).

Maybe if you provided more details, such as the information provided
in M-x report-emacs-bug, someone could help you.

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Bob Rogers]

   From: Chong Yidong <cyd@stupidchicken.com>
   Date: Sat, 24 Mar 2007 16:04:43 -0400

   Bob Rogers <rogers-emacs@rgrjr.dyndns.org> writes:

   > First bug:
   >
   >    3.  Move point anywhere on the line that says "More testing" and type
   > M-q.  Observe that nothing happens, when it should join up the next
   > line.  Typing M-q on the next line DTRT.

   I cannot reproduce this.  Typing M-q on the indicated line joins the
   lines, as it should.  This is with latest CVS, GNU Emacs 22.0.96.29
   (i686-pc-linux-gnu, GTK+ Version 2.10.6).

   Maybe if you provided more details, such as the information provided
   in M-x report-emacs-bug, someone could help you.

See below.

					-- Bob

------------------------------------------------------------------------
In GNU Emacs 22.0.96.1 (i686-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2007-03-23 on rgrjr
Windowing system distributor `The XFree86 Project, Inc', version 11.0.40300001
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US
  locale-coding-system: iso-8859-1
  default-enable-multibyte-characters: t

Major mode: Lisp

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent input:
M-x r e p o <tab> r <tab> <return>

Recent messages:
test-lisp-commment-fill.lisp has auto save data; consider M-x recover-this-file
Loading vc-svn...done
Loading vc...
Loading easymenu...done
Loading vc...done
Making completion list...
Loading help-mode...done
Loading emacsbug...
Loading regexp-opt...done
Loading emacsbug...done

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[martin rudalics]

I suppose `comment-end' is "" in you buffer.  Then the problem is with
the following stretch of code in `fill-paragraph':

       (and fill-paragraph-handle-comment
	   ;; Our code only handles \n-terminated comments right now.
	   comment-start (equal comment-end "")
	   (let ((fill-paragraph-handle-comment nil))
	     (fill-comment-paragraph arg)))

We have to find something more intelligent here.

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Bob Rogers]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 1242 bytes --]

   From: martin rudalics <rudalics@gmx.at>
   Date: Sat, 24 Mar 2007 23:07:28 +0100

   I suppose `comment-end' is "" in you buffer.

That seems to be the case, at least initially.

   Then the problem is with the following stretch of code in
   `fill-paragraph':

	  (and fill-paragraph-handle-comment
	      ;; Our code only handles \n-terminated comments right now.
	      comment-start (equal comment-end "")
	      (let ((fill-paragraph-handle-comment nil))
		(fill-comment-paragraph arg)))

   We have to find something more intelligent here.

Is this really the place?  If I add a call to error just before the
fill-comment-paragraph call, none of the four bugs cause it to be
called.

   It might be better to recognize Lisp "#|...|#" comments explicitly at
a point where we know we're dealing with Lisp.  The attached patch is a
total hack, but it seems to relieve the symptoms of the first three bugs
(and changes that of the fourth).  This style of comment can be nested,
so the hack will get confused if still inside a comment but after an
inner comment.  The changed symptoms of the fourth bug suggest that the
bounds passed to the fill-region-as-paragraph call are wrong.  But I'll
bet someone else can do better.

					-- Bob


[-- Attachment #2: lisp-comment-fill-1.patch --]
[-- Type: text/plain, Size: 1619 bytes --]

Index: lisp/emacs-lisp/lisp-mode.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/emacs-lisp/lisp-mode.el,v
retrieving revision 1.200
diff -c -r1.200 lisp-mode.el
*** lisp/emacs-lisp/lisp-mode.el	5 Mar 2007 03:20:58 -0000	1.200
--- lisp/emacs-lisp/lisp-mode.el	25 Mar 2007 00:13:20 -0000
***************
*** 1246,1252 ****
  paragraph of it that point is in, preserving the comment's indentation
  and initial semicolons."
    (interactive "P")
!   (or (fill-comment-paragraph justify)
        ;; Since fill-comment-paragraph returned nil, that means we're not in
        ;; a comment: Point is on a program line; we are interested
        ;; particularly in docstring lines.
--- 1246,1266 ----
  paragraph of it that point is in, preserving the comment's indentation
  and initial semicolons."
    (interactive "P")
!   (or (if (save-excursion
! 	    (and (re-search-backward "#|\\||#" nil t)
! 		 (equal (match-string 0) "#|")))
! 	  ;; We are in a "#|...|#" comment.  [Assuming we haven't been
! 	  ;; fooled by quoted strings or ;-comments.]
! 	  (let ((start (save-excursion
! 			 (forward-paragraph -1)
! 			 (point)))
! 		(end (save-excursion
! 		       (forward-paragraph 1)
! 		       (point))))
! 	    (message "trying it")
! 	    (fill-region-as-paragraph start end justify))
! 	  ;; Try "normal" comment paragraph fill.
! 	  (fill-comment-paragraph justify))
        ;; Since fill-comment-paragraph returned nil, that means we're not in
        ;; a comment: Point is on a program line; we are interested
        ;; particularly in docstring lines.

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[martin rudalics]

[-- Attachment #1: Type: text/plain, Size: 1992 bytes --]

 >    Then the problem is with the following stretch of code in
 >    `fill-paragraph':
 >
 > 	  (and fill-paragraph-handle-comment
 > 	      ;; Our code only handles \n-terminated comments right now.
 > 	      comment-start (equal comment-end "")
 > 	      (let ((fill-paragraph-handle-comment nil))
 > 		(fill-comment-paragraph arg)))
 >
 >    We have to find something more intelligent here.
 >
 > Is this really the place?  If I add a call to error just before the
 > fill-comment-paragraph call, none of the four bugs cause it to be
 > called.

Correct.  But what I wanted to express was that `fill-comment-paragraph'
will get it wrong for a block comment.  The following part is used to
find the end of the region to fill by searching for `comment-re' at the
begin of every line which fails for block comments.

	     ;; Find the beginning of the first line past the region to fill.
	     (save-excursion
	       (while (progn (forward-line 1)
			     (looking-at comment-re)))

 >    It might be better to recognize Lisp "#|...|#" comments explicitly at
 > a point where we know we're dealing with Lisp.  The attached patch is a
 > total hack, but it seems to relieve the symptoms of the first three bugs
 > (and changes that of the fourth).  This style of comment can be nested,
 > so the hack will get confused if still inside a comment but after an
 > inner comment.  The changed symptoms of the fourth bug suggest that the
 > bounds passed to the fill-region-as-paragraph call are wrong.  But I'll
 > bet someone else can do better.

I'm afraid this problem is not restricted to the Lisp case.  I think we
should modify `fill-comment-paragraph' to

(1) check whether the current major mode allows comments that are not
terminated by a newline,

(2) if (1) applies parse whether we are in such a comment or a nested
comment, and

(3) call `fill-comment-paragraph' iff (2) doesn't apply.

Anyway please try the attached patch.  I'm confident Stefan will find
a better solution soon.

[-- Attachment #2: lisp-mode.patch --]
[-- Type: text/plain, Size: 1058 bytes --]

*** lisp-mode.el	Fri Mar  9 06:48:38 2007
--- lisp-mode.el	Sun Mar 25 12:12:38 2007
***************
*** 1246,1252 ****
  paragraph of it that point is in, preserving the comment's indentation
  and initial semicolons."
    (interactive "P")
!   (or (fill-comment-paragraph justify)
        ;; Since fill-comment-paragraph returned nil, that means we're not in
        ;; a comment: Point is on a program line; we are interested
        ;; particularly in docstring lines.
--- 1246,1258 ----
  paragraph of it that point is in, preserving the comment's indentation
  and initial semicolons."
    (interactive "P")
!   (or (and (equal major-mode 'lisp-mode)
! 	   (let ((state (syntax-ppss (point))))
! 	     (and (nth 4 state) (eq (nth 7 state) t)))
! 	   (let ((comment-end "|#")) 
! 	     (fill-paragraph justify)
! 	     t))
!       (fill-comment-paragraph justify)
        ;; Since fill-comment-paragraph returned nil, that means we're not in
        ;; a comment: Point is on a program line; we are interested
        ;; particularly in docstring lines.

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Bob Rogers]

   From: martin rudalics <rudalics@gmx.at>
   Date: Sun, 25 Mar 2007 12:18:17 +0200

   . . .

    >    It might be better to recognize Lisp "#|...|#" comments explicitly at
    > a point where we know we're dealing with Lisp.  The attached patch is a
    > total hack, but it seems to relieve the symptoms of the first three bugs
    > (and changes that of the fourth).  This style of comment can be nested,
    > so the hack will get confused if still inside a comment but after an
    > inner comment.  The changed symptoms of the fourth bug suggest that the
    > bounds passed to the fill-region-as-paragraph call are wrong.  But I'll
    > bet someone else can do better.

   I'm afraid this problem is not restricted to the Lisp case . . .

   Anyway please try the attached patch.  I'm confident Stefan will find
   a better solution soon.

It corrects all four syndromes for me; thanks.  (In my ignorance, I had
tried rebinding comment-end, but around a fill-comment-paragraph call,
which did nothing.)

					-- Bob

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Stefan Monnier]

>    I found three such bugs.  To reproduce:

Yes, the current code has several limitations in this rest.
Could you describe how each case behaves when compared to the behavior in
Emacs-21.4?


        Stefan

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Bob Rogers]

   From: Stefan Monnier <monnier@iro.umontreal.ca>
   Date: Sun, 25 Mar 2007 15:53:43 -0400

   >    I found three such bugs.  To reproduce:

   Yes, the current code has several limitations in this rest.
   Could you describe how each case behaves when compared to the behavior in
   Emacs-21.4?

	   Stefan

Sorry; I don't have 21.4 available to test.  Here is a comparison of
what happens in Emacs 21.3 and CVS Emacs:

   1.  Move point anywhere on the line that says "More testing" and type
M-q.

   21.3:  The paragraph is joined up into a single line.

   CVS:  Nothing happens.  (Typing M-q on the next line DTRT.)

   2.  Move to any of the paragraphs in the second #|..| comment (though
only the one numbered 4 needs filling), and type M-q.

   21.3:  The paragraphs are filled.  Only the fourth one needs it; when
filled, the "and" moves from the second line to the first.

   CVS:  "Search failed" error.

   3.  Add a ";" comment line at the end of the file, move point to
anywhere in the *second* line of the paragraph numbered 4, and type M-q.

   21.3:  The paragraph is filled, as before.

   CVS:  Not only is this paragraph not filled, but nearly all of the
comment text is duplicated several times!

   4.  Move the point to the line before the paragraph numbered 1, and
type M-q.

   21.3:  The text is not changed, but the buffer-modified flag is set.

   CVS:  Instead of filling, this line is deleted, and an "Invalid
search bound (wrong side of point)" error is signalled.

					-- Bob

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Stefan Monnier]

>    I found three such bugs.  To reproduce:

Does the patch below help?


        Stefan


*** newcomment.el	29 jan 2007 13:35:18 -0500	1.100
--- newcomment.el	27 mar 2007 22:56:20 -0400	
***************
*** 391,397 ****
  		 pt (or limit (point-max)) nil nil
  		 (list nil nil nil (nth 3 s) nil nil nil nil)
  		 t)))
!       (if (not (and (nth 8 s) (not (nth 3 s))))
  	  (unless noerror (error "No comment"))
  	;; We found the comment.
  	(let ((pos (point))
--- 391,399 ----
  		 pt (or limit (point-max)) nil nil
  		 (list nil nil nil (nth 3 s) nil nil nil nil)
  		 t)))
!       (if (or (not (and (nth 8 s) (not (nth 3 s))))
! 	      ;; Make sure the comment starts after PT.
! 	      (< (nth 8 s) pt))
  	  (unless noerror (error "No comment"))
  	;; We found the comment.
  	(let ((pos (point))

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Bob Rogers]

   From: Stefan Monnier <monnier@iro.umontreal.ca>
   Date: Tue, 27 Mar 2007 22:57:15 -0400

   >    I found three such bugs.  To reproduce:

   Does the patch below help?

	   Stefan

Yest; it fixes all the major symptoms.  The only thing that is still not
quite right is that filling the paragraph numbered 4 doesn't seem to
obey fill-column.  If I do "C-h v fill-column RET" in the lisp-mode
buffer in "emacs -Q", it tells me that the value is 70, but what I get
after filling is this:

       4.  Invoke (recompile-everything) so that *functions-compiled*
    and *functions-called* reflect the runtime.

Without the quoting indentation, the first line is 65 characters long.
What I expect is this:

       4.  Invoke (recompile-everything) so that *functions-compiled* and
    *functions-called* reflect the runtime.

where the first line is 69 chars long.  This may be a separate issue,
since I get the same behavior for paragraphs inside quoted strings, with
and without your patch.  However, emacs 21.3 does it the way I expect,
both in quoted strings and in #|...|# comments.

   Thanks for fixing this (and sorry for not responding sooner),

					-- Bob

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Stefan Monnier]

> Without the quoting indentation, the first line is 65 characters long.

I guess it has to do with emacs-lisp-docstring-fill-column.
Looks like a separate bug.

Maybe the patch below fixes it,


        Stefan


--- lisp-mode.el	05 mar 2007 21:15:36 -0500	1.200
+++ lisp-mode.el	31 mar 2007 16:29:27 -0400	
@@ -1280,7 +1280,8 @@
 				     "\\|\\s-*\\([(;:\"]\\|`(\\|#'(\\)"))
 	    (paragraph-separate
 	     (concat paragraph-separate "\\|\\s-*\".*[,\\.]$"))
-            (fill-column (if (integerp emacs-lisp-docstring-fill-column)
+            (fill-column (if (and (integerp emacs-lisp-docstring-fill-column)
+                                  (derived-mode-p 'emacs-lisp-mode))
                              emacs-lisp-docstring-fill-column
                            fill-column)))
 	(fill-paragraph justify))

Re: Multiple bugs in lisp-mode M-q on paragraphs within #|..|# comments

[Bob Rogers]

   From: Stefan Monnier <monnier@iro.umontreal.ca>
   Date: Sat, 31 Mar 2007 16:30:17 -0400

   > Without the quoting indentation, the first line is 65 characters long.

   I guess it has to do with emacs-lisp-docstring-fill-column.
   Looks like a separate bug.

   Maybe the patch below fixes it,

Works like a charm; thank you.

					-- Bob