filling bug in text-mode

filling bug in text-mode

[Werner LEMBERG]

[GNU Emacs 22.0.50.1 (i686-pc-linux-gnu, GTK+ Version 2.2.4) of
2005-06-27]


Consider the following text in text-mode:

----

It's not necessary to buy a new computer just for a DVD
drive...  Today, the most important part of a computer is the amount
of RAM -- the more, the better.  It's far more important than a fast
CPU.

----

Now set the column width to 70 chars and execute `fill-paragraph' --
the string `drive...' should be moved to the first line, since the
last period is followed by two spaces, but nothing happens.  Why?
This looks like a bug to me.


    Werner

Re: filling bug in text-mode

[Juri Linkov]

> Consider the following text in text-mode:
> ----
> It's not necessary to buy a new computer just for a DVD
> drive...  Today, the most important part of a computer is the amount
> of RAM -- the more, the better.  It's far more important than a fast
> CPU.
> ----
> Now set the column width to 70 chars and execute `fill-paragraph' --
> the string `drive...' should be moved to the first line, since the
> last period is followed by two spaces, but nothing happens.  Why?
> This looks like a bug to me.

I looked into this bug.  It is caused by `sentence-end' not matching
the sentence ending with three periods.  The second rule in `fill-nobreak-p':

     ;; Another approach to the same problem.
     (save-excursion
       (skip-chars-backward ". ")
       (and (looking-at "\\.")
     	    (not (looking-at (sentence-end)))))

tries to match the end of the sentence, but fails.

One solution is to fix the regexp in `sentence-end' to match three
periods.  However, this might have an undesirable effect in other places.
A more localized fix is to change the rule above to skip only one period
backward before trying to match the sentence end.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: filling bug in text-mode

[Richard M. Stallman]

    One solution is to fix the regexp in `sentence-end' to match three
    periods.

I think that is the correct fix, because the sentence commands
_should_ stop after three periods (when followed by two spaces
or a newline).  It is a bug if they don't stop there.

Re: filling bug in text-mode

[Juri Linkov]

>     One solution is to fix the regexp in `sentence-end' to match three
>     periods.
>
> I think that is the correct fix, because the sentence commands
> _should_ stop after three periods (when followed by two spaces
> or a newline).

Should the sentence commands stop also after three question marks???
And sentences can also end with three exclamation marks!!!

> It is a bug if they don't stop there.

This depends on the operational semantics of `sentence-end' rather
than its formal definition.  The current value of `sentence-end' allows
the sentence commands to process sentences with three periods correctly
except one place that caused this bug.  The use of `sentence-end' is
very questionable in this place.  Please, look at it again:

     (save-excursion
       (skip-chars-backward ". ")
       (and (looking-at "\\.")
     	    (not (looking-at (sentence-end)))))

Why only periods are skipped and matched by `looking-at' while
sentences can end also with `!' and `?' which are included in
`sentence-end' along with the period?

I'm inclined to fix this place rather than to change the default value
of `sentence-end'.  There are functions like `fill-delete-newlines'
that rely on its current default value, so changing the default value
of `sentence-end' will break such functions.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: filling bug in text-mode

[Richard M. Stallman]

    Should the sentence commands stop also after three question marks???
    And sentences can also end with three exclamation marks!!!

Yes, they can--and the sentence commands already do stop there.
I just tried this test case:

  foo???  bar!!!  lose.

Re: filling bug in text-mode

[Juri Linkov]

>     Should the sentence commands stop also after three question marks???
>     And sentences can also end with three exclamation marks!!!
>
> Yes, they can--and the sentence commands already do stop there.
> I just tried this test case:
>
>   foo???  bar!!!  lose.

That means there is nothing wrong with the current default value of
`sentence-end' (at least, in regard to the reported bug).

The patch below fixes the bug.  It assumes that more than one
successive periods should allow breaking even if they are followed
by just one space.

Index: lisp/textmodes/fill.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/textmodes/fill.el,v
retrieving revision 1.181
diff -c -r1.181 fill.el
*** lisp/textmodes/fill.el	9 Aug 2005 14:01:29 -0000	1.181
--- lisp/textmodes/fill.el	10 Oct 2005 06:11:59 -0000
***************
*** 349,355 ****
       ;; Another approach to the same problem.
       (save-excursion
         (skip-chars-backward ". ")
!        (and (looking-at "\\.")
       	    (not (looking-at (sentence-end)))))
       ;; Don't split a line if the rest would look like a new paragraph.
       (unless use-hard-newlines
--- 349,355 ----
       ;; Another approach to the same problem.
       (save-excursion
         (skip-chars-backward ". ")
!        (and (looking-at "\\.[^.]")
       	    (not (looking-at (sentence-end)))))
       ;; Don't split a line if the rest would look like a new paragraph.
       (unless use-hard-newlines

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: filling bug in text-mode

[Werner LEMBERG]

> The patch below fixes the bug.  It assumes that more than one
> successive periods should allow breaking even if they are followed
> by just one space.

Thanks!  Will you check this in?


    Werner

Re: filling bug in text-mode

[Richard M. Stallman]

    The patch below fixes the bug.  It assumes that more than one
    successive periods should allow breaking even if they are followed
    by just one space.

That's not correct--it SHOULD demand two spaces or a newline,
after three periods, just as it does after one period.

Re: filling bug in text-mode

[Werner LEMBERG]

>     The patch below fixes the bug.  It assumes that more than one
>     successive periods should allow breaking even if they are
>     followed by just one space.
> 
> That's not correct--it SHOULD demand two spaces or a newline, after
> three periods, just as it does after one period.

This was my first reaction too, but after some thinking I can't
imagine a situation where you *really* want to prevent a line break
after two or periods.  The very reason to suppress line breaking is
only to have a correct handling of abbreviated names like J. User.
IMHO we should consider the case `period/question mark/exclamation
mark + single space' as the exception which prevents a break, and
everything else should allow it.  Doing the opposite (this is,
always insist on two spaces after those punctuation marks) appears
unnatural to me.


    Werner

Re: filling bug in text-mode

[Richard M. Stallman]

    >     The patch below fixes the bug.  It assumes that more than one
    >     successive periods should allow breaking even if they are
    >     followed by just one space.
    > 
    > That's not correct--it SHOULD demand two spaces or a newline, after
    > three periods, just as it does after one period.

    This was my first reaction too, but after some thinking I can't
    imagine a situation where you *really* want to prevent a line break
    after two or periods.

We're talking about what sentence-end should match; that is to say,
where M-e should stop.

It should stop after an ellipsis if the ellipsis is followed by two
spaces or a newline.  An ellipsis followed by just one space is not
the end of a sentence.

As for filling, it does not break the line after a period
followed by one space, because doing so would make it appear
to be the end of a sentence.

Re: filling bug in text-mode

[Juri Linkov]

> We're talking about what sentence-end should match; that is to say,
> where M-e should stop.
>
> It should stop after an ellipsis if the ellipsis is followed by two
> spaces or a newline.  An ellipsis followed by just one space is not
> the end of a sentence.
>
> As for filling, it does not break the line after a period
> followed by one space, because doing so would make it appear
> to be the end of a sentence.

While looking at fixing this, I noticed that there are more cases
where `fill-nobreak-p' doesn't allow breaking at the end of a sentence
(where M-e stops).

For example, filling:

It's not necessary to buy a new computer just for a DVD
drive. . .  Today, the most important part of a computer is the amount
of RAM -- the more, the better.  It's far more important than a fast
CPU.

doesn't break at two spaces between `. . .' and `Today'.  Whereas the
expected result of filling would be:

It's not necessary to buy a new computer just for a DVD drive. . .
Today, the most important part of a computer is the amount of RAM --
the more, the better.  It's far more important than a fast CPU.

There is another example:

It's not necessary to buy a new computer just for a DVD
drive.  . . Today, the most important part of a computer is the amount
of RAM -- the more, the better.  It's far more important than a fast
CPU.

where filling breaks at wrong point, turning this paragraph into:

It's not necessary to buy a new computer just for a DVD drive.  . .
Today, the most important part of a computer is the amount of RAM --
the more, the better.  It's far more important than a fast CPU.

while the expected result should have breaking at two spaces:

It's not necessary to buy a new computer just for a DVD drive.
. . Today, the most important part of a computer is the amount of
RAM -- the more, the better.  It's far more important than a fast CPU.

The patch below fixes these cases as well as the original problem:

Index: lisp/textmodes/fill.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/textmodes/fill.el,v
retrieving revision 1.181
diff -c -r1.181 fill.el
*** lisp/textmodes/fill.el	9 Aug 2005 14:01:29 -0000	1.181
--- lisp/textmodes/fill.el	14 Oct 2005 11:38:09 -0000
***************
*** 344,356 ****
       ;; it at the end of the line.
       (and sentence-end-double-space
  	  (save-excursion
! 	    (skip-chars-backward ". ")
! 	    (looking-at "\\. \\([^ ]\\|$\\)")))
       ;; Another approach to the same problem.
       (save-excursion
!        (skip-chars-backward ". ")
!        (and (looking-at "\\.")
!      	    (not (looking-at (sentence-end)))))
       ;; Don't split a line if the rest would look like a new paragraph.
       (unless use-hard-newlines
         (save-excursion
--- 344,357 ----
       ;; it at the end of the line.
       (and sentence-end-double-space
  	  (save-excursion
! 	    (skip-chars-backward " ")
! 	    (and (eq (preceding-char) ?.)
! 		 (looking-at " \\([^ ]\\|$\\)"))))
       ;; Another approach to the same problem.
       (save-excursion
!        (skip-chars-backward " ")
!        (and (eq (preceding-char) ?.)
! 	    (not (progn (forward-char -1) (looking-at (sentence-end))))))
       ;; Don't split a line if the rest would look like a new paragraph.
       (unless use-hard-newlines
         (save-excursion

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: filling bug in text-mode

[David Kastrup]

"Richard M. Stallman" <rms@gnu.org> writes:

>     The patch below fixes the bug.  It assumes that more than one
>     successive periods should allow breaking even if they are followed
>     by just one space.
>
> That's not correct--it SHOULD demand two spaces or a newline,
> after three periods, just as it does after one period.

Why?  A single period can signify a part of an abbreviation, like in
Richard M. Stallman, but I can't see three periods doing the same.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: filling bug in text-mode

[Stefan Monnier]

>> The patch below fixes the bug.  It assumes that more than one
>> successive periods should allow breaking even if they are followed
>> by just one space.
>> 
>> That's not correct--it SHOULD demand two spaces or a newline,
>> after three periods, just as it does after one period.

> Why?  A single period can signify a part of an abbreviation, like in
> Richard M. Stallman, but I can't see three periods doing the same.

IIRC the problem is what to do when filling

    foo bar...
    baz

Should it turn into

    foo bar... baz
or
    foo bar...  baz

By making "bar... baz" unbreakable we make sure that we can always choose
the second option.


        Stefan

Re: filling bug in text-mode

[Richard M. Stallman]

    IIRC the problem is what to do when filling

	foo bar...
	baz

    Should it turn into

	foo bar... baz
    or
	foo bar...  baz

It should be the latter, because that preserves the fact that it
counts as a sentence end.