bug#25685: fill-paragraph vs. \n vs. Chinese / English boundaries

bug#25685: fill-paragraph vs. \n vs. Chinese / English boundaries

[積丹尼 Dan Jacobson]

Big problem!
Put the cursor upon the following paragraph,

We proceeded to dig up 些 some Canna edulis Ker 食用美人蕉
when surprise surprise...

and do

M-h (translated from <escape> h) runs the command mark-paragraph
(found in global-map), which is an interactive compiled Lisp function
in ‘paragraphs.el’.

then

M-q (translated from <escape> q) runs the command fill-paragraph
(found in global-map), which is an interactive compiled Lisp function
in ‘fill.el’.

It becomes

We proceeded to dig up 些 some Canna edulis Ker 食用美人蕉when
surprise surprise...

I.e., the "食用美人蕉when" are now stuck together, making our text
look very unprofessional.

You might argue that that's the way the ball bounces. But then I would
say what about "up 些 some", you didn't glue that together, and on
only one side (when removing "\n") either.

So what do I want? This:

We proceeded to dig up 些 some Canna edulis Ker 食用美人蕉 when
surprise surprise...

bug#25685: fill-paragraph vs. \n vs. Chinese / English boundaries

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

On Sat, 11 Feb 2017 08:22:19 +0800, 積丹尼 Dan Jacobson wrote:
> We proceeded to dig up 些 some Canna edulis Ker 食用美人蕉
> when surprise surprise...

> and do  M-h  then  M-q
> It becomes

> We proceeded to dig up 些 some Canna edulis Ker 食用美人蕉when
> surprise surprise...

> I.e., the "食用美人蕉when" are now stuck together, making our text
> look very unprofessional.

Fixing such things one by one manually is my routine, too.  But
I first tried:

* lisp/textmodes/fill.el (fill-delete-newlines):
Don't delete leading and trailing space from CJK word.

I'm not sure if it is the right solution for every case, though.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 592 bytes --]

--- fill.el~	2017-01-04 22:17:04.000000000 +0000
+++ fill.el	2017-02-12 23:57:42.946118200 +0000
@@ -494,8 +494,8 @@
 	    (replace-match (get-text-property (match-beginning 0) 'fill-space))
 	  (let ((prev (char-before (match-beginning 0)))
 		(next (following-char)))
-	    (if (and (or (aref (char-category-set next) ?|)
-			 (aref (char-category-set prev) ?|))
+	    (if (and (aref (char-category-set next) ?|)
+		     (aref (char-category-set prev) ?|)
 		     (or (aref fill-nospace-between-words-table next)
 			 (aref fill-nospace-between-words-table prev)))
 		(delete-char -1))))))

bug#25685: fill-paragraph vs. \n vs. Chinese / English boundaries

[Katsumi Yamaoka]

On Mon, 13 Feb 2017 09:22:46 +0900, Katsumi Yamaoka wrote:
> I first tried:

> * lisp/textmodes/fill.el (fill-delete-newlines):
> Don't delete leading and trailing space from CJK word.

In Japan some people write an English word in Japanese text with
spaces, and others do it with no space:

日本語と English の混在
日本語とEnglishの混在

For the later case, some word processor separates them with thin
spaces automatically, but Emacs doesn't (I'm not wrong, am I?).
So, I am the former.  Anyway, if the patch is applied, it would
embarrass the later people, i.e.:

寿限無寿限無五劫の擦り切れ海砂利水魚の水行末雲来末風来末
eating寝るplaceに住む処やぶら小路の藪柑子パイポパイポ...
↓
寿限無寿限無五劫の擦り切れ海砂利水魚の水行末雲来末風来末 eating寝る
placeに住む処やぶら小路の藪柑子パイポパイポ...

So, there may want to be a user option, of which the default value
doesn't change the present behavior.  Is there a good name for it?

fill-multilinguala-words-with-spaces
                  text?
fill-separate-words-with-spaces-in-multilingual-text
fill-words-with-spaces-in-multilingual-text

↓describe its meaning in doc string
fill-words-with-spaces
fill-paragraph-with-spaces ← not necessarily used for paragraph?

bug#25685: fill-paragraph vs. \n vs. Chinese / English boundaries

[Katsumi Yamaoka]

On Tue, 14 Feb 2017 08:13:19 +0900, Katsumi Yamaoka wrote:
> So, there may want to be a user option, of which the default value
> doesn't change the present behavior.  Is there a good name for it?

I've installed it in the Emacs master
<http://lists.gnu.org/archive/html/emacs-diffs/2017-02/msg00185.html>
as:
,----[ C-h v fill-separate-heterogeneous-words-with-space RET ]
| fill-separate-heterogeneous-words-with-space is a variable defined in ‘fill.el’.
| Its value is t
| Original value was nil
| 
| Documentation:
| Non-nil means that use a space to separate words of different kind.
| This will be done with a word in the end of a line and a word in the
| beginning of the next line when concatenating them for filling those
| lines.  Whether to use a space is up to how the words are categorized.
| 
| You can customize this variable.
| 
| This variable was introduced, or its default value was changed, in
| version 26.1 of Emacs.
| 
| [back]
`----