Word wrap for non-whitespace-seperated language

Word wrap for non-whitespace-seperated language

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 1352 bytes --]

Emacs word wrap tries to only wrap at whitespace and tabs; that doesn’t work well when I mix, say, Chinese with English: since Chinese doesn’t have spaces between words, Emacs couldn’t find whitespaces to wrap a line, and it could happen that the line is wrapped very early, far from the right fringe. For example:

中英文混排中英文混排中英文混排中英文混排中英文混排中英文混排中英文混排中英文混排中英文混
排中英文混排中英文混排中英文混排 English English
中英文混排中英文混排中英文混排中英文混排中英文混排中英文混排中英文混排中英文混排中英文混

The ideal wrapping is

中英文混排中英文混排中英文混排中英文混排中英文混排中英文混排中英文混排中英文混排中英文混
排中英文混排中英文混排中英文混排 English English 中英文混排中英文混排中英文混排中英
文混排中英文混排中英文混排中英文混排中英文混排中英文混

It would be nice if word wrap knows when to wrap on whitespaces and when to simply wrap between characters. Is the wrapping feature implemented in redisplay engine? Or in lisp? The word-wrap variable is defined in buffer.c but I didn’t find the word wrap code in buffer.c. If it’s in lisp maybe I can hack on it.

Yuan



[-- Attachment #2: Type: text/html, Size: 2332 bytes --]

Re: Word wrap for non-whitespace-seperated language

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Wed, 4 Mar 2020 13:39:34 -0500
> 
> Is the wrapping feature implemented in redisplay engine?

Yes.

We have kinsoku.el, but it is used only for filling, not for word
wrap.

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

> On Mar 4, 2020, at 1:44 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Wed, 4 Mar 2020 13:39:34 -0500
>> 
>> Is the wrapping feature implemented in redisplay engine?
> 
> Yes.
> 
> We have kinsoku.el, but it is used only for filling, not for word
> wrap.

Thanks, where is it located? Is it deeply integrated?

Yuan

Re: Word wrap for non-whitespace-seperated language

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Wed, 4 Mar 2020 13:51:41 -0500
> Cc: emacs-devel@gnu.org
> 
> Thanks, where is it located? Is it deeply integrated?

You are asking about word wrap?  It's part of the basic layout code in
the display engine, search xdisp.c for WORD_WRAP.

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

> On Mar 4, 2020, at 2:16 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com>
>> Date: Wed, 4 Mar 2020 13:51:41 -0500
>> Cc: emacs-devel@gnu.org
>> 
>> Thanks, where is it located? Is it deeply integrated?
> 
> You are asking about word wrap?  It's part of the basic layout code in
> the display engine, search xdisp.c for WORD_WRAP.

Thanks, that’s what I meant. Seems to me that simply changing IT_DISPLAYING_WHITESPACE won’t give what I want, since I want the engine to wrap _anywhere_ between Chinese/Japanes/etc characters. Use fill.el and wrap with overlay might be a more tractable approach.

Yuan

Re: Word wrap for non-whitespace-seperated language

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Wed, 4 Mar 2020 15:34:45 -0500
> Cc: emacs-devel@gnu.org
> 
> > You are asking about word wrap?  It's part of the basic layout code in
> > the display engine, search xdisp.c for WORD_WRAP.
> 
> Thanks, that’s what I meant. Seems to me that simply changing IT_DISPLAYING_WHITESPACE won’t give what I want, since I want the engine to wrap _anywhere_ between Chinese/Japanes/etc characters. Use fill.el and wrap with overlay might be a more tractable approach.

AFAIK "anywhere" is not really correct, because kinsoku rules mandate
that some characters cannot appear at the beginning or end of a line.

I do agree that changing IT_DISPLAYING_WHITESPACE is not the right
approach for this, because it would make redisplay too slow.  We need
a smarter algorithm.

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

I wrote something that works. The goal is to have a word wrap effect for variable pitch words while breaking not only on white space characters but also between CJK characters. fill.el doesn’t work because it doesn’t work for variable pitch fonts, word-wrap doesn’t work because it can’t break between CJK characters. The key point is to find the place to break, my current method is forward char until the x position of point exceeds certain value (say 70*7 pixels). This has two flaws: first, forward char-by-char is very slow, although I added some simple optimizations, the lag is still noticeable even with small paragraphs. Second, I can’t get the position of point if it goes out of the window, in which case I need to recenter. 

Is there a better way to search forward for a point such that it’s x position is larger than some value? (Come think of it, is it possible to find a point in a window by x-y coordinate?) Thanks.

Yuan

RE: Word wrap for non-whitespace-seperated language

[Drew Adams]

> I wrote something that works. The goal is to have a word wrap effect for
> variable pitch words while breaking not only on white space characters but
> also between CJK characters. fill.el doesn’t work because it doesn’t work
> for variable pitch fonts, word-wrap doesn’t work because it can’t break
> between CJK characters. The key point is to find the place to break, my
> current method is forward char until the x position of point exceeds certain
> value (say 70*7 pixels). This has two flaws: first, forward char-by-char is
> very slow, although I added some simple optimizations, the lag is still
> noticeable even with small paragraphs. Second, I can’t get the position of
> point if it goes out of the window, in which case I need to recenter.
> 
> Is there a better way to search forward for a point such that it’s x
> position is larger than some value? (Come think of it, is it possible to
> find a point in a window by x-y coordinate?) Thanks.

I have no idea if it helps (apologies, if not),
but maybe take a look at library `find-where.el'.
It lets you find (forward or backward) the first
(or the Nth) buffer position where some predicate
holds.

---

https://www.emacswiki.org/emacs/FindWhere

https://www.emacswiki.org/emacs/download/find-where.el

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

> I have no idea if it helps (apologies, if not),
> but maybe take a look at library `find-where.el'.
> It lets you find (forward or backward) the first
> (or the Nth) buffer position where some predicate
> holds.

Thanks Adam. I actually find the way just now (by sheer luck). Now its reasonably fast.

For anyone that’s interested, here is the code. Set sfill-variale-pitch to t and M-x sfill-paragraph on a text paragraph, and see the magic happens ;-) It would be great if some one can have a look at it and give some suggestions, especially if it can be made faster. Next I want to try to make it automatic and only fills the visible portion just-in-time. Could some one point me to some library that does similar things? Thanks.


[-- Attachment #2: sfill.el --]
[-- Type: application/octet-stream, Size: 4243 bytes --]

;;; sfill.el --- Soft and smart fill      -*- lexical-binding: t; -*-

;; Author: Yuan Fu <casouri@gmail.com>

;;; This file is NOT part of GNU Emacs

;;; Commentary:
;;

;;; Code:
;;

(defvar-local sfill-column 70
  "Default fill Column for sfill.")

(defvar-local sfill-variable-pitch-column 70
  "Column used for variable pitch filling.")

(defvar-local sfill-variale-pitch nil
  "Set to non-nil and sfill will assume variable pitch when filling.")

(defface sfill-debug-face (let ((spec '(:inherit default))
                            (display t))
                        `((,display . ,spec)))
  "Face for highlighting sfill overlays."
  :group 'sfill)

(define-minor-mode sfill-debug-mode
  "Toggle debug mode for sfill."
  :lighter ""
  (if sfill-debug-mode
      (set-face-attribute 'sfill-debug-face nil :inherit 'highlight)
    (set-face-attribute 'sfill-debug-face nil :inherit 'default)))

(defun sfill-insert (string)
  "Insert STRING at point by overlay."
  ;; We shouldn’t need to break line at point-max.
  (if (or (eq (point) (point-max)))
      (error "Cannot insert at the end of visible buffer")
    (let* ((beg (point))
           (end (1+ (point)))
           (ov (make-overlay beg end)))
      (overlay-put ov 'sfill t)
      (overlay-put ov 'before-string string)
      (overlay-put ov 'evaporate t)
      (overlay-put ov 'face 'sfill-debug-face))))

(defun sfill-clear-overlay (beg end)
  "Clear overlays that `soft-insert' made between BEG and END."
  (let ((overlay-list (overlays-in beg end)))
    (dolist (ov overlay-list)
      (when (overlay-get ov 'sfill)
        (delete-overlay ov)))))

(defun sfill-clear-char (char beg end)
  "Remove CHAR(string) in the region from BEG to END."
  (save-excursion
    (goto-char beg)
    (while (re-search-forward char end t)
      ;; I can be more intelligent here, but since the break point
      ;; function is from fill.el, better keep in sync with it.
      ;; (see ‘fill-move-to-break-point’)
      (if (and (eq (char-charset (char-before (1- (point)))) 'ascii)
	       (eq (char-charset (char-after (point))) 'ascii))
          (replace-match " ")
        (replace-match "")))))

(defun sfill-forward-column (column)
  "Forward COLUMN columns."
  (while (>= column 0)
    (forward-char)
    (setq column (- column (char-width (char-before))))))

(defun sfill-forward-column-variable-pitch (column bound)
  "Forward COLUMN columns in variable pitch environment.
BOUND is point where we shouldn’t go beyond."
  ;; X offset from widow’s left edge in pixels. We want to break
  ;; around this position.
  (let* ((column-x-pos (* column (window-font-width)))
         (initial-y (cadr (pos-visible-in-window-p nil nil t))))
    (goto-char (min (posn-point (posn-at-x-y column-x-pos initial-y))
                    bound))))

(defun sfill-region (beg end)
  "Fill region between BEG and END. Assume mono space."
  (save-excursion
    (sfill-clear-overlay beg end)
    (sfill-clear-char "\n" beg end)
    (let (linebeg)
      (goto-char beg)
      (while (< (point) end)
        (setq linebeg (point))
        ;; We cannot use ‘move-to-column’ as fill.el does. Because we
        ;; break lines with overlays, so if we are at a fake newline,
        ;; ‘move-to-column’ doesn’t really go forward.
        (if sfill-variale-pitch
            (progn
              (unless (pos-visible-in-window-p)
                (recenter nil t))
              (sfill-forward-column-variable-pitch
               sfill-variable-pitch-column end))
          (sfill-forward-column sfill-column))
        ;; Check again if we are in the desired range.
        (when (< (point) end)
          (fill-move-to-break-point linebeg)
          (skip-chars-forward " \t")
          (sfill-insert "\n"))))))

(defun sfill-paragraph ()
  "Fill current paragraph."
  (interactive)
  (let (beg end)
    (save-excursion
      (backward-paragraph)
      (skip-chars-forward "\n")
      (setq beg (point))
      (forward-paragraph)
      (skip-chars-backward "\n")
      (setq end (point))
      (sfill-region beg end))))

(provide 'sfill)

;;; sfill.el ends here

[-- Attachment #3: Type: text/plain, Size: 6 bytes --]



Yuan

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

> For anyone that’s interested, here is the code. Set sfill-variale-pitch to t and M-x sfill-paragraph on a text paragraph, and see the magic happens ;-) It would be great if some one can have a look at it and give some suggestions, especially if it can be made faster. Next I want to try to make it automatic and only fills the visible portion just-in-time. Could some one point me to some library that does similar things? Thanks.

I feel like this can be added to font-lock with some tweak, let me see…

Yuan

Re: Word wrap for non-whitespace-seperated language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Would someone please make fill.el work for variable pitch fonts
and other things that vary from fixed columns?  It is not a small job
but it is vital for Emacs to advance in the area of formatting.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]


> On Mar 6, 2020, at 11:23 PM, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> Would someone please make fill.el work for variable pitch fonts
> and other things that vary from fixed columns?  It is not a small job
> but it is vital for Emacs to advance in the area of formatting.

I’ve got something mostly working, it provides on-the-fly word wrapping that supports variable pitch fonts and CJK characters. With minor changes it can be also used for filling but I don’t see the point for hard filling variable-pitch font text: they display differently in different fonts.

Here is a demo comparing the new wrapper (left) with word-wrap and fill-paragraph (right):


[-- Attachment #2: Pasted Graphic 3.png --]
[-- Type: image/png, Size: 711910 bytes --]

[-- Attachment #3: Type: text/plain, Size: 516 bytes --]



Now it works reasonably well in text-mode. But I haven’t give it a lot of test. You can M-x sfill-mode in any text buffer to use it.

One small issue is that it is not very fast when wrapping variable-pitch fonts. I think most of the time is spent around pos-visible-in-window-p and posn-at-x-y. I use them to find the rough place to break the line. Is there faster ways to to that? Basically I want to know the point at about certain column. Currently I’m using (posn-point (posn-at-x-y …)).

Yuan


[-- Attachment #4: sfill.el --]
[-- Type: application/octet-stream, Size: 10388 bytes --]

;;; sfill.el --- Soft and smart fill      -*- lexical-binding: t; -*-

;; Author: Yuan Fu <casouri@gmail.com>

;;; This file is NOT part of GNU Emacs

;;; Commentary:
;;
;; This package gives you word wrapping with more precision than the
;; default one. The default word wrapping (‘toggle-word-wrap’) can
;; only wrap on white spaces and tabs, thus is unable to wrap text
;; with both CJK characters and latin characters properly. Also it
;; can’t wrap on arbitrary columns. On the other hand,
;; ‘fill-paragraph’ can only work with mono spaced fonts, filling
;; variable pitch font usually gives sub-optimal result. (And, of
;; course, it destructively insert newlines, which may not be what you
;; want.)
;;
;; This package solves above problems. It wraps lines correctly no
;; matter the text is latin or CJK or both, and no matter it’s mono
;; spaces or variable pitch. It wraps on arbitrary columns and it
;; handles kinsoku correctly (thanks to kinsoku.el).
;;
;;   Usage
;;
;; 	M-x sfill-mode RET
;;
;;   Customization
;;
;; ‘sfill-column’.

;;; Code:
;;

(require 'subr-x)
;; Require solely for ‘buffer-face-mode’, so that we can guess we are
;; in variable pitch setting or mono space setting. This is necessary
;; because we can use a much faster function in mono space setting.
(require 'face-remap)

(defvar-local sfill-column 70
  "Fill Column for sfill.")

(defface sfill-debug-face (let ((spec '(:inherit default))
                                (display t))
                            `((,display . ,spec)))
  "Face for highlighting sfill overlays."
  :group 'sfill)

(define-minor-mode sfill-debug-mode
  "Toggle debug mode for sfill."
  :lighter ""
  :global t
  (if sfill-debug-mode
      (set-face-attribute 'sfill-debug-face nil :inherit 'highlight)
    (set-face-attribute 'sfill-debug-face nil :inherit 'default)))

(defun sfill-insert-newline ()
  "Insert newline at point by overlay."
  ;; We shouldn’t need to break line at point-max.
  (if (or (eq (point) (point-max)))
      (error "Cannot insert at the end of visible buffer")
    (let* ((beg (point))
           (end (1+ (point)))
           (ov (make-overlay beg end nil t)))
      (overlay-put ov 'sfill t)
      (overlay-put ov 'before-string "\n")
      (overlay-put ov 'evaporate t)
      (overlay-put ov 'face 'sfill-debug-face))))

(defun sfill-clear-overlay (beg end)
  "Clear overlays that `soft-insert' made between BEG and END."
  (let ((overlay-list (overlays-in beg end)))
    (dolist (ov overlay-list)
      (when (overlay-get ov 'sfill)
        (delete-overlay ov)))))

(defun sfill-delete-overlay-at (point)
  "Delete sfill overlay at POINT."
  (sfill-clear-overlay point (1+ point)))

(defun sfill-clear-newline (beg end)
  "Remove newlines in the region from BEG to END."
  (save-excursion
    (goto-char beg)
    (while (re-search-forward "\n" end t)
      ;; I can be more intelligent here, but since the break point
      ;; function is from fill.el, better keep in sync with it.
      ;; (see ‘fill-move-to-break-point’)
      (if (and (eq (char-charset (char-before (1- (point)))) 'ascii)
	       (eq (char-charset (char-after (point))) 'ascii))
          (replace-match " ")
        (replace-match "")))
    (put-text-property beg end 'sfill-bol nil)))

(defun sfill-forward-column (column)
  "Forward COLUMN columns.

This only works correctly in mono space setting."
  (condition-case nil
      (while (>= column 0)
        (forward-char)
        (setq column (- column (char-width (char-before)))))
    ('end-of-buffer nil)))

(defun sfill-move-to-column (column bound)
  "Go to COLUMN and return (point).

BOUND is point where we shouldn’t go beyond. So if the point at COLUMN
is beyond BOUND, stop at BOUND. If we go outside the visible portion of
the window before reaching BOUND, don’t move and return nil."
  ;; ‘column-x-pos’ is the x offset from widow’s left edge in pixels.
  ;; We want to break around this position.
  (when-let* ((column-x-pos (* column (window-font-width)))
              (initial-y (cadr (pos-visible-in-window-p nil nil t)))
              (point (posn-point (posn-at-x-y column-x-pos initial-y))))
    (if (eq point nil)
        nil
      (goto-char (min point bound)))))

(defun sfill-go-to-break-point (linebeg bound)
  "Move to the position where the line should be broken.
LINEBEG is the beginning of current visual line.
We don’t go beyond BOUND."
  (let ((break-point nil)
        (monop (not buffer-face-mode)))
    (if monop
        (sfill-forward-column sfill-column)
      (while (not break-point)
        (when (not (setq break-point
                         (sfill-move-to-column
                          sfill-column bound)))
          ;; If we moved out of the visible window,
          ;; ‘sfill-move-to-column’ returns nil. Recenter and try again.
          (recenter))))
    ;; If this (visual) line is the last line of the (visual) paragraph,
    ;; (point) would be equal to bound, and we want to stay there, so
    ;; that later we don’t insert newline incorrectly.
    (unless (>= (point) bound)
      (fill-move-to-break-point linebeg)
      (skip-chars-forward " \t"))))

(defsubst sfill-next-break (point bound)
  "Return the position of the first line break after POINT.
Don’t go beyond BOUND."
  (next-single-char-property-change
   (1+ point)
   'sfill nil bound))

(defsubst sfill-at-break (point)
  "Return non-nil if POINT is at a line break."
  (plist-get (text-properties-at point) 'sfill-bol))

(defsubst sfill-prev-break (point bound)
  "Return the position of the first line break before POINT.
Don’t go beyond BOUND."
  (1- (previous-single-char-property-change
       point 'sfill nil
       (1+ bound))))

(defun sfill-line (point &optional force)
  "Fill the line in where POINT is.
Return (BEG END) where the text is filled. BEG is the visual
beginning of current live. END is the actual end of line. If
FORCE is non-nil, update the whole line."
  (catch 'early-termination
    (save-window-excursion
      (save-excursion
        (if (eq point (point-max))
            (throw 'early-termination (cons point point)))
        (let* ((end (line-end-position))
               (prev-break (if (sfill-at-break point) point
                             (sfill-prev-break
                              point (line-beginning-position))))
               (prev-break (sfill-prev-break
                            prev-break (line-beginning-position)))
               next-existing-break
               (beg prev-break)
               (match-count 0))
          (goto-char beg)
          (while (< (point) end)
            (setq next-existing-break (sfill-next-break (point) end))
            (sfill-delete-overlay-at next-existing-break)
            (sfill-go-to-break-point (point) end)
            (unless (>= (point) end)
              (sfill-insert-newline))
            (if (eq next-existing-break (point))
                (setq match-count (1+ match-count)))
            (if (and (not force) (>= match-count 2))
                (throw 'early-termination (cons beg end))))
          (cons beg end))))))

;; Slightly faster but not completely correct
;;
;; (defun sfill-line (point &optional force)
;;   "Fill the line in where POINT is.
;; Return (BEG END) where the text is filled. BEG is the visual
;; beginning of current live. END is the actual end of line. If
;; FORCE is non-nil, update the whole line."
;;   (catch 'early-termination
;;     (save-window-excursion
;;       (save-excursion
;;         (if (eq point (point-max))
;;             (throw 'early-termination (cons point point)))
;;         (let* ((end (line-end-position))
;;                (prev-break (if (sfill-at-break point) point
;;                              (sfill-prev-break
;;                               point (line-beginning-position))))
;;                next-existing-break
;;                (beg prev-break))
;;           (goto-char beg)
;;           (while (< (point) end)
;;             (setq next-existing-break (sfill-next-break (point) end))
;;             (sfill-delete-overlay-at next-existing-break)
;;             (sfill-go-to-break-point (point) end)
;;             (unless (>= (point) end)
;;               (sfill-insert-newline))
;;             (if (and (not force) (eq next-existing-break (point)))
;;                 (throw 'early-termination (cons beg end))))
;;           (cons beg end))))))

(defun sfill-region (&optional beg end force)
  "Fill each line in the region from BEG to END.

If FORCE is non-nil, update the whole line. BEG and END default
to beginning and end of the buffer."
  (save-excursion
    (goto-char (or beg (point-min)))
    (sfill-line (point) force)
    (while (re-search-forward "\n" (or end (point-max)) t)
      (sfill-line (point) force))
    (cons (or beg (point-min)) (or end (point-max)))))

;; (defun sfill-paragraph ()
;;   "Fill current paragraph."
;;   (interactive)
;;   (let (beg end)
;;     (save-excursion
;;       (backward-paragraph)
;;       (skip-chars-forward "\n")
;;       (setq beg (point))
;;       (forward-paragraph)
;;       (skip-chars-backward "\n")
;;       (setq end (point))
;;       (sfill-region-destructive beg end))))

(defun sfill-unfill (&optional beg end)
  "Un-fill region from BEG to END, default to whole buffer."
  (sfill-clear-overlay (or beg (point-min)) (or end (point-max))))

(defun sfill-jit-lock-fn (beg end)
  "Fill line in region between BEG and END."
  (cons 'jit-lock-bounds (sfill-region beg end)))

(defvar sfill-mode-map (let ((map (make-sparse-keymap)))
                         ;; (define-key map (kbd "C-a") #'backward-sentence)
                         ;; (define-key map (kbd "C-e") #'forward-sentence)
                         map)
  "The keymap for minor mode ‘sfill-mode’.")

(define-minor-mode sfill-mode
  "Automatically wrap lines."
  :lighter ""
  :keymap 'sfill-mode-map
  (if sfill-mode
      (progn
        (jit-lock-register #'sfill-jit-lock-fn)
        (jit-lock-refontify))
    (jit-lock-unregister #'sfill-jit-lock-fn)
    (sfill-unfill)))

(provide 'sfill)

;;; sfill.el ends here

[-- Attachment #5: Type: text/plain, Size: 2 bytes --]

Re: Word wrap for non-whitespace-seperated language

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Sat, 7 Mar 2020 00:04:19 -0500
> Cc: eliz@gnu.org,
>  emacs-devel@gnu.org
> 
> > Would someone please make fill.el work for variable pitch fonts
> > and other things that vary from fixed columns?  It is not a small job
> > but it is vital for Emacs to advance in the area of formatting.
> 
> I’ve got something mostly working, it provides on-the-fly word wrapping that supports variable pitch fonts and CJK characters. With minor changes it can be also used for filling but I don’t see the point for hard filling variable-pitch font text: they display differently in different fonts.

Thank you for your work, but I don't think this is what Richard was
asking for.  A properly filled text according to what Richard wants
should have the right edge of the text almost completely justified to
the same pixel.

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

> Thank you for your work, but I don't think this is what Richard was
> asking for.  A properly filled text according to what Richard wants
> should have the right edge of the text almost completely justified to
> the same pixel.

I see, sfill.el is more wrapping than filling, maybe the name is misleading... Anyway it would be nice if sfill can be included in Emacs as it provides a nicer wrapping that is same as what you get in a WYSIWYG editor. It can be a (more capable but less efficient) alternative to the word-wrap feature. WDYT?

Btw I’ve sorted out my previous question about a faster way to get the point around column x. I seem to be incapable of solving a problem before asking it out loud in public, sorry for the noise. Now it is reasonably fast for daily use.

Yuan


[-- Attachment #2: sfill.el --]
[-- Type: application/octet-stream, Size: 9815 bytes --]

;;; sfill.el --- Soft and smart fill      -*- lexical-binding: t; -*-

;; Author: Yuan Fu <casouri@gmail.com>

;;; This file is NOT part of GNU Emacs

;;; Commentary:
;;
;; This package gives you word wrapping with more precision than the
;; default one. The default word wrapping (‘toggle-word-wrap’) can
;; only wrap on white spaces and tabs, thus is unable to wrap text
;; with both CJK characters and latin characters properly. Also it
;; can’t wrap on arbitrary columns. On the other hand,
;; ‘fill-paragraph’ can only work with mono spaced fonts, filling
;; variable pitch font usually gives sub-optimal result. (And, of
;; course, it destructively insert newlines, which may not be what you
;; want.)
;;
;; This package solves above problems. It wraps lines correctly no
;; matter the text is latin or CJK or both, and no matter it’s mono
;; spaces or variable pitch. It wraps on arbitrary columns and it
;; handles kinsoku correctly (thanks to kinsoku.el).
;;
;;   Usage
;;
;; 	M-x sfill-mode RET
;;
;;   Customization
;;
;; ‘sfill-column’.

;;; Code:
;;

(require 'subr-x)
(require 'cl-lib)

(defvar-local sfill-column 70
  "Fill Column for sfill.")

(defface sfill-debug-face (let ((spec '(:inherit highlight))
                                (display t))
                            `((,display . ,spec)))
  "Face for highlighting sfill overlays."
  :group 'sfill)

(define-minor-mode sfill-debug-mode
  "Toggle debug mode for sfill."
  :lighter ""
  :global t
  (sfill-region nil nil t))

(defun sfill-insert-newline ()
  "Insert newline at point by overlay."
  ;; We shouldn’t need to break line at point-max.
  (if (or (eq (point) (point-max)))
      (error "Cannot insert at the end of visible buffer")
    (let* ((beg (point))
           (end (1+ (point)))
           (ov (make-overlay beg end nil t)))
      (overlay-put ov 'sfill t)
      (overlay-put ov 'before-string "\n")
      (overlay-put ov 'evaporate t)
      (when sfill-debug-mode
        (overlay-put ov 'face 'sfill-debug-face)))))

(defun sfill-clear-overlay (beg end)
  "Clear overlays that `soft-insert' made between BEG and END."
  (let ((overlay-list (overlays-in beg end)))
    (dolist (ov overlay-list)
      (when (overlay-get ov 'sfill)
        (delete-overlay ov)))))

(defun sfill-delete-overlay-at (point)
  "Delete sfill overlay at POINT."
  (sfill-clear-overlay point (1+ point)))

(defun sfill-clear-newline (beg end)
  "Remove newlines in the region from BEG to END."
  (save-excursion
    (goto-char beg)
    (while (re-search-forward "\n" end t)
      ;; I can be more intelligent here, but since the break point
      ;; function is from fill.el, better keep in sync with it.
      ;; (see ‘fill-move-to-break-point’)
      (if (and (eq (char-charset (char-before (1- (point)))) 'ascii)
	       (eq (char-charset (char-after (point))) 'ascii))
          (replace-match " ")
        (replace-match "")))
    (put-text-property beg end 'sfill-bol nil)))

(defun sfill-forward-column (column)
  "Forward COLUMN columns.

This only works correctly in mono space setting."
  (condition-case nil
      (while (>= column 0)
        (forward-char)
        (setq column (- column (char-width (char-before)))))
    ('end-of-buffer nil)))

(defun sfill-forward-column-visual (column)
  "Forward COLUMN columns and return point.

Works for both mono space and variable pitch."
  ;; ‘column-x-pos’ is the x offset from widow’s left edge in pixels.
  ;; We want to break around this position.
  (condition-case nil
      (let ((column-x-pos (* column (window-font-width))))
        (while (>= column-x-pos 0)
          (forward-char)
          (setq column-x-pos
                (- column-x-pos
                   (car (mapcar (lambda (glyph) (aref glyph 4))
                                (font-get-glyphs (font-at (1- (point)))
                                                 (1- (point))(point)))))))
        (point))
    ('end-of-buffer (point))))

(defun sfill-go-to-break-point (linebeg bound)
  "Move to the position where the line should be broken.
LINEBEG is the beginning of current visual line.
We don’t go beyond BOUND."
  (if (display-multi-font-p)
      (sfill-forward-column-visual sfill-column)
    (sfill-forward-column sfill-column))
  (if (> (point) bound)
      (goto-char bound))
  ;; If this (visual) line is the last line of the (visual) paragraph,
  ;; (point) would be equal to bound, and we want to stay there, so
  ;; that later we don’t insert newline incorrectly.
  (unless (>= (point) bound)
    (fill-move-to-break-point linebeg)
    (skip-chars-forward " \t")))

(defsubst sfill-next-break (point bound)
  "Return the position of the first line break after POINT.
Don’t go beyond BOUND."
  (next-single-char-property-change
   (1+ point)
   'sfill nil bound))

(defsubst sfill-at-break (point)
  "Return non-nil if POINT is at a line break."
  (plist-get (text-properties-at point) 'sfill-bol))

(defsubst sfill-prev-break (point bound)
  "Return the position of the first line break before POINT.
Don’t go beyond BOUND."
  (1- (previous-single-char-property-change
       point 'sfill nil
       (1+ bound))))

(defun sfill-line (point &optional force)
  "Fill the line in where POINT is.
Return (BEG END) where the text is filled. BEG is the visual
beginning of current live. END is the actual end of line. If
FORCE is non-nil, update the whole line."
  (catch 'early-termination
    (save-window-excursion
      (save-excursion
        (if (eq point (point-max))
            (throw 'early-termination (cons point point)))
        (let* ((end (line-end-position))
               (prev-break (if (sfill-at-break point) point
                             (sfill-prev-break
                              point (line-beginning-position))))
               (prev-break (sfill-prev-break
                            prev-break (line-beginning-position)))
               next-existing-break
               (beg prev-break)
               (match-count 0))
          (goto-char beg)
          (while (< (point) end)
            (setq next-existing-break (sfill-next-break (point) end))
            (sfill-delete-overlay-at next-existing-break)
            (sfill-go-to-break-point (point) end)
            (unless (>= (point) end)
              (sfill-insert-newline))
            (if (eq next-existing-break (point))
                (setq match-count (1+ match-count)))
            (if (and (not force) (>= match-count 2))
                (throw 'early-termination (cons beg end))))
          (cons beg end))))))

;; Slightly faster but not completely correct
;;
;; (defun sfill-line (point &optional force)
;;   "Fill the line in where POINT is.
;; Return (BEG END) where the text is filled. BEG is the visual
;; beginning of current live. END is the actual end of line. If
;; FORCE is non-nil, update the whole line."
;;   (catch 'early-termination
;;     (save-window-excursion
;;       (save-excursion
;;         (if (eq point (point-max))
;;             (throw 'early-termination (cons point point)))
;;         (let* ((end (line-end-position))
;;                (prev-break (if (sfill-at-break point) point
;;                              (sfill-prev-break
;;                               point (line-beginning-position))))
;;                next-existing-break
;;                (beg prev-break))
;;           (goto-char beg)
;;           (while (< (point) end)
;;             (setq next-existing-break (sfill-next-break (point) end))
;;             (sfill-delete-overlay-at next-existing-break)
;;             (sfill-go-to-break-point (point) end)
;;             (unless (>= (point) end)
;;               (sfill-insert-newline))
;;             (if (and (not force) (eq next-existing-break (point)))
;;                 (throw 'early-termination (cons beg end))))
;;           (cons beg end))))))

(defun sfill-region (&optional beg end force)
  "Fill each line in the region from BEG to END.

If FORCE is non-nil, update the whole line. BEG and END default
to beginning and end of the buffer."
  (save-excursion
    (goto-char (or beg (point-min)))
    (sfill-line (point) force)
    (while (re-search-forward "\n" (or end (point-max)) t)
      (sfill-line (point) force))
    (cons (or beg (point-min)) (or end (point-max)))))

;; (defun sfill-paragraph ()
;;   "Fill current paragraph."
;;   (interactive)
;;   (let (beg end)
;;     (save-excursion
;;       (backward-paragraph)
;;       (skip-chars-forward "\n")
;;       (setq beg (point))
;;       (forward-paragraph)
;;       (skip-chars-backward "\n")
;;       (setq end (point))
;;       (sfill-region-destructive beg end))))

(defun sfill-unfill (&optional beg end)
  "Un-fill region from BEG to END, default to whole buffer."
  (sfill-clear-overlay (or beg (point-min)) (or end (point-max))))

(defun sfill-jit-lock-fn (beg end)
  "Fill line in region between BEG and END."
  (cons 'jit-lock-bounds (sfill-region beg end)))

(defvar sfill-mode-map (let ((map (make-sparse-keymap)))
                         ;; (define-key map (kbd "C-a") #'backward-sentence)
                         ;; (define-key map (kbd "C-e") #'forward-sentence)
                         map)
  "The keymap for minor mode ‘sfill-mode’.")

(define-minor-mode sfill-mode
  "Automatically wrap lines."
  :lighter ""
  :keymap 'sfill-mode-map
  (if sfill-mode
      (progn
        (jit-lock-register #'sfill-jit-lock-fn)
        (jit-lock-refontify))
    (jit-lock-unregister #'sfill-jit-lock-fn)
    (sfill-unfill)))

(provide 'sfill)

;;; sfill.el ends here

Re: Word wrap for non-whitespace-seperated language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Thank you for your work, but I don't think this is what Richard was
  > > asking for.  A properly filled text according to what Richard wants
  > > should have the right edge of the text almost completely justified to
  > > the same pixel.

Both styles are useful, so it would be good to support both.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Re: Word wrap for non-whitespace-seperated language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I’ve got something mostly working, it provides on-the-fly word
  > wrapping that supports variable pitch fonts and CJK
  > characters. With minor changes it can be also used for filling but
  > I don’t see the point for hard filling variable-pitch font text:
  > they display differently in different fonts.

Without truly filling the lines, do the commands C-a, C-n, etc work properly?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

> I’ve got something mostly working, it provides on-the-fly word
>> wrapping that supports variable pitch fonts and CJK
>> characters. With minor changes it can be also used for filling but
>> I don’t see the point for hard filling variable-pitch font text:
>> they display differently in different fonts.
> 
> Without truly filling the lines, do the commands C-a, C-n, etc work properly?

It depends on the commands: C-n/p works fine (given line-move-visual is t), but C-a/e moves to the end of buffer line, which is the beginning/end of the paragraph in a users eyes: same as what you get by default when Emacs wraps long lines.

Maybe I should call it “flywrap”, as it merely wraps lines visually (by overlay) rather than editing the buffer content. (It doesn’t justify either, as Eli pointed out.)

Yuan

Re: Word wrap for non-whitespace-seperated language

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Without truly filling the lines, do the commands C-a, C-n, etc work properly?

  > It depends on the commands: C-n/p works fine (given
  > line-move-visual is t), but C-a/e moves to the end of buffer line,
  > which is the beginning/end of the paragraph in a users eyes

I was afraid it would be like that.

I suppose that the rectangle commands don't work, and C-x = does
something unhelpful.  Is that so?  What about C-x TAB?

If all the commands that ought to make sense for such text
understand how the display shows the line-wrapping, and DTRT,
I suppose that will be as good as real wrapping.  Would you like to
work on that?




-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

> 
>>> Without truly filling the lines, do the commands C-a, C-n, etc work properly?
> 
>> It depends on the commands: C-n/p works fine (given
>> line-move-visual is t), but C-a/e moves to the end of buffer line,
>> which is the beginning/end of the paragraph in a users eyes
> 
> I was afraid it would be like that.
> 
> I suppose that the rectangle commands don't work, and C-x = does
> something unhelpful.  Is that so?  What about C-x TAB?
> 
> If all the commands that ought to make sense for such text
> understand how the display shows the line-wrapping, and DTRT,
> I suppose that will be as good as real wrapping.  Would you like to
> work on that?

For all the commands, the behavior of flywrap (I’m calling it flywrap now) should be identical to that of normally redisplay word wrapping: in both settings the text is just one line, but we display it across multiple lines on screen. E.g., C-x = works the same, C-x TAB only adds indentation to the beginning of the displayed “paragraph”, because that’s the beginning of the buffer line and there is really only one line. So in that regard flywrap is as good as redisplay wrapping. It’s much slower though (but fast enough to use).

I wrote two simple commands to go to the beginning/end of the displayed line, they are automatically bound when flywrap-mode is on.

Here is the latest version: 

Yuan


[-- Attachment #2: flywrap.el --]
[-- Type: application/octet-stream, Size: 11574 bytes --]

;;; flywrap.el --- Soft and smart fill      -*- lexical-binding: t; -*-

;; Author: Yuan Fu <casouri@gmail.com>

;;; This file is NOT part of GNU Emacs

;;; Commentary:
;;
;; This package gives you word wrapping with more precision than the
;; default one. The default word wrapping (‘toggle-word-wrap’) can
;; only wrap on white spaces and tabs, thus is unable to wrap text
;; with both CJK characters and latin characters properly. Also it
;; can’t wrap on arbitrary columns. On the other hand,
;; ‘fill-paragraph’ can only work with mono spaced fonts, filling
;; variable pitch font usually gives sub-optimal result. (And, of
;; course, it destructively insert newlines, which may not be what you
;; want.)
;;
;; This package solves above problems. It wraps lines correctly no
;; matter the text is latin or CJK or both, and no matter it’s mono
;; spaces or variable pitch. It wraps on arbitrary columns and it
;; handles kinsoku correctly (thanks to kinsoku.el).
;;
;;   Usage
;;
;; 	M-x flywrap-mode RET
;;
;;   Customization
;;
;; ‘flywrap-column’.

;;; Code:
;;

(require 'subr-x)
(require 'cl-lib)

(defvar-local flywrap-column 70
  "Fill Column for flywrap.")

;;; Backstage

(defface flywrap-debug-face (let ((spec '(:inherit highlight))
                                  (display t))
                              `((,display . ,spec)))
  "Face for highlighting flywrap overlays."
  :group 'flywrap)

(define-minor-mode flywrap-debug-mode
  "Toggle debug mode for flywrap."
  :lighter ""
  :global t
  (flywrap-region nil nil t))

(defun flywrap-insert-newline ()
  "Insert newline at point by overlay."
  ;; We shouldn’t need to break line at point-max.
  (if (or (eq (point) (point-max)))
      (error "Cannot insert at the end of visible buffer")
    (let* ((beg (point))
           (end (1+ (point)))
           (ov (make-overlay beg end nil t)))
      (overlay-put ov 'flywrap t)
      (overlay-put ov 'before-string "\n")
      (overlay-put ov 'evaporate t)
      (when flywrap-debug-mode
        (overlay-put ov 'face 'flywrap-debug-face)))))

(defun flywrap-clear-overlay (beg end)
  "Clear overlays that `soft-insert' made between BEG and END."
  (let ((overlay-list (overlays-in beg end)))
    (dolist (ov overlay-list)
      (when (overlay-get ov 'flywrap)
        (delete-overlay ov)))))

(defun flywrap-delete-overlay-at (point)
  "Delete flywrap overlay at POINT."
  (flywrap-clear-overlay point (1+ point)))

(defun flywrap-unfill (beg end)
  "Remove newlines in the region from BEG to END."
  (save-excursion
    (goto-char beg)
    (while (re-search-forward "\n" end t)
      (unless
          ;; If we are at point-max, ‘char-after’ returns nil.
          (eq (point) (point-max))
        ;; Regarding the 'ascii: I can be more intelligent here
        ;; (include iso-latin, etc), but since the break point function
        ;; is from fill.el, better keep in sync with it. (see
        ;; ‘fill-move-to-break-point’).
        ;; Don’t remove consecutive newlines.
        (cond ((or (eq (char-before (1- (point))) ?\n)
                   (eq (char-after (point)) ?\n))
               nil)
              ;; Separate ascii characters with space
              ((and (eq (char-charset (char-before (1- (point)))) 'ascii)
	            (eq (char-charset (char-after (point))) 'ascii))
               (replace-match " "))
              ;; Don’t separate CJK characters.
              (t (replace-match "")))))))

(defun flywrap-forward-column (column)
  "Forward COLUMN columns.

This only works correctly in mono space setting."
  (condition-case nil
      (while (>= column 0)
        (forward-char)
        (setq column (- column (char-width (char-before)))))
    ('end-of-buffer nil)))

(defun flywrap-forward-column-visual (column)
  "Forward COLUMN columns and return point.

Works for both mono space and variable pitch."
  ;; ‘column-x-pos’ is the x offset from widow’s left edge in pixels.
  ;; We want to break around this position.
  (condition-case nil
      (let ((column-x-pos (* column (window-font-width))))
        (while (>= column-x-pos 0)
          (forward-char)
          (unless (invisible-p (point))
            (setq column-x-pos
                  (- column-x-pos
                     (car (mapcar
                           (lambda (glyph) (aref glyph 4))
                           (font-get-glyphs (font-at (1- (point)))
                                            (1- (point))(point))))))))
        (point))
    ('end-of-buffer (point))))

(defun flywrap-go-to-break-point (linebeg bound)
  "Move to the position where the line should be broken.
LINEBEG is the beginning of current visual line.
We don’t go beyond BOUND."
  ;; Go to roughly the right place to break.
  (if (display-multi-font-p)
      (flywrap-forward-column-visual flywrap-column)
    (flywrap-forward-column flywrap-column))
  ;; If this (visual) line is the last line of the (visual) paragraph,
  ;; (point) would be equal to bound, and we want to stay there, so
  ;; that later we don’t insert newline incorrectly.
  (if (>= (point) bound)
      (goto-char bound))
  ;; Find the RIGHT place to break.
  (when (< (point) bound)
    (let ((fill-nobreak-invisible t))
      (fill-move-to-break-point linebeg))
    (skip-chars-forward " \t")))

(defsubst flywrap-next-break (point bound)
  "Return the position of the first line break after POINT.
Don’t go beyond BOUND."
  (if (eq point (point-max))
      point
    (next-single-char-property-change
     (1+ point)
     ;; If we pass a bound larger than point-max,
     ;; Emacs hangs.
     'flywrap nil (min bound (point-max)))))

(defsubst flywrap-at-break (point)
  "Return non-nil if POINT is at a line break."
  (plist-get (mapcan #'overlay-properties
                     (overlays-at point))
             'flywrap))

(defsubst flywrap-prev-break (point bound)
  "Return the position of the first line break before POINT.
Don’t go beyond BOUND."
  (max (1- (previous-single-char-property-change
            point 'flywrap nil
            (min (1+ bound) (poin-min))))
       (point-min)))

(defun flywrap-line (point &optional force)
  "Fill the line in where POINT is.
Return (BEG END) where the text is filled. BEG is the visual
beginning of current live. END is the actual end of line. If
FORCE is non-nil, update the whole line."
  (catch 'early-termination
    (save-window-excursion
      (save-excursion
        (if (eq point (point-max))
            (throw 'early-termination (cons point point)))
        (let* ((end (line-end-position))
               (prev-break (if (flywrap-at-break point) point
                             (flywrap-prev-break
                              point (line-beginning-position))))
               (prev-break (flywrap-prev-break
                            prev-break (line-beginning-position)))
               next-existing-break
               (beg prev-break)
               (match-count 0))
          (goto-char beg)
          (while (< (point) end)
            (setq next-existing-break (flywrap-next-break (point) end))
            (flywrap-delete-overlay-at next-existing-break)
            (flywrap-go-to-break-point (point) end)
            (unless (>= (point) end)
              (flywrap-insert-newline))
            (if (eq next-existing-break (point))
                (setq match-count (1+ match-count)))
            (if (and (not force) (>= match-count 2))
                (throw 'early-termination (cons beg end))))
          (cons beg end))))))

(defun flywrap-region (&optional beg end force)
  "Fill each line in the region from BEG to END.

If FORCE is non-nil, update the whole line. BEG and END default
to beginning and end of the buffer."
  (save-excursion
    (goto-char (or beg (point-min)))
    (flywrap-line (point) force)
    (while (re-search-forward "\n" (or end (point-max)) t)
      (flywrap-line (point) force))
    (cons (or beg (point-min)) (or end (point-max)))))

;; (defun flywrap-paragraph ()
;;   "Fill current paragraph."
;;   (interactive)
;;   (let (beg end)
;;     (save-excursion
;;       (backward-paragraph)
;;       (skip-chars-forward "\n")
;;       (setq beg (point))
;;       (forward-paragraph)
;;       (skip-chars-backward "\n")
;;       (setq end (point))
;;       (flywrap-region-destructive beg end))))

(defun flywrap-unwrap (&optional beg end)
  "Un-fill region from BEG to END, default to whole buffer."
  (flywrap-clear-overlay (or beg (point-min)) (or end (point-max))))

(defun flywrap-jit-lock-fn (beg end)
  "Fill line in region between BEG and END."
  (cons 'jit-lock-bounds (flywrap-region beg end)))

;; Currently not used.
(defun flywrap-after-change-fn (beg _ _)
  "Hook called after buffer content change.
See ‘after-change-functions’ for explanation on BEG END LEN."
  (flywrap-region beg (line-end-position))
  ;; (if (eq (- end beg) 1)
  ;;     ;; Self insert command, only wrap on space
  ;;     (when (member (char-after beg) '(?\s ?， ?。 ?、))
  ;;       (flywrap-region beg (line-end-position)))
  ;;   (flywrap-region beg (line-end-position)))
  )

;;; Userland

(defun flywrap-move-end-of-line (&optional arg)
  "Move to the end of current visual line.

With argument ARG not nil, move to the next ARG line end."
  (interactive "^p")
  (let ((arg (or arg 1))
        (point (point)))
    ;; This way hitting C-e at (visual) EOL doesn’t move to next line.
    (if (flywrap-at-break (1+ point))
        (setq point (1- point)))
    (while (> arg 0)
      (setq point (flywrap-next-break point (1+ (line-end-position))))
      (setq arg (1- arg)))
    (goto-char (1- point))))

(defun flywrap-move-beginning-of-line (arg)
  "Move to the beginning of current visual line.

With argument ARG not nil, move to the previous ARG line beginning."
  (interactive "^p")
  (let ((arg (or arg 1))
        (point (point)))
    ;; This way hitting C-a at (visual) BOL doesn’t move to previous line.
    (if (flywrap-at-break point)
        (setq point (+ 2 point)))
    (while (> arg 0)
      (setq point (flywrap-prev-break point (line-beginning-position)))
      (setq arg (1- arg)))
    (goto-char point)))

(defvar flywrap-mode-map
  (let ((map (make-sparse-keymap)))
    (define-key map (kbd "C-a") #'flywrap-move-beginning-of-line)
    (define-key map (kbd "C-e") #'flywrap-move-end-of-line)
    map)
  "The keymap for minor mode ‘flywrap-mode’.")

(define-minor-mode flywrap-mode
  "Automatically wrap lines."
  :lighter ""
  :keymap 'flywrap-mode-map
  (if flywrap-mode
      (progn
        ;; We want to control the depth of ‘flywrap-jit-lock-fn’ so it
        ;; runs hopefully after other functions. For example, let Org
        ;; mode’s fortifier to add invisible property (for links, etc)
        ;; before we wrap lines.
        (add-hook 'jit-lock-functions #'flywrap-jit-lock-fn 90 t)
        ;; Fix problem with incorrect wrapping when unfold a org
        ;; header.
        (advice-add #'org-flag-region :after
                    (lambda (from to _ _1) (when flywrap-mode
                                             (flywrap-region from to))))
        (jit-lock-refontify))
    (jit-lock-unregister #'flywrap-jit-lock-fn)
    (flywrap-unwrap)))

(provide 'flywrap)

;;; flywrap.el ends here

[-- Attachment #3: Type: text/plain, Size: 2 bytes --]

Re: Word wrap for non-whitespace-seperated language

[Eli Zaretskii]

> From: Yuan Fu <casouri@gmail.com>
> Date: Mon, 9 Mar 2020 11:44:51 -0400
> Cc: eliz@gnu.org,
>  emacs-devel@gnu.org
> 
> >>> Without truly filling the lines, do the commands C-a, C-n, etc work properly?
> > 
> >> It depends on the commands: C-n/p works fine (given
> >> line-move-visual is t), but C-a/e moves to the end of buffer line,
> >> which is the beginning/end of the paragraph in a users eyes
> > 
> > I was afraid it would be like that.
> > 
> > I suppose that the rectangle commands don't work, and C-x = does
> > something unhelpful.  Is that so?  What about C-x TAB?
> > 
> > If all the commands that ought to make sense for such text
> > understand how the display shows the line-wrapping, and DTRT,
> > I suppose that will be as good as real wrapping.  Would you like to
> > work on that?
> 
> For all the commands, the behavior of flywrap (I’m calling it flywrap now) should be identical to that of normally redisplay word wrapping:

No, with word-wrap C-a/C-e work as you'd expect: they move to the
beginning/end of the visual line, not the physical line.

Re: Word wrap for non-whitespace-seperated language

[Yuan Fu]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]



> On Mar 9, 2020, at 1:20 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Yuan Fu <casouri@gmail.com <mailto:casouri@gmail.com>>
>> Date: Mon, 9 Mar 2020 11:44:51 -0400
>> Cc: eliz@gnu.org <mailto:eliz@gnu.org>,
>> emacs-devel@gnu.org <mailto:emacs-devel@gnu.org>
>> 
>>>>> Without truly filling the lines, do the commands C-a, C-n, etc work properly?
>>> 
>>>> It depends on the commands: C-n/p works fine (given
>>>> line-move-visual is t), but C-a/e moves to the end of buffer line,
>>>> which is the beginning/end of the paragraph in a users eyes
>>> 
>>> I was afraid it would be like that.
>>> 
>>> I suppose that the rectangle commands don't work, and C-x = does
>>> something unhelpful.  Is that so?  What about C-x TAB?
>>> 
>>> If all the commands that ought to make sense for such text
>>> understand how the display shows the line-wrapping, and DTRT,
>>> I suppose that will be as good as real wrapping.  Would you like to
>>> work on that?
>> 
>> For all the commands, the behavior of flywrap (I’m calling it flywrap now) should be identical to that of normally redisplay word wrapping:
> 
> No, with word-wrap C-a/C-e work as you'd expect: they move to the
> beginning/end of the visual line, not the physical line.

Ah yes, I tried again and it DTRT.

Btw there is a typo in the previous file, here is the fixed one.

Yuan


[-- Attachment #2.1: Type: text/html, Size: 4329 bytes --]

[-- Attachment #2.2: flywrap.el --]
[-- Type: application/octet-stream, Size: 11575 bytes --]

;;; flywrap.el --- Soft and smart fill      -*- lexical-binding: t; -*-

;; Author: Yuan Fu <casouri@gmail.com>

;;; This file is NOT part of GNU Emacs

;;; Commentary:
;;
;; This package gives you word wrapping with more precision than the
;; default one. The default word wrapping (‘toggle-word-wrap’) can
;; only wrap on white spaces and tabs, thus is unable to wrap text
;; with both CJK characters and latin characters properly. Also it
;; can’t wrap on arbitrary columns. On the other hand,
;; ‘fill-paragraph’ can only work with mono spaced fonts, filling
;; variable pitch font usually gives sub-optimal result. (And, of
;; course, it destructively insert newlines, which may not be what you
;; want.)
;;
;; This package solves above problems. It wraps lines correctly no
;; matter the text is latin or CJK or both, and no matter it’s mono
;; spaces or variable pitch. It wraps on arbitrary columns and it
;; handles kinsoku correctly (thanks to kinsoku.el).
;;
;;   Usage
;;
;; 	M-x flywrap-mode RET
;;
;;   Customization
;;
;; ‘flywrap-column’.

;;; Code:
;;

(require 'subr-x)
(require 'cl-lib)

(defvar-local flywrap-column 70
  "Fill Column for flywrap.")

;;; Backstage

(defface flywrap-debug-face (let ((spec '(:inherit highlight))
                                  (display t))
                              `((,display . ,spec)))
  "Face for highlighting flywrap overlays."
  :group 'flywrap)

(define-minor-mode flywrap-debug-mode
  "Toggle debug mode for flywrap."
  :lighter ""
  :global t
  (flywrap-region nil nil t))

(defun flywrap-insert-newline ()
  "Insert newline at point by overlay."
  ;; We shouldn’t need to break line at point-max.
  (if (or (eq (point) (point-max)))
      (error "Cannot insert at the end of visible buffer")
    (let* ((beg (point))
           (end (1+ (point)))
           (ov (make-overlay beg end nil t)))
      (overlay-put ov 'flywrap t)
      (overlay-put ov 'before-string "\n")
      (overlay-put ov 'evaporate t)
      (when flywrap-debug-mode
        (overlay-put ov 'face 'flywrap-debug-face)))))

(defun flywrap-clear-overlay (beg end)
  "Clear overlays that `soft-insert' made between BEG and END."
  (let ((overlay-list (overlays-in beg end)))
    (dolist (ov overlay-list)
      (when (overlay-get ov 'flywrap)
        (delete-overlay ov)))))

(defun flywrap-delete-overlay-at (point)
  "Delete flywrap overlay at POINT."
  (flywrap-clear-overlay point (1+ point)))

(defun flywrap-unfill (beg end)
  "Remove newlines in the region from BEG to END."
  (save-excursion
    (goto-char beg)
    (while (re-search-forward "\n" end t)
      (unless
          ;; If we are at point-max, ‘char-after’ returns nil.
          (eq (point) (point-max))
        ;; Regarding the 'ascii: I can be more intelligent here
        ;; (include iso-latin, etc), but since the break point function
        ;; is from fill.el, better keep in sync with it. (see
        ;; ‘fill-move-to-break-point’).
        ;; Don’t remove consecutive newlines.
        (cond ((or (eq (char-before (1- (point))) ?\n)
                   (eq (char-after (point)) ?\n))
               nil)
              ;; Separate ascii characters with space
              ((and (eq (char-charset (char-before (1- (point)))) 'ascii)
	            (eq (char-charset (char-after (point))) 'ascii))
               (replace-match " "))
              ;; Don’t separate CJK characters.
              (t (replace-match "")))))))

(defun flywrap-forward-column (column)
  "Forward COLUMN columns.

This only works correctly in mono space setting."
  (condition-case nil
      (while (>= column 0)
        (forward-char)
        (setq column (- column (char-width (char-before)))))
    ('end-of-buffer nil)))

(defun flywrap-forward-column-visual (column)
  "Forward COLUMN columns and return point.

Works for both mono space and variable pitch."
  ;; ‘column-x-pos’ is the x offset from widow’s left edge in pixels.
  ;; We want to break around this position.
  (condition-case nil
      (let ((column-x-pos (* column (window-font-width))))
        (while (>= column-x-pos 0)
          (forward-char)
          (unless (invisible-p (point))
            (setq column-x-pos
                  (- column-x-pos
                     (car (mapcar
                           (lambda (glyph) (aref glyph 4))
                           (font-get-glyphs (font-at (1- (point)))
                                            (1- (point))(point))))))))
        (point))
    ('end-of-buffer (point))))

(defun flywrap-go-to-break-point (linebeg bound)
  "Move to the position where the line should be broken.
LINEBEG is the beginning of current visual line.
We don’t go beyond BOUND."
  ;; Go to roughly the right place to break.
  (if (display-multi-font-p)
      (flywrap-forward-column-visual flywrap-column)
    (flywrap-forward-column flywrap-column))
  ;; If this (visual) line is the last line of the (visual) paragraph,
  ;; (point) would be equal to bound, and we want to stay there, so
  ;; that later we don’t insert newline incorrectly.
  (if (>= (point) bound)
      (goto-char bound))
  ;; Find the RIGHT place to break.
  (when (< (point) bound)
    (let ((fill-nobreak-invisible t))
      (fill-move-to-break-point linebeg))
    (skip-chars-forward " \t")))

(defsubst flywrap-next-break (point bound)
  "Return the position of the first line break after POINT.
Don’t go beyond BOUND."
  (if (eq point (point-max))
      point
    (next-single-char-property-change
     (1+ point)
     ;; If we pass a bound larger than point-max,
     ;; Emacs hangs.
     'flywrap nil (min bound (point-max)))))

(defsubst flywrap-at-break (point)
  "Return non-nil if POINT is at a line break."
  (plist-get (mapcan #'overlay-properties
                     (overlays-at point))
             'flywrap))

(defsubst flywrap-prev-break (point bound)
  "Return the position of the first line break before POINT.
Don’t go beyond BOUND."
  (max (1- (previous-single-char-property-change
            point 'flywrap nil
            (min (1+ bound) (point-min))))
       (point-min)))

(defun flywrap-line (point &optional force)
  "Fill the line in where POINT is.
Return (BEG END) where the text is filled. BEG is the visual
beginning of current live. END is the actual end of line. If
FORCE is non-nil, update the whole line."
  (catch 'early-termination
    (save-window-excursion
      (save-excursion
        (if (eq point (point-max))
            (throw 'early-termination (cons point point)))
        (let* ((end (line-end-position))
               (prev-break (if (flywrap-at-break point) point
                             (flywrap-prev-break
                              point (line-beginning-position))))
               (prev-break (flywrap-prev-break
                            prev-break (line-beginning-position)))
               next-existing-break
               (beg prev-break)
               (match-count 0))
          (goto-char beg)
          (while (< (point) end)
            (setq next-existing-break (flywrap-next-break (point) end))
            (flywrap-delete-overlay-at next-existing-break)
            (flywrap-go-to-break-point (point) end)
            (unless (>= (point) end)
              (flywrap-insert-newline))
            (if (eq next-existing-break (point))
                (setq match-count (1+ match-count)))
            (if (and (not force) (>= match-count 2))
                (throw 'early-termination (cons beg end))))
          (cons beg end))))))

(defun flywrap-region (&optional beg end force)
  "Fill each line in the region from BEG to END.

If FORCE is non-nil, update the whole line. BEG and END default
to beginning and end of the buffer."
  (save-excursion
    (goto-char (or beg (point-min)))
    (flywrap-line (point) force)
    (while (re-search-forward "\n" (or end (point-max)) t)
      (flywrap-line (point) force))
    (cons (or beg (point-min)) (or end (point-max)))))

;; (defun flywrap-paragraph ()
;;   "Fill current paragraph."
;;   (interactive)
;;   (let (beg end)
;;     (save-excursion
;;       (backward-paragraph)
;;       (skip-chars-forward "\n")
;;       (setq beg (point))
;;       (forward-paragraph)
;;       (skip-chars-backward "\n")
;;       (setq end (point))
;;       (flywrap-region-destructive beg end))))

(defun flywrap-unwrap (&optional beg end)
  "Un-fill region from BEG to END, default to whole buffer."
  (flywrap-clear-overlay (or beg (point-min)) (or end (point-max))))

(defun flywrap-jit-lock-fn (beg end)
  "Fill line in region between BEG and END."
  (cons 'jit-lock-bounds (flywrap-region beg end)))

;; Currently not used.
(defun flywrap-after-change-fn (beg _ _)
  "Hook called after buffer content change.
See ‘after-change-functions’ for explanation on BEG END LEN."
  (flywrap-region beg (line-end-position))
  ;; (if (eq (- end beg) 1)
  ;;     ;; Self insert command, only wrap on space
  ;;     (when (member (char-after beg) '(?\s ?， ?。 ?、))
  ;;       (flywrap-region beg (line-end-position)))
  ;;   (flywrap-region beg (line-end-position)))
  )

;;; Userland

(defun flywrap-move-end-of-line (&optional arg)
  "Move to the end of current visual line.

With argument ARG not nil, move to the next ARG line end."
  (interactive "^p")
  (let ((arg (or arg 1))
        (point (point)))
    ;; This way hitting C-e at (visual) EOL doesn’t move to next line.
    (if (flywrap-at-break (1+ point))
        (setq point (1- point)))
    (while (> arg 0)
      (setq point (flywrap-next-break point (1+ (line-end-position))))
      (setq arg (1- arg)))
    (goto-char (1- point))))

(defun flywrap-move-beginning-of-line (arg)
  "Move to the beginning of current visual line.

With argument ARG not nil, move to the previous ARG line beginning."
  (interactive "^p")
  (let ((arg (or arg 1))
        (point (point)))
    ;; This way hitting C-a at (visual) BOL doesn’t move to previous line.
    (if (flywrap-at-break point)
        (setq point (+ 2 point)))
    (while (> arg 0)
      (setq point (flywrap-prev-break point (line-beginning-position)))
      (setq arg (1- arg)))
    (goto-char point)))

(defvar flywrap-mode-map
  (let ((map (make-sparse-keymap)))
    (define-key map (kbd "C-a") #'flywrap-move-beginning-of-line)
    (define-key map (kbd "C-e") #'flywrap-move-end-of-line)
    map)
  "The keymap for minor mode ‘flywrap-mode’.")

(define-minor-mode flywrap-mode
  "Automatically wrap lines."
  :lighter ""
  :keymap 'flywrap-mode-map
  (if flywrap-mode
      (progn
        ;; We want to control the depth of ‘flywrap-jit-lock-fn’ so it
        ;; runs hopefully after other functions. For example, let Org
        ;; mode’s fortifier to add invisible property (for links, etc)
        ;; before we wrap lines.
        (add-hook 'jit-lock-functions #'flywrap-jit-lock-fn 90 t)
        ;; Fix problem with incorrect wrapping when unfold a org
        ;; header.
        (advice-add #'org-flag-region :after
                    (lambda (from to _ _1) (when flywrap-mode
                                             (flywrap-region from to))))
        (jit-lock-refontify))
    (jit-lock-unregister #'flywrap-jit-lock-fn)
    (flywrap-unwrap)))

(provide 'flywrap)

;;; flywrap.el ends here

[-- Attachment #2.3: Type: text/html, Size: 226 bytes --]

Re: Word wrap for non-whitespace-seperated language

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Fri, 06 Mar 2020 23:23:00 -0500
> 
> Would someone please make fill.el work for variable pitch fonts
> and other things that vary from fixed columns?  It is not a small job
> but it is vital for Emacs to advance in the area of formatting.

I think shr.el already does that, see shr-fill-lines.  We just need to
lift that code from shr.el and use it in fill.el.

Lars, am I missing something?

Re: Word wrap for non-whitespace-seperated language

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Would someone please make fill.el work for variable pitch fonts
>> and other things that vary from fixed columns?  It is not a small job
>> but it is vital for Emacs to advance in the area of formatting.
>
> I think shr.el already does that, see shr-fill-lines.  We just need to
> lift that code from shr.el and use it in fill.el.
>
> Lars, am I missing something?

No, I thing that should be fairly straightforward to do.

I'm not quite sure it would be generally useful, though -- presumably
people who use variable-width fonts when writing text also set
`word-wrap' to t, because you can't really tell what font you're next
going to use when editing/looking at the text.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Word wrap for non-whitespace-seperated language

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rms@gnu.org,  casouri@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 07 Mar 2020 09:40:53 +0100
> 
> I'm not quite sure it would be generally useful, though -- presumably
> people who use variable-width fonts when writing text also set
> `word-wrap' to t, because you can't really tell what font you're next
> going to use when editing/looking at the text.

Well, it should be useful locally, I guess.  And perhaps enriched-mode
can also record the font (or we could augment it to do so), so that
this information could be passed to others.