bug#1808: 23.0.60; picture-mode not considering double-width characters alignment

bug#1808: 23.0.60; picture-mode not considering double-width characters alignment

[poppyer]

This is not a new bug of EMACS 23; but it is there in EMACS22 for a
long time.  In M-x picture-mode, emacs acts in a "replace" typing
mode, i.e. when you type a char, it replace the old one such that the
alignment is maintained.

But when mixing with double-width characters (e.g. CJK chars), one to
one char replacing become problematic, e.g. if we replace a
single-width char with a double-wdith char, the alignment will be
destroyed.

So, what I suggests is: if we replace a double-width to a
single-width, we should add a extra single-width space; and if we
replace a single-width to a double-witdh, we should check its
following char: if it is single-width, delete it; otherwise replace it
with a single-width space.



In GNU Emacs 23.0.60.1 (i386-apple-darwin9.6.0, NS apple-appkit-949.43)
 of 2008-12-25 on neutron.local
Windowing system distributor `Apple', version 97.112.112.108.101.45.97.112.112.107.105.116.45.57.52.57.46.52.51
configured using `configure  '--with-ns''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: zh_CN.UTF-8
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  value of $XMODIFIERS: @im=fcitx
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Group

Minor modes in effect:
  erc-log-mode: t
  erc-list-mode: t
  erc-menu-mode: t
  erc-autojoin-mode: t
  erc-ring-mode: t
  erc-networks-mode: t
  erc-pcomplete-mode: t
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-match-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-netsplit-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  gnus-undo-mode: t
  recentf-mode: t
  which-function-mode: t
  show-paren-mode: t
  mouse-sel-mode: t
  global-hl-line-mode: t
  pinbar-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  tool-bar-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p C-p 
9 RET C-n C-n SPC F C-\ x t u SPC g f SPC j e r r SPC 
w b SPC s h n SPC w d t SPC w f t c b SPC . RET y i 
SPC j SPC y u SPC d SPC 1 s o v SPC g n g DEL SPC 4 
s o v SPC u j SPC e SPC DEL e t SPC g SPC w h SPC A 
A P DEL DEL DEL k h t SPC m g SPC DEL m g j SPC DEL 
m g SPC DEL m h SPC t SPC g SPC w h SPC A P r a w k 
SPC g SPC g h SPC DEL w h SPC x g SPC ? DEL . RET RET 
ESC [ A ESC [ A C-e ESC [ D ESC [ D DEL t s SPC w y 
SPC u t h p SPC ESC [ C DEL ESC [ B C-x k C-g C-x k 
y q g C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
ESC < n RET SPC n SPC q ESC < ESC > ESC < ESC x n e 
w s C-p C-p RET U N o o o N o o o ESC x n e w s C-p 
C-p RET ESC 1 ESC < ESC > ESC < C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n 
C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n ESC x r 
e p o TAB r TAB RET

Recent messages:
Reading... done.
Preparing newsticker buffer...
Newsticker stopped!
Mark set [3 times]
Reading active file from gmail via nnimap...
nnimap: Checking mailboxes...done
Reading active file from freenews.netfront.net via nntp...
Reading active file from news.motzarella.org via nntp...
Checking new news...done
Making completion list...

bug#1808: 23.0.60; picture-mode not considering double-width characters alignment

[Alan J Third]

poppyer <poppyer@gmail.com> writes:

> This is not a new bug of EMACS 23; but it is there in EMACS22 for a
> long time.  In M-x picture-mode, emacs acts in a "replace" typing
> mode, i.e. when you type a char, it replace the old one such that the
> alignment is maintained.
>
> But when mixing with double-width characters (e.g. CJK chars), one to
> one char replacing become problematic, e.g. if we replace a
> single-width char with a double-wdith char, the alignment will be
> destroyed.

Sorry it's taken this long for someone to get back to you.

First, do you know if this is still a problem for you?

If so, how are you entering the CJK characters?

I find in Emacs 25 that if I try entering a greek alpha by typing:

C-x 8 RET GREEK SMALL LETTER ALPHA

It inserts the character, however when I bind the same character to a
key, eg.:

(global-set-key (kbd "C-c a") "α")

and enter it that way it works as expected.

Is this similar?

-- 
Alan Third

bug#1808: 23.0.60; picture-mode not considering double-width characters alignment

[Eli Zaretskii]

> From: Alan J Third <alan@idiocy.org>
> Date: Sat, 09 Jan 2016 22:43:36 +0000
> Cc: 1808@debbugs.gnu.org
> 
> > This is not a new bug of EMACS 23; but it is there in EMACS22 for a
> > long time.  In M-x picture-mode, emacs acts in a "replace" typing
> > mode, i.e. when you type a char, it replace the old one such that the
> > alignment is maintained.
> >
> > But when mixing with double-width characters (e.g. CJK chars), one to
> > one char replacing become problematic, e.g. if we replace a
> > single-width char with a double-wdith char, the alignment will be
> > destroyed.
> 
> Sorry it's taken this long for someone to get back to you.
> 
> First, do you know if this is still a problem for you?

I think I still see it in the current Emacs-25 branch.

> If so, how are you entering the CJK characters?

The point is double-width characters, not necessarily CJK character,
AFAIU.  There's a list of such characters in
lisp/international/character.el (search for "full-width").

Typing any of the characters for which the char-width-table entry
holds 2 should exhibit the problem.

> I find in Emacs 25 that if I try entering a greek alpha by typing:
> 
> C-x 8 RET GREEK SMALL LETTER ALPHA
> 
> It inserts the character, however when I bind the same character to a
> key, eg.:
> 
> (global-set-key (kbd "C-c a") "α")
> 
> and enter it that way it works as expected.

Yes, binding it to a key is the way to go.  And when inserting such a
character, it looks like picture-mode does TRT: you will see in
picture-insert that it looks at the character width.  But I think the
bug report is not about inserting double-width characters, it's about
replacing them with a single-width character.  If I type a
double-width character, the alignment is kept reasonably well
("reasonably" because the double-width characters don't always take up
exactly twice the number of pixels of a single-width character, the
exact ration depends on the font being used for the double-width
character).  To keep the alignment, picture-insert replaces 2
single-width characters with the double-width one.  However, if I then
replace it with a single-width character, the alignment is destroyed.
And I think this bug report is about that latter use case.

Thanks for working on this.

bug#1808: 23.0.60; picture-mode not considering double-width characters alignment

[Alan J Third]

Eli Zaretskii <eliz@gnu.org> writes:

> The point is double-width characters, not necessarily CJK character,
> AFAIU.  There's a list of such characters in
> lisp/international/character.el (search for "full-width").
>
> Typing any of the characters for which the char-width-table entry
> holds 2 should exhibit the problem.

I hadn't realised there was such a thing, I had assumed it was two byte
unicode type characters. Now I know different. :)

> Yes, binding it to a key is the way to go.  And when inserting such a
> character, it looks like picture-mode does TRT: you will see in
> picture-insert that it looks at the character width.  But I think the
> bug report is not about inserting double-width characters, it's about
> replacing them with a single-width character.  If I type a
> double-width character, the alignment is kept reasonably well
> ("reasonably" because the double-width characters don't always take up
> exactly twice the number of pixels of a single-width character, the
> exact ration depends on the font being used for the double-width
> character).  To keep the alignment, picture-insert replaces 2
> single-width characters with the double-width one.  However, if I then
> replace it with a single-width character, the alignment is destroyed.
> And I think this bug report is about that latter use case.

It turns out that insertion sometimes does TRT, but not in the case
where you have a single-width char followed by a double-width, and you
try to overwrite the single-width char with a double-width. Both the
single and double-width chars are overwritten, and everything to the
right moves left one column, therefore going out of alignment.

I've written a small patch that I think fixes all these issues.

It works out how many columns the characters that are about to be
over-written take up, then deletes them as before, and finally inserts a
number of spaces to fill the gap (if they're needed). Then the old code
inserts the new character.

Spaces seemed the like the safest option, but it could be easily changed.


diff -c /Users/alan/src/emacs/picture.el /Users/alan/src/emacs/new-picture.el
*** /Users/alan/src/emacs/picture.el	2016-01-10 18:26:02.000000000 +0000
--- /Users/alan/src/emacs/new-picture.el	2016-01-10 19:51:53.000000000 +0000
***************
*** 272,278 ****
  	(or (eolp)
  	    (let ((pos (point)))
  	      (move-to-column col t)
! 	      (delete-region pos (point)))))
        (insert ch)
        (forward-char -1)
        (picture-move))))
--- 272,282 ----
  	(or (eolp)
  	    (let ((pos (point)))
  	      (move-to-column col t)
! 	      (let ((old-width (string-width (buffer-substring pos (point)))))
! 		(delete-region pos (point))
! 		(when (> old-width width)
! 		  (insert-char ?  (- old-width width))
! 		  (goto-char pos))))))
        (insert ch)
        (forward-char -1)
        (picture-move))))

Diff finished.  Sun Jan 10 19:51:57 2016

-- 
Alan Third

bug#1808: 23.0.60; picture-mode not considering double-width characters alignment

[Eli Zaretskii]

> From: Alan J Third <alan@idiocy.org>
> Cc: 1808@debbugs.gnu.org,  poppyer@gmail.com
> Date: Sun, 10 Jan 2016 20:11:45 +0000
> 
> I've written a small patch that I think fixes all these issues.
> 
> It works out how many columns the characters that are about to be
> over-written take up, then deletes them as before, and finally inserts a
> number of spaces to fill the gap (if they're needed). Then the old code
> inserts the new character.
> 
> Spaces seemed the like the safest option, but it could be easily changed.

Thanks, this looks correct to me.  Please commit, and I think we can
close the bug.

bug#1808: 23.0.60; picture-mode not considering double-width characters alignment

[Eli Zaretskii]

> Date: Mon, 11 Jan 2016 20:28:07 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: poppyer@gmail.com, 1808@debbugs.gnu.org
> 
> > From: Alan J Third <alan@idiocy.org>
> > Cc: 1808@debbugs.gnu.org,  poppyer@gmail.com
> > Date: Sun, 10 Jan 2016 20:11:45 +0000
> > 
> > I've written a small patch that I think fixes all these issues.
> > 
> > It works out how many columns the characters that are about to be
> > over-written take up, then deletes them as before, and finally inserts a
> > number of spaces to fill the gap (if they're needed). Then the old code
> > inserts the new character.
> > 
> > Spaces seemed the like the safest option, but it could be easily changed.
> 
> Thanks, this looks correct to me.  Please commit, and I think we can
> close the bug.

Fixed with commit b70dba4 for Emacs 25.1.

Closing.