bug#24972: 26.0.50; Multibyte characters are misaligned in ibuffer

bug#24972: 26.0.50; Multibyte characters are misaligned in ibuffer

[Liu Hui]

[-- Attachment #1: Type: text/plain, Size: 3960 bytes --]

Hello,

As shown in the screenshot attached, multibyte characters (the second 
line) are misaligned in ibuffer. The problem seems at least relevant to 
the function `ibuffer-compile-format', which uses `length' rather than 
`string-width' for characters.


In GNU Emacs 26.0.50.2 (x86_64-pc-linux-gnu, GTK+ Version 3.18.9)
  of 2016-10-26 built on lcy01-25
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description:	Ubuntu 16.04.1 LTS

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Updating buffer list...
Formats have changed, recompiling...done
Updating buffer list...done
Commands: m, u, t, RET, g, k, S, D, Q; q to quit; h for help

Configured using:
  'configure --build=x86_64-linux-gnu --prefix=/usr
  '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
  '--infodir=${prefix}/share/info' --sysconfdir=/etc --localstatedir=/var
  --disable-silent-rules '--libdir=${prefix}/lib/x86_64-linux-gnu'
  '--libexecdir=${prefix}/lib/x86_64-linux-gnu' --disable-maintainer-mode
  --disable-dependency-tracking --prefix=/usr --sharedstatedir=/var/lib
  --program-suffix=-snapshot --with-modules=yes --with-x=yes
  --with-x-toolkit=gtk3 --with-xwidgets=yes 'CFLAGS=-g -O2
  -fstack-protector-strong -Wformat -Werror=format-security'
  'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
  'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES XWIDGETS LIBSYSTEMD

Important settings:
   value of $LANG: zh_CN.UTF-8
   value of $XMODIFIERS: @im=fcitx
   locale-coding-system: utf-8-unix

Major mode: IBuffer

Minor modes in effect:
   tooltip-mode: t
   global-eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   buffer-read-only: t
   line-number-mode: t
   transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message subr-x puny dired dired-loaddefs
format-spec rfc822 mml mml-sec password-cache epa derived epg epg-config
gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils ibuf-ext ibuffer
ibuffer-loaddefs seq byte-opt gv bytecomp byte-compile cl-extra
help-mode easymenu cconv cl-loaddefs pcase cl-lib time-date mule-util
china-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev
obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting system-font-setting font-render-setting
xwidget-internal move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 106143 9040)
  (symbols 48 21101 0)
  (miscs 40 546 154)
  (strings 32 20160 4361)
  (string-bytes 1 648450)
  (vectors 16 15319)
  (vector-slots 8 526212 5865)
  (floats 8 186 176)
  (intervals 56 401 8)
  (buffers 976 14)
  (heap 1024 32825 1228))


[-- Attachment #2: bug.png --]
[-- Type: image/png, Size: 19979 bytes --]

bug#24972: 26.0.50; Multibyte characters are misaligned in ibuffer

[Mark Oteiza]

Liu Hui <liuhui1610@gmail.com> writes:
> As shown in the screenshot attached, multibyte characters (the second
> line) are misaligned in ibuffer. The problem seems at least relevant
> to the function `ibuffer-compile-format', which uses `length' rather
> than `string-width' for characters.

Indeed, changing to string-width improves it. ibuffer should probably
also be using display properties to align the columns, but that's
another issue.

Committed in 7e2a8b1. Thanks for reporting

bug#24972: 26.0.50; Multibyte characters are misaligned in ibuffer

[Liu Hui]

在 2016年11月21日 02:58, Mark Oteiza 写道:
>
> Liu Hui <liuhui1610@gmail.com> writes:
>> As shown in the screenshot attached, multibyte characters (the second
>> line) are misaligned in ibuffer. The problem seems at least relevant
>> to the function `ibuffer-compile-format', which uses `length' rather
>> than `string-width' for characters.
>
> Indeed, changing to string-width improves it. ibuffer should probably
> also be using display properties to align the columns, but that's
> another issue.
>
> Committed in 7e2a8b1. Thanks for reporting
>

Thank you, but I find it does not work when the multibyte string is 
long. For example, if the string-width of buffer name, e.g. 
"一个很长很长的文件名", exceeds the default width of ibuffer column, M-x ibuffer 
will fail and issue an error.

According to the backtrace, it is because `ibuffer-compile-format' calls 
`ibuffer-compile-make-substring-form'/`ibuffer-compile-make-eliding-form', 
which use `substring' to truncate the multibyte string. A similar issue 
was reported in https://github.com/emacs-helm/helm/issues/170. Probably 
replacing `substring' solves the problem.

bug#24972: 26.0.50; Multibyte characters are misaligned in ibuffer

[Mark Oteiza]

On 21/11/16 at 02:00pm, Liu Hui wrote:
> 在 2016年11月21日 02:58, Mark Oteiza 写道:
> > Liu Hui <liuhui1610@gmail.com> writes:
> > > As shown in the screenshot attached, multibyte characters (the second
> > > line) are misaligned in ibuffer. The problem seems at least relevant
> > > to the function `ibuffer-compile-format', which uses `length' rather
> > > than `string-width' for characters.
> > 
> > Indeed, changing to string-width improves it. ibuffer should probably
> > also be using display properties to align the columns, but that's
> > another issue.
> > 
> > Committed in 7e2a8b1. Thanks for reporting
> 
> Thank you, but I find it does not work when the multibyte string is long.
> For example, if the string-width of buffer name, e.g.
> "一个很长很长的文件名", exceeds the default width of ibuffer column, M-x
> ibuffer will fail and issue an error.
> 
> According to the backtrace, it is because `ibuffer-compile-format' calls
> `ibuffer-compile-make-substring-form'/`ibuffer-compile-make-eliding-form',
> which use `substring' to truncate the multibyte string. A similar issue was
> reported in https://github.com/emacs-helm/helm/issues/170. Probably
> replacing `substring' solves the problem.

Ah, thanks. I pushed another change that fixes the error.  Still, if the
name is a particular mix of half- and full-width though (e.g.
"aーーーーーー"), the next column(s) might be off by one.

bug#24972: 26.0.50; Multibyte characters are misaligned in ibuffer

[Liu Hui]

在 2016年11月21日 17:35, Mark Oteiza 写道:
> On 21/11/16 at 02:00pm, Liu Hui wrote:
>> 在 2016年11月21日 02:58, Mark Oteiza 写道:
>>> Liu Hui <liuhui1610@gmail.com> writes:
>>>> As shown in the screenshot attached, multibyte characters (the second
>>>> line) are misaligned in ibuffer. The problem seems at least relevant
>>>> to the function `ibuffer-compile-format', which uses `length' rather
>>>> than `string-width' for characters.
>>>
>>> Indeed, changing to string-width improves it. ibuffer should probably
>>> also be using display properties to align the columns, but that's
>>> another issue.
>>>
>>> Committed in 7e2a8b1. Thanks for reporting
>>
>> Thank you, but I find it does not work when the multibyte string is long.
>> For example, if the string-width of buffer name, e.g.
>> "一个很长很长的文件名", exceeds the default width of ibuffer column, M-x
>> ibuffer will fail and issue an error.
>>
>> According to the backtrace, it is because `ibuffer-compile-format' calls
>> `ibuffer-compile-make-substring-form'/`ibuffer-compile-make-eliding-form',
>> which use `substring' to truncate the multibyte string. A similar issue was
>> reported in https://github.com/emacs-helm/helm/issues/170. Probably
>> replacing `substring' solves the problem.
>
> Ah, thanks. I pushed another change that fixes the error.  Still, if the
> name is a particular mix of half- and full-width though (e.g.
> "aーーーーーー"), the next column(s) might be off by one.
>

Thank you, but the trailing "..." is missing for long buffer names, no 
matter whether they contain multibyte characters. For example, 
"01234567890123456789" becomes "012345678901234567". The normal result 
should be "012345678901234...".

bug#24972: 26.0.50; Multibyte characters are misaligned in ibuffer

[Mark Oteiza]

On 22/11/16 at 11:51am, Liu Hui wrote:
> Thank you, but the trailing "..." is missing for long buffer names, no
> matter whether they contain multibyte characters. For example,
> "01234567890123456789" becomes "012345678901234567". The normal result
> should be "012345678901234...".

Would you try the following?

diff --git a/lisp/ibuffer.el b/lisp/ibuffer.el
index faadb67..c550b7d 100644
--- a/lisp/ibuffer.el
+++ b/lisp/ibuffer.el
@@ -1559,16 +1559,17 @@ ibuffer-compile-make-eliding-form
 		  `(concat ,ellipsis
 			   (substring ,strvar
 				      (string-width ibuffer-eliding-string)))
-		`(truncate-string-to-width
-		  ,strvar strlen nil nil
-		  ,ellipsis))
+		`(concat
+		  (truncate-string-to-width
+                   ,strvar (- strlen (string-width ,ellipsis)) nil ?.)
+                  ,ellipsis))
 	   ,strvar)
       strvar)))
 
 (defun ibuffer-compile-make-substring-form (strvar maxvar from-end-p)
   (if from-end-p
-      `(truncate-string-to-width str (string-width str) (- strlen ,maxvar))
-    `(truncate-string-to-width ,strvar ,maxvar)))
+      `(truncate-string-to-width str (string-width str) (- strlen ,maxvar) nil ?\s)
+    `(truncate-string-to-width ,strvar ,maxvar nil ?\s)))
 
 (defun ibuffer-compile-make-format-form (strvar widthform alignment)
   (let* ((left `(make-string tmp2 ?\s))

bug#24972: 26.0.50; Multibyte characters are misaligned in ibuffer

[Liu Hui]

在 2016年11月22日 13:12, Mark Oteiza 写道:
> On 22/11/16 at 11:51am, Liu Hui wrote:
>> Thank you, but the trailing "..." is missing for long buffer names, no
>> matter whether they contain multibyte characters. For example,
>> "01234567890123456789" becomes "012345678901234567". The normal result
>> should be "012345678901234...".
>
> Would you try the following?
>
> diff --git a/lisp/ibuffer.el b/lisp/ibuffer.el
> index faadb67..c550b7d 100644
> --- a/lisp/ibuffer.el
> +++ b/lisp/ibuffer.el
> @@ -1559,16 +1559,17 @@ ibuffer-compile-make-eliding-form
>  		  `(concat ,ellipsis
>  			   (substring ,strvar
>  				      (string-width ibuffer-eliding-string)))
> -		`(truncate-string-to-width
> -		  ,strvar strlen nil nil
> -		  ,ellipsis))
> +		`(concat
> +		  (truncate-string-to-width
> +                   ,strvar (- strlen (string-width ,ellipsis)) nil ?.)
> +                  ,ellipsis))
>  	   ,strvar)
>        strvar)))
>
>  (defun ibuffer-compile-make-substring-form (strvar maxvar from-end-p)
>    (if from-end-p
> -      `(truncate-string-to-width str (string-width str) (- strlen ,maxvar))
> -    `(truncate-string-to-width ,strvar ,maxvar)))
> +      `(truncate-string-to-width str (string-width str) (- strlen ,maxvar) nil ?\s)
> +    `(truncate-string-to-width ,strvar ,maxvar nil ?\s)))
>
>  (defun ibuffer-compile-make-format-form (strvar widthform alignment)
>    (let* ((left `(make-string tmp2 ?\s))
>

It works very well. Thank you.

bug#24972: 26.0.50; Multibyte characters are misaligned in ibuffer

[Mark Oteiza]

On 22/11/16 at 02:54pm, Liu Hui wrote:
> 
> 在 2016年11月22日 13:12, Mark Oteiza 写道:
> > On 22/11/16 at 11:51am, Liu Hui wrote:
> > > Thank you, but the trailing "..." is missing for long buffer names, no
> > > matter whether they contain multibyte characters. For example,
> > > "01234567890123456789" becomes "012345678901234567". The normal result
> > > should be "012345678901234...".
> > 
> > Would you try the following?
> > 
> > diff --git a/lisp/ibuffer.el b/lisp/ibuffer.el
> > index faadb67..c550b7d 100644
> > --- a/lisp/ibuffer.el
> > +++ b/lisp/ibuffer.el
> > @@ -1559,16 +1559,17 @@ ibuffer-compile-make-eliding-form
> >  		  `(concat ,ellipsis
> >  			   (substring ,strvar
> >  				      (string-width ibuffer-eliding-string)))
> > -		`(truncate-string-to-width
> > -		  ,strvar strlen nil nil
> > -		  ,ellipsis))
> > +		`(concat
> > +		  (truncate-string-to-width
> > +                   ,strvar (- strlen (string-width ,ellipsis)) nil ?.)
> > +                  ,ellipsis))
> >  	   ,strvar)
> >        strvar)))
> > 
> >  (defun ibuffer-compile-make-substring-form (strvar maxvar from-end-p)
> >    (if from-end-p
> > -      `(truncate-string-to-width str (string-width str) (- strlen ,maxvar))
> > -    `(truncate-string-to-width ,strvar ,maxvar)))
> > +      `(truncate-string-to-width str (string-width str) (- strlen ,maxvar) nil ?\s)
> > +    `(truncate-string-to-width ,strvar ,maxvar nil ?\s)))
> > 
> >  (defun ibuffer-compile-make-format-form (strvar widthform alignment)
> >    (let* ((left `(make-string tmp2 ?\s))
> > 
> 
> It works very well. Thank you.

Wahoo! Pushed, thanks.