bug#33688: UTF symbols in 25.2.2 under Ubuntu

bug#33688: UTF symbols in 25.2.2 under Ubuntu

[Sergei Meshveliani]

From: mechvel <mechvel@botik.ru>
To: bug-gnu-emacs@gnu.org
Subject: 25.2; UTF symbols

Dear Emacs team,

This is on  GNU Emacs 25.2.2  installed on Ubuntu Linux 18.04 LTS.
It draws wrongly some UTF symbols.

I see this when writing programs in Agda.
Agda is a programming language that uses some UTF symbols in the source
code. It has a certain Agda input method for UTF symbols.
For example, in
                    A\rB          (1)

\r is an UTF symbol \rightarrow in Agda programs.
(1) needs to be drawn somewht similar to
                                         A->B

But emacs draws the part `>' of the arrow inside B, so that the string
looks like
                    A-B

, and with with a certain garbage inside B. 

Another example: there is a particular arrow which is input by the
Agda input method as
                       \-->

It is intended to be as  \r  but a small bit different.
Emacs under Debian Linux draws it so that I cannot distinguish it from
drawing \r.  So that it draws
                               _Preserves_\r-->_
as close to                    _Preserves_->_
, which is mainly all right.

But  GNU Emacs 25.2.2 under Ubuntu 18.04  draws it as

                           _Preserves_-->
                                        _

, where the arrow is long, and its `>' part reaches even further than
the last underscore.


Comment about how UTF gets to Agda programs via emacs
=====================================================

To install the Agda input method one needs to put into  .emacs
a Lisp code somewhat of this kind

--------------------------------------------------------------------
(load-file (let ((coding-system-for-read 'utf-8))
                   (shell-command-to-string "agda-mode locate")))

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(agda2-abort-highlighting (quote interactive))
 '(agda2-program-args
   (quote
    ("-i" "." "-i" "/home/mechvel/agda/stLib/dec4-2018/src" "-i"
"/home/mechvel\
/agda/UNPrelude/src" "-i" "/home/mechvel/poly/0.1" "+RTS" "-K200m"
"-M15G" "-RT\
S"))))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 )
-------------------------------------------------------------------


Some part of it get there by the  emacs  commands

  load-library agda2-mode
  customize-group agda2
  ...

  > agda-mode compile


Regards,

------
Sergei



======================================================================
This text has been generated automatically by emacs.
Probably it describes the precise configuration:
 

In GNU Emacs 25.2.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.21)
 of 2017-09-22, modified by Debian built on lgw01-amd64-050
System Description:     Ubuntu 18.04.1 LTS

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes

--enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.2/sit\
e-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.2/site-lisp:/usr/sh\
are/emacs/site-lisp
 --with-sound=alsa --without-gconf --build x86_64-linux-gnu
 --prefix=/usr --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-pop=yes

--enable-locallisppath=/etc/emacs25:/etc/emacs:/usr/local/share/emacs/25.2/sit\
e-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/25.2/site-lisp:/usr/sh\
are/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-x=yes --with-x-toolkit=gtk3
 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs25-jYekUr/emacs25-25.2+1=.
-fstack-protector-st\
rong -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LANG: ru_RU.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Loading 00debian-vars...done
Loading /etc/emacs/site-start.d/50dictionaries-common.el (source)...
Loading debian-ispell...
Loading /var/cache/dictionaries-common/emacsen-ispell-default.el
(source)...don\
e
Loading debian-ispell...done
Loading /var/cache/dictionaries-common/emacsen-ispell-dicts.el
(source)...done
Loading /etc/emacs/site-start.d/50dictionaries-common.el (source)...done
Loading /etc/emacs/site-start.d/50tcsh.el (source)...done
Loading /home/mechvel/.cabal/share/x86_64-linux-ghc-8.0.2/Agda-2.6.0/emacs-mode\
/agda2.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
/usr/share/emacs/25.2/site-lisp/debian-startup
hides /usr/share/emacs/site-lisp\
/debian-startup

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode easymenu
cl-loaddefs pcase cl-lib mail-prsvr mail-utils term/xterm xterm
time-date agda2 mule-util cyril-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 91504 4904)
 (symbols 48 19952 0)
 (miscs 40 79 97)
 (strings 32 14896 4789)
 (string-bytes 1 422389)
 (vectors 16 9971)
 (vector-slots 8 389091 16959)
 (floats 8 166 371)
 (intervals 56 255 13)
 (buffers 976 19))

bug#33688: UTF symbols in 25.2.2 under Ubuntu

[Eli Zaretskii]

> From: Sergei Meshveliani <mechvel@botik.ru>
> Date: Sun, 09 Dec 2018 19:19:11 +0300
> 
> This is on  GNU Emacs 25.2.2  installed on Ubuntu Linux 18.04 LTS.
> It draws wrongly some UTF symbols.
> 
> I see this when writing programs in Agda.
> Agda is a programming language that uses some UTF symbols in the source
> code. It has a certain Agda input method for UTF symbols.
> For example, in
>                     A\rB          (1)
> 
> \r is an UTF symbol \rightarrow in Agda programs.
> (1) needs to be drawn somewht similar to
>                                          A->B
> 
> But emacs draws the part `>' of the arrow inside B, so that the string
> looks like
>                     A-B
> 
> , and with with a certain garbage inside B. 

If you go to that character between A and B, and type "C-u C-x =",
what does Emacs display in the *Help* buffer?  Please post the entire
contents of the *Help* buffer you get.

bug#33688: UTF symbols in 25.2.2 under Ubuntu

[Sergei Meshveliani]

On Sun, 2018-12-09 at 20:05 +0200, Eli Zaretskii wrote:
> > From: Sergei Meshveliani <mechvel@botik.ru>
> > Date: Sun, 09 Dec 2018 19:19:11 +0300
> > 
> > This is on  GNU Emacs 25.2.2  installed on Ubuntu Linux 18.04 LTS.
> > It draws wrongly some UTF symbols.
> > 
> > I see this when writing programs in Agda.
> > Agda is a programming language that uses some UTF symbols in the source
> > code. It has a certain Agda input method for UTF symbols.
> > For example, in
> >                     A\rB          (1)
> > 
> > \r is an UTF symbol \rightarrow in Agda programs.
> > (1) needs to be drawn somewht similar to
> >                                          A->B
> > 
> > But emacs draws the part `>' of the arrow inside B, so that the string
> > looks like
> >                     A-B
> > 
> > , and with with a certain garbage inside B. 
> 
> If you go to that character between A and B, and type "C-u C-x =",
> what does Emacs display in the *Help* buffer?  Please post the entire
> contents of the *Help* buffer you get.
> 


I print below what is shown in Debian + GNU Emacs 23.2.1
and
                            in Ubuntu 18.04 + GNU Emacs 25.2. 



**********************************************************************
In  Debian,  GNU Emacs 23.2.1   everything is drawn correct.

\r    Example:  A\rB   to be drawn close to   A->B
The codes are
        character: → (8594, #o20622, #x2192)
preferred charset: unicode (Unicode (ISO10646))
       code point: 0x2192
           syntax: w    which means: word
       category: .:Base, c:Chinese, h:Korean, j:Japanese
       to input: type
       "\r-" or "\rightarrow" or "\r" or "\longrightarrow" or "\to" or
"\->"
                   with Agda
      buffer code: #xE2 #x86 #x92
        file code: #xE2 #x86 #x92 (encoded by coding system utf-8-unix)
          display: terminal code #xE2 #x86 #x92

Character code properties: customize what to show
  name: RIGHTWARDS ARROW
  old-name: RIGHT ARROW
  general-category: Sm (Symbol, Math)

------------------------------------------------------------
\-->    Example:  Preserves_\-->_   to be drowun close to  Preserves_->_
 
The codes are

        character: ⟶ (10230, #o23766, #x27f6)
preferred charset: unicode (Unicode (ISO10646))
       code point: 0x27F6
           syntax: w    which means: word
         category: .:Base
         to input: type "\r--" or "\r" or "\-->" with Agda
      buffer code: #xE2 #x9F #xB6
        file code: #xE2 #x9F #xB6 (encoded by coding system utf-8-unix)
          display: terminal code #xE2 #x9F #xB6

Character code properties: customize what to show
  name: LONG RIGHTWARDS ARROW
  general-category: Sm (Symbol, Math)



************************************************************************
In Ubuntu 18.04,  Gnu Emacs 25.2 :

\r   is wrongly drawn in  A\rB

The codes are 
    character: → (displayed as →) (codepoint 8594, #o20622, #x2192)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x2192
               script: symbol
               syntax: w        which means: word
    category: .:Base, c:Chinese, h:Korean, j:Japanese
    to input: type "\r-" or "\rightarrow" or "\r" or "\longrightarrow"\
 or "\to" or "\->" with Agda input method
          buffer code: #xE2 #x86 #x92
    file code: #xE2 #x86 #x92 (encoded by coding system utf-8-unix)
              display: terminal code #xE2 #x86 #x92

Character code properties: customize what to show
  name: RIGHTWARDS ARROW
  old-name: RIGHT ARROW
  general-category: Sm (Symbol, Math)
  decomposition: (8594) ('→')

---------------------------------------------------------------
\-->   is very wrongly drawn in  Preserves_\-->_

The codes are
    character: ⟶ (displayed as ⟶) (codepoint 10230, #o23766, #x27f6)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x27F6
               script: symbol
               syntax: w        which means: word
             category: .:Base
       to input: type "\r--" or "\r" or "\-->" with Agda input method
       buffer code: #xE2 #x9F #xB6
       file code: #xE2 #x9F #xB6 (encoded by coding system utf-8-unix)
              display: terminal code #xE2 #x9F #xB6

Character code properties: customize what to show
  name: LONG RIGHTWARDS ARROW
  general-category: Sm (Symbol, Math)
  decomposition: (10230) ('⟶')

bug#33688: UTF symbols in 25.2.2 under Ubuntu

[Eli Zaretskii]

> From: Sergei Meshveliani <mechvel@botik.ru>
> Cc: 33688@debbugs.gnu.org
> Date: Mon, 10 Dec 2018 00:56:29 +0300
> 
> > > But emacs draws the part `>' of the arrow inside B, so that the string
> > > looks like
> > >                     A-B
> > > 
> > > , and with with a certain garbage inside B. 
> > 
> > If you go to that character between A and B, and type "C-u C-x =",
> > what does Emacs display in the *Help* buffer?  Please post the entire
> > contents of the *Help* buffer you get.
> > 
> 
> 
> I print below what is shown in Debian + GNU Emacs 23.2.1
> and
>                             in Ubuntu 18.04 + GNU Emacs 25.2. 

This shows that Emacs produces the same codepoints and the same byte
sequences in both cases.  It also shows that you are working on a text
terminal frame (as opposed to a GUI frame), where display of
characters is performed by the console or a terminal emulator.  So I
think either the terminal emulators used on the two systems are
different, or there's some problem with the newer version of the
terminal emulator or fonts used by that emulator.

IOW, this doesn't seem to be an Emacs problem, but a problem with
terminal emulator and/or fonts used by the terminal emulator.

Thanks.

bug#33688: UTF symbols in 25.2.2 under Ubuntu

[Sergei Meshveliani]

On Mon, 2018-12-10 at 08:09 +0200, Eli Zaretskii wrote:
> > From: Sergei Meshveliani <mechvel@botik.ru>
> > Cc: 33688@debbugs.gnu.org
> > Date: Mon, 10 Dec 2018 00:56:29 +0300
> > 
> > > > But emacs draws the part `>' of the arrow inside B, so that the string
> > > > looks like
> > > >                     A-B
> > > > 
> > > > , and with with a certain garbage inside B. 
> > > 
> > > If you go to that character between A and B, and type "C-u C-x =",
> > > what does Emacs display in the *Help* buffer?  Please post the entire
> > > contents of the *Help* buffer you get.
> > > 
> > 
> > 
> > I print below what is shown in Debian + GNU Emacs 23.2.1
> > and
> >                             in Ubuntu 18.04 + GNU Emacs 25.2. 
> 
> This shows that Emacs produces the same codepoints and the same byte
> sequences in both cases.  It also shows that you are working on a text
> terminal frame (as opposed to a GUI frame), where display of
> characters is performed by the console or a terminal emulator.  So I
> think either the terminal emulators used on the two systems are
> different, or there's some problem with the newer version of the
> terminal emulator or fonts used by that emulator.
> 
> IOW, this doesn't seem to be an Emacs problem, but a problem with
> terminal emulator and/or fonts used by the terminal emulator.
> 
> Thanks.
> 


I am sorry, I have forgotten of how do I call emacs!
I called it via alias as
                          emacs -nw

I Debian + emacs 23..  it works fine, 
but not in Ubuntu 18.04 + emacs 25.2.

Now, if I call it simply by
                            emacs 

then it is almost all right.
Only
 ∷   (\::)   is shown considerably larger than, for example, capital O,
 and ∣  ( \| )  is separated with half-blank, so that  ∣x  
 is drawn close to  ∣ x

(while  Debian + (emacs-23 -nw)   draws everything all right).

Ubuntu 18.04 + emacs 25.2  shows the following description:

******************************************************************
\::

 character: ∷ (displayed as ∷) (codepoint 8759, #o21067, #x2237)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x2237
               script: symbol
               syntax: w        which means: word
             category: .:Base, c:Chinese
             to input: type "\::" with Agda input method
          buffer code: #xE2 #x88 #xB7
      file code: #xE2 #x88 #xB7 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-GOOG-Noto Sans Mono CJK
    SC-normal-normal-normal-*-17-*-*-*-*-0-iso10646-1 (#x34B)

Character code properties: customize what to show
  name: PROPORTION
  general-category: Sm (Symbol, Math)
  decomposition: (8759) ('∷')

**********************************************************************
\|

character: ∣ (displayed as ∣) (codepoint 8739, #o21043, #x2223)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x2223
               script: symbol
               syntax: w        which means: word
             category: .:Base, c:Chinese
    to input: type "\|" or "\shortmid" or "\mid" with Agda input method
    buffer code: #xE2 #x88 #xA3
    file code: #xE2 #x88 #xA3 (encoded by coding system utf-8-unix)
              display: by this font (glyph code)
    xft:-GOOG-Noto Sans Mono CJK
      SC-normal-normal-normal-*-17-*-*-*-*-0-iso10646-1 (#x33D)

Character code properties: customize what to show
  name: DIVIDES
  general-category: Sm (Symbol, Math)
  decomposition: (8739) ('∣')
***********************************************************************


Thanks,

------
Sergei

bug#33688: UTF symbols in 25.2.2 under Ubuntu

[Eli Zaretskii]

> From: Sergei Meshveliani <mechvel@botik.ru>
> Cc: 33688@debbugs.gnu.org
> Date: Mon, 10 Dec 2018 23:43:54 +0300
> 
> > IOW, this doesn't seem to be an Emacs problem, but a problem with
> > terminal emulator and/or fonts used by the terminal emulator.
> > 
> > Thanks.
> > 
> 
> 
> I am sorry, I have forgotten of how do I call emacs!
> I called it via alias as
>                           emacs -nw
> 
> I Debian + emacs 23..  it works fine, 
> but not in Ubuntu 18.04 + emacs 25.2.
> 
> Now, if I call it simply by
>                             emacs 
> 
> then it is almost all right.
> Only
>  ∷   (\::)   is shown considerably larger than, for example, capital O,
>  and ∣  ( \| )  is separated with half-blank, so that  ∣x  
>  is drawn close to  ∣ x
> 
> (while  Debian + (emacs-23 -nw)   draws everything all right).
> 
> Ubuntu 18.04 + emacs 25.2  shows the following description:
> 
> ******************************************************************
> \::
> 
>  character: ∷ (displayed as ∷) (codepoint 8759, #o21067, #x2237)
>     preferred charset: unicode (Unicode (ISO10646))
> code point in charset: 0x2237
>                script: symbol
>                syntax: w        which means: word
>              category: .:Base, c:Chinese
>              to input: type "\::" with Agda input method
>           buffer code: #xE2 #x88 #xB7
>       file code: #xE2 #x88 #xB7 (encoded by coding system utf-8-unix)
>               display: by this font (glyph code)
>     xft:-GOOG-Noto Sans Mono CJK
>     SC-normal-normal-normal-*-17-*-*-*-*-0-iso10646-1 (#x34B)
> 
> Character code properties: customize what to show
>   name: PROPORTION
>   general-category: Sm (Symbol, Math)
>   decomposition: (8759) ('∷')
> 
> **********************************************************************
> \|
> 
> character: ∣ (displayed as ∣) (codepoint 8739, #o21043, #x2223)
>     preferred charset: unicode (Unicode (ISO10646))
> code point in charset: 0x2223
>                script: symbol
>                syntax: w        which means: word
>              category: .:Base, c:Chinese
>     to input: type "\|" or "\shortmid" or "\mid" with Agda input method
>     buffer code: #xE2 #x88 #xA3
>     file code: #xE2 #x88 #xA3 (encoded by coding system utf-8-unix)
>               display: by this font (glyph code)
>     xft:-GOOG-Noto Sans Mono CJK
>       SC-normal-normal-normal-*-17-*-*-*-*-0-iso10646-1 (#x33D)
> 
> Character code properties: customize what to show
>   name: DIVIDES
>   general-category: Sm (Symbol, Math)
>   decomposition: (8739) ('∣')
> ***********************************************************************

My only advice is to try using a different font for these characters,
not Noto Sans Mono CJK.

I don't think there's an Emacs bug here.

bug#33688: UTF symbols in 25.2.2 under Ubuntu

[Stefan Kangas]

tags 33688 + notabug
close 33688
thanks

Eli Zaretskii <eliz@gnu.org> writes:

> My only advice is to try using a different font for these characters,
> not Noto Sans Mono CJK.
>
> I don't think there's an Emacs bug here.

I'm therefore closing this bug report.

Best regards,
Stefan Kangas