position of line moves depending on visible chars

position of line moves depending on visible chars

[Yasushi SHOJI]

Hi all,

I'm sending this to ML instead of creating a bug report since I'm not
sure a) I can describe what I'm seeing well enough, b) whether it's a
feature or a bug.  It's annoying enough to write a report, though. Git
shows me that the commit ba5f83d is the tipping point.

I'm seeing the position of a text line in an Emacs buffer moves up and
down depending on chars in the visible area.

Let's say I'm running my Emacs on a system with two different fonts,
each of which has different height in the buffer.  In my case they are
Japanese chars and ASCII chars.

To reproduce this, put the following contents in an Emacs buffer and
make the width of the window narrow enough, so that the last long line
does not fit in the window.

あ
ああ
あああ
ああああ
あああああ
ああああああaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Enable truncate line (M-x toggle-truncate-lines) so the longest line
does not fold at the end.

Now, put your cursor at the last long line and do C-a / C-e to move
back and forth.  The position of the last line moves up and down a few
pixels depending on how much "あ" is visible/hidden. You can put more
"あ" vertically at the point-min and you see line moves more.

If "あ" is not tall enough, pick one of chars from HELLO buffer and
replace with it.

I'm using Emacs-25 branch (80e2044a7f19719720d330e2796c9dfe5471e100)
and this is not affect by -Q option, but rather height of fonts.  The
master branch behaves as 25.  Emacs 24 works just fine.

Git bisect points at:

> ba5f83dfe5dea1b9dd3fca5d21384afc92cd2060 is the first bad commit
> commit ba5f83dfe5dea1b9dd3fca5d21384afc92cd2060
> Author: Eli Zaretskii <eliz@gnu.org>
> Date:   Sat May 30 12:33:08 2015 +0300
> 
>     Fix display of cursor at end of empty lines
>     
>     * src/xdisp.c (normal_char_ascent_descent): Accept additional
>     argument: the character to use for metrics in case the font
>     declares too large ascent and descent values.  Add 1 pixel to
>     ascent and descent values.
>     (normal_char_height): Accept additional argument: the character to
>     use for metrics in case the font declares too large height value.
>     Call normal_char_ascent_descent instead of doing calculations for
>     a different default character.
>     (estimate_mode_line_height, handle_single_display_spec)
>     (calc_pixel_width_or_height, produce_stretch_glyph)
>     (calc_line_height_property, produce_glyphless_glyph): All callers
>     changed.
>     (append_space_for_newline): Make sure the space glyph produced at
>     end of line has correct ascent and descent values, and the glyph
>     row has correct height, even when it's empty.
> 
> :040000 040000 f16e7827c624deeeeac0a92497ed45c9088ca6a1 8bd29005315cbad4d465ae1f62a001f27acf4889 M	src


Here is my Emacs info:

In GNU Emacs 25.0.95.1 (x86_64-pc-linux-gnu, GTK+ Version 3.20.6)
 of 2016-07-11 built on leno
Repository revision: 658daf93e295dd00048d15001335f58f91e679f6
Windowing system distributor 'The X.Org Foundation', version 11.0.11803000
System Description:	Debian GNU/Linux unstable (sid)

Configured features:
XPM JPEG TIFF GIF PNG SOUND DBUS GSETTINGS NOTIFY ACL LIBSELINUX GNUTLS
LIBXML2 FREETYPE XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LC_COLLATE: C
  value of $LC_TIME: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  diff-auto-refine-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug thai-util thai-word lao-util vc-git
diff-mode grep compile rx vc-dir ewoc vc vc-dispatcher view eieio-opt
speedbar imenu sb-image ezimage org-element org-rmail org-mhe org-irc
org-info org-gnus org-docview doc-view subr-x jka-compr image-mode
org-bibtex bibtex org-bbdb org-w3m org org-macro org-footnote
org-pcomplete pcomplete org-list org-faces org-entities noutline outline
org-version ob-emacs-lisp ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-src ob-keys ob-comint comint ansi-color ob-core ob-eval org-compat
advice org-macs org-loaddefs find-func cal-menu calendar cal-loaddefs
gnus-sum gnus-group mm-url gnus-undo gnus-start gnus-cloud nnimap nnmail
mail-source imap tls gnutls utf7 netrc nnoo parse-time gnus-spec
gnus-int gnus-range gnus-win gnus message dired desktop frameset edmacro
kmacro format-spec rfc822 mml url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util url-parse
auth-source eieio byte-opt bytecomp byte-compile cconv eieio-core
url-vars mm-view mml-smime smime dig mailcap mml-sec password-cache epg
cl-extra epg-config mm-decode cl-seq mm-bodies mm-encode mail-parse
rfc2231 mailabbrev easy-mmode sendmail mailheader gnus-ems ring nnheader
gnus-util gmm-utils rmail dframe rfc2047 rfc2045 ietf-drums mail-utils
mm-util help-fns help-mode derived easymenu mail-prsvr wid-edit cl
cl-macs inline cl-loaddefs cl-lib gv time-date mule-util pcase tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 957739 124263)
 (symbols 48 50455 0)
 (miscs 40 705 517)
 (strings 32 78140 7017)
 (string-bytes 1 2789002)
 (vectors 16 14564)
 (vector-slots 8 683038 41376)
 (floats 8 475 331)
 (intervals 56 1749 375)
 (buffers 976 29)
 (heap 1024 67545 1888))
-- 
            yashi

Re: position of line moves depending on visible chars

[Eli Zaretskii]

> Date: Sun, 17 Jul 2016 22:31:10 +0900
> From: Yasushi SHOJI <yashi@atmark-techno.com>
> 
> I'm seeing the position of a text line in an Emacs buffer moves up and
> down depending on chars in the visible area.
> 
> Let's say I'm running my Emacs on a system with two different fonts,
> each of which has different height in the buffer.  In my case they are
> Japanese chars and ASCII chars.
> 
> To reproduce this, put the following contents in an Emacs buffer and
> make the width of the window narrow enough, so that the last long line
> does not fit in the window.
> 
> あ
> ああ
> あああ
> ああああ
> あああああ
> ああああああaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> 
> Enable truncate line (M-x toggle-truncate-lines) so the longest line
> does not fold at the end.
> 
> Now, put your cursor at the last long line and do C-a / C-e to move
> back and forth.  The position of the last line moves up and down a few
> pixels depending on how much "あ" is visible/hidden. You can put more
> "あ" vertically at the point-min and you see line moves more.

This is not a bug, but the intended behavior: Emacs lays out lines
according to what is actually shown.  When you scroll the window
horizontally by typing C-e, the lines became less tall, because the
Japanese characters are not shown.

> I'm using Emacs-25 branch (80e2044a7f19719720d330e2796c9dfe5471e100)
> and this is not affect by -Q option, but rather height of fonts.  The
> master branch behaves as 25.  Emacs 24 works just fine.
> 
> Git bisect points at:
> 
> > ba5f83dfe5dea1b9dd3fca5d21384afc92cd2060 is the first bad commit
> > commit ba5f83dfe5dea1b9dd3fca5d21384afc92cd2060
> > Author: Eli Zaretskii <eliz@gnu.org>
> > Date:   Sat May 30 12:33:08 2015 +0300
> > 
> >     Fix display of cursor at end of empty lines

Yes, this is a deliberate change in behavior, as explained in the
commit log message.

Thanks.

Re: position of line moves depending on visible chars

[Yasushi SHOJI]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

Hi Eli,

Thank your for your comment.

On Sun, Jul 17, 2016 at 11:07 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> This is not a bug, but the intended behavior: Emacs lays out lines
> according to what is actually shown.  When you scroll the window
> horizontally by typing C-e, the lines became less tall, because the
> Japanese characters are not shown.

That's what I observe.

Is there an option to get the old behavior back?
If not, would you accept such option, say for 25.2?

I'm not oppose to the change but I'm sure this affects many multilingual
writers.  A vibrating line while typing isn't a good experience.

Thanks,
-- 
              yashi

[-- Attachment #2: Type: text/html, Size: 817 bytes --]

Re: position of line moves depending on visible chars

[Eli Zaretskii]

> From: Yasushi SHOJI <yashi@atmark-techno.com>
> Date: Mon, 18 Jul 2016 01:40:45 +0900
> Cc: emacs-devel@gnu.org
> 
> Is there an option to get the old behavior back?

No.

> If not, would you accept such option, say for 25.2?

I'd need to see the code which implements such an option, before I
express my opinion.

The behavior was changed in order to avoid huge line spacing when a
font reports a preposterously large height.  Any option such as the
one you are talking about will have to keep the new behavior whereby
the display engine doesn't user the font height for drawing the cursor
on empty lines, because this will have an even worse effects.

> I'm not oppose to the change but I'm sure this affects many multilingual
> writers. A vibrating line while typing isn't a good experience.

It's impossible to keep lines at the same spacing when multiple fonts
are involved anyway, so the situation you are describing is not new.

Re: position of line moves depending on visible chars

[Yasushi SHOJI]

Hi,

On Mon, Jul 18, 2016 at 2:14 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Yasushi SHOJI <yashi@atmark-techno.com>
>> Date: Mon, 18 Jul 2016 01:40:45 +0900
>> Cc: emacs-devel@gnu.org
>>
> The behavior was changed in order to avoid huge line spacing when a
> font reports a preposterously large height.  Any option such as the
> one you are talking about will have to keep the new behavior whereby
> the display engine doesn't user the font height for drawing the cursor
> on empty lines, because this will have an even worse effects.

Are you saying that the change is to workaround bad fonts?

I apologize for my ignorance.

>> I'm not oppose to the change but I'm sure this affects many multilingual
>> writers. A vibrating line while typing isn't a good experience.
>
> It's impossible to keep lines at the same spacing when multiple fonts
> are involved anyway, so the situation you are describing is not new.

Would you enlighten me about this? (A link to bts would be fine.)

I'm not sure why it's impossible, even though Emacs 24 works just
fine.  Are you saying that the working Emacs 24 is a pure luck? (having
good fonts?)

Thanks,
-- 
            yashi

Re: position of line moves depending on visible chars

[Eli Zaretskii]

> From: Yasushi SHOJI <yashi@atmark-techno.com>
> Date: Fri, 22 Jul 2016 09:01:05 +0900
> Cc: emacs-devel@gnu.org
> 
> > The behavior was changed in order to avoid huge line spacing when a
> > font reports a preposterously large height.  Any option such as the
> > one you are talking about will have to keep the new behavior whereby
> > the display engine doesn't user the font height for drawing the cursor
> > on empty lines, because this will have an even worse effects.
> 
> Are you saying that the change is to workaround bad fonts?

Indirectly, yes.  The previous code used the font-global size, both
for layout of a line with text, and for empty lines.  With those "bad"
fonts, non-empty lines could be fixed by using the dimensions of the
individual characters, but empty lines didn't have even that.  So some
different algorithm needed to be used, and that's what you see now.
I agree that the results are sometimes less than satisfactory, so if
someone can come up with a solution that doesn't re-introduce the
original problems, I'm for it.

> > It's impossible to keep lines at the same spacing when multiple fonts
> > are involved anyway, so the situation you are describing is not new.
> 
> Would you enlighten me about this? (A link to bts would be fine.)

There are too many bits, unfortunately.  In a nutshell, the Emacs
display engine determines the height of each screen line and the
spacing between them after it traverses all the characters to be
displayed on that screen line.  Since characters in a line can have
different faces, including faces that define use of non-default fonts,
the results of this cannot be predicted in advance.

For a simple example of what that does start the Emacs Info browser,
and look at the chapter and section header lines: they are taller than
the rest.

> Are you saying that the working Emacs 24 is a pure luck? (having
> good fonts?)

In a way, yes.  Try setting your default font to Latin Modern Math,
and see what happens in Emacs 24.5 and in Emacs 25.  Bug#20628 has the
details.