bug#65015: 29.1; align-to on wrapped line regression

bug#65015: 29.1; align-to on wrapped line regression

[Axel Forsman]

I noticed that the interpretation of the hpos given to the :align-to
space specification property changed in Emacs 29.1 compared to 28.2,
without it being documented anywhere. In version 28 it counts relative
to the visual start of the line, whereas in version 29 it starts at the
logical start of the line.

That is, the following MWE exhibits different visual behavior in Emacs
28 contra 29:

    (insert
     (concat
      "\n"
      (make-string (round (* 1.25 (window-text-width))) ?x)
      (propertize " " 'display `(space :align-to ,(round
(window-text-width) 2)))
      "foo\n\n"))

(In 28 the text "foo" is centered correctly by the space. In 29 the
space has zero-width and no effect.)

The previous behavior makes more sense in the context of section 41.16.3
Pixel Specification for Spaces in the Emacs manual, and it would be
quite the breaking change so I am hoping it was unintentional.


Kind regards
Axel Forsman


In GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu)
Repository revision: emacs-29.1
Repository branch: master
System Description: NixOS 23.05 (Stoat)

Configured using:
 'configure
 --prefix=/nix/store/whazydpl0yj8i03aapsd2cyry70mng55-emacs-unstable-29.1-nox
 --disable-build-details --with-modules --with-gif=no --with-jpeg=no
 --with-png=no --with-tiff=no --with-x=no --with-xpm=no
 --with-native-compilation --with-tree-sitter'

Configured features:
DBUS GMP GNUTLS GPM JSON LIBSELINUX LIBSYSTEMD LIBXML2 MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER SECCOMP SOUND SQLITE3 THREADS
TREE_SITTER ZLIB

Important settings:
  value of $LC_MESSAGES: en_US.UTF-8
  value of $LANG: sv_SE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  undo-tree-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  hotfuzz-vertico-mode: t
  vertico-mode: t
  xclip-mode: t
  evil-mode: t
  evil-local-mode: t
  electric-pair-mode: t
  delete-selection-mode: t
  global-auto-revert-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/run/current-system/sw/share/emacs/site-lisp/site-start hides
/nix/store/whazydpl0yj8i03aapsd2cyry70mng55-emacs-unstable-29.1-nox/share/emacs/site-lisp/site-start

Features:
(shadow sort mail-extr emacsbug mule-util hotfuzz-module notmuch
notmuch-tree notmuch-jump notmuch-hello image wid-edit notmuch-show
notmuch-print notmuch-crypto notmuch-mua notmuch-message notmuch-draft
notmuch-maildir-fcc notmuch-address notmuch-company notmuch-parser
notmuch-wash diff-mode easy-mmode coolj goto-addr icalendar diary-lib
diary-loaddefs cal-menu calendar cal-loaddefs notmuch-tag crm
notmuch-lib notmuch-version notmuch-compat hl-line message sendmail
yank-media dired dnd dired-loaddefs rfc822 mml mailabbrev mail-utils
gmm-utils mailheader mm-view mml-smime mml-sec epa derived epg rfc6068
epg-config gnus-util text-property-search time-date smime password-cache
gnutls puny dig mm-decode mm-bodies mm-encode mailcap mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr term/tmux term/xterm xterm
lua-mode-autoloads nix-mode-autoloads julia-mode-autoloads
cmake-mode-autoloads yaml-mode-autoloads haskell-mode-autoloads
typescript-mode-autoloads markdown-mode-autoloads rust-mode-autoloads
mytheme-theme ws-butler-autoloads rmsbolt-autoloads corfu-autoloads
yasnippet undo-tree diff queue yasnippet-autoloads transient format-spec
eieio eieio-core magit-autoloads magit-section-autoloads
git-commit-autoloads with-editor-autoloads dash-autoloads
xterm-color-autoloads devdocs-autoloads hotfuzz vertico compat
hotfuzz-autoloads vertico-autoloads compat-autoloads xclip
xclip-autoloads evil evil-keybindings evil-integration evil-maps
evil-commands reveal evil-jumps evil-command-window evil-search evil-ex
evil-types evil-macros evil-repeat evil-states evil-core byte-opt comp
regexp-opt comp-cstr warnings icons subr-x rx cl-seq cl-macs gv cl-extra
help-mode tool-bar advice evil-common thingatpt rect evil-vars ring
edmacro kmacro undo-tree-autoloads queue-autoloads evil-autoloads
goto-chg-autoloads pcase elec-pair delsel autorevert filenotify
cl-loaddefs cl-lib ekipage bytecomp byte-compile rmc iso-transl tooltip
cconv eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify multi-tty make-network-process native-compile
emacs)

Memory information:
((conses 16 623896 226463)
 (symbols 48 15703 10)
 (strings 32 85936 57749)
 (string-bytes 1 4096111)
 (vectors 16 33101)
 (vector-slots 8 543541 137525)
 (floats 8 64 333)
 (intervals 56 62415 18186)
 (buffers 984 13))

bug#65015: 29.1; align-to on wrapped line regression

[Eli Zaretskii]

> From: Axel Forsman <axelsfor@gmail.com>
> Date: Wed, 2 Aug 2023 15:22:35 +0200
> 
> I noticed that the interpretation of the hpos given to the :align-to
> space specification property changed in Emacs 29.1 compared to 28.2,
> without it being documented anywhere. In version 28 it counts relative
> to the visual start of the line, whereas in version 29 it starts at the
> logical start of the line.
> 
> That is, the following MWE exhibits different visual behavior in Emacs
> 28 contra 29:
> 
>     (insert
>      (concat
>       "\n"
>       (make-string (round (* 1.25 (window-text-width))) ?x)
>       (propertize " " 'display `(space :align-to ,(round
> (window-text-width) 2)))
>       "foo\n\n"))
> 
> (In 28 the text "foo" is centered correctly by the space. In 29 the
> space has zero-width and no effect.)
> 
> The previous behavior makes more sense in the context of section 41.16.3
> Pixel Specification for Spaces in the Emacs manual, and it would be
> quite the breaking change so I am hoping it was unintentional.

It was intentional, since :align-to counts columns, and columns in
Emacs continue being counted in continuation lines, they don't start
from zero again at the point where the line wraps.  Cf current-column
and move-to-column.  What you seem to expect would make it impossible
to wrap lines with :align-to space display specs without losing the
alignment when the line wraps.

So this change fixed a bug, and it is therefore here to stay.  That is
also the reason why it is not in NEWS: we don't include bug fixes
there.

bug#65015: 29.1; align-to on wrapped line regression

[Axel Forsman]

> It was intentional, since :align-to counts columns, and columns in
> Emacs continue being counted in continuation lines, they don't start
> from zero again at the point where the line wraps.

But in that case this is still a documentation bug, since all
documentation refers to the value of :align-to as being a hpos and
not a col.

Where does the notion that :align-to takes a column come from?

> What you seem to expect would make it impossible
> to wrap lines with :align-to space display specs without losing the
> alignment when the line wraps.

More like "make it impossible to specify a :align-to value greater than
the width of the text area", because as you worded it, I would argue
the opposite holds. According to the docs,

    (concat "..." (propertize " " 'display `(space :align-to (- right
4))) "foo")

should right align the string "foo", however that alignment is lost now
in version 29 when the line has been wrapped, but not in 28.

Could you please link to the discussion where the old behavior was
termed a bug so I can read up?, because just from reading the Emacs
Lisp manual I am not so sure I agree.

Since it is easier to get the version 29 behavior using the min-width
property instead of :align-to (except that does not work in overlays?)
than the version 28 behavior, is there any chance this could be reverted?

I had an implementation of overlay-based popups working in Emacs 28
that used vertical-motion and :align-to to render the popup on the right
xy-positions regardless of where logical line breaks were. It cannot easily
be adapted for Emacs 29 since a lot of the work that the display engine
otherwise was doing now has to be done manually due to :align-to being
dumber.
So I have a bit of a bias toward the old :align-to workings.


/Axel Forsman

On Wed, Aug 2, 2023 at 4:40 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Axel Forsman <axelsfor@gmail.com>
> > Date: Wed, 2 Aug 2023 15:22:35 +0200
> >
> > I noticed that the interpretation of the hpos given to the :align-to
> > space specification property changed in Emacs 29.1 compared to 28.2,
> > without it being documented anywhere. In version 28 it counts relative
> > to the visual start of the line, whereas in version 29 it starts at the
> > logical start of the line.
> >
> > That is, the following MWE exhibits different visual behavior in Emacs
> > 28 contra 29:
> >
> >     (insert
> >      (concat
> >       "\n"
> >       (make-string (round (* 1.25 (window-text-width))) ?x)
> >       (propertize " " 'display `(space :align-to ,(round
> > (window-text-width) 2)))
> >       "foo\n\n"))
> >
> > (In 28 the text "foo" is centered correctly by the space. In 29 the
> > space has zero-width and no effect.)
> >
> > The previous behavior makes more sense in the context of section 41.16.3
> > Pixel Specification for Spaces in the Emacs manual, and it would be
> > quite the breaking change so I am hoping it was unintentional.
>
> It was intentional, since :align-to counts columns, and columns in
> Emacs continue being counted in continuation lines, they don't start
> from zero again at the point where the line wraps.  Cf current-column
> and move-to-column.  What you seem to expect would make it impossible
> to wrap lines with :align-to space display specs without losing the
> alignment when the line wraps.
>
> So this change fixed a bug, and it is therefore here to stay.  That is
> also the reason why it is not in NEWS: we don't include bug fixes
> there.

bug#65015: 29.1; align-to on wrapped line regression

[Eli Zaretskii]

> From: Axel Forsman <axelsfor@gmail.com>
> Date: Wed, 2 Aug 2023 23:19:02 +0200
> Cc: 65015@debbugs.gnu.org
> 
> > It was intentional, since :align-to counts columns, and columns in
> > Emacs continue being counted in continuation lines, they don't start
> > from zero again at the point where the line wraps.
> 
> But in that case this is still a documentation bug, since all
> documentation refers to the value of :align-to as being a hpos and
> not a col.

Well, it doesn't really say what HPOS is.  I have clarified it now,
thanks.

> Where does the notion that :align-to takes a column come from?

What else can it be?  :align-to is used for indenting and aligning
text, and is equivalent to using TABs, so column numbers are natural
when doing that.  People expect current-column to agree with
:align-to, and rightfully so.

> > What you seem to expect would make it impossible
> > to wrap lines with :align-to space display specs without losing the
> > alignment when the line wraps.
> 
> More like "make it impossible to specify a :align-to value greater than
> the width of the text area", because as you worded it, I would argue
> the opposite holds. According to the docs,
> 
>     (concat "..." (propertize " " 'display `(space :align-to (- right
> 4))) "foo")
> 
> should right align the string "foo", however that alignment is lost now
> in version 29 when the line has been wrapped, but not in 28.

The above just makes no sense in wrapped lines, that's all.  It only
makes sense in lines that are narrower than the window width.

> Could you please link to the discussion where the old behavior was
> termed a bug so I can read up?

The change was the result of fixing bug#56176, although there isn't
much of discussion there, and the situation didn't involve
continuation lines, it involved horizontal scrolling and line
truncation.  But the behavior should be consistent; in particular,
when a line is hscrolled, it would be wrong to reset :align-to to
count from the visual left edge of the line on display, instead of
from the (hidden) beginning of line.

> because just from reading the Emacs Lisp manual I am not so sure I
> agree.

It's okay to disagree, but my opinion on this is quite firm.

> Since it is easier to get the version 29 behavior using the min-width
> property instead of :align-to (except that does not work in overlays?)
> than the version 28 behavior, is there any chance this could be reverted?

Reverting the change is out of the question, because it fixed a real
bug, see bug#56176.  We could perhaps consider some changes in this
area, but to make any such changes, we'd need a consistent idea of
behavior that covers line continuation, line truncation, and
horizontal scrolling, and is at least in some sense consistent with
current-column.  TBH, I don't believe something like this is possible,
unless we keep the behavior of Emacs 29.

> I had an implementation of overlay-based popups working in Emacs 28
> that used vertical-motion and :align-to to render the popup on the right
> xy-positions regardless of where logical line breaks were. It cannot easily
> be adapted for Emacs 29 since a lot of the work that the display engine
> otherwise was doing now has to be done manually due to :align-to being
> dumber.
> So I have a bit of a bias toward the old :align-to workings.

I hear you, but building features on top of such dark corners of the
display-engine behavior was and always will be fraught with some risk
of breakage.

AFAIU, to adapt to this change, your code will have to call
current-column to compute the adjustment of the value you pass to
:align-to.  Something like the below:

  (save-excursion
    (beginning-of-visual-line)
    (or (= (char-before) ?\n)
        (backward-char))
    (current-column))

will tell you what to add to the "visual-based" value of :align-to to
get what you want.

bug#65015: 29.1; align-to on wrapped line regression

[Axel Forsman]

> Well, it doesn't really say what HPOS is.  I have clarified it now,
> thanks.

The docstring of compute-motion already uses HPOS to signify the
horizontal position relative to the visual left edge of the text area.
I have not read your change yet, but please use COL for the value
of :align-to instead of HPOS to avoid overloading terminology. That's
my suggestion at least. (Though vertical-motion also talks about
COLS in the HPOS sense, but it is at least clear about its interpretation.)

> What else can it be?

I read into the fact that HPOS is already defined elsewhere and that it
takes millimeter-values, and took it as a pixel x-offset from the current
left edge of the text area. Is that really so outlandish?

> The above just makes no sense in wrapped lines, that's all.

Why do you feel the need to paint me as an idiot? Under the semantics
imposed by HPOS being as described in the compute-motion docstring,
the previous behavior is just what anyone would expect. Clearly it does
make sense.

> The change was the result of fixing bug#56176, although there isn't
> much of discussion there, and the situation didn't involve
> continuation lines, it involved horizontal scrolling and line
> truncation.  But the behavior should be consistent; in particular,
> when a line is hscrolled, it would be wrong to reset :align-to to
> count from the visual left edge of the line on display, instead of
> from the (hidden) beginning of line.
> [...]
> We could perhaps consider some changes in this
> area, but to make any such changes, we'd need a consistent idea of
> behavior that covers line continuation, line truncation, and
> horizontal scrolling, and is at least in some sense consistent with
> current-column.  TBH, I don't believe something like this is possible,
> unless we keep the behavior of Emacs 29.

To be clear, I think tying :align-to to columns (as in current-column) is a
perfectly fine choice, and there is nothing that needs changing to that end.
However I do not agree that the previous behavior was necessarily a bug.
Pardon my persistence, but would it be possible to reintroduce the old
behavior under a new space spec property name, to give everyone (read:
me) a clear upgrade path? Evidently it was useful.

> AFAIU, to adapt to this change, your code will have to call
> current-column to compute the adjustment of the value you pass to
> :align-to.

Well yeah, but if that is now what the best implementation has to do, then
that is clearly less than satisfactory. I think overlay popups are an important
use case for everyone using Emacs text-frames.


/Axel Forsman

On Thu, Aug 3, 2023 at 7:49 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Axel Forsman <axelsfor@gmail.com>
> > Date: Wed, 2 Aug 2023 23:19:02 +0200
> > Cc: 65015@debbugs.gnu.org
> >
> > > It was intentional, since :align-to counts columns, and columns in
> > > Emacs continue being counted in continuation lines, they don't start
> > > from zero again at the point where the line wraps.
> >
> > But in that case this is still a documentation bug, since all
> > documentation refers to the value of :align-to as being a hpos and
> > not a col.
>
> Well, it doesn't really say what HPOS is.  I have clarified it now,
> thanks.
>
> > Where does the notion that :align-to takes a column come from?
>
> What else can it be?  :align-to is used for indenting and aligning
> text, and is equivalent to using TABs, so column numbers are natural
> when doing that.  People expect current-column to agree with
> :align-to, and rightfully so.
>
> > > What you seem to expect would make it impossible
> > > to wrap lines with :align-to space display specs without losing the
> > > alignment when the line wraps.
> >
> > More like "make it impossible to specify a :align-to value greater than
> > the width of the text area", because as you worded it, I would argue
> > the opposite holds. According to the docs,
> >
> >     (concat "..." (propertize " " 'display `(space :align-to (- right
> > 4))) "foo")
> >
> > should right align the string "foo", however that alignment is lost now
> > in version 29 when the line has been wrapped, but not in 28.
>
> The above just makes no sense in wrapped lines, that's all.  It only
> makes sense in lines that are narrower than the window width.
>
> > Could you please link to the discussion where the old behavior was
> > termed a bug so I can read up?
>
> The change was the result of fixing bug#56176, although there isn't
> much of discussion there, and the situation didn't involve
> continuation lines, it involved horizontal scrolling and line
> truncation.  But the behavior should be consistent; in particular,
> when a line is hscrolled, it would be wrong to reset :align-to to
> count from the visual left edge of the line on display, instead of
> from the (hidden) beginning of line.
>
> > because just from reading the Emacs Lisp manual I am not so sure I
> > agree.
>
> It's okay to disagree, but my opinion on this is quite firm.
>
> > Since it is easier to get the version 29 behavior using the min-width
> > property instead of :align-to (except that does not work in overlays?)
> > than the version 28 behavior, is there any chance this could be reverted?
>
> Reverting the change is out of the question, because it fixed a real
> bug, see bug#56176.  We could perhaps consider some changes in this
> area, but to make any such changes, we'd need a consistent idea of
> behavior that covers line continuation, line truncation, and
> horizontal scrolling, and is at least in some sense consistent with
> current-column.  TBH, I don't believe something like this is possible,
> unless we keep the behavior of Emacs 29.
>
> > I had an implementation of overlay-based popups working in Emacs 28
> > that used vertical-motion and :align-to to render the popup on the right
> > xy-positions regardless of where logical line breaks were. It cannot easily
> > be adapted for Emacs 29 since a lot of the work that the display engine
> > otherwise was doing now has to be done manually due to :align-to being
> > dumber.
> > So I have a bit of a bias toward the old :align-to workings.
>
> I hear you, but building features on top of such dark corners of the
> display-engine behavior was and always will be fraught with some risk
> of breakage.
>
> AFAIU, to adapt to this change, your code will have to call
> current-column to compute the adjustment of the value you pass to
> :align-to.  Something like the below:
>
>   (save-excursion
>     (beginning-of-visual-line)
>     (or (= (char-before) ?\n)
>         (backward-char))
>     (current-column))
>
> will tell you what to add to the "visual-based" value of :align-to to
> get what you want.

bug#65015: 29.1; align-to on wrapped line regression

[Eli Zaretskii]

> From: Axel Forsman <axelsfor@gmail.com>
> Date: Thu, 3 Aug 2023 12:48:26 +0200
> Cc: 65015@debbugs.gnu.org
> 
> > Well, it doesn't really say what HPOS is.  I have clarified it now,
> > thanks.
> 
> The docstring of compute-motion already uses HPOS to signify the
> horizontal position relative to the visual left edge of the text area.
> I have not read your change yet, but please use COL for the value
> of :align-to instead of HPOS to avoid overloading terminology.

That would be sub-optimal, since HPOS can also be a pixel-wise
specification.

> I read into the fact that HPOS is already defined elsewhere and that it
> takes millimeter-values, and took it as a pixel x-offset from the current
> left edge of the text area. Is that really so outlandish?

Not outlandish, no.  Just inconsistent with the rest of the behavior
when text alignment is required.  The :align-to alignment should work
as expected, i.e. align the text in the same manner, when the window
width changes, when lines are truncated or wrapped, and when truncated
lines are hscrolled.  If the :align-to argument is measured from the
visual beginning of a screen line, this can never work, don't you
agree?

> > The above just makes no sense in wrapped lines, that's all.
> 
> Why do you feel the need to paint me as an idiot?

Whatever made you say that, I wonder?  When I say "makes no sense", I
mean it makes no sense to me.

I guess we interpret the geometries of the Emacs lines very
differently if doing that in wrapped lines makes sense to you.

> Under the semantics imposed by HPOS being as described in the
> compute-motion docstring, the previous behavior is just what anyone
> would expect. Clearly it does make sense.

It does make sense in other contexts, but not in the context of
aligning text that should hold when the window dimensions change and
horizontal scrolling is used.  The idea of :align-to is to keep the
text aligned in those circumstances.  Think, for example, about
tabulated-list-mode which is used by quite a few Emacs features: it
uses :align-to to align columns in a table-like display, and that
display should not become messed up when the window is hscrolled or
changes its width.

This is the kind of applications that :align-to is used for, and they
must be supported correctly.

> Pardon my persistence, but would it be possible to reintroduce the old
> behavior under a new space spec property name, to give everyone (read:
> me) a clear upgrade path? Evidently it was useful.

A new display spec would be possible, especially if the problems the
pre-29 implementation caused are not considered important (e.g., what
do you expect to happen when the window is hscrolled?).  But someone
will have to write the code and document it.

> > AFAIU, to adapt to this change, your code will have to call
> > current-column to compute the adjustment of the value you pass to
> > :align-to.
> 
> Well yeah, but if that is now what the best implementation has to do, then
> that is clearly less than satisfactory.

Why is it not satisfactory?  The calculation is quite simple, IIUC.

> I think overlay popups are an important use case for everyone using
> Emacs text-frames.

FTR: I have nothing against overlay popups.