bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Sebastien Vauban]

Hello,

Now that I've changed some Gnus markers to UTF8 "drawings" [1], I don't
have my Gnus summary buffer correctly aligned anymore.

See http://screencast.com/t/s33eYYg7Cn.

Is there a solution to that, to guarantee that the alignment can be
correct?

Worse, it seems that the same UTF8 char can have a "correct" width in
some fonts, and not in others... Not speaking that some UTF8 chars exist
in some fonts, and not in others...

Best regards,
  Seb

[1] Here is my setup:

--8<---------------cut here---------------start------------->8---
(setq gnus-unread-mark ?✉)
(setq gnus-del-mark ?✗)
(setq gnus-read-mark ?✓)
(setq gnus-killed-mark ?☠)
(setq gnus-unseen-mark ?✩)
--8<---------------cut here---------------end--------------->8---

-- 
Sebastien Vauban

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Eli Zaretskii]

> From: Sebastien Vauban <sva-news@mygooglest.com>
> Date: Sun, 15 Feb 2015 10:05:58 +0100
> 
> Now that I've changed some Gnus markers to UTF8 "drawings" [1], I don't
> have my Gnus summary buffer correctly aligned anymore.
> 
> See http://screencast.com/t/s33eYYg7Cn.
> 
> Is there a solution to that, to guarantee that the alignment can be
> correct?

Only if Gnus will align text using the :align-to display property,
instead of inserting whitespace characters.

> Worse, it seems that the same UTF8 char can have a "correct" width in
> some fonts, and not in others...

Of course: it depends on the dimensions of the glyphs in each font.

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Sebastien Vauban]

Eli Zaretskii wrote:
>> From: Sebastien Vauban <sva-news-D0wtAvR13HarG/iDocfnWg@public.gmane.org>
>> Date: Sun, 15 Feb 2015 10:05:58 +0100
>> 
>> Now that I've changed some Gnus markers to UTF8 "drawings" [1], I don't
>> have my Gnus summary buffer correctly aligned anymore.
>> 
>> See http://screencast.com/t/s33eYYg7Cn.
>> 
>> Is there a solution to that, to guarantee that the alignment can be
>> correct?
>
> Only if Gnus will align text using the :align-to display property,
> instead of inserting whitespace characters.

So, I take it for granted that it doesn't use it yet.  Is this quite
new?

>> Worse, it seems that the same UTF8 char can have a "correct" width in
>> some fonts, and not in others...
>
> Of course: it depends on the dimensions of the glyphs in each font.

Yes, but I was wondering (or hoping) if there was a mechanism in Emacs
to sort of zoom in/out the characters so that they'd take the same space
regardless of their (buggy?) definition (buggy when in
a non-proportional font)...

PS- This problem may occur, maybe, because of automatic font replacement
for characters not found in my default font (Consolas)?

Best regards,
  Seb

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Eli Zaretskii]

> From: Sebastien Vauban <sva-news@mygooglest.com>
> Cc: 19872@debbugs.gnu.org
> Date: Mon, 16 Feb 2015 10:45:14 +0100
> 
> >> Is there a solution to that, to guarantee that the alignment can be
> >> correct?
> >
> > Only if Gnus will align text using the :align-to display property,
> > instead of inserting whitespace characters.
> 
> So, I take it for granted that it doesn't use it yet.

I guess, judging by your description.  I don't use Gnus, but you can
easily verify that by looking at each character in the offending line
with "C-x =": if all of the whitespace characters are simple blanks or
TABs, then you know.

> Is this quite new?

You mean, :align-to?  It was new 15 years or so ago.

> >> Worse, it seems that the same UTF8 char can have a "correct" width in
> >> some fonts, and not in others...
> >
> > Of course: it depends on the dimensions of the glyphs in each font.
> 
> Yes, but I was wondering (or hoping) if there was a mechanism in Emacs
> to sort of zoom in/out the characters so that they'd take the same space

No, it doesn't.  (Is that at all possible?  I'm not expert on fonts.)
Emacs simply chooses a font whose size matches the best what it needs.

> regardless of their (buggy?) definition (buggy when in
> a non-proportional font)...

They are not buggy.  The font designer(s) decided which size to use
for each glyph.  It's a profession and a discipline.

> PS- This problem may occur, maybe, because of automatic font replacement
> for characters not found in my default font (Consolas)?

Yes, and I'm guessing this is what happened in your case.  You can see
which font each character came from by using "C-u C-x =".

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Stefan Monnier]

> No, it doesn't.  (Is that at all possible?  I'm not expert on fonts.)

Yes, it's definitely possible by scaling the font appropriately (and
not necessarily by the same amount in X as in Y).  To do that we'd first
have to introduce some kind of annotation/variable to indicate that the
text is intended to be fixed width.


        Stefan

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Lars Ingebrigtsen]

>> > Only if Gnus will align text using the :align-to display property,
>> > instead of inserting whitespace characters.
>> 
>> So, I take it for granted that it doesn't use it yet.

Using align-to in the summary buffer wouldn't help much directly,
because Gnus also has to limit the width of the strings.  That can be
done with vertical-motion and stuff, but would make generating the
buffer slow.

If Emacs had a version of align-to that really aligned to the specified
pixel position, even if the text displayed there is wider than that
position, then that would be nice.

(That is, it would have to truncate the text if it's too wide.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Sebastien Vauban <sva-news@mygooglest.com>, 19872@debbugs.gnu.org
> Date: Tue, 17 Feb 2015 15:17:28 +1100
> 
> Using align-to in the summary buffer wouldn't help much directly,
> because Gnus also has to limit the width of the strings.

No, you just need to place the display property on the first character
that exceeds the width limit.

> That can be done with vertical-motion and stuff, but would make
> generating the buffer slow.

I don't see why would you need to do all that.  First, you already do
these calculations, to know how many blanks to insert, right?  So you
already know whether a string is too long, at least in terms of
characters, right?  And :align-to can work in character units as well
as in pixels.

And second, AFAIU you are talking about an additional feature.  The OP
presented a use case where no string is too long, AFAICT.  So it would
get you bonus points to handle long strings as well, but that's not
what this bug report is about: the same problem exists with the
current "alignment" using whitespace, right?

> If Emacs had a version of align-to that really aligned to the specified
> pixel position, even if the text displayed there is wider than that
> position, then that would be nice.
> 
> (That is, it would have to truncate the text if it's too wide.)

The things people expect the display engine to do...

I think it would be wrong for the display engine to automatically
truncate text in this case, since there's also the use case where the
alignment is only meant to be in effect when preceding text is not
long enough, similar to minimum width specification in formatted
output.

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't see why would you need to do all that.  First, you already do
> these calculations, to know how many blanks to insert, right?  So you
> already know whether a string is too long, at least in terms of
> characters, right?  And :align-to can work in character units as well
> as in pixels.

Well, the problem here is that some fonts are wider than others.  If
Gnus says "this should be 20 characters wide", then if some of the
glyphs are wider than the normal 20 characters, then things won't line
up any more.

You could reserve more space for these instances, but that would leave
too much white space normally.

> And second, AFAIU you are talking about an additional feature.  The OP
> presented a use case where no string is too long, AFAICT.  So it would
> get you bonus points to handle long strings as well, but that's not
> what this bug report is about: the same problem exists with the
> current "alignment" using whitespace, right?

Gnus truncates the strings if they're too long and inserts spaces if
they're too short.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: sva-news@mygooglest.com,  19872@debbugs.gnu.org
> Date: Wed, 18 Feb 2015 11:43:26 +1100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't see why would you need to do all that.  First, you already do
> > these calculations, to know how many blanks to insert, right?  So you
> > already know whether a string is too long, at least in terms of
> > characters, right?  And :align-to can work in character units as well
> > as in pixels.
> 
> Well, the problem here is that some fonts are wider than others.  If
> Gnus says "this should be 20 characters wide", then if some of the
> glyphs are wider than the normal 20 characters, then things won't line
> up any more.

AFAIR, :align-to works in units of canonical character width, so this
problem does not exist.

> > And second, AFAIU you are talking about an additional feature.  The OP
> > presented a use case where no string is too long, AFAICT.  So it would
> > get you bonus points to handle long strings as well, but that's not
> > what this bug report is about: the same problem exists with the
> > current "alignment" using whitespace, right?
> 
> Gnus truncates the strings if they're too long and inserts spaces if
> they're too short.

Then I think you already have everything in place, just replace
insertion of blanks with a :align-to display property.

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAIR, :align-to works in units of canonical character width, so this
> problem does not exist.

I don't know what you mean by "canonical character width".

If we've reserved space for 20 default-width characters, and we have a
strings like

  [12345678901234567890] Foo
  [12345678901234567890] Foo
  [12345678901234567890] Foo
  [廣東話廣東話廣東話廣東話廣東話廣東話廣東] Foo
  [12345678901234567890] Foo
  [12345678901234567890] Foo
  [12345678901234567890] Foo
  [12345678901234567890] Foo

no amount of align-to will make these columns line up.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: sva-news@mygooglest.com,  19872@debbugs.gnu.org
> Date: Thu, 19 Feb 2015 16:49:53 +1100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > AFAIR, :align-to works in units of canonical character width, so this
> > problem does not exist.
> 
> I don't know what you mean by "canonical character width".

It's the average width of the current frame's 'default' face's font.
IOW, the value returned by 'frame-char-width'.  Display-related
functions usually count "columns" in these units.

> If we've reserved space for 20 default-width characters, and we have a
> strings like
> 
>   [12345678901234567890] Foo
>   [12345678901234567890] Foo
>   [12345678901234567890] Foo
>   [廣東話廣東話廣東話廣東話廣東話廣東話廣東] Foo
>   [12345678901234567890] Foo
>   [12345678901234567890] Foo
>   [12345678901234567890] Foo
>   [12345678901234567890] Foo
> 
> no amount of align-to will make these columns line up.

That's a separate problem.  I thought you said that you were
truncating too long strings, so I thought these cases were already
taken care of.

Do you use something like string-width or char-width to measure the
width of strings on display while accounting for wide characters like
the ones above and for zero-width combining characters?  E.g., in this
case string-width says that the string of Kanji characters is
40-column wide, even though it consists of only 20 characters.  And
for a string such as "ẛ̣", string-width returns 1, even though there
are 3 characters there: u+017f, u+0323, and u+0307, because Emacs
composes them on display into a single glyph (a.k.a. "grapheme
cluster").  Since these strings typically use different fonts, the
results are only approximately correct, but they are a much better
approximation than if you count each character as 1 column on display.

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Stefan Monnier]

>> [12345678901234567890] Foo
>> [12345678901234567890] Foo
>> [12345678901234567890] Foo
>> [廣東話廣東話廣東話廣東話廣東話廣東話廣東] Foo
>> [12345678901234567890] Foo
>> [12345678901234567890] Foo
>> [12345678901234567890] Foo
>> [12345678901234567890] Foo
>> 
>> no amount of align-to will make these columns line up.

> That's a separate problem.  I thought you said that you were
> truncating too long strings, so I thought these cases were already
> taken care of.

The problem is that truncating based on display width is difficult and
costly if we want to handle proportional fonts.

That's why Lars said:

   Using align-to in the summary buffer wouldn't help much directly,
   because Gnus also has to limit the width of the strings.  That can be
   done with vertical-motion and stuff, but would make generating the
   buffer slow.


-- Stefan

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, sva-news@mygooglest.com,
>         19872@debbugs.gnu.org
> Date: Thu, 19 Feb 2015 08:49:32 -0500
> 
> >> [12345678901234567890] Foo
> >> [12345678901234567890] Foo
> >> [12345678901234567890] Foo
> >> [廣東話廣東話廣東話廣東話廣東話廣東話廣東] Foo
> >> [12345678901234567890] Foo
> >> [12345678901234567890] Foo
> >> [12345678901234567890] Foo
> >> [12345678901234567890] Foo
> >> 
> >> no amount of align-to will make these columns line up.
> 
> > That's a separate problem.  I thought you said that you were
> > truncating too long strings, so I thought these cases were already
> > taken care of.
> 
> The problem is that truncating based on display width is difficult and
> costly if we want to handle proportional fonts.

I agree in general, but cannot see why this particular case couldn't
be resolved reasonably well, e.g. by adding one more column "just in
case", before the :align-to property.

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Stefan Monnier]

> If Emacs had a version of align-to that really aligned to the specified
> pixel position, even if the text displayed there is wider than that
> position, then that would be nice.

Indeed, I already mentioned this need for the redisplay to provide
a feature that makes it truncate data for columns (as in
tabulated-list-mode).


        Stefan

bug#19872: 24.4; UTF8 characters of unusual width (Gnus markers)

[Lars Ingebrigtsen]

Emacs now has functions to determine the pixel width of text, so Gnus
could use those to limit the lengths, and use :align-to to do the
padding.

So I think this would finally be possible to fix.  It might make
generating large summary buffers a bit slower, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no