Unicode combining characters

Unicode combining characters

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

Hindi Devanagari script has lot of unicode combining characters which
results in misalignment in a rectangular overlay for constant number of
characters  (screenshot )
<https://1.bp.blogspot.com/-P2ZnFePOpOo/YK0cNJ4B5II/AAAAAAAAJJs/t-MADtxUeps3S_WXZ_rFWjf9daH49sr9QCLcBGAsYHQ/s421/combining.png>
What would be a recommended way to tackle this in Emacs?

[-- Attachment #2: Type: text/html, Size: 429 bytes --]

Re: Unicode combining characters

[Stefan Monnier]

> Hindi Devanagari script has lot of unicode combining characters which
> results in misalignment in a rectangular overlay for constant number of
> characters  (screenshot )
> <https://1.bp.blogspot.com/-P2ZnFePOpOo/YK0cNJ4B5II/AAAAAAAAJJs/t-MADtxUeps3S_WXZ_rFWjf9daH49sr9QCLcBGAsYHQ/s421/combining.png>
> What would be a recommended way to tackle this in Emacs?

In a GUI session, the usual answer is to use posframe, AFAIK.


        Stefan

Re: Unicode combining characters

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Tue, 25 May 2021 21:26:44 +0530
> 
> Hindi Devanagari script has lot of unicode combining characters which results in misalignment in a
> rectangular overlay for constant number of characters (screenshot ) 
> What would be a recommended way to tackle this in Emacs?

Use align-to 'space' display spec and/or the window-text-pixel-size
function, which will account for the actual size of the text on
display.  string-width can also be used, but it only gives an
approximation, as it is oblivious of the actual size of the font
glyphs.

Re: Unicode combining characters

[Clément Pit-Claudel]

On 5/25/21 1:24 PM, Eli Zaretskii wrote:
>> From: Anand Tamariya <atamariya@gmail.com>
>> Date: Tue, 25 May 2021 21:26:44 +0530
>>
>> Hindi Devanagari script has lot of unicode combining characters which results in misalignment in a
>> rectangular overlay for constant number of characters (screenshot ) 
>> What would be a recommended way to tackle this in Emacs?
> 
> Use align-to 'space' display spec and/or the window-text-pixel-size
> function, which will account for the actual size of the text on
> display. 

Will this work? The misaligned specs are already part of a replacing dipsplay spec, so the additional align-to would be ignored, no?

(IIRC, there is no way to say "replace this text by this string followed by this specified space; it's one or the other, right?)

Re: Unicode combining characters

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 25 May 2021 14:15:33 -0400
> 
> On 5/25/21 1:24 PM, Eli Zaretskii wrote:
> >> From: Anand Tamariya <atamariya@gmail.com>
> >> Date: Tue, 25 May 2021 21:26:44 +0530
> >>
> >> Hindi Devanagari script has lot of unicode combining characters which results in misalignment in a
> >> rectangular overlay for constant number of characters (screenshot ) 
> >> What would be a recommended way to tackle this in Emacs?
> > 
> > Use align-to 'space' display spec and/or the window-text-pixel-size
> > function, which will account for the actual size of the text on
> > display. 
> 
> Will this work? The misaligned specs are already part of a replacing dipsplay spec, so the additional align-to would be ignored, no?

I don't understand, but maybe you know about the particular use case
more than I do.  I just mentioned two devices that can be accurate to
1 pixel wrt to the X coordinate.

> (IIRC, there is no way to say "replace this text by this string followed by this specified space; it's one or the other, right?)

Again, I don't think I follow.  If you have "this text", you can
calculate its width on display, and then know how many pixels of white
space you will need after "this string" replaces that text.  So,
unless I'm missing something, specifying the space width is redundant,
and actually makes a solvable problem unsolvable.

But I might be talking nonsense because I don't understand what
problem the OP wants to solve.

Re: Unicode combining characters

[Clément Pit-Claudel]

On 5/25/21 2:39 PM, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 25 May 2021 14:15:33 -0400
>>
>> On 5/25/21 1:24 PM, Eli Zaretskii wrote:
>>>> From: Anand Tamariya <atamariya@gmail.com>
>>>> Date: Tue, 25 May 2021 21:26:44 +0530
>>>>
>>>> Hindi Devanagari script has lot of unicode combining characters which results in misalignment in a
>>>> rectangular overlay for constant number of characters (screenshot ) 
>>>> What would be a recommended way to tackle this in Emacs?
>>>
>>> Use align-to 'space' display spec and/or the window-text-pixel-size
>>> function, which will account for the actual size of the text on
>>> display. 
>>
>> Will this work? The misaligned specs are already part of a replacing dipsplay spec, so the additional align-to would be ignored, no?
> 
> I don't understand, but maybe you know about the particular use case
> more than I do.  I just mentioned two devices that can be accurate to
> 1 pixel wrt to the X coordinate.
> 
>> (IIRC, there is no way to say "replace this text by this string followed by this specified space; it's one or the other, right?)
> 
> Again, I don't think I follow.  If you have "this text", you can
> calculate its width on display, and then know how many pixels of white
> space you will need after "this string" replaces that text.  So,
> unless I'm missing something, specifying the space width is redundant,
> and actually makes a solvable problem unsolvable.

Based on the screenshot this is an issue with Company.  Company displays its "pop-ups" by putting a replacing 'display property on the text following the point (and on the next few lines).  So if the buffer contains

ABC XYZ DEF GHI
JKL MNO PQR STU

and the point is after XYZ, then company puts a replacing display spec from " DEF" to "STU".
To display completions "XYZ1233" and "XYZ456", the replacing display spec contains "123| GHI\nJKL XYZ456| STU", so the final display is

ABC XYZ123| GHI
JKL XYZ456| STU

The OP's issue is that "123" and "456" don't have the same length.  As far as I know, there is no way to add extra space after 123 or 456 so that they reach the same X coordinate, given that they are already part of a display spec.

Clément.

Re: Unicode combining characters

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 25 May 2021 15:30:21 -0400
> 
> Based on the screenshot this is an issue with Company.  Company displays its "pop-ups" by putting a replacing 'display property on the text following the point (and on the next few lines).  So if the buffer contains
> 
> ABC XYZ DEF GHI
> JKL MNO PQR STU
> 
> and the point is after XYZ, then company puts a replacing display spec from " DEF" to "STU".
> To display completions "XYZ1233" and "XYZ456", the replacing display spec contains "123| GHI\nJKL XYZ456| STU", so the final display is
> 
> ABC XYZ123| GHI
> JKL XYZ456| STU
> 
> The OP's issue is that "123" and "456" don't have the same length.  As far as I know, there is no way to add extra space after 123 or 456 so that they reach the same X coordinate, given that they are already part of a display spec.

First, the OP said "overlay", and overlay strings can have display
properties.

And second, I'd expect the current code to use string-width to compute
how much whitespace will be needed after each completion candidate,
and string-width already accounts for composed (a.k.a "combined")
characters.  Yes, string-width provides only an approximation for the
true pixel width of the string, but that's not specific to
compositions, and the whole technique is somewhat of a kludge anyway,
for this reason and others.

Re: Unicode combining characters

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

Thanks Eli - align-to 'space' display spec seems helpful.

Though it's a company specific issue related to unicode character
composition, here's some more details on the issue for record should
somebody else stumble upon the same.

Let's call the first character in the screenshot as shr (single glyph) and
the second one as sh-r (two glyphs).
(setq shr  (string 2358 2381 2352))
(setq sh-r (string 2358 2352))

(string-width shr)  ;; 2
(string-width sh-r) ;; 2

To create the rectangular region, we need to pad the strings with
appropriate number of spaces. align-to 'space' display spec seems helpful
in this case as shown below. You will notice that character "a" is aligned
in both cases. Now I need to figure out how to use the same within company.

(insert (concat shr
(let ((sp " "))
 (font-lock-append-text-property 0 1 'display `(space . (:align-to 10)) sp)
 sp)
"a"))

(insert (concat sh-r
(let ((sp " "))
 (font-lock-append-text-property 0 1 'display `(space . (:align-to 10)) sp)
 sp)
"a"))

[-- Attachment #2: Type: text/html, Size: 1391 bytes --]

Re: Unicode combining characters

[Joost Kremers]

On Wed, May 26 2021, Anand Tamariya wrote:
> Thanks Eli - align-to 'space' display spec seems helpful.
>
> Though it's a company specific issue related to unicode character
> composition, here's some more details on the issue for record should
> somebody else stumble upon the same.

At the risk of posting something irrelevant: the effect shown in the screen shot
you posted also occurs if you use company in a buffer with variable-pitch-mode
(which I do in e.g., LaTeX buffers). I don't know if that's the same problem,
but if it is, a solution would be applicable beyond combining characters. 

-- 
Joost Kremers
Life has its moments

Re: Unicode combining characters

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Wed, 26 May 2021 15:21:05 +0530
> Cc: emacs-devel@gnu.org
> 
> Let's call the first character in the screenshot as shr (single glyph) and the second one as sh-r (two glyphs).
> (setq shr  (string 2358 2381 2352))
> (setq sh-r (string 2358 2352))
> 
> (string-width shr)  ;; 2
> (string-width sh-r) ;; 2

Sorry, it turns out I've misremembered: string-width doesn't account
for "automatic compositions", the ones that happen due to
composition-function-table (as opposed to "static compositions" which
happen due to the 'composition' text property).  So this case
currently cannot be handled correctly by string-width; we should fix
that.

Re: Unicode combining characters

[Eli Zaretskii]

> Date: Wed, 26 May 2021 15:54:43 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > (setq shr  (string 2358 2381 2352))
> > (setq sh-r (string 2358 2352))
> > 
> > (string-width shr)  ;; 2
> > (string-width sh-r) ;; 2
> 
> Sorry, it turns out I've misremembered: string-width doesn't account
> for "automatic compositions", the ones that happen due to
> composition-function-table (as opposed to "static compositions" which
> happen due to the 'composition' text property).  So this case
> currently cannot be handled correctly by string-width; we should fix
> that.

Please try the latest master branch, I hope I fixed this now.

Re: Unicode combining characters

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

>
> Please try the latest master branch, I hope I fixed this now.
>

The fix works for the given example. However, here's another one that
ideally should be one composed glyph (validated by moving the cursor over
the glyph) but counts as 2 in string-width.

(setq ra (string 2352 2366))

(string-width ra) ; 2

;; Glyph in a word
(setq shankar (string 2358 2306 2325 2352 2366))

[-- Attachment #2: Type: text/html, Size: 715 bytes --]

Re: Unicode combining characters

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Thu, 27 May 2021 12:30:04 +0530
> Cc: emacs-devel@gnu.org
> 
>  Please try the latest master branch, I hope I fixed this now.
> 
> The fix works for the given example. However, here's another one that ideally should be one composed glyph
> (validated by moving the cursor over the glyph) but counts as 2 in string-width.
> 
> (setq ra (string 2352 2366))
> 
> (string-width ra) ; 2

OK, I improved this case now on master, please take a look.

However, please note that getting this right makes string-width more
dependent on the selected-frame's font used by the default face for
the characters of the string.  In particular, if that font is unable
to combine the characters that should be composed, you will now get
width which could be different from the value on other frames with
other fonts.  Also, the new code only works in interactive sessions on
GUI frames, because we need the shaping engine (a.k.a. "font driver")
to compose characters.

Re: Unicode combining characters

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

> OK, I improved this case now on master, please take a look

I'm seeing a couple of warnings:

character.c: In function ‘lisp_string_width’:
character.c:397:16: warning: assignment to ‘int’ from ‘Lisp_Object’
  {aka ‘struct Lisp_X *’} makes integer from pointer without a cast
  [-Wint-conversion]
  397 |     font_width = AREF (font_info, 11);
      |                ^
character.c:398:19: warning: ordered comparison of pointer with integer
  zero [-Wextra]
  398 |     if (font_info <= 0)
      |                   ^~
character.c:399:18: warning: assignment to ‘int’ from ‘Lisp_Object’ {aka
  ‘struct Lisp_X *’} makes integer from pointer without a cast
  [-Wint-conversion]
  399 |       font_width = AREF (font_info, 10);

Do the font_info elements need to be untagged, and font_width rather
than font_info checked for being positive?

Thanks,

-- 
Basil

gcc (Debian 10.2.1-6) 10.2.1 20210110

Re: Unicode combining characters

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: Anand Tamariya <atamariya@gmail.com>,  emacs-devel@gnu.org
> Date: Thu, 27 May 2021 11:34:55 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > OK, I improved this case now on master, please take a look
> 
> I'm seeing a couple of warnings:

Oops! sorry, should be fixed now.

Re: Unicode combining characters

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

> > The fix works for the given example. However, here's another one that
> ideally should be one composed glyph
> > (validated by moving the cursor over the glyph) but counts as 2 in
> string-width.
> >
> > (setq ra (string 2352 2366))
> >
> > (string-width ra) ; 2
>
> OK, I improved this case now on master, please take a look.
>
> Wonderful!! It works. Thanks.
Do you think (current-column) should also return a value conforming to the
display logic? e.g. if 'ra' above is the first character in the line and
point next to it, should it report 1 or 2?

[-- Attachment #2: Type: text/html, Size: 831 bytes --]

Re: Unicode combining characters

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Thu, 27 May 2021 18:57:58 +0530
> Cc: emacs-devel@gnu.org
> 
>  OK, I improved this case now on master, please take a look.
> 
> Wonderful!! It works. Thanks.

Thanks for testing.

> Do you think (current-column) should also return a value conforming to the display logic? e.g. if 'ra' above is
> the first character in the line and point next to it, should it report 1 or 2?

That'd be too much, IMO.  current-column is called in many places, and
it would be unexpected for it to return different values depending on
the font and the frame.  The correspondence between these two
functions is not 100% now anyway (e.g., current-column is sensitive to
auto-composition-mode, whereas string-width isn't).

Lisp programs that need 100% accuracy in these matters should call
window-text-pixel-size.