Re: Ligatures (was: Unify the Platforms: Cairo+FreeType+Harfbuzz Everywhere (except TTY))

[Pip Cet <pipcet@gmail.com>]

Hi, Eli,

On Wed, May 20, 2020 at 3:31 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Pip Cet <pipcet@gmail.com>
> > Date: Tue, 19 May 2020 21:43:49 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, Alan Third <alan@idiocy.org>, emacs-devel@gnu.org
> >
> > And I'm afraid the difference is much more obvious with box cursors
> > than it is with carets. I'm attaching a screenshot of a patched Emacs
> > displaying "ffi", with point on the second f, in the "Linux Libertine
> > Display O" font (using approximately equal slices).
> >
> > I think this is a bit of a worst-case scenario, a three-letter
> > ligature in a font using ligatures and overhangs very
> > enthusiastically. It might be okay for other fonts.
>
> I'm not sure this is the worst case.  It might be the worst case if we
> are talking about ligatures that involve only ASCII characters, and
> don't involve symbols like ==> that gets converted to ⇒.  But in
> general, there are worse cases, like á (two codepoints).  And for
> kicks see the Khmer hello in etc/HELLO, where you can find 4
> codepoints that produce a grapheme cluster made of 3 glyphs.

You're correct: I'm simply not dealing with Khmer or composed
characters (which are different from ligatures, of course) in the
patch, and I'm not certain how to deal with them in theory, either.

> If we only want this feature for ASCII ligatures, then it sounds like
> a limitation to me (and frankly, somewhat unclean as features go),

Not "only for ASCII ligatures", but not "any conceivable combination
of codepoints into glyphs" either. Just those supported by the font
and Harfbuzz.

> but
> if we really want this only for these limited cases, we will need to
> somehow indicate to the display engine which ligatures are to be
> handled like this and which aren't.

Well, we now know that fonts can provide information about how a
ligature is to be split into one-dimensional slices; I filed a pull
request against Harfbuzz (since merged) that would actually make the
corresponding API work, at least for the "Libertinus" font family.

Of course that means that Emacs behavior would depend on the font
tables in ways it currently doesn't. That's a problem.

> > My remaining idea is to stretch characters so we can break up a
> > ligature without changing its total width. I'm not sure how to do
> > that, though.
>
> I don't think I understand what you'd like to do.  Can you elaborate?

My idea was to display "ffi" with the point on the second f by
condensing an "f" glyph to cover the middle third of the "ffi" glyph.
However, I might have been too critical of how good the simple
solution deals with this case.

> > (I'm also attaching the patch, for the morbidly curious; it isn't
> > clean, readable, or finished in any way, and contains at least one
> > obvious bug. It's just good enough to produce the screenshot, and
> > maybe it can serve as a hint as to which files need changing for
> > ligatures to work; but such changes would have to be done very
> > differently from the patch.).
>
> Right, the actual implementation will have to be different.  In
> particular, I think that if ligatures will use automatic compositions,
> the information you need is already stored in the composition table
> and reachable from the glyph string, so you don't need to invoke the
> shaper again.

Well, I'm sorry to bring up a different (though somewhat related
issue), but kerning is also an issue: we need a shaper to get that
right, not just a composition table, right?

> I see you implemented this for static compositions, which are
> semi-obsolete.

I'm sorry, I'm afraid I don't understand. This should handle any
composition the shaper does, and only those, but slices up everything
horizontally by default.

> Also, I don't see the code which moves point inside
> the ligature; Emacs will not allow doing that by default.  In
> particular, how did you tell the display code to show the cursor on
> the middle 'f', not on the first one?  Did I miss something?

I produce three "struct glyph"s for "ffi": each has width one third of
the actual font glyph, and stores, in convoluted form, information
about which slice of the font glyph is to be actually drawn.

> And finally, you said you intended to do this via row->clip, but this
> patch does something very different.  What changed your mind?

I was surprised this no longer seemed to be strictly necessary: as far
as the display code is concerned, we're dealing with three separate
glyphs with overhang areas, and those are already handled by the
cursor-drawing code.

Clipping is still needed: to deal with double-drawing issues, and to
deal with such crimes as making part of a ligature have a different
foreground color.

I'm sorry it's not particularly obvious from the patch, but the
approach I took yesterday is this:

1. every struct glyph has a "context", which specifies the character
for the struct glyph and some surrounding text.
2. every struct glyph is converted to a slice of (currently) a single
font glyph, by sending the context through the shaper and cutting out
the relevant bits
3. struct glyphs are displayed one by one

Problems:
1. ligatures can cross line boundaries
2. the context has to be updated, and trigger redisplay of the struct glyph
3. clipping is necessary
4. there are N clipped drawing operations for a single glyph covering
N struct glyphs.
5. corner cases can have ambiguous context: for example, a string of
many "f"s would be paired into "ff" glyphs, and simply cutting off the
context after a certain number of characters might result in the wrong
pairing

On the other hand, it deals with kerning as well as ligatures. And
other problems (right now, we call the shaper on 64 characters for
every character we actually display, which makes things noticeably
slow) are fixable.

Overall, I'd like to think more about alternative approaches to the
"context string" one before implementing anything. How would that work
for kerning, in particular?