Re: Ligatures (was: Unify the Platforms: Cairo+FreeType+Harfbuzz Everywhere (except TTY))

[Pip Cet <pipcet@gmail.com>]

On Thu, May 21, 2020 at 2:11 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Pip Cet <pipcet@gmail.com>
> > Date: Thu, 21 May 2020 10:01:03 +0000
> > Cc: cpitclaudel@gmail.com, alan@idiocy.org, emacs-devel@gnu.org
> > > but
> > > if we really want this only for these limited cases, we will need to
> > > somehow indicate to the display engine which ligatures are to be
> > > handled like this and which aren't.
> >
> > Well, we now know that fonts can provide information about how a
> > ligature is to be split into one-dimensional slices;
>
> The question is: do we want to show those carets for all the character
> compositions, even if the information is provided?  If not, we will
> have to indicate somehow whether they should or shouldn't be shown for
> each particular grapheme cluster.

Oh. I hadn't thought about fonts providing such caret information in
cases where they shouldn't, but of course that's a valid concern.

> > Of course that means that Emacs behavior would depend on the font
> > tables in ways it currently doesn't. That's a problem.
>
> It isn't a problem to depend on that if most fonts provide this
> information.

> Then we could simply say this is not supported when the
> information is not in the font.

I'm not sure how simple that would be: we could treat ligatures
without carets as atomic, or we could tell harfbuzz not to apply
ligatures without carets, or maybe make that decision depend on
whether the ligature is required or discretionary...

> But if many fonts that support
> ligatures don't provide this information, we will need to have some
> fallback, like assume that every codepoint has the same share of the
> ligature's width.  the fact that other applications use a simplistic
> heuristic and not the information in the fonts suggests that either
> the information is not readily available or there are some other
> problems with using it.

Correct, it does. I'm not sure which one is the case.

> > > Right, the actual implementation will have to be different.  In
> > > particular, I think that if ligatures will use automatic compositions,
> > > the information you need is already stored in the composition table
> > > and reachable from the glyph string, so you don't need to invoke the
> > > shaper again.
> >
> > Well, I'm sorry to bring up a different (though somewhat related
> > issue), but kerning is also an issue: we need a shaper to get that
> > right, not just a composition table, right?
>
> Automatic compositions already use the shaper, see autocmp_chars.

I'm not sure I understand how kerning would work using automatic compositions.

> > > I see you implemented this for static compositions, which are
> > > semi-obsolete.
> >
> > I'm sorry, I'm afraid I don't understand. This should handle any
> > composition the shaper does, and only those, but slices up everything
> > horizontally by default.
>
> I'm talking about the changes in gui_produce_glyphs.  Its high-level
> structure is basically
>
>   if (it->what == IT_CHARACTER)
>     {
>     ... /* handles character glyphs */
>     }
>   else if (it->what == IT_COMPOSITION && it->cmp_it.ch < 0)
>     {
>     ... /* A static compositions. */
>     }
>   else if (it->what == IT_COMPOSITION)
>     {
>       /* A dynamic (automatic) composition.  */
>     }
>   [...]
>
> You made changes only in the "static compositions" part.

No. I didn't touch the "static compositions" part at all, except for
passing an extra NULL pointer to an API I'd extended. (At least,
that's what I intended, for all the changes to be in the IT_CHARACTER
part).

> That code
> handles compositions created by compose-region.  The "modern" way of
> composing text in Emacs uses automatic compositions, which are
> controlled by data in composition-function-table.  This is where we
> call the shaping engine to produce the glyphs according to rules
> stored in the font.  I don't see in your patch any changes that affect
> ligatures created by automatic compositions; did I miss something?

I don't think so; I went for a third route, that of leaving all
compositions handling to the shaper and doing none of it in Emacs
itself.

> If you use the automatic compositions route, then the information you
> need, i.e. the number of clusters in the shaped text and the overall
> width of the ligature, is already produced by the shaper and stored in
> the "gstring" object in the composition table, see the description of
> that object in the doc string of composition-get-gstring.  So there
> should be no need to invoke the shaper inside gui_produce_glyphs and
> elsewhere.  (If we want to use the carets information from the font,
> we will probably need to extend the gstring object to store that as
> well, and extend the shape method to extract this information when
> available.)

Yes, and that seemed too complicated for me for something that I
thought wouldn't handle kerning anyway...

> > > Also, I don't see the code which moves point inside
> > > the ligature; Emacs will not allow doing that by default.  In
> > > particular, how did you tell the display code to show the cursor on
> > > the middle 'f', not on the first one?  Did I miss something?
> >
> > I produce three "struct glyph"s for "ffi": each has width one third of
> > the actual font glyph, and stores, in convoluted form, information
> > about which slice of the font glyph is to be actually drawn.
>
> Ah, okay, I missed that.  But producing 3 glyphs instead of just one
> is not necessarily the best idea, I think.

I agree! I'd be happy to hear better ideas, and I think for now "use
fixed-width fonts" is a better idea...

> As you point out, one
> problem will be with splitting the ligature across lines.  Another
> problem is more expensive display.

You mean the actual "copy the glyph bitmap to the glass" display?
Because I don't think that's relevant. Overall redisplay() time really
goes up calling the shaper on 32 characters for every character
displayed, though, so that's a concern I agree with.

> And we won't be able to display
> the ligature as a single glyph, for those who want that, at least not
> easily.

But that's what they can do now, with the IT_COMPOSITION case, right?
Because I did not touch that code so I didn't expect that to break
(famous last words).

> > > And finally, you said you intended to do this via row->clip, but this
> > > patch does something very different.  What changed your mind?
> >
> > I was surprised this no longer seemed to be strictly necessary: as far
> > as the display code is concerned, we're dealing with three separate
> > glyphs with overhang areas, and those are already handled by the
> > cursor-drawing code.
>
> Yes.  But if we return to a single glyph, then we'd need to do some
> clipping.

As I said, we need to do the clipping to render antialiased pixels properly.

It's just two lines of code in ftcrfont_draw:

  cairo_rectangle (cr, x, y - FONT_BASE (face->font),
           s->width, FONT_HEIGHT (face->font));
  cairo_clip (cr);

> > On the other hand, it deals with kerning as well as ligatures.
>
> You mean, kerning of simple characters, for which we don't produce
> ligatures?

Yes, that's what I mean.

> Or kerning within ligatures?  If the latter, then I don't
> see why we'd need that: font designers already design the ligatures to
> have the optimal kerning, no?

It's certainly not our job to fix that if they don't!

Perhaps I can digress a little and describe what I think the
interaction with the shaper should be like:

Emacs: I'd like to display codepoint 'f'
Harfbuzz: you'll have to tell me the codepoint before that
Emacs: 'f'
Harfbuzz: and the one after those two
Emacs: 'i'
Harfbuzz: and the one before all of those
Emacs: That's too expensive for me to compute / it's the beginning of
paragraph / a bidi boundary / an object without an assigned codepoint
/ ...
Harfbuzz: okay, display it as the middle slice of the "ffi" glyph

I.e., I'd like Harfbuzz to be asynchronous, and request more
information, parsimoniously, about the context of the codepoint we're
describing, rather than working in one go from "complete" information
to an indefinitely-long line of glyphs. And deal well with us deciding
it's too expensive to perform that much look-back/look-ahead. (Because
in real life, ligatures depend on knowing some amount of the context,
but not all of it, or people could never start writing.)

Of course, all this doesn't change that the "struct it" design is
somewhat difficult to extend to handling look-ahead: it's easy enough
to create a copy of the iterator and advance that while leaving the
actual iterator intact, but it's also really slow. In fact I suspect
the best way would be to make struct it a heap-allocated pseudovector
(not necessarily one ordinarily garbage-collected, though), and cache
"future" iterator states once we compute them.

You're correct when you say that some major redesign is needed in this
area, but I don't think that's the subject of the current discussion.