Re: Ligatures (was: Unify the Platforms: Cairo+FreeType+Harfbuzz Everywhere (except TTY))

[Pip Cet <pipcet@gmail.com>]

On Sat, May 23, 2020 at 2:08 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Pip Cet <pipcet@gmail.com>
> > Date: Sat, 23 May 2020 12:36:56 +0000
> > Cc: cpitclaudel@gmail.com, alan@idiocy.org, emacs-devel@gnu.org
> >
> > > > You write: "(b) is not really feasible without redesigning the entire
> > > > Emacs display engine". I don't see how that's true at all. All we need
> > > > is some limited look-ahead.
> > >
> > > We already have look-ahead: that's what the regexp part of the
> > > composition rules are about.  That is not the crucial problem.
> >
> > But it's the only problem I see!
>
> Then maybe I don't understand what you mean by look-ahead.  Is that
> the decision how to choose those 32 characters of "context"?

Yes.

> Then why
> not use the current regexp-based approach, which is already much
> smarter than just blindly taking a fixed amount of surrounding text?

Because I do not know the regexp to use?

> > When you see an IT_CHARACTER, you get some context, hand it to
> > HarfBuzz, slice up the relevant glyphs, and display them.
>
> The problem is, of course, in the "some context" part.  Your patch
> used an arbitrary 32-character chunk of text around the character to
> shape, which is of course not what the shaping engines want: they want
> _all_ of the surrounding text, the entire paragraph.

Which is clearly too expensive to actually give them, which is
something I didn't think it was necessary to even spell out.

> Your patch also invokes the shaper twice, on the same 32 characters,
> once in encode_char method and again in the text_extents method, which
> is another waste.  The code in composite.c caches the composed
> characters to avoid that, but you bypass it.

Absolutely.

> This is okay for showing the concept, but we cannot use this in
> production.  There are too many arbitrary decisions and inefficient
> expensive operations.

I agree, of course! In fact, the 32-character limit was chosen as a
reminder to myself that things would inherently be inefficient.

> > It doesn't involve composite.c at all, and that's good, because for
> > those tricky special cases composite.c does a better job than standard
> > shaping, and we need to keep that feature. It just shouldn't be the
> > regular route.
>
> Of course, you never tell how to distinguish between the "tricky
> special cases" for which we still need to use composite.c and friends,
> and the other kind.

The tricky special cases get handled as before, and come in with the
iterator .what set to IT_COMPOSITE. The standard cases come in with
.what set to IT_CHARACTER.

> Moreover, the HarfBuzz guys clearly say that what we do now is wrong
> for those "tricky" cases as well, so if we are going to fix that, why
> fix it only for ligatures made out of ASCII characters?

There's no such limitation, but, yes, ideally people would find they
don't need automatic compositions anymore...

> > > The crucial problem is that we currently perform layout decisions one
> > > grapheme cluster at a time, whereas what HarfBuzz people say is that
> > > we should basically do that one screen line at a time.
> >
> > I think we're going to have to compromise: that's why my patch used a
> > 32-character context rather than an entire line or just a single
> > character.
>
> If we are going to compromise, then why not compromise on what we
> already have, which is much less than 32 characters?

0 characters?

> Why should we
> enormously complicate and slow down our code without actually solving
> the problem?

We shouldn't.

> Did you ever see ligatures that are 32-character long?

"Zapfino" is the longest I've seen.

> > Ideally, of course, in most real cases we'd use whitespace-delimited
> > words as chunks. That's mere optimization, though.
>
> That'd be the wrong optimization, AFAIK.

Sure, but since it is exclusively an optimization, it's performance
considerations alone that will decide whether it is.

> E.g., some scripts don't
> have whitespace separated words at all, and still need shaping.

Thus "most".

> And
> what exactly is whitespace for this purpose? e.g., does it include
> Unicode control characters such as ZWJ?

Thankfully, that doesn't matter much: it's just a question of what we
optimize for, not one of what the results will look like.

So I'd say " ", "\t", and "\n" are enough, which is what the display
engine already handles specially.

> > > A secondary (but important) problem is that character composition
> > > involves calls to Lisp, which is relatively slow.  This precludes
> > > calling the shaper for too many characters at once, too many times for
> > > each redisplay cycle of a window.
> >
> > I agree we shouldn't go through Lisp. My patch didn't.
>
> Your patch hard-codes arbitrary numbers without any way to control
> that from Lisp.

Yes.

> Such code will never fly in Emacs.

Of course not.

> > Calling the shaper less often is an important optimization, too. For
> > whitespace-delimited words, we only need to call it once.
>
> This doesn't work when the produced sequence of glyphs doesn't fit on
> the screen line.

> What the current layout code does in this case won't
> work well when you need to break a long sequence of glyphs in the
> middle and then continue on the next line from where you left off on
> this one.

You mean in visual-mode? Because what the current layout code does by
default is to break along any glyph boundary, and I don't see how
that's broken in any way.

> The longer the sequence of glyphs you get from the shaper
> in one go, the higher the probability of hitting this issue.

You break between the glyphs. It doesn't depend on whether you have
two or 20 or 100.

> The bottom line of this is that I think you will find very quickly
> that the basic assumptions of the current design -- that we produce
> single glyphs or very short sequences of them for each call to the
> shaper -- that these assumptions bite you on every step, because the
> code which deals with layout implicitly assumes this.

The shaper interface I described would actually return a single glyph
for each top-level call, with a number of callbacks to provide
context. So that assumption would hold up very well indeed...

> In short, I really don't see how this could ever work, except in a
> very limited set of simple use cases.  E.g., what do you do with
> bidirectional text? ignore it?

A bidi boundary is a hard boundary for HarfBuzz, and no shaping
happens across it. Is that what you mean by "ignore it"?

> > > I don't think there's any disagreements on this high and abstract
> > > level.
> >
> > I think there are: if we treat fonts as programs, we need to let them
> > do their job, which involves kerning, substitutions, ligatures, and
> > even crazy stuff like randomizing the glyph used for each character to
> > get a more hand-written appearance. We don't need to know about
> > ligatures, we just let the font do it. No Lisp callbacks, just a call
> > to harfbuzz.
>
> I think this is a simplistic view of how the display engine works,

Quite possibly :-)

> and
> I don't see how it could work in production while supporting all the
> use cases we already do.

It only comes in for use cases not handled otherwise, i.e. those where
the iterator is at an IT_CHARACTER. All other use cases are
unaffected, because they mean we're overriding the font decision
anyway.

As I said, the problem I have is to get look-ahead working, which you
think isn't a problem. I've got an idea for it, but it doesn't work
(yet); my theory is the bidi.c code fails to keep its state in the
iterator and can't deal with multiple parallel iterators.

> I could be wrong, though, so I'm looking
> forward to see you present a series of patches that do support the
> existing use cases and the ligatures as well, and don't cause any
> slowdown in redisplay.

As I said, what's stopping me is the look-ahead problem, and in
particular some code in bidi.c that doesn't play along well with
look-ahead.