unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Kerning and ligatures
@ 2020-05-30 22:13 Pip Cet
  2020-06-08 18:29 ` Yagnesh Raghava Yakkala
  0 siblings, 1 reply; 2+ messages in thread
From: Pip Cet @ 2020-05-30 22:13 UTC (permalink / raw)
  To: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 3050 bytes --]

This is a snapshot of my work on supporting kerning and ligatures; the
goal of this patch is to support those features for variable-pitch
fonts in English, but I'm not ignoring RTL languages, or scripts that
require shaping.

Like the first patch, this is meant to demonstrate the behavior I
would like to see, not the implementation. It's not meant for
immediate inclusion; perhaps it can serve as a starting point for
someone wishing to work on this further. That said, this patch does
not re-use the composite.c code. I tried to, several times, but what
it does is simply too different to allow useful code-sharing.

What's it all about is entering text like "affiliation" or "AVATAR"
while using a variable-pitch font. The proper way of displaying this,
for many OpenType fonts with variable pitch, involves ligatures and
kerning.

A ligature is an OpenType glyph that represents several characters;
for example, "ffi" in "affiliation".  Kerning is the determination of
each character's x advance based on its context; for example, in
"AVATAR", the distance between "A", "V", and the following "A" is
reduced.

None of this complicated logic is in the patch; instead, it uses the
HarfBuzz library.

This is somewhat unfortunate because the HarfBuzz API is fairly
limited: instead of requesting context incrementally from the calling
program, it expects to be provided with a "complete" context which it
then shapes. Doing that is probably too expensive, so we provide
"enough" context based on a heuristic. (This is complicated because
getting context is expensive, and too much of it can be harmful; the
current approach is to take a lot of context so the horrible
performance reminds me to do something about it :-) ).

This code supports M-to-N correspondences between code points and
glyphs points; it does so by first producing a single struct glyph
covering all (N) glyphs, then slicing it into one slice for each of
the (M) code points.

Important differences between this code and the existing composite.c code:

1. You can "enter" ligatures, such as by moving point to the second
"f" in "affiliation". This also applies to clusters of glyphs not
usually thought of as ligatures: For accented letters written as two
codepoints, such as á, this may be useful.  For שָׁ, it's irritating.

2. This code does not allow HarfBuzz to default to the language your
system is set to use. That prevents something as innocuous as a
screenshot from revealing your likely location. (The current language
default is "en").

3. Ligatures and kerning work with display strings or overlay strings,
rather than ending there or even revealing replaced buffer text as the
existing code does.

4. This code breaks lines after every glyph slice. That is suboptimal
for many languages.

Can someone help with the Telugu "hello"? I wonder whether there
should be a gap, and how much of one, between స్కా and రం in నమస్కారం?

PS: Sorry for the amount of recent noise.

[-- Attachment #2: 0001-Snapshot-of-kerning-ligatures-work.-Do-not-merge.patch --]
[-- Type: text/x-patch, Size: 64441 bytes --]

From d56171e67fdbf6a56cb134bf07916b54c5d6426e Mon Sep 17 00:00:00 2001
From: Pip Cet <pipcet@gmail.com>
Date: Sat, 30 May 2020 21:58:41 +0000
Subject: [PATCH] Snapshot of kerning/ligatures work. Do not merge.

---
 src/alloc.c      |  13 +-
 src/composite.c  |  10 +-
 src/data.c       |   1 +
 src/dispextern.h |  28 ++-
 src/font.c       |  31 ++--
 src/font.h       |  42 ++++-
 src/ftcrfont.c   | 423 +++++++++++++++++++++++++++++++++++++++------
 src/hbfont.c     |   2 +-
 src/lisp.h       |   1 +
 src/xdisp.c      | 438 +++++++++++++++++++++++++++++++++++++----------
 src/xfont.c      |  54 +++---
 src/xterm.c      |   4 +-
 12 files changed, 855 insertions(+), 192 deletions(-)

diff --git a/src/alloc.c b/src/alloc.c
index e241b9933a..cc73097957 100644
--- a/src/alloc.c
+++ b/src/alloc.c
@@ -6224,6 +6224,9 @@ DEFUN ("garbage-collect", Fgarbage_collect, Sgarbage_collect, 0, 0, "",
   return CALLMANY (Flist, total);
 }
 
+static void
+mark_vectorlike (union vectorlike_header *header);
+
 /* Mark Lisp objects in glyph matrix MATRIX.  Currently the
    only interesting objects referenced from glyphs are strings.  */
 
@@ -6243,9 +6246,13 @@ mark_glyph_matrix (struct glyph_matrix *matrix)
 	    struct glyph *end_glyph = glyph + row->used[area];
 
 	    for (; glyph < end_glyph; ++glyph)
-	      if (STRINGP (glyph->object)
-		  && !string_marked_p (XSTRING (glyph->object)))
-		mark_object (glyph->object);
+	      {
+		if (STRINGP (glyph->object)
+		    && !string_marked_p (XSTRING (glyph->object)))
+		  mark_object (glyph->object);
+		if (glyph->context != NULL)
+		  mark_vectorlike (&glyph->context->header);
+	      }
 	  }
       }
 }
diff --git a/src/composite.c b/src/composite.c
index 518502be49..4c35d2714e 100644
--- a/src/composite.c
+++ b/src/composite.c
@@ -816,7 +816,7 @@ fill_gstring_body (Lisp_Object gstring)
   ptrdiff_t len = LGSTRING_CHAR_LEN (gstring);
   ptrdiff_t i;
   struct font *font = NULL;
-  unsigned int code;
+  Lisp_Object code = Qnil;
 
   if (FONT_OBJECT_P (font_object))
     font = XFONT_OBJECT (font_object);
@@ -836,10 +836,10 @@ fill_gstring_body (Lisp_Object gstring)
       LGLYPH_SET_CHAR (g, c);
 
       if (font != NULL)
-        code = font->driver->encode_char (font, LGLYPH_CHAR (g));
+        code = font->driver->encode_char (font, LGLYPH_CHAR (g), NULL);
       else
-        code = FONT_INVALID_CODE;
-      if (code != FONT_INVALID_CODE)
+        code = Qnil;
+      if (!NILP (code))
         {
 	  font_fill_lglyph_metrics (g, font, code);
         }
@@ -1223,7 +1223,7 @@ composition_reseat_it (struct composition_it *cmp_it, ptrdiff_t charpos,
 	      if (! VECTORP (elt) || ASIZE (elt) != 3
 		  || ! FIXNUMP (AREF (elt, 1)))
 		continue;
-	      if (XFIXNAT (AREF (elt, 1)) != cmp_it->lookback)
+	      if (XFIXNUM (AREF (elt, 1)) != cmp_it->lookback)
 		goto no_composition;
 	      lgstring = autocmp_chars (elt, charpos, bytepos, endpos,
 					w, face, string, direction);
diff --git a/src/data.c b/src/data.c
index 1db0a983b4..ea2604b379 100644
--- a/src/data.c
+++ b/src/data.c
@@ -267,6 +267,7 @@ DEFUN ("type-of", Ftype_of, Stype_of, 1, 1, 0,
 	case PVEC_MISC_PTR:
         case PVEC_OTHER:
         case PVEC_SUB_CHAR_TABLE:
+	case PVEC_GLYPH_CONTEXT:
         case PVEC_FREE: ;
         }
       emacs_abort ();
diff --git a/src/dispextern.h b/src/dispextern.h
index 0b1f3d14ae..bbb11962f3 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -397,6 +397,14 @@ #define SET_GLYPH_FROM_GLYPH_CODE(glyph, gc)				\
 };
 
 
+struct glyph_context
+{
+  union vectorlike_header header;
+  Lisp_Object string;
+  Lisp_Object script;
+  Lisp_Object position;
+};
+
 /* Glyphs.
 
    Be extra careful when changing this structure!  Esp. make sure that
@@ -567,6 +575,8 @@ #define FACE_ID_BITS	20
     /* Used to compare all bit-fields above in one step.  */
     unsigned val;
   } u;
+
+  struct glyph_context *context;
 };
 
 
@@ -1329,7 +1339,7 @@ #define MATRIX_ROW_OVERLAPS_SUCC_P(ROW)		\
   enum glyph_row_area area;
 
   /* Characters to be drawn, and number of characters.  */
-  unsigned *char2b;
+  Lisp_Object *char2b;
   int nchars;
 
   /* A face-override for drawing cursors, mouse face and similar.  */
@@ -2272,12 +2282,28 @@ #define IT_STACK_SIZE 5
   int width;
 };
 
+/* This is way too much. */
+#define IT_HISTORY_SIZE 128
+struct it_history
+{
+  ptrdiff_t len;
+  int c[IT_HISTORY_SIZE];
+};
+
 struct it
 {
   /* The window in which we iterate over current_buffer (or a string).  */
   Lisp_Object window;
   struct window *w;
 
+  struct it_history history;
+  struct
+  {
+    ptrdiff_t next_transition;
+    ptrdiff_t age;
+    struct glyph_context *context;
+  } context_cache;
+
   /* The window's frame.  */
   struct frame *f;
 
diff --git a/src/font.c b/src/font.c
index ab00402b40..1069c7713b 100644
--- a/src/font.c
+++ b/src/font.c
@@ -184,8 +184,8 @@ font_make_object (int size, Lisp_Object entity, int pixelsize)
 {
   Lisp_Object font_object;
   struct font *font
-    = (struct font *) allocate_pseudovector (size, FONT_OBJECT_MAX,
-					     FONT_OBJECT_MAX, PVEC_FONT);
+    = (struct font *) allocate_pseudovector (size, FONT_OBJECT_MAX + 1,
+					     FONT_OBJECT_MAX + 1, PVEC_FONT);
   int i;
 
   /* GC can happen before the driver is set up,
@@ -3010,7 +3010,7 @@ font_has_char (struct frame *f, Lisp_Object font, int c)
       if (result >= 0)
 	return result;
     }
-  return (fontp->driver->encode_char (fontp, c) != FONT_INVALID_CODE);
+  return !NILP (fontp->driver->encode_char (fontp, c, NULL));
 }
 
 
@@ -3023,7 +3023,7 @@ font_encode_char (Lisp_Object font_object, int c)
 
   eassert (FONT_OBJECT_P (font_object));
   font = XFONT_OBJECT (font_object);
-  return font->driver->encode_char (font, c);
+  return XFIXNUM (font->driver->encode_char (font, c, NULL));
 }
 
 
@@ -4413,11 +4413,11 @@ DEFUN ("clear-font-cache", Fclear_font_cache, Sclear_font_cache, 0, 0, 0,
 
 \f
 void
-font_fill_lglyph_metrics (Lisp_Object glyph, struct font *font, unsigned int code)
+font_fill_lglyph_metrics (Lisp_Object glyph, struct font *font, Lisp_Object code)
 {
   struct font_metrics metrics;
 
-  LGLYPH_SET_CODE (glyph, code);
+  LGLYPH_SET_CODE (glyph, XFIXNUM (code));
   font->driver->text_extents (font, &code, 1, &metrics);
   LGLYPH_SET_LBEARING (glyph, metrics.lbearing);
   LGLYPH_SET_RBEARING (glyph, metrics.rbearing);
@@ -4638,12 +4638,12 @@ DEFUN ("internal-char-font", Finternal_char_font, Sinternal_char_font, 1, 2, 0,
   struct face *face = FACE_FROM_ID (f, face_id);
   if (! face->font)
     return Qnil;
-  unsigned code = face->font->driver->encode_char (face->font, c);
-  if (code == FONT_INVALID_CODE)
+  Lisp_Object code = face->font->driver->encode_char (face->font, c, NULL);
+  if (NILP (code))
     return Qnil;
   Lisp_Object font_object;
   XSETFONT (font_object, face->font);
-  return Fcons (font_object, INT_TO_INTEGER (code));
+  return Fcons (font_object, code);
 }
 
 #if 0
@@ -4962,11 +4962,11 @@ DEFUN ("font-get-glyphs", Ffont_get_glyphs, Sfont_get_glyphs, 3, 4, 0,
     {
       Lisp_Object g;
       int c = XFIXNAT (chars[i]);
-      unsigned code;
+      Lisp_Object code;
       struct font_metrics metrics;
 
-      code = font->driver->encode_char (font, c);
-      if (code == FONT_INVALID_CODE)
+      code = font->driver->encode_char (font, c, NULL);
+      if (NILP (code))
 	{
 	  ASET (vec, i, Qnil);
 	  continue;
@@ -4975,7 +4975,12 @@ DEFUN ("font-get-glyphs", Ffont_get_glyphs, Sfont_get_glyphs, 3, 4, 0,
       LGLYPH_SET_FROM (g, i);
       LGLYPH_SET_TO (g, i);
       LGLYPH_SET_CHAR (g, c);
-      LGLYPH_SET_CODE (g, code);
+      if (FIXNUMP (code))
+	LGLYPH_SET_CODE (g, XFIXNUM (code));
+      else if (VECTORP (code) && FIXNUMP (AREF (code, 4)))
+	LGLYPH_SET_CODE (g, XFIXNUM (AREF (code, 4)));
+      else
+	LGLYPH_SET_CODE (g, FONT_INVALID_CODE);
       font->driver->text_extents (font, &code, 1, &metrics);
       LGLYPH_SET_WIDTH (g, metrics.width);
       LGLYPH_SET_LBEARING (g, metrics.lbearing);
diff --git a/src/font.h b/src/font.h
index 8614e7fa10..eee5e4c81e 100644
--- a/src/font.h
+++ b/src/font.h
@@ -276,6 +276,8 @@ #define FONT_ENCODING_NOT_DECIDED 255
 
   Lisp_Object props[FONT_OBJECT_MAX];
 
+  Lisp_Object hb_cache;
+
   /* Beyond here, there should be no more Lisp_Object components.  */
 
   /* Minimum and maximum glyph widths, in pixels.  Some font backends,
@@ -456,7 +458,7 @@ GC_FONT_ENTITY_P (Lisp_Object x)
 INLINE bool
 FONT_OBJECT_P (Lisp_Object x)
 {
-  return FONTP (x) && PVSIZE (x) == FONT_OBJECT_MAX;
+  return FONTP (x) && PVSIZE (x) == FONT_OBJECT_MAX + 1;
 }
 
 /* Like FONT_OBJECT_P, but can be used in the garbage collector.  */
@@ -565,6 +567,8 @@ #define FONT_PIXEL_SIZE_QUANTUM 1
 
 #define FONT_INVALID_CODE 0xFFFFFFFF
 
+struct glyph_context;
+
 /* Font driver.  Members specified as "optional" can be NULL.  */
 
 struct font_driver
@@ -645,13 +649,15 @@ #define FONT_INVALID_CODE 0xFFFFFFFF
 
   /* Return a glyph code of FONT for character C (Unicode code point).
      If FONT doesn't have such a glyph, return FONT_INVALID_CODE.  */
-  unsigned (*encode_char) (struct font *font, int c);
+  Lisp_Object (*encode_char) (struct font *font, int c, struct glyph_context *context);
+
+  Lisp_Object (*guess_script) (struct font *font, int c);
 
   /* Compute the total metrics of the NGLYPHS glyphs specified by
      the font FONT and the sequence of glyph codes CODE, and store the
      result in METRICS.  */
   void (*text_extents) (struct font *font,
-			const unsigned *code, int nglyphs,
+			const Lisp_Object *code, int nglyphs,
 			struct font_metrics *metrics);
 
 #ifdef HAVE_WINDOW_SYSTEM
@@ -886,7 +892,7 @@ valid_font_driver (struct font_driver const *d)
 extern Lisp_Object font_range (ptrdiff_t, ptrdiff_t, ptrdiff_t *,
 			       struct window *, struct face *,
 			       Lisp_Object);
-extern void font_fill_lglyph_metrics (Lisp_Object, struct font *, unsigned int);
+extern void font_fill_lglyph_metrics (Lisp_Object, struct font *, Lisp_Object);
 
 extern Lisp_Object font_put_extra (Lisp_Object font, Lisp_Object prop,
                                    Lisp_Object val);
@@ -1006,6 +1012,34 @@ font_data_structures_may_be_ill_formed (void)
   return false;
 #endif
 }
+#define HB_GLYPH_GLYPH 0
+#define HB_GLYPH_CODE 1
+#define HB_GLYPH_X_ADVANCE 2
+#define HB_GLYPH_X_OFF 3
+#define HB_GLYPH_Y_OFF 4
+#define HB_GLYPH_CODEPOINT 5
+#define HB_GLYPH_LENGTH 6
+
+#define HB_SLICE_SLICE 0
+#define HB_SLICE_X_ADVANCE 1
+#define HB_SLICE_X_OFFSET 5
+#define HB_SLICE_I 2
+#define HB_SLICE_N 3
+#define HB_SLICE_GLYPHS 4
+#define HB_SLICE_LENGTH 6
+
+#define HB_SHAPED_CODE 0
+#define HB_SHAPED_CLUSTER 1
+#define HB_SHAPED_METRICS 2
+#define HB_SHAPED_CARETS 3
+#define HB_SHAPED_RTL 4
+#define HB_SHAPED_LENGTH 5
+
+#define HB_SHAPED_METRICS_X_ADVANCE 0
+#define HB_SHAPED_METRICS_Y_ADVANCE 1
+#define HB_SHAPED_METRICS_X_OFFSET 2
+#define HB_SHAPED_METRICS_Y_OFFSET 3
+#define HB_SHAPED_METRICS_LENGTH 4
 
 INLINE_HEADER_END
 
diff --git a/src/ftcrfont.c b/src/ftcrfont.c
index 7832d4f5ce..b51d3e61bd 100644
--- a/src/ftcrfont.c
+++ b/src/ftcrfont.c
@@ -218,7 +218,7 @@ ftcrfont_open (struct frame *f, Lisp_Object entity, int pixel_size)
   font->average_width /= 95;
 
   cairo_scaled_font_extents (ftcrfont_info->cr_scaled_font, &extents);
-  font->ascent = lround (extents.ascent);
+  font->ascent = ceil (extents.ascent);
   val = assq_no_quit (QCminspace, AREF (entity, FONT_EXTRA_INDEX));
   if (!(CONSP (val) && NILP (XCDR (val))))
     {
@@ -322,64 +322,353 @@ ftcrfont_has_char (Lisp_Object font, int c)
   return -1;
 }
 
-static unsigned
-ftcrfont_encode_char (struct font *font, int c)
+static Lisp_Object
+ftcrfont_guess_script (struct font *font, int c)
+{
+  block_input ();
+  hb_buffer_t *hb_buf = hb_buffer_create ();
+  hb_buffer_set_content_type (hb_buf, HB_BUFFER_CONTENT_TYPE_UNICODE);
+  hb_buffer_add (hb_buf, c, 0);
+  hb_buffer_guess_segment_properties (hb_buf);
+  /* Force the language to a known and system-independent value, so as
+     not to make Emacs's behavior depend on LC_CTYPE. */
+  hb_buffer_set_language (hb_buf, hb_language_from_string ("en", -1));
+  hb_script_t script =
+    hb_buffer_get_script (hb_buf);
+  Lisp_Object ret;
+
+  if (script == HB_SCRIPT_UNKNOWN ||
+      script == HB_SCRIPT_COMMON ||
+      script == HB_SCRIPT_INHERITED ||
+      script == HB_SCRIPT_INVALID)
+    ret = Qnil;
+  else
+    {
+      char string[5];
+      string[4] = 0;
+      string[3] = script & 0xff;
+      string[2] = (script>>8) & 0xff;
+      string[1] = (script>>16) & 0xff;
+      string[0] = (script>>24) & 0xff;
+      ret = build_string (string);
+    }
+
+  unblock_input ();
+
+  return ret;
+}
+
+static hb_codepoint_t
+hb_mirror (hb_unicode_funcs_t *ufuncs,
+	   hb_codepoint_t codepoint,
+	   void *user_data)
+{
+  /* HarfBuzz expects unmirrored codepoints, so if we mirrored it,
+     don't mirror it again.  */
+  if (bidi_mirror_char (codepoint) != codepoint)
+    return codepoint;
+  return hb_unicode_mirroring (hb_unicode_funcs_get_parent (ufuncs), codepoint);
+}
+
+static Lisp_Object
+get_shaped_context (struct font *font, Lisp_Object string, Lisp_Object script)
 {
   struct font_info *ftcrfont_info = (struct font_info *) font;
-  unsigned code = FONT_INVALID_CODE;
-  unsigned char utf8[MAX_MULTIBYTE_LENGTH];
-  int utf8len = CHAR_STRING (c, utf8);
-  cairo_glyph_t stack_glyph;
-  cairo_glyph_t *glyphs = &stack_glyph;
-  int num_glyphs = 1;
-
-  if (cairo_scaled_font_text_to_glyphs (ftcrfont_info->cr_scaled_font, 0, 0,
-					(char *) utf8, utf8len,
-					&glyphs, &num_glyphs,
-					NULL, NULL, NULL)
-      == CAIRO_STATUS_SUCCESS)
+  if (NILP (font->hb_cache))
+    font->hb_cache = CALLN (Fmake_hash_table, QCtest, Qequal, QCweakness, Qkey);
+
+  Lisp_Object cached = Fgethash (Fcons (string, script), font->hb_cache, Qnil);
+
+  if (!NILP (cached))
+    return cached;
+  block_input ();
+  unsigned int num_glyphs = 0;
+  hb_buffer_t *hb_buf = hb_buffer_create ();
+  hb_unicode_funcs_t *ufuncs = hb_unicode_funcs_create (hb_buffer_get_unicode_funcs (hb_buf));
+  hb_unicode_funcs_set_mirroring_func (ufuncs, hb_mirror, NULL, NULL);
+  hb_buffer_set_unicode_funcs (hb_buf, ufuncs);
+  hb_buffer_set_cluster_level (hb_buf, HB_BUFFER_CLUSTER_LEVEL_MONOTONE_CHARACTERS);
+  bool rtl = false;
+  for (int i = 0; i < SCHARS (string); i++)
+    {
+      int c = XFIXNUM (Faref (string, make_fixnum (i)));
+      hb_buffer_add (hb_buf, c, i);
+    }
+  hb_buffer_set_content_type (hb_buf, HB_BUFFER_CONTENT_TYPE_UNICODE);
+  if (STRINGP (script) && SBYTES (script) >= 4)
+    {
+      hb_script_t scriptint = 0;
+      scriptint += SREF (script, 0) << 24;
+      scriptint += SREF (script, 1) << 16;
+      scriptint += SREF (script, 2) <<  8;
+      scriptint += SREF (script, 3);
+      hb_buffer_set_script (hb_buf, scriptint);
+    }
+  hb_buffer_guess_segment_properties (hb_buf);
+  /* Force the language to a known and system-independent value, so as
+     not to make Emacs's behavior depend on LC_CTYPE. */
+  hb_buffer_set_language (hb_buf, hb_language_from_string ("en", -1));
+  rtl = hb_buffer_get_direction (hb_buf) == HB_DIRECTION_RTL;
+  hb_font_t *hb_font = hb_ft_font_create_referenced
+    (cairo_ft_scaled_font_lock_face (ftcrfont_info->cr_scaled_font));
+  hb_shape (hb_font, hb_buf, NULL, 0);
+  hb_glyph_info_t *glyph_info = hb_buffer_get_glyph_infos
+    (hb_buf, &num_glyphs);
+  hb_glyph_position_t *glyph_pos = hb_buffer_get_glyph_positions
+    (hb_buf, &num_glyphs);
+
+  /* Reverse glyphs for RTL. This is because HarfBuzz reverses them,
+     not because we want to bypass the code in bidi.c. */
+  if (rtl)
+    {
+      for (int i = 0; i < num_glyphs/2; i++)
+	{
+	  hb_glyph_position_t temp_pos = glyph_pos[num_glyphs - i - 1];
+	  glyph_pos[num_glyphs - i - 1] = glyph_pos[i];
+	  glyph_pos[i] = temp_pos;
+
+	  hb_glyph_info_t temp_info = glyph_info[num_glyphs - i - 1];
+	  glyph_info[num_glyphs - i - 1] = glyph_info[i];
+	  glyph_info[i] = temp_info;
+	}
+    }
+
+  /* Merge non-spacing glyphs into the previous cluster. */
+  int last_spacing_cluster = 0;
+  for (int i = 0; i < num_glyphs; i++)
     {
-      if (glyphs != &stack_glyph)
-	cairo_glyph_free (glyphs);
-      else if (stack_glyph.index)
-	code = stack_glyph.index;
+      if (glyph_pos[i].x_advance == 0
+	  && glyph_info[i].cluster > last_spacing_cluster)
+	glyph_info[i].cluster = last_spacing_cluster;
+      else
+	last_spacing_cluster = glyph_info[i].cluster;
     }
 
+  /* For RTL, un-reverse glyphs that ended up forming a cluster. */
+  if (rtl)
+    {
+      for (int i = 0; i < num_glyphs;)
+	{
+	  int j;
+	  for (j = i + 1; j < num_glyphs; j++)
+	    {
+	      if (glyph_info[j].cluster != glyph_info[i].cluster)
+		break;
+	    }
+	  for (int k = i; k < i + (j - i)/2; k++)
+	    {
+	      int k2 = i + j - k - 1;
+	      hb_glyph_position_t temp_pos = glyph_pos[k2];
+	      glyph_pos[k2] = glyph_pos[k];
+	      glyph_pos[k] = temp_pos;
+
+	      hb_glyph_info_t temp_info = glyph_info[k2];
+	      glyph_info[k2] = glyph_info[k];
+	      glyph_info[k] = temp_info;
+	    }
+	  i = j;
+	}
+    }
+
+  Lisp_Object ret = make_nil_vector (num_glyphs);
+  for (int i = 0; i < num_glyphs; i++)
+    {
+      Lisp_Object glyph = make_nil_vector (HB_SHAPED_LENGTH);
+      ASET (glyph, HB_SHAPED_CODE, make_fixnum (glyph_info[i].codepoint));
+      ASET (glyph, HB_SHAPED_CLUSTER, make_fixnum (glyph_info[i].cluster));
+      {
+	Lisp_Object v = make_nil_vector (HB_SHAPED_METRICS_LENGTH);
+	ASET (v, HB_SHAPED_METRICS_X_ADVANCE,
+	      make_fixnum ((glyph_pos[i].x_advance + 32)/ 64));
+	ASET (v, HB_SHAPED_METRICS_Y_ADVANCE,
+	      make_fixnum ((glyph_pos[i].y_advance + 32) / 64));
+	ASET (v, HB_SHAPED_METRICS_X_OFFSET,
+	      make_fixnum ((glyph_pos[i].x_offset) / 64));
+	ASET (v, HB_SHAPED_METRICS_Y_OFFSET,
+	      make_fixnum ((glyph_pos[i].y_offset) / 64));
+	ASET (glyph, HB_SHAPED_METRICS, v);
+      }
+#if 0
+      Lisp_Object v = Qnil;
+      if (false) {
+	hb_position_t carets[1024];
+	unsigned int caret_count = 1022;
+	if (hb_ot_layout_get_ligature_carets
+	    (hb_font, HB_DIRECTION_LTR /* XXX */,
+	     glyph_info[i].codepoint, 0,
+	     &caret_count, carets + 1))
+	  {
+	    carets[0] = 0;
+	    caret_count++;
+	    carets[caret_count++] = glyph_pos[i].x_advance;
+	    v = make_nil_vector (caret_count);
+	    for (ptrdiff_t i = 0; i < caret_count; i++)
+	      ASET (v, i, make_fixnum (carets[i]));
+	  }
+      }
+#endif
+      ASET (glyph, HB_SHAPED_CARETS, Qnil);
+      ASET (glyph, HB_SHAPED_RTL, rtl ? Qt : Qnil);
+      ASET (ret, i, glyph);
+    }
+
+  Fputhash (Fcons (string, script), ret, font->hb_cache);
+  unblock_input ();
+  return ret;
+}
+
+static Lisp_Object
+ftcrfont_encode_char (struct font *font, int c, struct glyph_context *context)
+{
+  Lisp_Object code = Qnil;
+
+  if (context == NULL)
+    {
+      context = ALLOCATE_PSEUDOVECTOR (struct glyph_context, position, PVEC_GLYPH_CONTEXT);
+      context->script = Qnil;
+      context->string = CALLN (Fstring, make_fixnum (c));
+      context->position = make_fixnum (0);
+    }
+
+  bool rtl = false;
+  Lisp_Object shaped = get_shaped_context (font, context->string, context->script);
+  unsigned int num_glyphs = ASIZE (shaped);
+  rtl = ! NILP (AREF (AREF (shaped, 0), HB_SHAPED_RTL));
+  int glyph0 = 0, glyph1 = num_glyphs;
+  int cluster0 = 0, cluster1 = XFIXNUM (Flength (context->string));
+  int char0 = 0, char1 = XFIXNUM (Flength (context->string));
+  for (int i = 0; i < num_glyphs; i++)
+    if (XFIXNUM (AREF (AREF (shaped, i), HB_SHAPED_CLUSTER))
+	<= XFIXNUM (context->position))
+      {
+	cluster0 = XFIXNUM (AREF (AREF (shaped, i), HB_SHAPED_CLUSTER));
+	int i1 = i;
+	while (i1 > 0
+	       && XFIXNUM (AREF (AREF (shaped, i1 - 1), HB_SHAPED_CLUSTER))
+	       == cluster0)
+	  i1--;
+	glyph0 = i1;
+	char0 = cluster0;
+      }
+  for (int i = 0; i < num_glyphs; i++)
+    if (XFIXNUM (AREF (AREF (shaped, i), HB_SHAPED_CLUSTER))
+	> XFIXNUM (context->position))
+      {
+	glyph1 = i;
+	cluster1 = XFIXNUM (AREF (AREF (shaped, i), HB_SHAPED_CLUSTER));
+	char1 = cluster1;
+	break;
+      }
+  int i = XFIXNUM (context->position) - char0;
+  if (rtl)
+    i = char1 - char0 - 1 - i;
+  Lisp_Object glyphs = make_nil_vector (glyph1 - glyph0);
+  int x_advance = 0;
+  for (ptrdiff_t i = glyph0; i < glyph1; i++)
+    {
+      Lisp_Object glyph = make_nil_vector (HB_GLYPH_LENGTH);
+      ASET (glyph, HB_GLYPH_GLYPH, Qglyph);
+      Lisp_Object code = AREF (AREF (shaped, i), HB_SHAPED_CODE);
+      /* This appears to be the only sign from HarfBuzz that our font
+	 is missing a glyph ... */
+      if (XFIXNUM (code) == 0)
+	return Qnil;
+      ASET (glyph, HB_GLYPH_CODE, AREF (AREF (shaped, i), HB_SHAPED_CODE));
+      ASET (glyph, HB_GLYPH_X_OFF, AREF (AREF (AREF (shaped, i), 2), 2));
+      ASET (glyph, HB_GLYPH_Y_OFF, CALLN (Fminus, AREF (AREF (AREF (shaped, i), 2), 3)));
+      ASET (glyph, HB_GLYPH_CODEPOINT, make_fixnum (c));
+      ASET (glyph, HB_GLYPH_X_ADVANCE, AREF (AREF (AREF (shaped, i), 2), 0));
+      ASET (glyphs, i - glyph0, glyph);
+      x_advance += XFIXNUM (AREF (AREF (AREF (shaped, i), 2), 0));
+    }
+
+  code = make_nil_vector (HB_SLICE_LENGTH);
+  ASET (code, HB_SLICE_SLICE, Qslice);
+  ASET (code, HB_SLICE_I, make_fixnum (i));
+  ASET (code, HB_SLICE_N, make_fixnum (char1 - char0));
+  int x0 = (char1 - char0 - i - 1) * x_advance / (char1 - char0);
+  int x1 = (char1 - char0 - i) * x_advance / (char1 - char0);
+  ASET (code, HB_SLICE_X_ADVANCE, make_fixnum (x1 - x0));
+  ASET (code, HB_SLICE_X_OFFSET, make_fixnum (x_advance - x1));
+  ASET (code, HB_SLICE_GLYPHS, glyphs);
+
   return code;
 }
 
 static void
 ftcrfont_text_extents (struct font *font,
-                       const unsigned *code,
+                       const Lisp_Object *code,
                        int nglyphs,
                        struct font_metrics *metrics)
 {
-  int width, i;
+  int width = 0;
 
   block_input ();
-  width = ftcrfont_glyph_extents (font, code[0], metrics);
-  for (i = 1; i < nglyphs; i++)
+  if (metrics)
     {
-      struct font_metrics m;
-      int w = ftcrfont_glyph_extents (font, code[i], metrics ? &m : NULL);
-
-      if (metrics)
+      for (int i = 0; i < nglyphs; i++)
 	{
-	  if (width + m.lbearing < metrics->lbearing)
-	    metrics->lbearing = width + m.lbearing;
-	  if (width + m.rbearing > metrics->rbearing)
-	    metrics->rbearing = width + m.rbearing;
-	  if (m.ascent > metrics->ascent)
-	    metrics->ascent = m.ascent;
-	  if (m.descent > metrics->descent)
-	    metrics->descent = m.descent;
+	  if (VECTORP (code[i]) && EQ (AREF (code[i], 0), Qslice))
+	    {
+	      int slice_ascent = 0;
+	      int slice_descent = 0;
+	      int slice_lbearing = 0;
+	      int slice_rbearing = 0;
+	      int slice_i = XFIXNUM (AREF (code[i], HB_SLICE_I));
+	      int slice_n = XFIXNUM (AREF (code[i], HB_SLICE_N));
+	      Lisp_Object glyphs = AREF (code[i], HB_SLICE_GLYPHS);
+	      int slice_width = 0;
+	      for (ptrdiff_t gi = 0; gi < ASIZE (glyphs); gi++)
+		{
+		  Lisp_Object glyph = AREF (glyphs, gi);
+		  struct font_metrics m;
+		  ftcrfont_glyph_extents (font, XFIXNUM (AREF (glyph, HB_GLYPH_CODE)), &m);
+		  int glyph_lbearing =
+		    slice_width + m.lbearing + XFIXNUM (AREF (glyph, HB_GLYPH_X_OFF));
+		  int glyph_rbearing =
+		    slice_width + m.rbearing + XFIXNUM (AREF (glyph, HB_GLYPH_X_OFF));
+		  int glyph_ascent =
+		    m.ascent + XFIXNUM (AREF (glyph, HB_GLYPH_Y_OFF)); // XXX -?
+
+		  int glyph_descent =
+		    m.descent + XFIXNUM (AREF (glyph, HB_GLYPH_Y_OFF)); // XXX -?
+
+		  if (gi == 0 || glyph_lbearing < slice_lbearing)
+		    slice_lbearing = glyph_lbearing;
+		  if (gi == 0 || glyph_rbearing > slice_rbearing)
+		    slice_rbearing = glyph_rbearing;
+		  if (gi == 0 || glyph_ascent > slice_ascent)
+		    slice_ascent = glyph_ascent;
+		  if (gi == 0 || glyph_descent > slice_descent)
+		    slice_descent = glyph_descent;
+
+		  slice_width += XFIXNUM (AREF (glyph, HB_GLYPH_X_ADVANCE));
+		}
+
+	      slice_width = XFIXNUM (AREF (code[i], HB_SLICE_X_ADVANCE));
+	      if (slice_i < slice_n - 1)
+		slice_rbearing = XFIXNUM (AREF (code[i], HB_SLICE_X_ADVANCE))
+		  + XFIXNUM (AREF (code[i], HB_SLICE_X_OFFSET));
+	      if (slice_i > 0)
+		slice_lbearing = XFIXNUM (AREF (code[i], HB_SLICE_X_OFFSET));
+
+	      if (i == 0 || slice_lbearing < metrics->lbearing)
+		metrics->lbearing = slice_lbearing;
+	      if (i == 0 || slice_rbearing > metrics->rbearing)
+		metrics->rbearing = slice_rbearing;
+	      if (i == 0 || slice_ascent > metrics->ascent)
+		metrics->ascent = slice_ascent;
+	      if (i == 0 || slice_descent > metrics->descent)
+		metrics->descent = slice_descent;
+	      width += slice_width;
+	    }
+	  else if (FIXNUMP (code[i]))
+	    {
+	    }
 	}
-      width += w;
+      metrics->width = width;
     }
   unblock_input ();
-
-  if (metrics)
-    metrics->width = width;
 }
 
 static int
@@ -486,7 +775,6 @@ ftcrfont_draw (struct glyph_string *s,
   struct face *face = s->face;
   struct font_info *ftcrfont_info = (struct font_info *) s->font;
   cairo_t *cr;
-  cairo_glyph_t *glyphs;
   int len = to - from;
   int i;
 
@@ -502,20 +790,54 @@ ftcrfont_draw (struct glyph_string *s,
       cairo_fill (cr);
     }
 
-  glyphs = alloca (sizeof (cairo_glyph_t) * len);
+  x_set_cr_source_with_gc_foreground (f, s->gc);
+  cairo_set_scaled_font (cr, ftcrfont_info->cr_scaled_font);
   for (i = 0; i < len; i++)
     {
-      glyphs[i].index = s->char2b[from + i];
-      glyphs[i].x = x;
-      glyphs[i].y = y;
-      x += (s->padding_p ? 1 : ftcrfont_glyph_extents (s->font,
-                                                       glyphs[i].index,
-                                                       NULL));
+      Lisp_Object code = s->char2b[from + i];
+      if (VECTORP (code))
+	{
+	  if (EQ (AREF (code, 0), Qslice))
+	    {
+	      int xtmp = x;
+	      cairo_save (cr);
+	      struct font_metrics metrics;
+	      ftcrfont_text_extents (s->font, &code, 1, &metrics);
+	      double x0 = x + metrics.lbearing - XFIXNUM (AREF (code, HB_SLICE_X_OFFSET));
+	      double x1 = x + metrics.rbearing - XFIXNUM (AREF (code, HB_SLICE_X_OFFSET));
+	      double y0 = y - metrics.ascent;
+	      double y1 = y + metrics.descent;
+	      x0--; y0--; x1++; y1++; /* FIXME */
+	      cairo_rectangle (cr, x0, y0, x1 - x0, y1 - y0);
+	      cairo_clip (cr);
+
+	      Lisp_Object glyphs = AREF (code, HB_SLICE_GLYPHS);
+	      for (ptrdiff_t gi = 0; gi < ASIZE (glyphs); gi++)
+		{
+		  Lisp_Object glyph = AREF (glyphs, gi);
+		  cairo_glyph_t cglyph;
+		  cglyph.index = XFIXNUM (AREF (glyph, HB_GLYPH_CODE));
+		  cglyph.x = x + XFIXNUM (AREF (glyph, HB_GLYPH_X_OFF));
+		  cglyph.y = y + XFIXNUM (AREF (glyph, HB_GLYPH_Y_OFF));
+		  cglyph.x -= XFIXNUM (AREF (code, HB_SLICE_X_OFFSET));
+		  x += XFIXNUM (AREF (glyph, HB_GLYPH_X_ADVANCE));
+		  cairo_show_glyphs (cr, &cglyph, 1);
+		}
+	      cairo_restore (cr);
+	      x = xtmp + XFIXNUM (AREF (code, HB_SLICE_X_ADVANCE));
+	    }
+	}
+      else if (FIXNUMP (code))
+	{
+	  cairo_glyph_t cglyph;
+	  cglyph.index = XFIXNUM (code);
+	  cglyph.x = x;
+	  cglyph.y = y;
+	  x += (s->padding_p ? 0 : 0);
+	  cairo_show_glyphs (cr, &cglyph, 1);
+	}
     }
 
-  x_set_cr_source_with_gc_foreground (f, s->gc);
-  cairo_set_scaled_font (cr, ftcrfont_info->cr_scaled_font);
-  cairo_show_glyphs (cr, glyphs, len);
 
   x_end_cr_clip (f);
 
@@ -583,6 +905,7 @@ ftcrhbfont_end_hb_font (struct font *font, hb_font_t *hb_font)
   .draw = ftcrfont_draw,
   .get_bitmap = ftcrfont_get_bitmap,
   .anchor_point = ftcrfont_anchor_point,
+  .guess_script = ftcrfont_guess_script,
 #ifdef HAVE_LIBOTF
   .otf_capability = ftcrfont_otf_capability,
 #endif
diff --git a/src/hbfont.c b/src/hbfont.c
index 576c5fe7f6..a863d0375e 100644
--- a/src/hbfont.c
+++ b/src/hbfont.c
@@ -577,7 +577,7 @@ hbfont_shape (Lisp_Object lgstring, Lisp_Object direction)
       LGLYPH_SET_CHAR (lglyph, chars[char_idx]);
       LGLYPH_SET_CODE (lglyph, info[i].codepoint);
 
-      unsigned code = info[i].codepoint;
+      Lisp_Object code = make_fixnum (info[i].codepoint);
       font->driver->text_extents (font, &code, 1, &metrics);
       LGLYPH_SET_WIDTH (lglyph, metrics.width);
       LGLYPH_SET_LBEARING (lglyph, metrics.lbearing);
diff --git a/src/lisp.h b/src/lisp.h
index 3442699088..a702f27d6d 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -1098,6 +1098,7 @@ DEFINE_GDB_SYMBOL_END (PSEUDOVECTOR_FLAG)
   PVEC_MUTEX,
   PVEC_CONDVAR,
   PVEC_MODULE_FUNCTION,
+  PVEC_GLYPH_CONTEXT,
 
   /* These should be last, for internal_equal and sxhash_obj.  */
   PVEC_COMPILED,
diff --git a/src/xdisp.c b/src/xdisp.c
index db0ec68315..9a9ab5db93 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -439,6 +439,7 @@ Copyright (C) 1985-1988, 1993-1995, 1997-2020 Free Software Foundation,
 #include "lisp.h"
 #include "atimer.h"
 #include "composite.h"
+#include "category.h"
 #include "keyboard.h"
 #include "sysstdio.h"
 #include "systime.h"
@@ -1046,7 +1047,7 @@ #define THIN_SPACE_WIDTH 1
 static bool get_overlay_strings_1 (struct it *, ptrdiff_t, bool);
 static void next_overlay_string (struct it *);
 static void reseat (struct it *, struct text_pos, bool);
-static void reseat_1 (struct it *, struct text_pos, bool);
+static void reseat_1 (struct it *, struct text_pos, bool, bool);
 static bool next_element_from_display_vector (struct it *);
 static bool next_element_from_string (struct it *);
 static bool next_element_from_c_string (struct it *);
@@ -4860,7 +4861,7 @@ handle_invisible_prop (struct it *it)
 		  bidi_dir_t pdir = it->bidi_it.paragraph_dir;
 
 		  SET_TEXT_POS (tpos, newpos, bpos);
-		  reseat_1 (it, tpos, false);
+		  reseat_1 (it, tpos, false, false);
 		  /* If we reseat on a newline/ZV, we need to prep the
 		     bidi iterator for advancing to the next character
 		     after the newline/EOB, keeping the current paragraph
@@ -6979,7 +6980,7 @@ reseat (struct it *it, struct text_pos pos, bool force_p)
 {
   ptrdiff_t original_pos = IT_CHARPOS (*it);
 
-  reseat_1 (it, pos, false);
+  reseat_1 (it, pos, false, false);
 
   /* Determine where to check text properties.  Avoid doing it
      where possible because text property lookup is very expensive.  */
@@ -7021,7 +7022,7 @@ reseat (struct it *it, struct text_pos pos, bool force_p)
    IT->stop_pos to POS, also.  */
 
 static void
-reseat_1 (struct it *it, struct text_pos pos, bool set_stop_p)
+reseat_1 (struct it *it, struct text_pos pos, bool set_stop_p, bool preserve_history)
 {
   /* Don't call this function when scanning a C string.  */
   eassert (it->s == NULL);
@@ -7041,6 +7042,10 @@ reseat_1 (struct it *it, struct text_pos pos, bool set_stop_p)
   it->object = it->w->contents;
   it->area = TEXT_AREA;
   it->multibyte_p = !NILP (BVAR (current_buffer, enable_multibyte_characters));
+  memset (&it->context_cache, 0, sizeof it->context_cache);
+  if (!preserve_history)
+    memset (&it->history, 0, sizeof it->history);
+
   it->sp = 0;
   it->string_from_display_prop_p = false;
   it->string_from_prefix_prop_p = false;
@@ -7834,6 +7839,22 @@ get_next_display_element (struct it *it)
 void
 set_iterator_to_next (struct it *it, bool reseat_p)
 {
+  if (it->what == IT_CHARACTER
+      && it->method != GET_FROM_DISPLAY_VECTOR)
+    {
+      int c = it->char_to_display;
+      if (it->history.len >= IT_HISTORY_SIZE)
+	{
+	  it->history.len = IT_HISTORY_SIZE - 1;
+	  memmove (it->history.c, it->history.c + 1, it->history.len * sizeof (it->history.c[0]));
+	}
+      it->history.c[it->history.len++] = c;
+      it->context_cache.next_transition -=
+	it->context_cache.next_transition >= 0;
+      it->context_cache.age++;
+    }
+  else
+    it->history.len = 0;
 
   switch (it->method)
     {
@@ -8656,7 +8677,7 @@ compute_stop_pos_backwards (struct it *it)
       it->end_charpos = min (charpos + 1, ZV);
       charpos = max (charpos - SCAN_BACK_LIMIT, BEGV);
       SET_TEXT_POS (pos, charpos, CHAR_TO_BYTE (charpos));
-      reseat_1 (it, pos, false);
+      reseat_1 (it, pos, false, true);
       compute_stop_pos (it);
       /* We must advance forward, right?  */
       if (it->stop_charpos <= charpos)
@@ -8702,7 +8723,7 @@ handle_stop_backwards (struct it *it, ptrdiff_t charpos)
       if (bufp)
 	{
 	  SET_TEXT_POS (pos1, charpos, CHAR_TO_BYTE (charpos));
-	  reseat_1 (it, pos1, false);
+	  reseat_1 (it, pos1, false, true);
 	}
       else
 	it->current.string_pos = string_pos (charpos, it->string);
@@ -10089,7 +10110,7 @@ move_it_vertically_backward (struct it *it, int dy)
      reseat to skip forward over invisible text, set up the iterator
      to deliver from overlay strings at the new position etc.  So,
      use reseat_1 here.  */
-  reseat_1 (it, it->current.pos, true);
+  reseat_1 (it, it->current.pos, true, false);
 
   /* We are now surely at a line start.  */
   it->current_x = it->hpos = 0;	/* FIXME: this is incorrect when bidi
@@ -10399,7 +10420,7 @@ move_it_by_lines (struct it *it, ptrdiff_t dvpos)
 	      back_to_previous_visible_line_start (it);
 	      it->vpos--;
 	    }
-	  reseat_1 (it, it->current.pos, true);
+	  reseat_1 (it, it->current.pos, true, false);
 	}
       else
 	RESTORE_IT (it, it, it2data);
@@ -14930,13 +14951,13 @@ text_outside_line_unchanged_p (struct window *w,
   if (window_outdated (w))
     {
       /* Gap in the line?  */
-      if (GPT < start || Z - GPT < end)
+      if (GPT - IT_HISTORY_SIZE < start || Z - GPT - IT_HISTORY_SIZE < end)
 	unchanged_p = false;
 
       /* Changes start in front of the line, or end after it?  */
       if (unchanged_p
-	  && (BEG_UNCHANGED < start - 1
-	      || END_UNCHANGED < end))
+	  && (BEG_UNCHANGED - IT_HISTORY_SIZE < start - 1
+	      || END_UNCHANGED - IT_HISTORY_SIZE < end))
 	unchanged_p = false;
 
       /* If selective display, can't optimize if changes start at the
@@ -14944,7 +14965,8 @@ text_outside_line_unchanged_p (struct window *w,
       if (unchanged_p
 	  && FIXNUMP (BVAR (current_buffer, selective_display))
 	  && XFIXNUM (BVAR (current_buffer, selective_display)) > 0
-	  && (BEG_UNCHANGED < start || GPT <= start))
+	  && (BEG_UNCHANGED - IT_HISTORY_SIZE < start
+	      || GPT - IT_HISTORY_SIZE <= start))
 	unchanged_p = false;
 
       /* If there are overlays at the start or end of the line, these
@@ -15514,7 +15536,7 @@ #define AINC(a,i)							\
       && CHARPOS (tlbufpos) > 0
       && !w->update_mode_line
       && !current_buffer->clip_changed
-      && !current_buffer->prevent_redisplay_optimizations_p
+      && false
       && FRAME_VISIBLE_P (XFRAME (w->frame))
       && !FRAME_OBSCURED_P (XFRAME (w->frame))
       && !XFRAME (w->frame)->cursor_type_changed
@@ -17402,7 +17424,7 @@ try_scrolling (Lisp_Object window, bool just_this_one_p,
       /* Maybe forget recorded base line for line number display.  */
       if (!just_this_one_p
 	  || current_buffer->clip_changed
-	  || BEG_UNCHANGED < CHARPOS (startp))
+	  || BEG_UNCHANGED - IT_HISTORY_SIZE < CHARPOS (startp))
 	w->base_line_number = 0;
 
       /* If cursor ends up on a partially visible line,
@@ -18110,7 +18132,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
   update_mode_line = (w->update_mode_line
 		      || update_mode_lines
 		      || buffer->clip_changed
-		      || buffer->prevent_redisplay_optimizations_p);
+		      || true);
 
   if (!just_this_one_p)
     /* If `just_this_one_p' is set, we apparently set must_be_updated_p more
@@ -18163,12 +18185,12 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
   current_matrix_up_to_date_p
     = (w->window_end_valid
        && !current_buffer->clip_changed
-       && !current_buffer->prevent_redisplay_optimizations_p
+       && false
        && !window_outdated (w)
        && !hscrolling_current_line_p (w));
 
-  beg_unchanged = BEG_UNCHANGED;
-  end_unchanged = END_UNCHANGED;
+  beg_unchanged = max (BEG_UNCHANGED - IT_HISTORY_SIZE, 0);
+  end_unchanged = max (END_UNCHANGED - IT_HISTORY_SIZE, 0);
 
   SET_TEXT_POS (opoint, PT, PT_BYTE);
 
@@ -20104,7 +20126,7 @@ #define GIVE_UP(X) return 0
      It would be nice to further
      reduce the number of cases where this prevents try_window_id.  */
   if (current_buffer->clip_changed
-      || current_buffer->prevent_redisplay_optimizations_p)
+      || true)
     GIVE_UP (3);
 
   /* Window must either use window-based redisplay or be full width.  */
@@ -20187,8 +20209,12 @@ #define GIVE_UP(X) return 0
     }
 
   /* The position of the first and last character that has been changed.  */
-  first_changed_charpos = BEG + BEG_UNCHANGED;
-  last_changed_charpos  = Z - END_UNCHANGED;
+  first_changed_charpos = BEG + BEG_UNCHANGED - IT_HISTORY_SIZE;
+  last_changed_charpos  = Z - END_UNCHANGED + IT_HISTORY_SIZE;
+  if (first_changed_charpos < BEG)
+    first_changed_charpos = BEG;
+  if (last_changed_charpos > Z)
+    last_changed_charpos = Z;
 
   /* If window starts after a line end, and the last change is in
      front of that newline, then changes don't affect the display.
@@ -24264,7 +24290,7 @@ #define ROW_GLYPH_NEWLINE_P(ROW,GLYPH)		\
       && !windows_or_buffers_changed
       && b
       && !b->clip_changed
-      && !b->prevent_redisplay_optimizations_p
+      && false
       && !window_outdated (w)
       /* We rely below on the cursor coordinates to be up to date, but
 	 we cannot trust them if some command moved point since the
@@ -24730,7 +24756,7 @@ DEFUN ("bidi-resolved-levels", Fbidi_resolved_levels,
       && !windows_or_buffers_changed
       && b
       && !b->clip_changed
-      && !b->prevent_redisplay_optimizations_p
+      && false
       && !window_outdated (w)
       && nrow >= 0
       && nrow < w->current_matrix->nrows
@@ -27432,7 +27458,7 @@ init_glyph_string (struct glyph_string *s,
 #ifdef HAVE_NTGUI
 		   HDC hdc,
 #endif
-		   unsigned *char2b, struct window *w, struct glyph_row *row,
+		   Lisp_Object *char2b, struct window *w, struct glyph_row *row,
 		   enum glyph_row_area area, int start, enum draw_glyphs_face hl)
 {
   memset (s, 0, sizeof *s);
@@ -27511,20 +27537,15 @@ append_glyph_string (struct glyph_string **head, struct glyph_string **tail,
 
 static struct face *
 get_char_face_and_encoding (struct frame *f, int c, int face_id,
-			    unsigned *char2b, bool display_p)
+			    Lisp_Object *char2b, bool display_p,
+			    struct glyph_context *context)
 {
   struct face *face = FACE_FROM_ID (f, face_id);
-  unsigned code = 0;
+  Lisp_Object code = Qnil;
 
   if (face->font)
-    {
-      code = face->font->driver->encode_char (face->font, c);
-
-      if (code == FONT_INVALID_CODE)
-	code = 0;
-    }
-  /* Ensure that the code is only 2 bytes wide.  */
-  *char2b = code & 0xFFFF;
+    code = face->font->driver->encode_char (face->font, c, context);
+  *char2b = code;
 
   /* Make sure X resources of the face are allocated.  */
 #ifdef HAVE_X_WINDOWS
@@ -27545,10 +27566,10 @@ get_char_face_and_encoding (struct frame *f, int c, int face_id,
 
 static struct face *
 get_glyph_face_and_encoding (struct frame *f, struct glyph *glyph,
-			     unsigned *char2b)
+			     Lisp_Object *char2b, struct glyph_context *context)
 {
   struct face *face;
-  unsigned code = 0;
+  Lisp_Object code = Qnil;
 
   eassert (glyph->type == CHAR_GLYPH);
   face = FACE_FROM_ID (f, glyph->face_id);
@@ -27559,16 +27580,13 @@ get_glyph_face_and_encoding (struct frame *f, struct glyph *glyph,
   if (face->font)
     {
       if (CHAR_BYTE8_P (glyph->u.ch))
-	code = CHAR_TO_BYTE8 (glyph->u.ch);
+	code = make_fixnum (CHAR_TO_BYTE8 (glyph->u.ch));
       else
-	code = face->font->driver->encode_char (face->font, glyph->u.ch);
-
-      if (code == FONT_INVALID_CODE)
-	code = 0;
+	code = face->font->driver->encode_char (face->font, glyph->u.ch,
+						context);
     }
 
-  /* Ensure that the code is only 2 bytes wide.  */
-  *char2b = code & 0xFFFF;
+  *char2b = code;
   return face;
 }
 
@@ -27577,20 +27595,21 @@ get_glyph_face_and_encoding (struct frame *f, struct glyph *glyph,
    Return true iff FONT has a glyph for C.  */
 
 static bool
-get_char_glyph_code (int c, struct font *font, unsigned *char2b)
+get_char_glyph_code (int c, struct font *font, Lisp_Object *char2b,
+		     struct glyph_context *context)
 {
-  unsigned code;
+  Lisp_Object code = Qnil;
 
   if (CHAR_BYTE8_P (c))
-    code = CHAR_TO_BYTE8 (c);
+    code = make_fixnum (CHAR_TO_BYTE8 (c));
   else
-    code = font->driver->encode_char (font, c);
+    code = font->driver->encode_char (font, c, context);
 
-  if (code == FONT_INVALID_CODE)
+  if (NILP (code))
     return false;
 
   /* Ensure that the code is only 2 bytes wide.  */
-  *char2b = code & 0xFFFF;
+  *char2b = code;
   return true;
 }
 
@@ -27632,7 +27651,8 @@ fill_composite_glyph_string (struct glyph_string *s, struct face *base_face,
 				       -1, Qnil);
 
 	  face = get_char_face_and_encoding (s->f, c, face_id,
-					     s->char2b + i, true);
+					     s->char2b + i, true,
+					     NULL);
 	  if (face)
 	    {
 	      if (! s->face)
@@ -27710,7 +27730,8 @@ fill_gstring_glyph_string (struct glyph_string *s, int face_id,
       unsigned code = LGLYPH_CODE (lglyph);
 
       /* Ensure that the code is only 2 bytes wide.  */
-      s->char2b[i] = code & 0xFFFF;
+      memset (s->char2b + i, 0, sizeof (s->char2b[i]));
+      s->char2b[i] = make_fixnum (code);
     }
 
   return glyph - s->row->glyphs[s->area];
@@ -27789,7 +27810,8 @@ fill_glyph_string (struct glyph_string *s, int face_id,
 	 && glyph->glyph_not_available_p == glyph_not_available_p)
     {
       s->face = get_glyph_face_and_encoding (s->f, glyph,
-					     s->char2b + s->nchars);
+					     s->char2b + s->nchars,
+					     glyph->context);
       ++s->nchars;
       eassert (s->nchars <= end - start);
       s->width += glyph->pixel_width;
@@ -27889,13 +27911,13 @@ fill_stretch_glyph_string (struct glyph_string *s, int start, int end)
 }
 
 static struct font_metrics *
-get_per_char_metric (struct font *font, const unsigned *char2b)
+get_per_char_metric (struct font *font, Lisp_Object *char2b)
 {
   static struct font_metrics metrics;
 
   if (! font)
     return NULL;
-  if (*char2b == FONT_INVALID_CODE)
+  if (NILP (*char2b))
     return NULL;
 
   font->driver->text_extents (font, char2b, 1, &metrics);
@@ -27916,11 +27938,11 @@ normal_char_ascent_descent (struct font *font, int c, int *ascent, int *descent)
 
   if (FONT_TOO_HIGH (font))
     {
-      unsigned char2b;
+      Lisp_Object char2b;
 
       /* Get metrics of C, defaulting to a reasonably sized ASCII
 	 character.  */
-      if (get_char_glyph_code (c >= 0 ? c : '{', font, &char2b))
+      if (get_char_glyph_code (c >= 0 ? c : '{', font, &char2b, NULL))
 	{
 	  struct font_metrics *pcm = get_per_char_metric (font, &char2b);
 
@@ -27963,11 +27985,13 @@ gui_get_glyph_overhangs (struct glyph *glyph, struct frame *f, int *left, int *r
 
   if (glyph->type == CHAR_GLYPH)
     {
-      unsigned char2b;
-      struct face *face = get_glyph_face_and_encoding (f, glyph, &char2b);
+      Lisp_Object char2b;
+      struct face *face = get_glyph_face_and_encoding (f, glyph, &char2b,
+						       glyph->context);
       if (face->font)
 	{
-	  struct font_metrics *pcm = get_per_char_metric (face->font, &char2b);
+	  struct font_metrics *pcm = get_per_char_metric (face->font,
+							  &char2b);
 	  if (pcm)
 	    {
 	      if (pcm->rbearing > pcm->width)
@@ -28273,21 +28297,21 @@ #define BUILD_IMAGE_GLYPH_STRING(START, END, HEAD, TAIL, HL, X, LAST_X) \
    is DRAW_CURSOR if a cursor has to be drawn.  LAST_X is the
    right-most x-position of the drawing area.  */
 
-#define BUILD_CHAR_GLYPH_STRINGS(START, END, HEAD, TAIL, HL, X, LAST_X)	   \
-     do									   \
-       {								   \
-	 int face_id;							   \
-	 unsigned *char2b;					   \
-									   \
-	 face_id = (row)->glyphs[area][START].face_id;			   \
-									   \
-	 s = alloca (sizeof *s);					   \
-	 SAFE_NALLOCA (char2b, 1, (END) - (START));			   \
-	 INIT_GLYPH_STRING (s, char2b, w, row, area, START, HL);	   \
-	 append_glyph_string (&HEAD, &TAIL, s);				   \
-	 s->x = (X);							   \
-	 START = fill_glyph_string (s, face_id, START, END, overlaps);	   \
-       }								   \
+#define BUILD_CHAR_GLYPH_STRINGS(START, END, HEAD, TAIL, HL, X, LAST_X)	\
+  do									\
+       {								\
+	 int face_id;							\
+	 Lisp_Object *char2b;						\
+									\
+	 face_id = (row)->glyphs[area][START].face_id;			\
+									\
+	 s = alloca (sizeof *s);					\
+	 SAFE_ALLOCA_LISP (char2b, (END) - (START));			\
+	 INIT_GLYPH_STRING (s, char2b, w, row, area, START, HL);	\
+	 append_glyph_string (&HEAD, &TAIL, s);				\
+	 s->x = (X);							\
+	 START = fill_glyph_string (s, face_id, START, END, overlaps);	\
+       }								\
      while (false)
 
 
@@ -28306,11 +28330,11 @@ #define BUILD_COMPOSITE_GLYPH_STRING(START, END, HEAD, TAIL, HL, X, LAST_X) \
     struct face *base_face = FACE_FROM_ID (f, face_id);		    \
     ptrdiff_t cmp_id = (row)->glyphs[area][START].u.cmp.id;		    \
     struct composition *cmp = composition_table[cmp_id];		    \
-    unsigned *char2b;						            \
+    Lisp_Object *char2b;						\
     struct glyph_string *first_s = NULL;				    \
     int n;								    \
     									    \
-    SAFE_NALLOCA (char2b, 1, cmp->glyph_len);				    \
+    SAFE_ALLOCA_LISP (char2b, cmp->glyph_len);				    \
     									    \
     /* Make glyph_strings for each glyph sequence that is drawable by	    \
        the same face, and append them to HEAD/TAIL.  */			    \
@@ -28338,14 +28362,14 @@ #define BUILD_COMPOSITE_GLYPH_STRING(START, END, HEAD, TAIL, HL, X, LAST_X) \
 #define BUILD_GSTRING_GLYPH_STRING(START, END, HEAD, TAIL, HL, X, LAST_X) \
   do {									  \
     int face_id;							  \
-    unsigned *char2b;						          \
+    Lisp_Object *char2b;						\
     Lisp_Object gstring;						  \
     									  \
     face_id = (row)->glyphs[area][START].face_id;			  \
     gstring = (composition_gstring_from_id				  \
 	       ((row)->glyphs[area][START].u.cmp.id));			  \
     s = alloca (sizeof *s);						  \
-    SAFE_NALLOCA (char2b, 1, LGSTRING_GLYPH_LEN (gstring));		  \
+    SAFE_ALLOCA_LISP (char2b, LGSTRING_GLYPH_LEN (gstring));		\
     INIT_GLYPH_STRING (s, char2b, w, row, area, START, HL);		  \
     append_glyph_string (&(HEAD), &(TAIL), s);				  \
     s->x = (X);								  \
@@ -28767,7 +28791,7 @@ #define IT_EXPAND_MATRIX_WIDTH(it, area)		\
    Called from gui_produce_glyphs when IT->glyph_row is non-null.  */
 
 static void
-append_glyph (struct it *it)
+append_glyph (struct it *it, struct glyph_context *context)
 {
   struct glyph *glyph;
   enum glyph_row_area area = it->area;
@@ -28791,6 +28815,7 @@ append_glyph (struct it *it)
 	}
       glyph->charpos = CHARPOS (it->position);
       glyph->object = it->object;
+      glyph->context = context;
       if (it->pixel_width > 0)
 	{
 	  eassert (it->pixel_width <= SHRT_MAX);
@@ -28873,6 +28898,7 @@ append_composite_glyph (struct it *it)
 	}
       glyph->charpos = it->cmp_it.charpos;
       glyph->object = it->object;
+      glyph->context = NULL;
       eassert (it->pixel_width <= SHRT_MAX);
       glyph->pixel_width = it->pixel_width;
       glyph->ascent = it->ascent;
@@ -29084,6 +29110,7 @@ produce_image_glyph (struct it *it)
 	{
 	  glyph->charpos = CHARPOS (it->position);
 	  glyph->object = it->object;
+	  glyph->context = NULL;
 	  glyph->pixel_width = clip_to_bounds (-1, it->pixel_width, SHRT_MAX);
 	  glyph->ascent = glyph_ascent;
 	  glyph->descent = it->descent;
@@ -29190,6 +29217,7 @@ produce_xwidget_glyph (struct it *it)
 	{
 	  glyph->charpos = CHARPOS (it->position);
 	  glyph->object = it->object;
+	  glyph->context = NULL;
 	  glyph->pixel_width = clip_to_bounds (-1, it->pixel_width, SHRT_MAX);
 	  glyph->ascent = glyph_ascent;
 	  glyph->descent = it->descent;
@@ -29276,6 +29304,7 @@ append_stretch_glyph (struct it *it, Lisp_Object object,
 	}
       glyph->charpos = CHARPOS (it->position);
       glyph->object = object;
+      glyph->context = NULL;
       /* FIXME: It would be better to use TYPE_MAX here, but
 	 __typeof__ is not portable enough...  */
       glyph->pixel_width = clip_to_bounds (-1, width, SHRT_MAX);
@@ -29730,6 +29759,7 @@ append_glyphless_glyph (struct it *it, int face_id, bool for_no_font, int len,
 	}
       glyph->charpos = CHARPOS (it->position);
       glyph->object = it->object;
+      glyph->context = NULL;
       eassert (it->pixel_width <= SHRT_MAX);
       glyph->pixel_width = it->pixel_width;
       glyph->ascent = it->ascent;
@@ -29829,7 +29859,7 @@ produce_glyphless_glyph (struct it *it, bool for_no_font, Lisp_Object acronym)
     {
       char buf[7];
       const char *str;
-      unsigned int code[6];
+      Lisp_Object code[6];
       int upper_len;
       int ascent, descent;
       struct font_metrics metrics_upper, metrics_lower;
@@ -29853,7 +29883,7 @@ produce_glyphless_glyph (struct it *it, bool for_no_font, Lisp_Object acronym)
 	  str = buf;
 	}
       for (len = 0; str[len] && ASCII_CHAR_P (str[len]) && len < 6; len++)
-	code[len] = font->driver->encode_char (font, str[len]);
+	code[len] = font->driver->encode_char (font, str[len], NULL);
       upper_len = (len + 1) / 2;
       font->driver->text_extents (font, code, upper_len,
 				  &metrics_upper);
@@ -29948,6 +29978,229 @@ #define IT_APPLY_FACE_BOX(it, face)				\
       }								\
     } while (false)
 
+#define IS_DELIMITER(c) ((c) <= ' ')
+#define it_lookahead(it) 128
+
+static struct glyph_context *
+make_context (struct it *it_orig, struct font *font)
+{
+  if (it_orig->context_cache.next_transition > 0)
+    {
+      struct glyph_context *context = ALLOCATE_PSEUDOVECTOR (struct glyph_context, position, PVEC_GLYPH_CONTEXT);
+      context->string = it_orig->context_cache.context->string;
+      context->script = it_orig->context_cache.context->script;
+      context->position = make_fixnum (XFIXNUM (it_orig->context_cache.context->position) + it_orig->context_cache.age);
+      return context;
+    }
+
+  Lisp_Object script = Qnil;
+  if (font->driver->guess_script)
+    script = font->driver->guess_script (font, it_orig->char_to_display);
+
+  bool seen_rtl = false;
+  bool seen_ltr = false;
+
+  if (CHAR_HAS_CATEGORY (it_orig->char_to_display, 'L'))
+    seen_ltr = true;
+  else if (CHAR_HAS_CATEGORY (it_orig->char_to_display, 'R'))
+    seen_rtl = true;
+
+  bool bidi_cache_shelved = false;
+  void *bidi_cache = NULL;
+  struct it it;
+  struct glyph_context *context = ALLOCATE_PSEUDOVECTOR (struct glyph_context, position, PVEC_GLYPH_CONTEXT);
+  int len = it_orig->history.len;
+  int past_len = len;
+  Lisp_Object position = Qnil;
+  int string_len = 5 * (IT_HISTORY_SIZE + it_lookahead (it_orig));
+  unsigned char *string = xmalloc (string_len + 1);
+  ptrdiff_t charpos = 0;
+  ptrdiff_t bytepos = 0;
+  ptrdiff_t start_charpos = 0;
+  ptrdiff_t start_bytepos = 0;
+  ptrdiff_t delim_bytepos = -1;
+  ptrdiff_t delim_charpos = -1;
+  ptrdiff_t next_transition = 0;
+
+  start_charpos = past_len;
+  int transitions = 0;
+  bool was_delimiter = IS_DELIMITER (it_orig->char_to_display);
+  while (start_charpos > 0)
+    {
+      int c = it_orig->history.c[start_charpos - 1];
+      if (c == '\n')
+	break;
+      bool is_delimiter = IS_DELIMITER (c);
+      transitions += is_delimiter != was_delimiter;
+      was_delimiter = is_delimiter;
+      start_charpos--;
+      if (transitions > 2)
+	break;
+    }
+
+  if (was_delimiter)
+    {
+      delim_bytepos = 0;
+      delim_charpos = start_charpos;
+    }
+  for (charpos = start_charpos; charpos < past_len + it_lookahead (it_orig);)
+    {
+      int c;
+      if (charpos < past_len)
+	c = it_orig->history.c[charpos];
+      else
+	{
+	  if (charpos == past_len)
+	    {
+	      bidi_cache = bidi_shelve_cache ();
+	      bidi_cache_shelved = true;
+	      it = *it_orig;
+	      position = make_fixnum (charpos - start_charpos);
+	    }
+
+	  if (it.end_charpos >= BEG
+	      && it.end_charpos == IT_CHARPOS (it)
+	      && it.method == GET_FROM_BUFFER
+	      && it.end_charpos < ZV)
+	    it.end_charpos++;
+	  if (!get_next_display_element (&it)
+	      || it.method == GET_FROM_DISPLAY_VECTOR
+	      || it.what != IT_CHARACTER
+	      || it.char_to_display < ' ')
+	    {
+	      if (charpos > past_len && next_transition == 0)
+		next_transition = charpos;
+
+	      break;
+	    }
+
+	  c = it.char_to_display;
+	  if (NILP (script)
+	      && font->driver->guess_script)
+	    script = font->driver->guess_script (font, c);
+
+	  set_iterator_to_next (&it, false);
+	}
+
+      if (c == '\n')
+	break;
+
+      if (delim_bytepos >= 0 && !IS_DELIMITER (c))
+	{
+	  if (charpos > past_len && next_transition == 0)
+	    next_transition = charpos;
+	  if (charpos < past_len)
+	    {
+	      start_bytepos = bytepos;
+	      start_charpos = charpos;
+	    }
+	  else if (delim_charpos > past_len)
+	    break;
+	  delim_bytepos = -1;
+	  delim_charpos = -1;
+	}
+      else if (delim_bytepos < 0 && IS_DELIMITER (c))
+	{
+	  if (charpos > past_len && next_transition == 0)
+	    next_transition = charpos;
+	  if (charpos > past_len)
+	    break;
+	  else if (charpos < past_len)
+	    {
+	      delim_bytepos = bytepos;
+	      delim_charpos = charpos;
+	    }
+	}
+      if ((seen_ltr && CHAR_HAS_CATEGORY (c, 'R'))
+	  || (seen_rtl && CHAR_HAS_CATEGORY (c, 'L')))
+	{
+	  if (charpos > past_len && next_transition == 0)
+	    next_transition = charpos;
+	  if (charpos >= past_len)
+	    break;
+	}
+
+      bytepos += CHAR_STRING (c, string + bytepos);
+      charpos++;
+
+      if ((seen_ltr && CHAR_HAS_CATEGORY (c, 'R'))
+	  || (seen_rtl && CHAR_HAS_CATEGORY (c, 'L')))
+	{
+	  if (charpos <= past_len)
+	    {
+	      start_bytepos = bytepos;
+	      start_charpos = charpos;
+	    }
+	}
+
+      if (charpos > past_len)
+	{
+	  if (CHAR_HAS_CATEGORY (c, 'R'))
+	    seen_rtl = true;
+	  else if (CHAR_HAS_CATEGORY (c, 'L'))
+	    seen_ltr = true;
+	}
+    }
+
+  if (charpos == past_len)
+    {
+      xfree (string);
+      if (bidi_cache_shelved)
+	bidi_unshelve_cache (bidi_cache, false);
+      return NULL;
+    }
+
+  if (delim_charpos > past_len)
+    {
+      bytepos = delim_bytepos;
+      charpos = delim_charpos;
+    }
+
+  int clen;
+
+  while (start_bytepos < bytepos
+	 && IS_DELIMITER (string_char_and_length (string + start_bytepos,
+						  &clen))
+	 && XFIXNUM (position) > 0)
+    {
+      start_bytepos += clen;
+      start_charpos++;
+      position = make_fixnum (XFIXNUM (position) - 1);
+    }
+
+  if (start_charpos == charpos)
+    {
+      xfree (string);
+      if (bidi_cache_shelved)
+	bidi_unshelve_cache (bidi_cache, false);
+      return NULL;
+    }
+  else
+    {
+      string[bytepos] = 0;
+      context->string = make_string ((char *)string + start_bytepos,
+				     bytepos - start_bytepos);
+      context->script = script;
+      xfree (string);
+    }
+  context->position = position;
+  if (seen_ltr != seen_rtl &&
+      seen_rtl != (it_orig->bidi_it.paragraph_dir == R2L))
+    {
+      context->string = Fnreverse (context->string);
+      context->position = make_fixnum (charpos - start_charpos - 1
+				       - XFIXNUM (context->position));
+    }
+  if (bidi_cache_shelved)
+    bidi_unshelve_cache (bidi_cache, false);
+
+  it_orig->context_cache.next_transition =
+    next_transition - past_len - XFIXNUM (context->position);
+  it_orig->context_cache.age = 0;
+  it_orig->context_cache.context = context;
+  return context;
+}
+
 /* RIF:
    Produce glyphs/get display metrics for the display element IT is
    loaded with.  See the description of struct it in dispextern.h
@@ -29962,7 +30215,7 @@ gui_produce_glyphs (struct it *it)
 
   if (it->what == IT_CHARACTER)
     {
-      unsigned char2b;
+      Lisp_Object char2b;
       struct face *face = FACE_FROM_ID (it->f, it->face_id);
       struct font *font = face->font;
       struct font_metrics *pcm = NULL;
@@ -29985,6 +30238,7 @@ gui_produce_glyphs (struct it *it)
       if (font->vertical_centering)
 	boff = VCENTER_BASELINE_OFFSET (font, it->f) - boff;
 
+      struct glyph_context *context = NULL;
       if (it->char_to_display != '\n' && it->char_to_display != '\t')
 	{
 	  it->nglyphs = 1;
@@ -30001,12 +30255,11 @@ gui_produce_glyphs (struct it *it)
  	      it->descent = FONT_DESCENT (font) - boff;
  	    }
 
-	  if (get_char_glyph_code (it->char_to_display, font, &char2b))
+	  context = make_context (it, font);
+	  if (get_char_glyph_code (it->char_to_display, font, &char2b,
+				   context))
 	    {
 	      pcm = get_per_char_metric (font, &char2b);
-	      if (pcm->width == 0
-		  && pcm->rbearing == 0 && pcm->lbearing == 0)
-		pcm = NULL;
 	    }
 
 	  if (pcm)
@@ -30091,9 +30344,10 @@ gui_produce_glyphs (struct it *it)
 				/ FONT_HEIGHT (font));
 		  append_stretch_glyph (it, it->object, it->pixel_width,
 					it->ascent + it->descent, ascent);
+		  it->glyph_row->glyphs[it->area][it->glyph_row->used[it->area] - 1].context = NULL;
 		}
 	      else
-		append_glyph (it);
+		append_glyph (it, context);
 
 	      /* If characters with lbearing or rbearing are displayed
 		 in this line, record that fact in a flag of the
@@ -30245,7 +30499,7 @@ gui_produce_glyphs (struct it *it)
 	      it->nglyphs = 1;
 	      if (FONT_TOO_HIGH (font))
 		{
-		  if (get_char_glyph_code (' ', font, &char2b))
+		  if (get_char_glyph_code (' ', font, &char2b, NULL))
 		    {
 		      pcm = get_per_char_metric (font, &char2b);
 		      if (pcm->width == 0
@@ -30342,7 +30596,7 @@ gui_produce_glyphs (struct it *it)
 	  int lbearing, rbearing;
 	  int i, width, ascent, descent;
 	  int c;
-	  unsigned char2b;
+	  Lisp_Object char2b;
 	  struct font_metrics *pcm;
 	  ptrdiff_t pos;
 
@@ -30384,7 +30638,7 @@ gui_produce_glyphs (struct it *it)
 	  if (! font_not_found_p)
 	    {
 	      get_char_face_and_encoding (it->f, c, it->face_id,
-					  &char2b, false);
+					  &char2b, false, NULL);
 	      pcm = get_per_char_metric (font, &char2b);
 	    }
 
@@ -30445,7 +30699,8 @@ gui_produce_glyphs (struct it *it)
 	      else
 		{
 		  get_char_face_and_encoding (it->f, ch, face_id,
-					      &char2b, false);
+					      &char2b, false,
+					      NULL);
 		  pcm = get_per_char_metric (font, &char2b);
 		}
 	      if (! pcm)
@@ -34345,6 +34600,7 @@ syms_of_xdisp (void)
   DEFSYM (Qdisplay, "display");
   DEFSYM (Qspace_width, "space-width");
   DEFSYM (Qraise, "raise");
+  DEFSYM (Qglyph, "glyph");
   DEFSYM (Qslice, "slice");
   DEFSYM (Qspace, "space");
   DEFSYM (Qmargin, "margin");
diff --git a/src/xfont.c b/src/xfont.c
index 1563b43bf9..3542e528b4 100644
--- a/src/xfont.c
+++ b/src/xfont.c
@@ -921,9 +921,10 @@ xfont_has_char (Lisp_Object font, int c)
   return (ENCODE_CHAR (repertory, c) != CHARSET_INVALID_CODE (repertory));
 }
 
-static unsigned
-xfont_encode_char (struct font *font, int c)
+static Lisp_Object
+xfont_encode_char (struct font *font, int c, struct glyph_context *context)
 {
+  Lisp_Object ret;
   XFontStruct *xfont = ((struct xfont_info *) font)->xfont;
   struct charset *charset;
   unsigned code;
@@ -931,18 +932,19 @@ xfont_encode_char (struct font *font, int c)
   charset = CHARSET_FROM_ID (font->encoding_charset);
   code = ENCODE_CHAR (charset, c);
   if (code == CHARSET_INVALID_CODE (charset))
-    return FONT_INVALID_CODE;
+    return Qnil;
   if (font->repertory_charset >= 0)
     {
       charset = CHARSET_FROM_ID (font->repertory_charset);
-      return (ENCODE_CHAR (charset, c) != CHARSET_INVALID_CODE (charset)
-	      ? code : FONT_INVALID_CODE);
+      ret = ENCODE_CHAR (charset, c) != CHARSET_INVALID_CODE (charset)
+	? make_fixnum (code) : Qnil;
+      return ret;
     }
-  return (xfont_get_pcm (xfont, code) ? code : FONT_INVALID_CODE);
+  return (xfont_get_pcm (xfont, code) ? make_fixnum (code) : Qnil);
 }
 
 static void
-xfont_text_extents (struct font *font, const unsigned int *code,
+xfont_text_extents (struct font *font, const Lisp_Object *code,
 		    int nglyphs, struct font_metrics *metrics)
 {
   XFontStruct *xfont = ((struct xfont_info *) font)->xfont;
@@ -953,9 +955,9 @@ xfont_text_extents (struct font *font, const unsigned int *code,
     {
       static XCharStruct *pcm;
 
-      if (code[i] >= 0x10000)
+      if (XFIXNUM (code[i]) >= 0x10000)
 	continue;
-      pcm = xfont_get_pcm (xfont, code[i]);
+      pcm = xfont_get_pcm (xfont, XFIXNUM (code[i]));
       if (! pcm)
 	continue;
       if (first)
@@ -1005,7 +1007,7 @@ xfont_draw (struct glyph_string *s, int from, int to, int x, int y,
       USE_SAFE_ALLOCA;
       char *str = SAFE_ALLOCA (len);
       for (i = 0; i < len ; i++)
-	str[i] = s->char2b[from + i] & 0xFF;
+	str[i] = XFIXNUM (s->char2b[from + i]) & 0xFF;
       block_input ();
       if (with_background)
 	{
@@ -1038,21 +1040,25 @@ xfont_draw (struct glyph_string *s, int from, int to, int x, int y,
       if (s->padding_p)
 	for (i = 0; i < len; i++)
           {
-            const unsigned code = s->char2b[from + i];
-            const XChar2b char2b = { .byte1 = code >> 8,
-                                     .byte2 = code & 0xFF };
+	    Lisp_Object code = s->char2b[from + i];
+	    const XChar2b char2b = {
+	      .byte1 = XFIXNUM (code) >> 8,
+	      .byte2 = XFIXNUM (code) & 0xff
+	    };
             XDrawImageString16 (display, FRAME_X_DRAWABLE (s->f),
                                 gc, x + i, y, &char2b, 1);
           }
       else
         {
           USE_SAFE_ALLOCA;
-          const unsigned *code = s->char2b + from;
+          const Lisp_Object *code = s->char2b + from;
           XChar2b *char2b;
           SAFE_NALLOCA (char2b, 1, len);
           for (int i = 0; i < len; ++i)
-            char2b[i] = (XChar2b) { .byte1 = code[i] >> 8,
-                                    .byte2 = code[i] & 0xFF };
+            char2b[i] = (XChar2b) {
+	      .byte1 = XFIXNUM (code[i]) >> 8,
+	      .byte2 = XFIXNUM (code[i]) & 0xFF
+	    };
           XDrawImageString16 (display, FRAME_X_DRAWABLE (s->f),
                               gc, x, y, char2b, len);
           SAFE_FREE ();
@@ -1063,21 +1069,25 @@ xfont_draw (struct glyph_string *s, int from, int to, int x, int y,
       if (s->padding_p)
 	for (i = 0; i < len; i++)
           {
-            const unsigned code = s->char2b[from + i];
-            const XChar2b char2b = { .byte1 = code >> 8,
-                                     .byte2 = code & 0xFF };
+            const Lisp_Object code = s->char2b[from + i];
+            const XChar2b char2b = {
+	      .byte1 = XFIXNUM (code) >> 8,
+	      .byte2 = XFIXNUM (code) & 0xFF
+	    };
             XDrawString16 (display, FRAME_X_DRAWABLE (s->f),
                            gc, x + i, y, &char2b, 1);
           }
       else
         {
           USE_SAFE_ALLOCA;
-          const unsigned *code = s->char2b + from;
+          const Lisp_Object *code = s->char2b + from;
           XChar2b *char2b;
           SAFE_NALLOCA (char2b, 1, len);
           for (int i = 0; i < len; ++i)
-            char2b[i] = (XChar2b) { .byte1 = code[i] >> 8,
-                                    .byte2 = code[i] & 0xFF };
+            char2b[i] = (XChar2b) {
+	      .byte1 = XFIXNUM (code[i]) >> 8,
+	      .byte2 = XFIXNUM (code[i]) & 0xFF
+	    };
           XDrawString16 (display, FRAME_X_DRAWABLE (s->f),
                          gc, x, y, char2b, len);
           SAFE_FREE ();
diff --git a/src/xterm.c b/src/xterm.c
index 7989cecec7..0c8c22cd9b 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -1993,7 +1993,7 @@ x_draw_composite_glyph_string_foreground (struct glyph_string *s)
 x_draw_glyphless_glyph_string_foreground (struct glyph_string *s)
 {
   struct glyph *glyph = s->first_glyph;
-  unsigned char2b[8];
+  Lisp_Object char2b[8];
   int x, i, j;
 
   /* If first glyph of S has a left box line, start drawing the text
@@ -2047,7 +2047,7 @@ x_draw_glyphless_glyph_string_foreground (struct glyph_string *s)
 
 	  /* It is assured that all LEN characters in STR is ASCII.  */
 	  for (j = 0; j < len; j++)
-            char2b[j] = s->font->driver->encode_char (s->font, str[j]) & 0xFFFF;
+            char2b[j] = s->font->driver->encode_char (s->font, str[j], NULL);
 	  s->font->driver->draw (s, 0, upper_len,
 				 x + glyph->slice.glyphless.upper_xoff,
 				 s->ybase + glyph->slice.glyphless.upper_yoff,
-- 
2.27.0.rc0


^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: Kerning and ligatures
  2020-05-30 22:13 Kerning and ligatures Pip Cet
@ 2020-06-08 18:29 ` Yagnesh Raghava Yakkala
  0 siblings, 0 replies; 2+ messages in thread
From: Yagnesh Raghava Yakkala @ 2020-06-08 18:29 UTC (permalink / raw)
  To: Pip Cet; +Cc: emacs-devel


Hello Pip,

> Can someone help with the Telugu "hello"? I wonder whether there
> should be a gap, and how much of one, between స్కా and రం in నమస్కారం?

No, there shouldn't be any gap as they are part of single word. 

Thanks.,
--
YYR



^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2020-06-08 18:29 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-30 22:13 Kerning and ligatures Pip Cet
2020-06-08 18:29 ` Yagnesh Raghava Yakkala

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).