Question about display engine

Question about display engine

[Ergus]

Hi:

Sorry to bother with this.

Fixing the issue 36858 in text mode I found that the
extend_face_to_end_of_line uses the same face for the last char in the
line.

This is useful to extend selection face the whole line, but this created
a difference with gui emacs when the last face was underlined or
overlined because in tui the underline is extended for the entire line.

To fix this I need to create a new face to extend until the end of the
line that has the same properties than it->face_id except that the
underline and overline properties will be disabled.

I can produce the desired effect doing:

```Lisp
(defface my_new_face
  '((t :weight normal :slant normal
       :underline nil :overline nil :strike-through nil
       :box nil :inverse-video nil :stipple nil)))
```

```C
DEFSYM (my_new_face, "my-new-face");

it->face_id = merge_faces (it->w, my-new-face, 0, it->face_id)
```

But this seems very dirty.

How can I produce the same effect in the right way? (I mean create a new
face_id based on it->face_id but with :underline nil :overline
nil... etc?) And only with C code.

Any help, suggestion?
Thanks in advance,
Ergus

Re: Question about display engine

[Eli Zaretskii]

> Date: Wed, 7 Aug 2019 02:54:11 +0200
> From: Ergus <spacibba@aol.com>
> Cc: emacs-devel@gnu.org
> 
> Sorry to bother with this.

Sorry for not answering sooner: I needed to find time to re-read the
relevant code and recollect the consequences.

> Fixing the issue 36858 in text mode I found that the
> extend_face_to_end_of_line uses the same face for the last char in the
> line.

Yes.

> This is useful to extend selection face the whole line, but this created
> a difference with gui emacs when the last face was underlined or
> overlined because in tui the underline is extended for the entire line.

The issue is more general, and not limited to the underline
attribute.  See below.

> To fix this I need to create a new face to extend until the end of the
> line that has the same properties than it->face_id except that the
> underline and overline properties will be disabled.

Why only underline and overline?  And why do you think you can reset
these attributes at will in this case?  Suppose someone defines the
'region' face to use underline -- we definitely cannot reset this
attribute in that case, because then the region will appear
non-contiguous, right?

> I can produce the desired effect doing:
> 
> ```Lisp
> (defface my_new_face
>   '((t :weight normal :slant normal
>        :underline nil :overline nil :strike-through nil
>        :box nil :inverse-video nil :stipple nil)))
> ```
> 
> ```C
> DEFSYM (my_new_face, "my-new-face");
> 
> it->face_id = merge_faces (it->w, my-new-face, 0, it->face_id)
> ```
> 
> But this seems very dirty.
> 
> How can I produce the same effect in the right way? (I mean create a new
> face_id based on it->face_id but with :underline nil :overline
> nil... etc?) And only with C code.

You need to call realize_face after copying the attributes of the
default face and resetting some of the attributes to Qunspecified.
But this is the mechanical part of the issue; I think the conceptual
part is more problematic.

First, we indeed behave inconsistently in GUI and TTY frames regarding
faces that straddle the newline.  On GUI frames, some attributes, such
as colors and the box attribute, extend all the way to the window's
edge; others, like underline, overline, and strike-through only affect
the single glyph that stands for the newline (which the display engine
adds so it will have a place to display the cursor).  By contrast, on
TTY frames, every attribute we support in text mode extends to the
window's edge.

Emacs behaved like that since v21.

The question is: which of the 2 is the correct display, if there is a
correct one?  When trying to answer this question, please keep in mind
two special use cases: the use case with the region, and the use case
where the same attribute continues on the next screen line.  Maybe
there are other relevant use cases as well.

I myself don't have a definitive answer.  The TTY behavior makes more
sense to me, but people frequently complain about it saying it's ugly,
so I guess "makes sense" doesn't necessarily cut it.

Re: Question about display engine

[Ergus]

On Wed, Aug 07, 2019 at 06:01:12PM +0300, Eli Zaretskii wrote:
>> Date: Wed, 7 Aug 2019 02:54:11 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: emacs-devel@gnu.org
>>
>> Sorry to bother with this.
>
>Sorry for not answering sooner: I needed to find time to re-read the
>relevant code and recollect the consequences.
>
>> Fixing the issue 36858 in text mode I found that the
>> extend_face_to_end_of_line uses the same face for the last char in the
>> line.
>
>Yes.
>
>> This is useful to extend selection face the whole line, but this created
>> a difference with gui emacs when the last face was underlined or
>> overlined because in tui the underline is extended for the entire line.
>
>The issue is more general, and not limited to the underline
>attribute.  See below.
>
>> To fix this I need to create a new face to extend until the end of the
>> line that has the same properties than it->face_id except that the
>> underline and overline properties will be disabled.
>
>Why only underline and overline?  And why do you think you can reset
>these attributes at will in this case?  Suppose someone defines the
>'region' face to use underline -- we definitely cannot reset this
>attribute in that case, because then the region will appear
>non-contiguous, right?
>
>> I can produce the desired effect doing:
>>
>> ```Lisp
>> (defface my_new_face
>>   '((t :weight normal :slant normal
>>        :underline nil :overline nil :strike-through nil
>>        :box nil :inverse-video nil :stipple nil)))
>> ```
>>
>> ```C
>> DEFSYM (my_new_face, "my-new-face");
>>
>> it->face_id = merge_faces (it->w, my-new-face, 0, it->face_id)
>> ```
>>
>> But this seems very dirty.
>>
>> How can I produce the same effect in the right way? (I mean create a new
>> face_id based on it->face_id but with :underline nil :overline
>> nil... etc?) And only with C code.
>
>You need to call realize_face after copying the attributes of the
>default face and resetting some of the attributes to Qunspecified.
>But this is the mechanical part of the issue; I think the conceptual
>part is more problematic.
>
>First, we indeed behave inconsistently in GUI and TTY frames regarding
>faces that straddle the newline.  On GUI frames, some attributes, such
>as colors and the box attribute, extend all the way to the window's
>edge; others, like underline, overline, and strike-through only affect
>the single glyph that stands for the newline (which the display engine
>adds so it will have a place to display the cursor).  By contrast, on
>TTY frames, every attribute we support in text mode extends to the
>window's edge.
>
>Emacs behaved like that since v21.
>
>The question is: which of the 2 is the correct display, if there is a
>correct one?  When trying to answer this question, please keep in mind
>two special use cases: the use case with the region, and the use case
>where the same attribute continues on the next screen line.  Maybe
>there are other relevant use cases as well.
>
>I myself don't have a definitive answer.  The TTY behavior makes more
>sense to me, but people frequently complain about it saying it's ugly,
>so I guess "makes sense" doesn't necessarily cut it.
>

Hi:

I am not sure about what will be the community opinion; so I'll wait for
some comments (and the usual complains) before implementing anything.

Maybe it makes some sense to consider the region as a corner case
here. So a possible solution is just to add a condition to reset the
underline (and other attributes) when the last glyph was not in the
active region.

The actual TUI behavior is annoying in some modes; so maybe the best is
to reproduce the gui behavior as it is now in TUI too.

But:

After thinking on that a little bit more since yesterday; maybe it is
possible to add another basic face for the rest of the line. That face
will be merged with the previous face as in the example code, so if it
specifies :underline then merging should work as specified; else, it
will just use the :underline from the latest glyph.

So the user could potentially customize the desired behavior. The
arguments will come about what the defaults should be, but at least we
don't limit that.

Does it makes sense?
Best,
Ergus

Re: Question about display engine

[Eli Zaretskii]

> Date: Wed, 7 Aug 2019 17:32:20 +0200
> From: Ergus <spacibba@aol.com>
> Cc: emacs-devel@gnu.org
> 
> After thinking on that a little bit more since yesterday; maybe it is
> possible to add another basic face for the rest of the line. That face
> will be merged with the previous face as in the example code, so if it
> specifies :underline then merging should work as specified; else, it
> will just use the :underline from the latest glyph.

Such a face will not be a fixed face, it will have to be recomputed
whenever the face of the text changes, right?  E.g., if the face of
the text specifies some color, you'd want this additional face to have
the same colors, right?

So it doesn't seem to be a face that can be customized in the usual
sense.  We could let the users specify face attributes they don't want
to see in face extension, though.

Re: Question about display engine

[Ergus]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

On Wed, Aug 07, 2019 at 06:45:47PM +0300, Eli Zaretskii wrote:
>> Date: Wed, 7 Aug 2019 17:32:20 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: emacs-devel@gnu.org
>>
>> After thinking on that a little bit more since yesterday; maybe it is
>> possible to add another basic face for the rest of the line. That face
>> will be merged with the previous face as in the example code, so if it
>> specifies :underline then merging should work as specified; else, it
>> will just use the :underline from the latest glyph.
>
>Such a face will not be a fixed face, it will have to be recomputed
>whenever the face of the text changes, right?  E.g., if the face of
>the text specifies some color, you'd want this additional face to have
>the same colors, right?
>
We don't use the face itself, just to merge with the previous glyph.

>So it doesn't seem to be a face that can be customized in the usual
>sense.  We could let the users specify face attributes they don't want
>to see in face extension, though.

Please look the proposed patch. It may need some improvements, but at
least the functional part is a decent solution for all the issues in my
opinion.


[-- Attachment #2: fix_36858.patch --]
[-- Type: text/plain, Size: 8407 bytes --]

diff --git a/lisp/faces.el b/lisp/faces.el
index 5193c216d0..9c3eba0fff 100644
--- a/lisp/faces.el
+++ b/lisp/faces.el
@@ -2510,7 +2510,7 @@ unwanted effects."
 
 ;; Definition stolen from display-line-numbers.
 (defface fill-column-indicator
-  '((t :inherit shadow :weight normal :slant normal
+  '((t :inherit extend-to-end-of-line :weight normal :slant normal
        :underline nil :overline nil :strike-through nil
        :box nil :inverse-video nil :stipple nil))
   "Face for displaying fill column indicator.
@@ -2694,12 +2694,20 @@ the same as `window-divider' face."
   :group 'basic-faces)
 
 (defface internal-border
-    '((t nil))
+  '((t nil))
   "Basic face for the internal border."
   :version "26.1"
   :group 'frames
   :group 'basic-faces)
 
+(defface extend-to-end-of-line
+  '((t :weight normal :slant normal
+       :underline nil :overline nil :strike-through nil
+       :box nil :inverse-video nil :stipple nil))
+  "Basic face to extend to end if line."
+  :version "27.1"
+  :group 'basic-faces)
+
 (defface minibuffer-prompt
   '((((background dark)) :foreground "cyan")
     ;; Don't use blue because many users of the MS-DOS port customize
diff --git a/src/dispextern.h b/src/dispextern.h
index 4e947daa25..9a8bad6d08 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -1780,6 +1780,7 @@ enum face_id
   WINDOW_DIVIDER_FIRST_PIXEL_FACE_ID,
   WINDOW_DIVIDER_LAST_PIXEL_FACE_ID,
   INTERNAL_BORDER_FACE_ID,
+  EXTEND_TO_END_OF_LINE_FACE_ID,
   BASIC_FACE_ID_SENTINEL
 };
 
diff --git a/src/xdisp.c b/src/xdisp.c
index 7338d2b7d4..738a6ca129 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -386,6 +386,7 @@ fill_column_indicator_column (struct it *it)
 {
   if (Vdisplay_fill_column_indicator
       && it->continuation_lines_width == 0
+      && IT_CHARPOS (*it) < ZV
       && CHARACTERP (Vdisplay_fill_column_indicator_character))
     {
       Lisp_Object col = (EQ (Vdisplay_fill_column_indicator_column, Qt)
@@ -20468,6 +20469,9 @@ extend_face_to_end_of_line (struct it *it)
       it->face_id = FACE_FOR_CHAR (f, face, 0, -1, Qnil);
     }
 
+  const int extend_face_merged_id =
+    merge_faces (it->w, Qextend_to_end_of_line, 0, it->face_id);
+
 #ifdef HAVE_WINDOW_SYSTEM
   if (FRAME_WINDOW_P (f))
     {
@@ -20519,12 +20523,14 @@ extend_face_to_end_of_line (struct it *it)
 	      const int char_width = (font->average_width
 				      ? font->average_width
 				      : font->space_width);
-	      int column_x;
-
-	      if (!INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
-		  && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
-		  && column_x >= it->current_x
-		  && column_x <= it->last_visible_x)
+	      int indicator_column_x;
+
+	      if (!INT_MULTIPLY_WRAPV (indicator_column,
+	                               char_width, &indicator_column_x)
+		  && !INT_ADD_WRAPV (it->lnum_pixel_width,
+		                     indicator_column_x, &indicator_column_x)
+		  && indicator_column_x >= it->current_x
+		  && indicator_column_x <= it->last_visible_x)
 	        {
 	          const char saved_char = it->char_to_display;
 	          const struct text_pos saved_pos = it->position;
@@ -20533,14 +20539,15 @@ extend_face_to_end_of_line (struct it *it)
 	          const bool saved_box_start = it->start_of_box_run_p;
 	          Lisp_Object save_object = it->object;
 
-	          /* The stretch width needs to considet the latter
+	          /* The stretch width needs to consider the latter
 	             added glyph.  */
 	          const int stretch_width
-		    = column_x - it->current_x - char_width;
+		    = indicator_column_x - it->current_x - char_width;
 
 	          memset (&it->position, 0, sizeof it->position);
 	          it->avoid_cursor_p = true;
 	          it->object = Qnil;
+		  it->face_id = extend_face_merged_id;
 
 	          /* Only generate a stretch glyph if there is distance
 	             between current_x and and the indicator position.  */
@@ -20548,6 +20555,7 @@ extend_face_to_end_of_line (struct it *it)
 		    {
 		      int stretch_ascent = (((it->ascent + it->descent)
 		                             * FONT_BASE (font)) / FONT_HEIGHT (font));
+
 		      append_stretch_glyph (it, Qnil, stretch_width,
 		                            it->ascent + it->descent,
 		                            stretch_ascent);
@@ -20555,24 +20563,29 @@ extend_face_to_end_of_line (struct it *it)
 
 	          /* Generate the glyph indicator only if
 	             append_space_for_newline didn't already.  */
-	          if (it->current_x < column_x)
+	          if (it->current_x < indicator_column_x)
 	            {
+		      it->face_id
+			= merge_faces (it->w, Qextend_to_end_of_line,
+			               0, extend_face_merged_id);
+
 		      it->char_to_display
 			= XFIXNAT (Vdisplay_fill_column_indicator_character);
-	              it->face_id
-			= merge_faces (it->w, Qfill_column_indicator,
-				       0, saved_face_id);
 	              PRODUCE_GLYPHS (it);
-	            }
 
-	          /* Restore the face after the indicator was generated.  */
-	          it->face_id = saved_face_id;
+		      it->face_id = extend_face_merged_id;
+	            }
 
 	          /* If there is space after the indicator generate an
 	             extra empty glyph to restore the face.  Issue was
 	             observed in X systems.  */
-	          it->char_to_display = ' ';
-	          PRODUCE_GLYPHS (it);
+		  if (it->current_x < it->last_visible_x) {
+		    it->char_to_display = ' ';
+		    PRODUCE_GLYPHS (it);
+		  }
+
+	          /* Restore the face after the indicator was generated.  */
+	          it->face_id = saved_face_id;
 
 	          it->char_to_display = saved_char;
 	          it->position = saved_pos;
@@ -20697,26 +20710,27 @@ extend_face_to_end_of_line (struct it *it)
       /* The last row's blank glyphs should get the default face, to
 	 avoid painting the rest of the window with the region face,
 	 if the region ends at ZV.  */
+
       if (it->glyph_row->ends_at_zv_p)
 	it->face_id = default_face->id;
       else
-	it->face_id = face->id;
+	it->face_id = extend_face_merged_id;
 
       /* Display fill-column indicator if needed.  */
       int indicator_column = fill_column_indicator_column (it);
       if (indicator_column >= 0
-	  && INT_ADD_WRAPV (it->lnum_pixel_width, indicator_column,
+          && INT_ADD_WRAPV (it->lnum_pixel_width, indicator_column,
 			    &indicator_column))
 	indicator_column = -1;
       do
 	{
 	  int saved_face_id;
-	  bool indicate = it->current_x == indicator_column;
+	  const bool indicate = it->current_x == indicator_column;
 	  if (indicate)
 	    {
 	      saved_face_id = it->face_id;
 	      it->face_id
-		= merge_faces (it->w, Qfill_column_indicator, 0, saved_face_id);
+		= merge_faces (it->w, Qfill_column_indicator, 0, extend_face_merged_id);
 	      it->c = it->char_to_display
 		= XFIXNAT (Vdisplay_fill_column_indicator_character);
 	    }
diff --git a/src/xfaces.c b/src/xfaces.c
index c3cae7e2a6..a0b0d9c6f8 100644
--- a/src/xfaces.c
+++ b/src/xfaces.c
@@ -4598,6 +4598,7 @@ lookup_basic_face (struct window *w, struct frame *f, int face_id)
     case WINDOW_DIVIDER_FIRST_PIXEL_FACE_ID:	name = Qwindow_divider_first_pixel;	break;
     case WINDOW_DIVIDER_LAST_PIXEL_FACE_ID:	name = Qwindow_divider_last_pixel;	break;
     case INTERNAL_BORDER_FACE_ID:	name = Qinternal_border; 	break;
+    case EXTEND_TO_END_OF_LINE_FACE_ID: name = Qextend_to_end_of_line;  break;
 
     default:
       emacs_abort (); /* the caller is supposed to pass us a basic face id */
@@ -5293,6 +5294,7 @@ realize_basic_faces (struct frame *f)
       realize_named_face (f, Qwindow_divider_last_pixel,
 			  WINDOW_DIVIDER_LAST_PIXEL_FACE_ID);
       realize_named_face (f, Qinternal_border, INTERNAL_BORDER_FACE_ID);
+      realize_named_face (f, Qextend_to_end_of_line, EXTEND_TO_END_OF_LINE_FACE_ID);
 
       /* Reflect changes in the `menu' face in menu bars.  */
       if (FRAME_FACE_CACHE (f)->menu_face_changed_p)
@@ -6592,6 +6594,7 @@ syms_of_xfaces (void)
   DEFSYM (Qwindow_divider_first_pixel, "window-divider-first-pixel");
   DEFSYM (Qwindow_divider_last_pixel, "window-divider-last-pixel");
   DEFSYM (Qinternal_border, "internal-border");
+  DEFSYM (Qextend_to_end_of_line, "extend-to-end-of-line");
 
   /* TTY color-related functions (defined in tty-colors.el).  */
   DEFSYM (Qtty_color_desc, "tty-color-desc");

Re: Question about display engine

[Eli Zaretskii]

> Date: Wed, 7 Aug 2019 17:57:38 +0200
> From: Ergus <spacibba@aol.com>
> Cc: emacs-devel@gnu.org
> 
> >Such a face will not be a fixed face, it will have to be recomputed
> >whenever the face of the text changes, right?  E.g., if the face of
> >the text specifies some color, you'd want this additional face to have
> >the same colors, right?
> >
> We don't use the face itself, just to merge with the previous glyph.
> 
> >So it doesn't seem to be a face that can be customized in the usual
> >sense.  We could let the users specify face attributes they don't want
> >to see in face extension, though.
> 
> Please look the proposed patch. It may need some improvements, but at
> least the functional part is a decent solution for all the issues in my
> opinion.

First, let's not mix this with the display-fill-column-indicator-mode,
let's keep the changes for these two separate, OK?

And second, I don't think I understand what we expect users to do with
this face's customization.  Suppose the user customizes this face to
set the :underline attribute, will the effect be reasonable?

I thought we wanted to let users determine which attributes will be
_reset_, not _set_.  But faces, as we use them normally, don't allow
resetting attributes.

And, of course, this leaves the more general problem I described in my
message: what is the right behavior for extending the face that
crosses a newline.  (Note that the display engine in general doesn't
know whether a face that doesn't end before a newline will or won't
continue on the next screen line.)

Thanks.

Re: Question about display engine

[martin rudalics]

 > And, of course, this leaves the more general problem I described in my
 > message: what is the right behavior for extending the face that
 > crosses a newline.  (Note that the display engine in general doesn't
 > know whether a face that doesn't end before a newline will or won't
 > continue on the next screen line.)

Recalling your proposal

(defcustom face-extend-to-window-edge t
    "Non-nil means extend face of last character on line to window edge.

Certain face attributes, if present in the face of the last character
of a line and different from those of the default face, cause the
empty space following the end of text on the line to be drawn with
those attributes, to give the empty space appearance similar to that
of the preceding text.  These attributes are those which affect the
background of a face: `:background', `:stipple', `:box', `:underline',
`:overline', and `:strike-through'.  By default, if the face of a
line's last character has any of these attributes, and the value is
different from that of the default face, the empty space following the
line's text will be drawn in the face of the last character.

This variable allows fine-tuning which attributes trigger the face
extension.  The default value of t means any of the mentioned
attributes will cause face extension.  The value of nil means face
extension is turned off.  A value that is a list of attributes will
extend the face only if any of the attributes from the list are
present in the last character's face.  Note that only attributes from
the above list are meaningful in list values of this variable.")

in the discussion of Bug#23574.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Wed, 7 Aug 2019 18:25:37 +0200
> 
>  > And, of course, this leaves the more general problem I described in my
>  > message: what is the right behavior for extending the face that
>  > crosses a newline.  (Note that the display engine in general doesn't
>  > know whether a face that doesn't end before a newline will or won't
>  > continue on the next screen line.)
> 
> Recalling your proposal
> 
> (defcustom face-extend-to-window-edge t
>     "Non-nil means extend face of last character on line to window edge.

Thanks for the reminder.

> Certain face attributes, if present in the face of the last character
> of a line and different from those of the default face, cause the
> empty space following the end of text on the line to be drawn with
> those attributes, to give the empty space appearance similar to that
> of the preceding text.  These attributes are those which affect the
> background of a face: `:background', `:stipple', `:box', `:underline',
> `:overline', and `:strike-through'.  By default, if the face of a
> line's last character has any of these attributes, and the value is
> different from that of the default face, the empty space following the
> line's text will be drawn in the face of the last character.

This is inaccurate, I think: on GUI frames only :background and :box
are extended all the way, the rest only "infect" the glyph we add for
displaying the newline.

> This variable allows fine-tuning which attributes trigger the face
> extension.  The default value of t means any of the mentioned
> attributes will cause face extension.  The value of nil means face
> extension is turned off.  A value that is a list of attributes will
> extend the face only if any of the attributes from the list are
> present in the last character's face.  Note that only attributes from
> the above list are meaningful in list values of this variable.")
> 
> in the discussion of Bug#23574.

And I think we need to consider the case of the region using one of
the attributes that are not in the list.

Re: Question about display engine

[martin rudalics]

 > And I think we need to consider the case of the region using one of
 > the attributes that are not in the list.

I now think it would make more sense to add an 'extend-to-window-edge'
attribute in the face definition itself.  Also, when looking at what
applications like Thunderbird or Firefox do, I nowhere see their
"region faces" extend to the right edges of their "windows" in "normal
text".  Admittedly, these two applications do not highlight an empty
space at the left edge of a window either and I'm fully aware of the
fact that I'm comparing apples and oranges here.  Still I think that
our users should be given the possibility to customize their regions
in a similar way as they do.

martin

Re: Question about display engine

[Ergus]

On Thu, Aug 08, 2019 at 09:25:45AM +0200, martin rudalics wrote:
>> And I think we need to consider the case of the region using one of
>> the attributes that are not in the list.
>
>I now think it would make more sense to add an 'extend-to-window-edge'
>attribute in the face definition itself.

I like that option as a concept, but it adds a new flag to a general
struct like the face for something that only affects the region face
now. But it is very general, so maybe could be useful in the future; but
it worth to make the face even more complex?.

BTW:

Looking at the merge_face function could anyone please explain better
what means: realized face and lface_id and please direct me to where is
documented how emacs uses the faces internally; the functionalities
available and specially the merge rules?

>Also, when looking at what
>applications like Thunderbird or Firefox do, I nowhere see their
>"region faces" extend to the right edges of their "windows" in "normal
>text".  Admittedly, these two applications do not highlight an empty
>space at the left edge of a window either and I'm fully aware of the
>fact that I'm comparing apples and oranges here.  Still I think that
>our users should be given the possibility to customize their regions
>in a similar way as they do.
>
>martin
>

Re: Question about display engine

[martin rudalics]

 >> I now think it would make more sense to add an 'extend-to-window-edge'
 >> attribute in the face definition itself.
 >
 > I like that option as a concept, but it adds a new flag to a general
 > struct like the face for something that only affects the region face
 > now. But it is very general, so maybe could be useful in the future; but
 > it worth to make the face even more complex?.

Here I'm not interested in the region at all.  What bothers me are
doc-strings and comments which I distinguish by setting their
background color.  And having that color extend to the end of the line
is annoying so I had to invent some resource consuming workarounds.

martin

Re: Question about display engine

[Ergus]

On August 8, 2019 10:45:53 AM GMT+02:00, martin rudalics <rudalics@gmx.at> wrote:
>>> I now think it would make more sense to add an
>'extend-to-window-edge'
> >> attribute in the face definition itself.
> >
> > I like that option as a concept, but it adds a new flag to a general
> > struct like the face for something that only affects the region face
>> now. But it is very general, so maybe could be useful in the future;
>but
> > it worth to make the face even more complex?.
>
>Here I'm not interested in the region at all.  What bothers me are
>doc-strings and comments which I distinguish by setting their
>background color.  And having that color extend to the end of the line
>is annoying so I had to invent some resource consuming workarounds.
>
>martin

Yes, that's exactly the point. The only face I see that needs to be extended so far is the region. If only the region is extended (assuming we won't stop extending that one too) you won't need your workarounds, extra settings, another flag in the face structure, or call extend face to end of line most of the time. 



-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Question about display engine

[martin rudalics]

 > Yes, that's exactly the point. The only face I see that needs to be
 > extended so far is the region. If only the region is extended
 > (assuming we won't stop extending that one too) you won't need your
 > workarounds, extra settings, another flag in the face structure, or
 > call extend face to end of line most of the time.

I'm afraid things are not that simple.  We have at least the secondary
selection and 'hl-line-mode' to take care of.  Moreover, there might
be users who do prefer the current way of extending (and not
extending) faces to window edges.  And I have no idea whether image or
rectangular regions require special treatment too.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>
> From: martin rudalics <rudalics@gmx.at>
> Date: Thu, 8 Aug 2019 15:05:37 +0200
> 
>  > Yes, that's exactly the point. The only face I see that needs to be
>  > extended so far is the region. If only the region is extended
>  > (assuming we won't stop extending that one too) you won't need your
>  > workarounds, extra settings, another flag in the face structure, or
>  > call extend face to end of line most of the time.
> 
> I'm afraid things are not that simple.  We have at least the secondary
> selection and 'hl-line-mode' to take care of.

Indeed, nothing is ever as simple in the display code, due to the
sheer amount of different use cases.  I think at least one other face
attribute that's special in this regard is :box, in particular (but
not only) because extend_face_to_end_of_line is called from the
function which redisplays the mode line and the header line.

> Moreover, there might be users who do prefer the current way of
> extending (and not extending) faces to window edges.  And I have no
> idea whether image or rectangular regions require special treatment
> too.

Yes, I think we will have to provide some backward compatibility shims
for these and other use cases.

Re: Question about display engine

[Ergus]

On Thu, Aug 08, 2019 at 04:59:54PM +0300, Eli Zaretskii wrote:
>> Cc: Eli Zaretskii <eliz@gnu.org>
>> From: martin rudalics <rudalics@gmx.at>
>> Date: Thu, 8 Aug 2019 15:05:37 +0200
>>
>>  > Yes, that's exactly the point. The only face I see that needs to be
>>  > extended so far is the region. If only the region is extended
>>  > (assuming we won't stop extending that one too) you won't need your
>>  > workarounds, extra settings, another flag in the face structure, or
>>  > call extend face to end of line most of the time.
>>
>> I'm afraid things are not that simple.  We have at least the secondary
>> selection and 'hl-line-mode' to take care of.
>
>Indeed, nothing is ever as simple in the display code, due to the
>sheer amount of different use cases.  I think at least one other face
>attribute that's special in this regard is :box, in particular (but
>not only) because extend_face_to_end_of_line is called from the
>function which redisplays the mode line and the header line.
>
Yes I have seen :p. That's why I will vote for simplicity+efficiency
over more complex customization that 99% of the users won't
need/know/use.

I only want a uniform behavior between gui and tui. Because bigger
changes I have understand that are close to impossible.

>> Moreover, there might be users who do prefer the current way of
>> extending (and not extending) faces to window edges.  And I have no
>> idea whether image or rectangular regions require special treatment
>> too.
>
>Yes, I think we will have to provide some backward compatibility shims
>for these and other use cases.
>
So finally what's the agreement about this? Does Eli (or anyone) have
the time to implement it?? (else I can try with some hints of
course). Can we add maybe a deadline to decide? I'm just wondering that
this can be forgotten like the discussion in Bug#23574 if nobody starts
working on it.

When and who decides/approves the changes to do?

I vote for:

Reproduce in TUI the gui behavior as is now by default but:

1) With a not-extend-by-default policy

2.0) With some condition checks to extend the mentioned exceptions
(secondary, region, hl_line_mode etc) maybe this last can be set in a
customizable variable as there will be relatively few elements.

xor

2.1) Add the "extensible" flag to the face.

3) Add an extra face to extend (like in my previous code) that needs to
be merged with the last face in the line conditionally and can be used
in case the user wants extend but removing the underline and keeping
the background color (for example).

This seems to be the most general solution in my opinion.

I am specially concerned about this because org-mode behavior in
terminal affects many users. 

Re: Question about display engine

[Eli Zaretskii]

> Date: Thu, 8 Aug 2019 18:43:19 +0200
> From: Ergus <spacibba@aol.com>
> Cc: martin rudalics <rudalics@gmx.at>, emacs-devel@gnu.org
> 
> So finally what's the agreement about this? Does Eli (or anyone) have
> the time to implement it?? (else I can try with some hints of
> course). Can we add maybe a deadline to decide? I'm just wondering that
> this can be forgotten like the discussion in Bug#23574 if nobody starts
> working on it.

We were talking about it for only a single day.  We should let others
chime in, so let's wait for a few days.

> I am specially concerned about this because org-mode behavior in
> terminal affects many users. 

How is Org mode relevant to this issue?  I think this is the first
time Org mode is mentioned in this context.

Re: Question about display engine

[Ergus]

On Thu, Aug 08, 2019 at 08:50:38PM +0300, Eli Zaretskii wrote:
>> Date: Thu, 8 Aug 2019 18:43:19 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: martin rudalics <rudalics@gmx.at>, emacs-devel@gnu.org
>>
>> So finally what's the agreement about this? Does Eli (or anyone) have
>> the time to implement it?? (else I can try with some hints of
>> course). Can we add maybe a deadline to decide? I'm just wondering that
>> this can be forgotten like the discussion in Bug#23574 if nobody starts
>> working on it.
>
>We were talking about it for only a single day.  We should let others
>chime in, so let's wait for a few days.
>
>> I am specially concerned about this because org-mode behavior in
>> terminal affects many users.
>
>How is Org mode relevant to this issue?  I think this is the first
>time Org mode is mentioned in this context.
>
In org-mode, many faces by default are underlined to indicate a section
limit or just to highlight a section's header (I just started to use
org-mode). So inserting the dfci functionality will expose the issue
with this underlines too much.

Re: Question about display engine

[Eli Zaretskii]

> Date: Fri, 9 Aug 2019 00:37:51 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >How is Org mode relevant to this issue?  I think this is the first
> >time Org mode is mentioned in this context.
> >
> In org-mode, many faces by default are underlined to indicate a section
> limit or just to highlight a section's header (I just started to use
> org-mode). So inserting the dfci functionality will expose the issue
> with this underlines too much.

I expect Org mode to underline only the headers, excluding the
following newline, in which case the issue we are discussing shouldn't
matter.  Isn't this the case?

Re: Question about display engine

[Ergus]

On Fri, Aug 09, 2019 at 09:28:51AM +0300, Eli Zaretskii wrote:
>> Date: Fri, 9 Aug 2019 00:37:51 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >How is Org mode relevant to this issue?  I think this is the first
>> >time Org mode is mentioned in this context.
>> >
>> In org-mode, many faces by default are underlined to indicate a section
>> limit or just to highlight a section's header (I just started to use
>> org-mode). So inserting the dfci functionality will expose the issue
>> with this underlines too much.
>
>I expect Org mode to underline only the headers, excluding the
>following newline, in which case the issue we are discussing shouldn't
>matter.  Isn't this the case?

No.

In gui it is if no dfci mode is enabled.

But in tui the underline extends until the end of the line always.

Re: Question about display engine

[Eli Zaretskii]

> Date: Fri, 9 Aug 2019 11:08:25 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >> In org-mode, many faces by default are underlined to indicate a section
> >> limit or just to highlight a section's header (I just started to use
> >> org-mode). So inserting the dfci functionality will expose the issue
> >> with this underlines too much.
> >
> >I expect Org mode to underline only the headers, excluding the
> >following newline, in which case the issue we are discussing shouldn't
> >matter.  Isn't this the case?
> 
> No.
> 
> In gui it is if no dfci mode is enabled.
> 
> But in tui the underline extends until the end of the line always.

Sorry, I don't understand.  It sounds like you are answering not the
question I asked, but some other question.

Re: Question about display engine

[Ergus]

On Fri, Aug 09, 2019 at 12:40:19PM +0300, Eli Zaretskii wrote:
>> Date: Fri, 9 Aug 2019 11:08:25 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >> In org-mode, many faces by default are underlined to indicate a section
>> >> limit or just to highlight a section's header (I just started to use
>> >> org-mode). So inserting the dfci functionality will expose the issue
>> >> with this underlines too much.
>> >
>> >I expect Org mode to underline only the headers, excluding the
>> >following newline, in which case the issue we are discussing shouldn't
>> >matter.  Isn't this the case?
>>
>> No.
>>
>> In gui it is if no dfci mode is enabled.
>>
>> But in tui the underline extends until the end of the line always.
>
>Sorry, I don't understand.  It sounds like you are answering not the
>question I asked, but some other question.
>
Hi:

The dfci mode exposes the issue 36858 in org-mode because:

Org mode underlines only the headers in gui if no dfci mode is
enabled, else the underline is extended until the indicator+1 character.

But in tui the underline extends until the end of the line always, so
the indicator looks pretty bad. That's what started this thread
actually. The incoherence between both interfaces.

Re: Question about display engine

[Eli Zaretskii]

> Date: Fri, 9 Aug 2019 13:31:34 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> The dfci mode exposes the issue 36858 in org-mode because:
> 
> Org mode underlines only the headers in gui if no dfci mode is
> enabled, else the underline is extended until the indicator+1 character.
> 
> But in tui the underline extends until the end of the line always, so
> the indicator looks pretty bad. That's what started this thread
> actually. The incoherence between both interfaces.

But this is unrelated to the issue of extending a face that straddles
a newline, right?  The issue being discussed in this thread is about
faces that cover the newline; those cases that don't cover the newline
are simply bugs in display-fill-column-indicator-mode, and should be
fixed regardless of what we are talking here.

It is possible that some solutions for the issue we are discussing
here would also fix bug#36858 as side effect, but we should not delay
solving that bug until the issue in this thread is resolved, because
this issue is much more general and has much wider implications.  So
it might be that we decide not to change the more general behavior, or
change it in a way that doesn't solve bug#36858.

Or am I missing something?

Re: Question about display engine

[Ergus]

On Fri, Aug 09, 2019 at 05:04:14PM +0300, Eli Zaretskii wrote:
>> Date: Fri, 9 Aug 2019 13:31:34 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> The dfci mode exposes the issue 36858 in org-mode because:
>>
>> Org mode underlines only the headers in gui if no dfci mode is
>> enabled, else the underline is extended until the indicator+1 character.
>>
>> But in tui the underline extends until the end of the line always, so
>> the indicator looks pretty bad. That's what started this thread
>> actually. The incoherence between both interfaces.
>
>But this is unrelated to the issue of extending a face that straddles
>a newline, right?  The issue being discussed in this thread is about
>faces that cover the newline; those cases that don't cover the newline
>are simply bugs in display-fill-column-indicator-mode, and should be
>fixed regardless of what we are talking here.
>
>It is possible that some solutions for the issue we are discussing
>here would also fix bug#36858 as side effect, but we should not delay
>solving that bug until the issue in this thread is resolved, because
>this issue is much more general and has much wider implications.  So
>it might be that we decide not to change the more general behavior, or
>change it in a way that doesn't solve bug#36858.
>
>Or am I missing something?
>
Hi Eli:

The problem is the incoherence, so the solution will be very different
depending of If we keep the tui behavior then there will be needed a
design choice about what to do with the indicator, if we keep the GUI I
have already a fix patch for that.

Re: Question about display engine

[martin rudalics]

 > I only want a uniform behavior between gui and tui. Because bigger
 > changes I have understand that are close to impossible.

Such uniform behavior should be a major goal.

 > Reproduce in TUI the gui behavior as is now by default but:
 >
 > 1) With a not-extend-by-default policy

I'm not sure what you mean here: not-extend-by-default in TUI only or
not-extend-by-default everywhere?

 > 3) Add an extra face to extend (like in my previous code) that needs to
 > be merged with the last face in the line conditionally and can be used
 > in case the user wants extend but removing the underline and keeping
 > the background color (for example).

I suppose this extra face would be suppressed for the region or
for tooltips.

martin

Re: Question about display engine

[Ergus]

On Fri, Aug 09, 2019 at 10:59:47AM +0200, martin rudalics wrote:
>> I only want a uniform behavior between gui and tui. Because bigger
>> changes I have understand that are close to impossible.
>
>Such uniform behavior should be a major goal.
>
>> Reproduce in TUI the gui behavior as is now by default but:
>>
>> 1) With a not-extend-by-default policy
>
>I'm not sure what you mean here: not-extend-by-default in TUI only or
>not-extend-by-default everywhere?
>
not-extend-by-default everywhere except in some conditions. Opposite to
the actual policy which is to extend always.

The behavior must be the same in both cases from the user point of
view. That's my main concern on this; I will actually be fine enough
with any decision that unifies behaviors.

>> 3) Add an extra face to extend (like in my previous code) that needs to
>> be merged with the last face in the line conditionally and can be used
>> in case the user wants extend but removing the underline and keeping
>> the background color (for example).
>
>I suppose this extra face would be suppressed for the region or
>for tooltips.
>
Yes, the merge step needs to be suppressed in this and many other
scenarios.

>martin

Re: Question about display engine

[Ergus]

On Fri, Aug 09, 2019 at 10:59:47AM +0200, martin rudalics wrote:
>> I only want a uniform behavior between gui and tui. Because bigger
>> changes I have understand that are close to impossible.
>
>Such uniform behavior should be a major goal.
>
>> Reproduce in TUI the gui behavior as is now by default but:
>>
>> 1) With a not-extend-by-default policy
>
>I'm not sure what you mean here: not-extend-by-default in TUI only or
>not-extend-by-default everywhere?
>
>> 3) Add an extra face to extend (like in my previous code) that needs to
>> be merged with the last face in the line conditionally and can be used
>> in case the user wants extend but removing the underline and keeping
>> the background color (for example).
>
>I suppose this extra face would be suppressed for the region or
>for tooltips.
>
tooltips yes, not needed for the region. Actually the region is it's
main use case to enable a simple customization when the user wants to
extend only the color but not underlined. (To have an idea about this
extra face give a look how I use it in my initial row patch.)

>martin
>

Re: Question about display engine

[Eli Zaretskii]

> Date: Thu, 8 Aug 2019 18:43:19 +0200
> From: Ergus <spacibba@aol.com>
> Cc: martin rudalics <rudalics@gmx.at>, emacs-devel@gnu.org
> 
> >Indeed, nothing is ever as simple in the display code, due to the
> >sheer amount of different use cases.  I think at least one other face
> >attribute that's special in this regard is :box, in particular (but
> >not only) because extend_face_to_end_of_line is called from the
> >function which redisplays the mode line and the header line.
> >
> Yes I have seen :p. That's why I will vote for simplicity+efficiency
> over more complex customization that 99% of the users won't
> need/know/use.

We cannot take the simplicity+efficiency escape if it will cause
inconsistent behavior in valid use cases.

> I only want a uniform behavior between gui and tui. Because bigger
> changes I have understand that are close to impossible.

Making the behavior of TTY and GUI frames consistent should be easy
once we decide which one we want to keep, provided we also decide not
to extend the background color on GUI (and TTY) frames.  But if we
decide to keep the TTY behavior, it will make GUI display slightly
less efficient, especially in wide windows.

> I vote for:
> 
> Reproduce in TUI the gui behavior as is now by default but:
> 
> 1) With a not-extend-by-default policy

Including non-extension of background color?  Because if you still
want to extend the color, I don't see how to make TTY and GUI
consistent, at least not easily.

> 2.0) With some condition checks to extend the mentioned exceptions
> (secondary, region, hl_line_mode etc) maybe this last can be set in a
> customizable variable as there will be relatively few elements.
> 
> xor
> 
> 2.1) Add the "extensible" flag to the face.

I think both of these possibilities aren't workable due to face
merging.

> 3) Add an extra face to extend (like in my previous code) that needs to
> be merged with the last face in the line conditionally and can be used
> in case the user wants extend but removing the underline and keeping
> the background color (for example).

This can only work if we assume users will want the same attribute be
extended or not irrespective of the face.  Such an assumption is not
necessarily valid: users could want the hl-line face, for example, be
always extended, regardless of its attributes, but would like at the
same time not see other faces' :underline attribute extended.

IOW, such a solution (and similar ones proposed here) look like
kludgey workarounds, something that IMO is not really appropriate for
a major feature such as faces.

> I am specially concerned about this because org-mode behavior in
> terminal affects many users. 

We could claim this to be a bug in Org.  No?  Why does Org insist on
covering the newline with the face?

Re: Question about display engine

[martin rudalics]

 > This can only work if we assume users will want the same attribute be
 > extended or not irrespective of the face.  Such an assumption is not
 > necessarily valid: users could want the hl-line face, for example, be
 > always extended, regardless of its attributes, but would like at the
 > same time not see other faces' :underline attribute extended.

Reiterating what I said elsewhere (and hoping not to bore anyone with
my ideas): With the face-based approach the user would have to set
background, underline and extend for the highlight face to get that
behavior.  With the attributes-based approach, a user would have to
include highlight both in 'extend-background' and 'extend-underline'.

martin

Re: Question about display engine

[martin rudalics]

 > Indeed, nothing is ever as simple in the display code, due to the
 > sheer amount of different use cases.  I think at least one other face
 > attribute that's special in this regard is :box, in particular (but
 > not only) because extend_face_to_end_of_line is called from the
 > function which redisplays the mode line and the header line.

Newlines within boxes are a pain.  Ideally, the entire text should be
enclosed within one single box.  Which obviously fails when the text
does not form a rectangular region on screen.  This is also evident
with buttons - you cannot draw a button with multiple lines of text.

As for the mode-/header-line or tooltip faces we'd probably have these
always extend to the end of line, overriding any extension property
specified for the underlying face.  There's also the case of remapped
backgrounds that IIRC cannot extend to the bottom edge of a window -
they stop with the last text line displayed (I have to look into this
matter again to say more).

martin

Re: Question about display engine

[Ergus]

On Thu, Aug 08, 2019 at 03:05:37PM +0200, martin rudalics wrote:
>> Yes, that's exactly the point. The only face I see that needs to be
>> extended so far is the region. If only the region is extended
>> (assuming we won't stop extending that one too) you won't need your
>> workarounds, extra settings, another flag in the face structure, or
>> call extend face to end of line most of the time.
>
>I'm afraid things are not that simple.  We have at least the secondary
>selection and 'hl-line-mode' to take care of.  Moreover, there might
>be users who do prefer the current way of extending (and not
>extending) faces to window edges.  And I have no idea whether image or
>rectangular regions require special treatment too.
>
>martin
>
You are right, I ignored those use cases, but I still don't think that
the faces are the right place to flag that. The line extension maybe
needs to be decided based on another text property. Maybe there are
already some conditions we can check dynamically. Because adding a flag
is a bit error prone when there are already some conditions.

There is also the case when the face to use comes from FACE_FOR_CHAR or
another is merged over that. Or when there is a highlight inside the
region at the end of the line text... That, in the display engine, I am
not clear yet how are handled.

About the user preferences I think it needs to be accepted the Eli's
choice (I will favor the uniformity+simplicity over over-specification,
but I am not very "emacsy" in that point). Because we won't make happy
everyone in any case.

Re: Question about display engine

[martin rudalics]

 > You are right, I ignored those use cases, but I still don't think that
 > the faces are the right place to flag that. The line extension maybe
 > needs to be decided based on another text property. Maybe there are
 > already some conditions we can check dynamically. Because adding a flag
 > is a bit error prone when there are already some conditions.
 >
 > There is also the case when the face to use comes from FACE_FOR_CHAR or
 > another is merged over that. Or when there is a highlight inside the
 > region at the end of the line text... That, in the display engine, I am
 > not clear yet how are handled.

OTOH, the display engine could simply delegate some design decisions
to the face specification apparatus and attribute any faults to the
latter.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Fri, 9 Aug 2019 10:59:33 +0200
> 
>  > You are right, I ignored those use cases, but I still don't think that
>  > the faces are the right place to flag that. The line extension maybe
>  > needs to be decided based on another text property. Maybe there are
>  > already some conditions we can check dynamically. Because adding a flag
>  > is a bit error prone when there are already some conditions.
>  >
>  > There is also the case when the face to use comes from FACE_FOR_CHAR or
>  > another is merged over that. Or when there is a highlight inside the
>  > region at the end of the line text... That, in the display engine, I am
>  > not clear yet how are handled.
> 
> OTOH, the display engine could simply delegate some design decisions
> to the face specification apparatus and attribute any faults to the
> latter.

I don't think this is workable, because of face merging.  The actual
face used to display the last character on a screen line can (and
frequently does) come from merging several faces, and there's no
meaningful answer to the question: which face did this attribute come
from?  For a face merged from 2 or more faces defined via defface, how
do you tell whether or not to extend it?  Thus, such delegation can
only yield inconsistent behavior and more bug reports.

Re: Question about display engine

[martin rudalics]

 >> OTOH, the display engine could simply delegate some design decisions
 >> to the face specification apparatus and attribute any faults to the
 >> latter.
 >
 > I don't think this is workable, because of face merging.  The actual
 > face used to display the last character on a screen line can (and
 > frequently does) come from merging several faces, and there's no
 > meaningful answer to the question: which face did this attribute come
 > from?

Answering this question should not be the task of a (face agnostic)
display engine.  The face merging algorithm would have to decide
whether the :extend attribute of the winning face should cause an
extension of any attribute specified by that face.  Which means that
if the region is specified as both underlining and background, _both_
would extend if the extend attribute of the region face is set.  It's
the user's responsibility to either not set underline or not request
extension of the region if such behavior is unwanted.

 > For a face merged from 2 or more faces defined via defface, how
 > do you tell whether or not to extend it?  Thus, such delegation can
 > only yield inconsistent behavior and more bug reports.

But there's only one face that decides whether and how the background
shall be set and there's only face that decides that for underlining.
Those faces' :extend attribute would specify how to extend the
background and how to extend the underline.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Sun, 11 Aug 2019 10:14:33 +0200
> 
>  > I don't think this is workable, because of face merging.  The actual
>  > face used to display the last character on a screen line can (and
>  > frequently does) come from merging several faces, and there's no
>  > meaningful answer to the question: which face did this attribute come
>  > from?
> 
> Answering this question should not be the task of a (face agnostic)
> display engine.  The face merging algorithm would have to decide
> whether the :extend attribute of the winning face should cause an
> extension of any attribute specified by that face.

What is the "winning face" in this context?

>  > For a face merged from 2 or more faces defined via defface, how
>  > do you tell whether or not to extend it?  Thus, such delegation can
>  > only yield inconsistent behavior and more bug reports.
> 
> But there's only one face that decides whether and how the background
> shall be set and there's only face that decides that for underlining.

Not necessarily true: two or more faces could specify the same value
of a particular attribute, including background and underlining.  The
:extend attribute could be different in some of these faces.

Re: Question about display engine

[martin rudalics]

 >> Answering this question should not be the task of a (face agnostic)
 >> display engine.  The face merging algorithm would have to decide
 >> whether the :extend attribute of the winning face should cause an
 >> extension of any attribute specified by that face.
 >
 > What is the "winning face" in this context?

IIUC the last one found by face_at_buffer_position that explicitly
specified a value for the attribute in question.  That is, the face
whose attribute is actually used by the display engine.

 > Not necessarily true: two or more faces could specify the same value
 > of a particular attribute, including background and underlining.  The
 > :extend attribute could be different in some of these faces.

Suppose the iterator is at a newline character within some highlighted
text within the region and wants to know the :background attribute at
that position.

With a face-based solution, face_at_buffer_position will have looked
for a :background value provided by the default face, the highlight
face and the region face, in this order.  The value of the :extend
attribute provided by the last face found that way is passed (in some
extend_background variable I presume) to the display engine so the
latter can determine whether the background shall be extended to the
window edge or not.

With an attribute-based solution, face_at_buffer_position will have
provided an appropriate value when the last face "found that way" is a
member of the 'extend-background' Lisp variable.

A similar approach would be used to decide whether the rest of the
line should be underlined, overlined, boxed, appear in inverse-video
or whatever someone considers important (I suppose that none of these
should ever extend).

As stated earlier, the face-based solution will intuitively not DTRT
when we allow underline to extend and a user sets, for example, all
:underline, :background and :extend for the region face.  In that
case, both background and underline of the region will extend.  With
an attribute-based solution, a user would just have added 'region' to
the 'extend-background' variable and this "problem" would be avoided.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Mon, 12 Aug 2019 10:59:52 +0200
> 
>  >> Answering this question should not be the task of a (face agnostic)
>  >> display engine.  The face merging algorithm would have to decide
>  >> whether the :extend attribute of the winning face should cause an
>  >> extension of any attribute specified by that face.
>  >
>  > What is the "winning face" in this context?
> 
> IIUC the last one found by face_at_buffer_position that explicitly
> specified a value for the attribute in question.  That is, the face
> whose attribute is actually used by the display engine.

Nitpicking: the display engine has no idea whose attributes it is
using, see below.

>  > Not necessarily true: two or more faces could specify the same value
>  > of a particular attribute, including background and underlining.  The
>  > :extend attribute could be different in some of these faces.
> 
> Suppose the iterator is at a newline character within some highlighted
> text within the region and wants to know the :background attribute at
> that position.
> 
> With a face-based solution, face_at_buffer_position will have looked
> for a :background value provided by the default face, the highlight
> face and the region face, in this order.  The value of the :extend
> attribute provided by the last face found that way is passed (in some
> extend_background variable I presume) to the display engine so the
> latter can determine whether the background shall be extended to the
> window edge or not.

If two or more faces of those merged specify :underline, and only one
of them has a non-nil :extend, will extending the underline do what
the users expect?

(And shouldn't we simply disregard entirely a face whose :extend
attribute is nil?  What can such a face possibly contribute in this
case?)

> With an attribute-based solution, face_at_buffer_position will have
> provided an appropriate value when the last face "found that way" is a
> member of the 'extend-background' Lisp variable.
> 
> A similar approach would be used to decide whether the rest of the
> line should be underlined, overlined, boxed, appear in inverse-video
> or whatever someone considers important (I suppose that none of these
> should ever extend).

You seem to assume that the iterator examines faces at every buffer
position it passes, and in particular on the newline at EOL.  But
that's not what happens, because examining and merging faces is
expensive, so Emacs avoids doing that unless necessary.  We only
examine faces where they change, and use next-single-property-change
to find the next position where we should again examine the faces.

In addition, face_at_buffer_position doesn't look for face attributes
one by one.  Instead, it finds all the source of the 'face' property
that are in effect at the given position -- the default face, the face
from text properties, and from all the overlays at that position --
and merges these attributes into a single attribute vector.  Then it
looks up a realized face with identical attributes, or realizes a new
face if no such face exists.  Thereafter, the iterator just uses that
face until the next checkpoint.  IOW, face_at_buffer_position returns
an ID of a realized (i.e. fully specified) face created by merging
several relevant sources of face information, and that realized face
has no references to the names of the individual faces from which it
was created, nor any memory of which non-unspecified attributes came
from which face source.

So to implement something like above, we will have to:

 . force face merging when we get to a newline (btw, what about
   continuation lines in this context? do we apply the extension logic
   to them as well?)
 . modify face_at_buffer_position and its subroutines to behave
   specially when called on a newline, and decide whether to merge or
   not merge attributes based on whatever data structures describe the
   preferences for extending those attributes; this would go at least
   two levels below face_at_buffer_position
 . do something similar in face_at_string_position, for display and
   overlay strings with embedded newlines

Sounds like fun project.  Volunteers are welcome.

Re: Question about display engine

[Stefan Monnier]

>> IIUC the last one found by face_at_buffer_position that explicitly
>> specified a value for the attribute in question.  That is, the face
>> whose attribute is actually used by the display engine.
>
> Nitpicking: the display engine has no idea whose attributes it is
> using, see below.

I don't know enough of the display engine to say something intelligent,
I'm afraid, but my naive understanding is that the way "it should" work
is that the face used on the "rest of the line" should be computed by
merging the various faces that apply to the corresponding LF character
but where the new `extend-to-end-of-line` property is obeyed
(i.e. a face is skipped if that property is nil).  IOW the
extend-to-end-of-line property is applied *during* merging rather than
after it.

Most likely this can't be mapped to the way things are done, tho.


        Stefan

Re: Question about display engine

[martin rudalics]

 > ... the face used on the "rest of the line" should be computed by
 > merging the various faces that apply to the corresponding LF character
 > but where the new `extend-to-end-of-line` property is obeyed
 > (i.e. a face is skipped if that property is nil).  IOW the
 > extend-to-end-of-line property is applied *during* merging rather than
 > after it.

That's the idea.  The display engine would then simply have to look at
a single bit to tell whether to extend a specific attribute to the end
of the line or not.

 > Most likely this can't be mapped to the way things are done, tho.

Most likely the solution will be disappointingly simple, once Eli
found it.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Tue, 13 Aug 2019 10:17:52 +0200
> 
> > ... the face used on the "rest of the line" should be computed by
> > merging the various faces that apply to the corresponding LF character
> > but where the new `extend-to-end-of-line` property is obeyed
> > (i.e. a face is skipped if that property is nil).  IOW the
> > extend-to-end-of-line property is applied *during* merging rather than
> > after it.
> 
> That's the idea.

Are you sure?  It sounds like Stefan was saying the same thing I was
saying: that the decision whether to use or not to use an attribute is
made when we merge the faces, not when we apply the face attributes to
produce the display on the glass.

Or maybe it's me who misunderstood what Stefan meant...

Re: Question about display engine

[Stefan Monnier]

> Are you sure?  It sounds like Stefan was saying the same thing I was
> saying: that the decision whether to use or not to use an attribute is
> made when we merge the faces, not when we apply the face attributes to
> produce the display on the glass.

Sounds about right, yes.


        Stefan

Re: Question about display engine

[martin rudalics]

 > Are you sure?  It sounds like Stefan was saying the same thing I was
 > saying: that the decision whether to use or not to use an attribute is
 > made when we merge the faces, not when we apply the face attributes to
 > produce the display on the glass.

If I ever gave you the impression that I meant something to the
contrary, that impression was wrong and I apologize for it.

 > Or maybe it's me who misunderstood what Stefan meant...

I think you misunderstood what I meant...

martin

Re: Question about display engine

[martin rudalics]

 >> IIUC the last one found by face_at_buffer_position that explicitly
 >> specified a value for the attribute in question.  That is, the face
 >> whose attribute is actually used by the display engine.
 >
 > Nitpicking: the display engine has no idea whose attributes it is
 > using, see below.

In an earlier mail I labelled the display engine as "face agnostic" so
I am very well aware of that.  What I tried to point out (and failed)
so far is precisely how to work around that issue.

 > If two or more faces of those merged specify :underline, and only one
 > of them has a non-nil :extend, will extending the underline do what
 > the users expect?

It's the last face whose (1) :underline attribute is merged in _and_
whose (2) :extend attribute is set (nil or non-nil) that determines
whether the underline extends or not.  Whether that is what the users
expect is beyond my knowledge.  We can only try to provide a fairly
consistent way of specifying it.

 > (And shouldn't we simply disregard entirely a face whose :extend
 > attribute is nil?  What can such a face possibly contribute in this
 > case?)

Consider a user who sets the :extend attribute to non-nil for the
'default' face and wants a 'link' background to not extend to the end
of line.  Such a user would want to set the :extend attribute of the
'link' face to nil to get the desired effect.

 > You seem to assume that the iterator examines faces at every buffer
 > position it passes, and in particular on the newline at EOL.

I certainly do not assume such a thing.  But when trying to find out
how face attributes are merged and realized, face_at_buffer_position
was the first point of reference where I found that (and I still don't
know a better one).

 > But
 > that's not what happens, because examining and merging faces is
 > expensive, so Emacs avoids doing that unless necessary.  We only
 > examine faces where they change, and use next-single-property-change
 > to find the next position where we should again examine the faces.

That's exactly what I expected but did not find immediately.  If you
tell me where to look for that (Fnext_single_property_change _is_
called by face_at_buffer_position) "examine faces where they change"
and in particular where a :background change is found and applied, I
probably could tell you more.

 > In addition, face_at_buffer_position doesn't look for face attributes
 > one by one.  Instead, it finds all the source of the 'face' property
 > that are in effect at the given position -- the default face, the face
 > from text properties, and from all the overlays at that position --
 > and merges these attributes into a single attribute vector.  Then it
 > looks up a realized face with identical attributes, or realizes a new
 > face if no such face exists.  Thereafter, the iterator just uses that
 > face until the next checkpoint.  IOW, face_at_buffer_position returns
 > an ID of a realized (i.e. fully specified) face created by merging
 > several relevant sources of face information, and that realized face
 > has no references to the names of the individual faces from which it
 > was created, nor any memory of which non-unspecified attributes came
 > from which face source.

When face_at_buffer_position puts some 'background' value into that
single attribute vector, it can simply set an 'extend_background' bit
in that vector which tells the display engine whether the background
specified by that vector shall be extended or not.  Note that that
'extend_background' bit would be left alone when the face specifying
:background does not also specify :extend.  In that case the value of
:extend of the last face merged in that specified both would persist.

 > So to implement something like above, we will have to:
 >
 >   . force face merging when we get to a newline (btw, what about
 >     continuation lines in this context? do we apply the extension logic
 >     to them as well?)

I never show continuation lines (with exception of the minibuffer
where I have no choice) but I suppose we should apply the extension
there as well.  Think of the region spanning several continued lines.

 >   . modify face_at_buffer_position and its subroutines to behave
 >     specially when called on a newline,

I strongly doubt that this "called on a newline" is needed.  Setting
the 'extend_background' indicator is strongly tied to the last setting
of the 'background' indicator in the attribute vector as done by the
face merging algorithm.

 > and decide whether to merge or
 >     not merge attributes based on whatever data structures describe the
 >     preferences for extending those attributes; this would go at least
 >     two levels below face_at_buffer_position
 >   . do something similar in face_at_string_position, for display and
 >     overlay strings with embedded newlines
 >
 > Sounds like fun project.  Volunteers are welcome.

If all these were really necessary, I'd never vote for such a change.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Tue, 13 Aug 2019 10:17:32 +0200
> 
> > (And shouldn't we simply disregard entirely a face whose :extend
> > attribute is nil?  What can such a face possibly contribute in this
> > case?)
> 
> Consider a user who sets the :extend attribute to non-nil for the
> 'default' face and wants a 'link' background to not extend to the end
> of line.  Such a user would want to set the :extend attribute of the
> 'link' face to nil to get the desired effect.

Yes, and my point is that in that case we can simply ignore the 'link'
face when extending the face of the last character of a line.

> > You seem to assume that the iterator examines faces at every buffer
> > position it passes, and in particular on the newline at EOL.
>
> I certainly do not assume such a thing.  But when trying to find out
> how face attributes are merged and realized, face_at_buffer_position
> was the first point of reference where I found that (and I still don't
> know a better one).

face_at_buffer_position is the right place, but it isn't called for
every buffer position.  And your description seemed to hint that you
thought it was called for every buffer position.

> > But
> > that's not what happens, because examining and merging faces is
> > expensive, so Emacs avoids doing that unless necessary.  We only
> > examine faces where they change, and use next-single-property-change
> > to find the next position where we should again examine the faces.
> 
> That's exactly what I expected but did not find immediately.  If you
> tell me where to look for that (Fnext_single_property_change _is_
> called by face_at_buffer_position) "examine faces where they change"

This has to do with how the display iterator works in general.  It
starts by determining where the next "stop position" is.  A "stop
position" (stored in it->stop_charpos) is the position where iteration
must stop and examine all the possible sources of affecting what and
how should be displayed next.  This includes faces, on-the-fly
fontification, invisible text, display properties, overlays, switching
from iteration over a buffer to iteration of a string and back, etc.
At that stop position, the iterator examines all these potential
game-changers, and computes and stores the results in its fields.
That is where face_at_buffer_position is called, and the result is
it->face_id, a numerical index into the frame's face cache which
defines the realized face to be used from now on for displaying
characters.  The iterator will not call face_at_buffer_position and
won't consider faces until it gets to the next stop position.

The next stop position is computed in compute_stop_pos.  You will see
that it doesn't call Fnext_single_property_change, but instead
accesses the interval tree directly, because that's more efficient:
you find changes in _any_ text properties that way.  It also considers
overlay changes and changes in character composition.  The next stop
position is the closest one of those provided by any of these sources.

When we get to the stop position, we invoke a series of known
handlers, each one responsible to handle the relevant Lisp data
structures that might affect the display.  See handle_stop.  In
particular, the handler of face properties is handle_face_prop, which
calls face_at_buffer_position or face_at_string_position.  Each
handler updates the relevant fields of the iterator structure as
necessary.  Having handled everything that needs to be handled at a
particular stop position, we call compute_stop_pos to compute the next
stop position, and then proceed producing glyphs using the updated
iterator information until we get to that next stop position, never
examining any of the above factors again until we get to that place.

> and in particular where a :background change is found and applied, I
> probably could tell you more.

Neither :background nor any other face attribute is "found and
applied" in the main iterator loop.  The iterator simply records in
each glyph the face ID to be used for displaying that glyph.  That
face ID is the value returned by the last call to
face_at_buffer_position.  The code which examines these attributes is
in the terminal-specific backends: xterm.c, w32term.c, nsterm.m, etc.
Only there we extract the colors, the underline, the strike-through,
etc. attributes, and use them to produce the corresponding visual
appearance.  For example, for the background color, that is where we
instruct the GUI system to "clear to EOL" with that color.

(The last paragraph is slightly over-simplified: you will see in a
couple of places that we do look at face->background in xdisp.c,
notably in extend_face_to_end_of_line, but only in order to compare it
with the frame's default background.)

> > In addition, face_at_buffer_position doesn't look for face attributes
> > one by one.  Instead, it finds all the source of the 'face' property
> > that are in effect at the given position -- the default face, the face
> > from text properties, and from all the overlays at that position --
> > and merges these attributes into a single attribute vector.  Then it
> > looks up a realized face with identical attributes, or realizes a new
> > face if no such face exists.  Thereafter, the iterator just uses that
> > face until the next checkpoint.  IOW, face_at_buffer_position returns
> > an ID of a realized (i.e. fully specified) face created by merging
> > several relevant sources of face information, and that realized face
> > has no references to the names of the individual faces from which it
> > was created, nor any memory of which non-unspecified attributes came
> > from which face source.
> 
> When face_at_buffer_position puts some 'background' value into that
> single attribute vector, it can simply set an 'extend_background' bit
> in that vector which tells the display engine whether the background
> specified by that vector shall be extended or not.

So you propose to have an "extend" bit for every face attribute?
There are 18 of them.  (Maybe some of the attributes won't need that
bit, but quite a few will.)

Moreover, given the description above of where the attributes are
acted upon, you are saying that xterm.c, w32term.c etc. will have to
consider whether the glyphs they display are at the end of a line or
not, and act accordingly, i.e. ignore some of the face attributes
under certain conditions, right?

> >   . modify face_at_buffer_position and its subroutines to behave
> >     specially when called on a newline,
> 
> I strongly doubt that this "called on a newline" is needed.  Setting
> the 'extend_background' indicator is strongly tied to the last setting
> of the 'background' indicator in the attribute vector as done by the
> face merging algorithm.

But the merged face will be used for displaying both the "extension"
part of the line and "normal" characters.  When displaying "normal"
characters, the background color should be used unconditionally,
whereas while displaying the "extension" part it should be used only
if its 'extend' bit is set.  Right?  What I wrote above assumed that
we make these decisions at face merge time, and will actually have 2
different realized faces; what you are suggesting is that we use a
single realized face and make those decisions when we use the faces to
write to the glass.  That doesn't strike me as a simpler
implementation.

> > Sounds like fun project.  Volunteers are welcome.
> 
> If all these were really necessary, I'd never vote for such a change.

I'm afraid I still don't see a simple solution.  Hopefully someone
will show me the light.

Re: Question about display engine

[martin rudalics]

 >> Consider a user who sets the :extend attribute to non-nil for the
 >> 'default' face and wants a 'link' background to not extend to the end
 >> of line.  Such a user would want to set the :extend attribute of the
 >> 'link' face to nil to get the desired effect.
 >
 > Yes, and my point is that in that case we can simply ignore the 'link'
 > face when extending the face of the last character of a line.

How would the display engine then know that it can "simply ignore"
the background of that face for the rest of the line?

 > face_at_buffer_position is the right place, but it isn't called for
 > every buffer position.  And your description seemed to hint that you
 > thought it was called for every buffer position.

My bad.  But understanding face implementatin _without_ the text you
wrote below would take a couple of days at least for a humble reader
like me.

 > This has to do with how the display iterator works in general.  It
 > starts by determining where the next "stop position" is.  A "stop
 > position" (stored in it->stop_charpos) is the position where iteration
 > must stop and examine all the possible sources of affecting what and
 > how should be displayed next.  This includes faces, on-the-fly
 > fontification, invisible text, display properties, overlays, switching
 > from iteration over a buffer to iteration of a string and back, etc.
 > At that stop position, the iterator examines all these potential
 > game-changers, and computes and stores the results in its fields.
 > That is where face_at_buffer_position is called, and the result is
 > it->face_id, a numerical index into the frame's face cache which
 > defines the realized face to be used from now on for displaying
 > characters.  The iterator will not call face_at_buffer_position and
 > won't consider faces until it gets to the next stop position.
 >
 > The next stop position is computed in compute_stop_pos.  You will see
 > that it doesn't call Fnext_single_property_change, but instead
 > accesses the interval tree directly, because that's more efficient:
 > you find changes in _any_ text properties that way.  It also considers
 > overlay changes and changes in character composition.  The next stop
 > position is the closest one of those provided by any of these sources.
 >
 > When we get to the stop position, we invoke a series of known
 > handlers, each one responsible to handle the relevant Lisp data
 > structures that might affect the display.  See handle_stop.  In
 > particular, the handler of face properties is handle_face_prop, which
 > calls face_at_buffer_position or face_at_string_position.  Each
 > handler updates the relevant fields of the iterator structure as
 > necessary.  Having handled everything that needs to be handled at a
 > particular stop position, we call compute_stop_pos to compute the next
 > stop position, and then proceed producing glyphs using the updated
 > iterator information until we get to that next stop position, never
 > examining any of the above factors again until we get to that place.

Thanks for the lucid description.  This is hopefully clear to me now.

 >> and in particular where a :background change is found and applied, I
 >> probably could tell you more.
 >
 > Neither :background nor any other face attribute is "found and
 > applied" in the main iterator loop.  The iterator simply records in
 > each glyph the face ID to be used for displaying that glyph.  That
 > face ID is the value returned by the last call to
 > face_at_buffer_position.

IIUC the latter will (unless the face was cached already) end up
calling merge_face_ref and somewhere there I find the lines

	      else if (EQ (keyword, QCbackground))
		{
		  if (STRINGP (value))
		    to[LFACE_BACKGROUND_INDEX] = value;
		  else
		    err = true;
		}

Hence IIUC in the

	  while (CONSP (face_ref) && CONSP (XCDR (face_ref)))

loop we could manage three booleans background, extend and
extend_value and instead of the above write

	      else if (EQ (keyword, QCbackground))
		{
		  if (STRINGP (value))
		    {
		      to[LFACE_BACKGROUND_INDEX] = value;

		      if (extend)
			to[LFACE_EXTEND_BACKGROUND] = extend_value;

		      background = true;
		    }
		  else
		    err = true;
		}
	      else if (EQ (keyword, QCextend_background))
		{
		  extend = true;
		  extend_value = !NILP (value);

		  if (background)
		    to[LFACE_EXTEND_BACKGROUND] = extend_value;
		}

 > The code which examines these attributes is
 > in the terminal-specific backends: xterm.c, w32term.c, nsterm.m, etc.
 > Only there we extract the colors, the underline, the strike-through,
 > etc. attributes, and use them to produce the corresponding visual
 > appearance.  For example, for the background color, that is where we
 > instruct the GUI system to "clear to EOL" with that color.
 >
 > (The last paragraph is slightly over-simplified: you will see in a
 > couple of places that we do look at face->background in xdisp.c,
 > notably in extend_face_to_end_of_line, but only in order to compare it
 > with the frame's default background.)

All these (the platform specific parts and the xdisp code) are already
too late for what I have in mind.

 >> When face_at_buffer_position puts some 'background' value into that
 >> single attribute vector, it can simply set an 'extend_background' bit
 >> in that vector which tells the display engine whether the background
 >> specified by that vector shall be extended or not.
 >
 > So you propose to have an "extend" bit for every face attribute?
 > There are 18 of them.  (Maybe some of the attributes won't need that
 > bit, but quite a few will.)

I'd propose to add these lazily if there is any need.  AFAIAC
:background is the only attribute where an extend bit sounds useful.
Some people here have explictily stated that they do not consider it
useful to extend the :underline attribute and I agree.  So if you
think that other attributes will need an extend bit, please tell me
which ones you have in mind.

 > Moreover, given the description above of where the attributes are
 > acted upon, you are saying that xterm.c, w32term.c etc. will have to
 > consider whether the glyphs they display are at the end of a line or
 > not, and act accordingly, i.e. ignore some of the face attributes
 > under certain conditions, right?

By no means.  The extend_background bit would be set in merge_face_ref
as sketched above (because this is the one that can read the
:background and :extend attributes of any face) and examined in
extend_face_to_end_of_line wherever face->background is involved
(because this is the one that knows that we want to display a
newline).

 > But the merged face will be used for displaying both the "extension"
 > part of the line and "normal" characters.  When displaying "normal"
 > characters, the background color should be used unconditionally,

Definitely so.

 > whereas while displaying the "extension" part it should be used only
 > if its 'extend' bit is set.  Right?

The extend_background bit, more precisely (the :extend face attribute
conceptually applies to all attributes so we can eventually handle
:underline and :box as well if somone wants that).

 > What I wrote above assumed that
 > we make these decisions at face merge time, and will actually have 2
 > different realized faces;

We should have only one realized face.

 > what you are suggesting is that we use a
 > single realized face and make those decisions when we use the faces to
 > write to the glass.  That doesn't strike me as a simpler
 > implementation.

But that decision should have become simpler now that all we need is
to examine that extend_background bit to tell whether the background
shall extend to EOL or not.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Wed, 14 Aug 2019 10:58:13 +0200
> 
>  >> Consider a user who sets the :extend attribute to non-nil for the
>  >> 'default' face and wants a 'link' background to not extend to the end
>  >> of line.  Such a user would want to set the :extend attribute of the
>  >> 'link' face to nil to get the desired effect.
>  >
>  > Yes, and my point is that in that case we can simply ignore the 'link'
>  > face when extending the face of the last character of a line.
> 
> How would the display engine then know that it can "simply ignore"
> the background of that face for the rest of the line?

By seeing that it has the :extend attribute set to nil.

> understanding face implementatin _without_ the text you wrote below
> would take a couple of days at least for a humble reader like me.

And I didn't even mention the complications related to bidi ;-)

>  > Neither :background nor any other face attribute is "found and
>  > applied" in the main iterator loop.  The iterator simply records in
>  > each glyph the face ID to be used for displaying that glyph.  That
>  > face ID is the value returned by the last call to
>  > face_at_buffer_position.
> 
> IIUC the latter will (unless the face was cached already) end up
> calling merge_face_ref and somewhere there I find the lines
> 
> 	      else if (EQ (keyword, QCbackground))
> 		{
> 		  if (STRINGP (value))
> 		    to[LFACE_BACKGROUND_INDEX] = value;
> 		  else
> 		    err = true;
> 		}
> 
> Hence IIUC in the
> 
> 	  while (CONSP (face_ref) && CONSP (XCDR (face_ref)))
> 
> loop we could manage three booleans background, extend and
> extend_value and instead of the above write
> 
> 	      else if (EQ (keyword, QCbackground))
> 		{
> 		  if (STRINGP (value))
> 		    {
> 		      to[LFACE_BACKGROUND_INDEX] = value;
> 
> 		      if (extend)
> 			to[LFACE_EXTEND_BACKGROUND] = extend_value;
> 
> 		      background = true;
> 		    }
> 		  else
> 		    err = true;
> 		}
> 	      else if (EQ (keyword, QCextend_background))
> 		{
> 		  extend = true;
> 		  extend_value = !NILP (value);
> 
> 		  if (background)
> 		    to[LFACE_EXTEND_BACKGROUND] = extend_value;
> 		}

This just copies the :extend flag to the merged face.  It sill copies
the :background attribute itself, and so the merged face will have a
non-default background color.

>  > So you propose to have an "extend" bit for every face attribute?
>  > There are 18 of them.  (Maybe some of the attributes won't need that
>  > bit, but quite a few will.)
> 
> I'd propose to add these lazily if there is any need.  AFAIAC
> :background is the only attribute where an extend bit sounds useful.

IMO, it makes very little sense to allow this only for the background
color.  For starters, please keep in mind that this whole discussion
started because we were considering to make TTY frames behave like GUI
frames wrt face extension, and TTY frames currently extend also the
other attributes, notably the :underline.  Some people also complain
about :underline being ugly on GUI frames when it crosses the newline
and continues on the next line.

I think we need to support this for :background, :underline,
:overline, :strike-through, :box, and :stipple.

> The extend_background bit would be set in merge_face_ref as sketched
> above (because this is the one that can read the :background and
> :extend attributes of any face) and examined in
> extend_face_to_end_of_line wherever face->background is involved
> (because this is the one that knows that we want to display a
> newline).

What do you want extend_face_to_end_of_line to do with the
extend_background bit?  extend_face_to_end_of_line doesn't apply the
background, it just produces glyphs, and has only face IDs to work
with.  So if the face ID of the last character specifies a face with a
background color, and the extend_background bit says not to extend the
background color, how would extend_face_to_end_of_line "remove" the
background color attribute from the face for which it only has the ID?

The only way to do that is to make a new face, by merging all the
attributes except the background color, and then use that new face's
ID for producing glyphs that extend the face.  This is why I said
earlier that I assumed we make the extension decisions at face merge
time, and will have 2 different realized faces, not one.  But you
disagreed with that conclusion:

>  > What I wrote above assumed that
>  > we make these decisions at face merge time, and will actually have 2
>  > different realized faces;
> 
> We should have only one realized face.

AFAIU, that's impossible, not on the level on which
extend_face_to_end_of_line (or any code in xdisp.c, really) works.  It
cannot "apply a face sans some attributes", there are no such
capabilities on this level.

Re: Question about display engine

[martin rudalics]

 >> understanding face implementatin _without_ the text you wrote below
 >> would take a couple of days at least for a humble reader like me.
 >
 > And I didn't even mention the complications related to bidi ;-)

Are there any bidi complications related to that particular text?  No
need for you to point anything out in detail: I suppose it's that
finding the next "stop" position gets complicated when the display
direction changes.  Now in my naive imagination buffer text per se is
one-directional so Fnext_single_property_change progresses always in
one direction.  Does the inteval tree gets botched in some sense?

 > This just copies the :extend flag to the merged face.

Yes.

 > It sill copies
 > the :background attribute itself, and so the merged face will have a
 > non-default background color.

But isn't that the whole idea of that step?  Or do you mean that since
the display engine can't cope with the extend bit and thus suppress
the background by demand, doing that just won't have any effect?

 > I'd propose to add these lazily if there is any need.  AFAIAC
 >> :background is the only attribute where an extend bit sounds useful.
 >
 > IMO, it makes very little sense to allow this only for the background
 > color.  For starters, please keep in mind that this whole discussion
 > started because we were considering to make TTY frames behave like GUI
 > frames wrt face extension, and TTY frames currently extend also the
 > other attributes, notably the :underline.  Some people also complain
 > about :underline being ugly on GUI frames when it crosses the newline
 > and continues on the next line.

That's why in a first step I would disregard the others and by default
not extend them.  Then we would see what is missing and fill in the
remaining attributes of this list ...

 > I think we need to support this for :background, :underline,
 > :overline, :strike-through, :box, and :stipple.

... to possibly extend to end of line as well.

 > What do you want extend_face_to_end_of_line to do with the
 > extend_background bit?  extend_face_to_end_of_line doesn't apply the
 > background, it just produces glyphs, and has only face IDs to work
 > with.

Since I wasn't aware of that fact ("I'm learning something new about
Emacs almost every day") ...

 > So if the face ID of the last character specifies a face with a
 > background color, and the extend_background bit says not to extend the
 > background color, how would extend_face_to_end_of_line "remove" the
 > background color attribute from the face for which it only has the ID?
 >
 > The only way to do that is to make a new face, by merging all the
 > attributes except the background color, and then use that new face's
 > ID for producing glyphs that extend the face.  This is why I said
 > earlier that I assumed we make the extension decisions at face merge
 > time, and will have 2 different realized faces, not one.  But you
 > disagreed with that conclusion:

... I obviously have to discontinue disagreeing with that conclusion.
But wouldn't that significantly increase the size of the face cache?
I have no idea about the typical size of that cache with a moderately
customized Emacs ...

 >>   > What I wrote above assumed that
 >>   > we make these decisions at face merge time, and will actually have 2
 >>   > different realized faces;
 >>
 >> We should have only one realized face.
 >
 > AFAIU, that's impossible, not on the level on which
 > extend_face_to_end_of_line (or any code in xdisp.c, really) works.  It
 > cannot "apply a face sans some attributes", there are no such
 > capabilities on this level.

Since extend_face_to_end_of_line already switches to the default face
when no extension of the region is wanted, it could also switch to the
default face when the extend_background bit is false.  Right?

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Thu, 15 Aug 2019 10:13:21 +0200
> 
>  >> understanding face implementatin _without_ the text you wrote below
>  >> would take a couple of days at least for a humble reader like me.
>  >
>  > And I didn't even mention the complications related to bidi ;-)
> 
> Are there any bidi complications related to that particular text?  No
> need for you to point anything out in detail: I suppose it's that
> finding the next "stop" position gets complicated when the display
> direction changes.

Exactly.  More accurately, we have 2 problems: (1) how do you know you
hit the stop point while going back; and (2) where do you go to
compute the "next" stop point.  You don't want to know the answers...

> Now in my naive imagination buffer text per se is
> one-directional so Fnext_single_property_change progresses always in
> one direction.  Does the inteval tree gets botched in some sense?

No, nothing terrible like that.  It's just those 2 problems mentioned
above.

>  > It sill copies
>  > the :background attribute itself, and so the merged face will have a
>  > non-default background color.
> 
> But isn't that the whole idea of that step?  Or do you mean that since
> the display engine can't cope with the extend bit and thus suppress
> the background by demand, doing that just won't have any effect?

The latter, yes.  IOW, doing that will simply defer the decision for
later.

>  > So if the face ID of the last character specifies a face with a
>  > background color, and the extend_background bit says not to extend the
>  > background color, how would extend_face_to_end_of_line "remove" the
>  > background color attribute from the face for which it only has the ID?
>  >
>  > The only way to do that is to make a new face, by merging all the
>  > attributes except the background color, and then use that new face's
>  > ID for producing glyphs that extend the face.  This is why I said
>  > earlier that I assumed we make the extension decisions at face merge
>  > time, and will have 2 different realized faces, not one.  But you
>  > disagreed with that conclusion:
> 
> ... I obviously have to discontinue disagreeing with that conclusion.
> But wouldn't that significantly increase the size of the face cache?

It will increase the cache, yes. Whether it will be significant is
another question; I'm not sure.

>  >> We should have only one realized face.
>  >
>  > AFAIU, that's impossible, not on the level on which
>  > extend_face_to_end_of_line (or any code in xdisp.c, really) works.  It
>  > cannot "apply a face sans some attributes", there are no such
>  > capabilities on this level.
> 
> Since extend_face_to_end_of_line already switches to the default face
> when no extension of the region is wanted

It does? where?

> it could also switch to the default face when the extend_background
> bit is false.  Right?

No.  A face is not just its background color, it's quite a few other
things.  You only want to reset the background color.  For example, if
the face of the last character used a larger font, you still want the
extension to use that larger font, e.g. for the fill-column indicator
character.

So switching to the default face in this case is wrong, we really need
to make a face exactly like the one we used, but without the attribute
that should not be extended.

Re: Question about display engine

[martin rudalics]

 > Exactly.  More accurately, we have 2 problems: (1) how do you know you
 > hit the stop point while going back; and (2) where do you go to
 > compute the "next" stop point.  You don't want to know the answers...

I'm not sure.  If that algorithm is newline agnostic, it will have to
be changed if an eager (see below) solution is implemented.

 >> ... I obviously have to discontinue disagreeing with that conclusion.
 >> But wouldn't that significantly increase the size of the face cache?
 >
 > It will increase the cache, yes. Whether it will be significant is
 > another question; I'm not sure.

In the worst case it would double the size of the cache for
implementing the background extension alone.  This looks like a
veritable ordeal to me.  IIUC we do all this to keep the display
optimizations (where we compare the glyph matrices) simple and
efficient.  Or is there an additional twist to it?

One could argue that if, in the current implementation, the display
element stops right before a newline character and that newline is,
presumably, a separate display element, the overhead remains
unchanged.  But AFAICT, font locking does not necessarily pay immense
attention to newlines (Lisp comments and C strings excluded) so at
least for displaying programming languages the overhead increase might
be significant.

 >> Since extend_face_to_end_of_line already switches to the default face
 >> when no extension of the region is wanted
 >
 > It does? where?

I thought it did so here

       /* The last row's blank glyphs should get the default face, to
	 avoid painting the rest of the window with the region face,
	 if the region ends at ZV.  */
       if (it->glyph_row->ends_at_zv_p)
	it->face_id = default_face->id;
       else
	it->face_id = face->id;

but apparently that's only a red herring.

 >> it could also switch to the default face when the extend_background
 >> bit is false.  Right?
 >
 > No.  A face is not just its background color, it's quite a few other
 > things.  You only want to reset the background color.  For example, if
 > the face of the last character used a larger font, you still want the
 > extension to use that larger font, e.g. for the fill-column indicator
 > character.
 >
 > So switching to the default face in this case is wrong, we really need
 > to make a face exactly like the one we used, but without the attribute
 > that should not be extended.

I currently see two ways to accomplish that: Eagerly, during face
merging, we would have to search for and possibly create the extend
face variation whenever the next stop point finding algorithm
encounters a newline character within the object it examines.  This
means that any display element that contains a newline character also
ends before that character and the entire logic of finding display
elements changes.

A lazy variant would have the display engine itself create the extend
face by demand whenever it encounters a newline character within the
current display element.  This would strike me as more elegant but
also means that the display engine has to scan the face cache and
potentially realize a new face here.

The worst aspect of all this is that there is no simpler solution even
if we attempted a different implementation of the extend logic.
Suppose I used just one single, global variable to turn off/on any
face extensions.  For turning it off, I'd still have to, at the end of
every line, assure that a separate realized face does implement the
"off" interpretation while the "on" interpretation is already provided
by the last face on that line.  Right?

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Fri, 16 Aug 2019 09:29:02 +0200
> 
>  > Exactly.  More accurately, we have 2 problems: (1) how do you know you
>  > hit the stop point while going back; and (2) where do you go to
>  > compute the "next" stop point.  You don't want to know the answers...
> 
> I'm not sure.  If that algorithm is newline agnostic, it will have to
> be changed if an eager (see below) solution is implemented.

Thankfully and mercifully, all the bidi calculations and hidden state
variables are reset at newlines.

>  > It will increase the cache, yes. Whether it will be significant is
>  > another question; I'm not sure.
> 
> In the worst case it would double the size of the cache for
> implementing the background extension alone.  This looks like a
> veritable ordeal to me.  IIUC we do all this to keep the display
> optimizations (where we compare the glyph matrices) simple and
> efficient.  Or is there an additional twist to it?

Well, a face cache is rarely too large, it mostly holds a few dozen
faces.  We also clear it from time to time, so I don't think this
would be a catastrophe.

> One could argue that if, in the current implementation, the display
> element stops right before a newline character and that newline is,
> presumably, a separate display element, the overhead remains
> unchanged.  But AFAICT, font locking does not necessarily pay immense
> attention to newlines (Lisp comments and C strings excluded) so at
> least for displaying programming languages the overhead increase might
> be significant.

Actually, IME the situation where a face crosses a newline is somewhat
rare.

>  >> Since extend_face_to_end_of_line already switches to the default face
>  >> when no extension of the region is wanted
>  >
>  > It does? where?
> 
> I thought it did so here
> 
>        /* The last row's blank glyphs should get the default face, to
> 	 avoid painting the rest of the window with the region face,
> 	 if the region ends at ZV.  */
>        if (it->glyph_row->ends_at_zv_p)
> 	it->face_id = default_face->id;
>        else
> 	it->face_id = face->id;
> 
> but apparently that's only a red herring.

This is a corner case, for the screen lines beyond EOB.  It is none of
our business for the purposes of this discussion.

> I currently see two ways to accomplish that: Eagerly, during face
> merging, we would have to search for and possibly create the extend
> face variation whenever the next stop point finding algorithm
> encounters a newline character within the object it examines.

Not my preference.  It will disrupt the neat algorithm of walking the
interval tree, or at least will force us to also search for a newline
(which is very fast, but still a complication).

> This means that any display element that contains a newline
> character also ends before that character and the entire logic of
> finding display elements changes.

What you call "display element" is actually called a "glyph string": a
sequence of glyphs that have the same face.  The division of glyphs
into glyph strings happens as part of preparing glyphs for display,
and will automatically pay attention to changes in the face ID.  So no
logic changes here.  The changes are as I described above: where we
compute the next stop position.  there' we will have to see whether
the stretch of text till the next stop includes a newline.

> A lazy variant would have the display engine itself create the extend
> face by demand whenever it encounters a newline character within the
> current display element.

I'd prefer this method.  Two reasons: (1) it is localized to the code
which may need such a face; and (2) it scales better, because the
display code is frequently invoked on short portions of the text, so
there's no guarantee that it will actually get to producing glyphs
with the "extension" variant of the face, so realizing that face in
advance might well be waste of unneeded effort, because the additional
face will never be used.

> The worst aspect of all this is that there is no simpler solution even
> if we attempted a different implementation of the extend logic.
> Suppose I used just one single, global variable to turn off/on any
> face extensions.  For turning it off, I'd still have to, at the end of
> every line, assure that a separate realized face does implement the
> "off" interpretation while the "on" interpretation is already provided
> by the last face on that line.  Right?

Yes, we will typically need that "on" face for the next screen line.

Re: Question about display engine

[martin rudalics]

 > I'd prefer this method.  Two reasons: (1) it is localized to the code
 > which may need such a face; and (2) it scales better, because the
 > display code is frequently invoked on short portions of the text, so
 > there's no guarantee that it will actually get to producing glyphs
 > with the "extension" variant of the face, so realizing that face in
 > advance might well be waste of unneeded effort, because the additional
 > face will never be used.

The following should capture what emerged from this discussion so far:

(1) Provide an :extend face attribute with the semantics to extend, if
     set, any "background-related" attributes like :background,
     :underline, :box ... specified by this face.

(2) When merging faces, set an extend-background, extend-underline,
     extend-box ... bit for all background-related attributes whenever
     the face merged in has both the :extend attribute non-nil and the
     corresponding background-related attribute set.

(3) When the display engine encounters a newline character and the
     current face has one of the extend-* bits set, either reuse or
     create a new realized face based on the current face, removing
     from a new realized face any background-related information for
     which the current face that has the corresponding extend-* bit
     set.  For example, if the current face specifies a background,
     remove that in the new face if the extend-background bit is unset.

(4) Use the face found or made in (3) for glyphs on the rest of the
     current line.

Conversely, we could use a :no-extend attribute and/or
no-extend-background bits in the realized faces if that's simpler or
more intuitive.

martin

Re: Question about display engine

[Ergus]

Hi:

After a couple of weeks with this discussion I am back to see if there
 is finally an agreement or more people interested. But I see there are
 mainly only Eli and Martin involved. The proposed solutions I see here
 seems to be a bit complex (to implements and to execute for the display
 engine). It seems that no applications or developers are interested on
 this detail enough to comment or propose solutions here.

I understand that the proposed are the most general solutions, but does
 it worth the complexity, overhead, and customization vs the benefit? Do
 we really need all these functionalities and control in detail? Are we
 really expecting that the users will be allowed||interested in
 customizing all this in such granular level?

Extending the face is a detail that most users won't even notice a
change while it works somehow. Actually we have had different behaviors
between gui and tui for years and nobody complained up to now.

Extending the underline|overline to the right of the line in the region
is a fancy detail, but I don't find it useful at all (underlining empty
long spaces is conceptually an error), and in the actual gui it is not
happening (not even extend_face_to_end_of_line in gui is executed as the
background color is extended automatically somewhere else) and there
haven't been complains either, so there is not people using that, so why
do we provide a complex solution implementation for a problem nobody
really cares and that potentially will produce overheads, code
complexity and more issues?

I am wondering about over-specifications and over-engineering for such a
detail, when most of the users only need to extend the background color.

On Sat, Aug 17, 2019 at 10:25:13AM +0200, martin rudalics wrote:
>> I'd prefer this method.  Two reasons: (1) it is localized to the code
>> which may need such a face; and (2) it scales better, because the
>> display code is frequently invoked on short portions of the text, so
>> there's no guarantee that it will actually get to producing glyphs
>> with the "extension" variant of the face, so realizing that face in
>> advance might well be waste of unneeded effort, because the additional
>> face will never be used.
>
>The following should capture what emerged from this discussion so far:
>
>(1) Provide an :extend face attribute with the semantics to extend, if
>    set, any "background-related" attributes like :background,
>    :underline, :box ... specified by this face.
>
>(2) When merging faces, set an extend-background, extend-underline,
>    extend-box ... bit for all background-related attributes whenever
>    the face merged in has both the :extend attribute non-nil and the
>    corresponding background-related attribute set.
>
>(3) When the display engine encounters a newline character and the
>    current face has one of the extend-* bits set, either reuse or
>    create a new realized face based on the current face, removing
>    from a new realized face any background-related information for
>    which the current face that has the corresponding extend-* bit
>    set.  For example, if the current face specifies a background,
>    remove that in the new face if the extend-background bit is unset.
>
>(4) Use the face found or made in (3) for glyphs on the rest of the
>    current line.
>
>Conversely, we could use a :no-extend attribute and/or
>no-extend-background bits in the realized faces if that's simpler or
>more intuitive.
>
>martin
>

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 19 Aug 2019 18:13:05 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Extending the face is a detail that most users won't even notice a
> change while it works somehow. Actually we have had different behaviors
> between gui and tui for years and nobody complained up to now.

I think you over-simplify the situation.  First, both GUI and TTY
frames behave the same when extension of face background color is
concerned, they only differ in how they handle extension of other
attributes (of which only the underline is relevant to TTY frames).
(And I did hear over the years a couple of complaints about how TTY
frames extend the underline attribute.)

And second, I refer you to the renewed discussion of bug#15934 a day
or two ago, from which my take is that users will notice and do care
about such changes in at least a couple of important use cases, as a
soon-to-be-pushed changes will prove.  So I don't think we can change
this behavior at will on the assumption that "no one will notice".

> I am wondering about over-specifications and over-engineering for such a
> detail, when most of the users only need to extend the background color.

This discussion established that even supporting just the background
color non-extension as a user option will require to have almost all
of the machinery in place: the extend bit, the generation of a special
face without the background color, etc.  And once we have that, adding
other face attributes to the soup is relatively easy and won't require
any design changes.

Re: Question about display engine

[Ergus]

On Mon, Aug 19, 2019 at 07:50:36PM +0300, Eli Zaretskii wrote:
>> Date: Mon, 19 Aug 2019 18:13:05 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>>
>> Extending the face is a detail that most users won't even notice a
>> change while it works somehow. Actually we have had different behaviors
>> between gui and tui for years and nobody complained up to now.
>
>I think you over-simplify the situation.  First, both GUI and TTY
>frames behave the same when extension of face background color is
>concerned, they only differ in how they handle extension of other
>attributes (of which only the underline is relevant to TTY frames).
>(And I did hear over the years a couple of complaints about how TTY
>frames extend the underline attribute.)
>
>And second, I refer you to the renewed discussion of bug#15934 a day
>or two ago, from which my take is that users will notice and do care
>about such changes in at least a couple of important use cases, as a
>soon-to-be-pushed changes will prove.  So I don't think we can change
>this behavior at will on the assumption that "no one will notice".
>
Hi Eli:

If soon-to-be-pushed means that you have had already some time to work on
this and it will be fixed before emacs 27 then I'll be happy with that
(I'll be allowed to fix the dfci issue). I was actually wondering
if the discussion was not going anywhere as I didn't see any comment
in a couple of days.

Unrelated with this I wanted to ask you if you think we should continue
with the indentation highlight implementation... because that discussion
never ended. Or you think it does not worth the effort.

I have seen that elpy already have something like what we want to
implement in the lisp level. So they will actually switch for sure to
ours if available. 

What do you think?

>> I am wondering about over-specifications and over-engineering for such a
>> detail, when most of the users only need to extend the background color.
>
>This discussion established that even supporting just the background
>color non-extension as a user option will require to have almost all
>of the machinery in place: the extend bit, the generation of a special
>face without the background color, etc.  And once we have that, adding
>other face attributes to the soup is relatively easy and won't require
>any design changes.
>

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 19 Aug 2019 23:30:24 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >And second, I refer you to the renewed discussion of bug#15934 a day
> >or two ago, from which my take is that users will notice and do care
> >about such changes in at least a couple of important use cases, as a
> >soon-to-be-pushed changes will prove.  So I don't think we can change
> >this behavior at will on the assumption that "no one will notice".
> >
> Hi Eli:
> 
> If soon-to-be-pushed means that you have had already some time to work on
> this and it will be fixed before emacs 27 then I'll be happy with that
> (I'll be allowed to fix the dfci issue).

No, that was about bug#15934, where a change will be soon pushed that
forces extension of a face to the edge of the window.

> I was actually wondering if the discussion was not going anywhere as
> I didn't see any comment in a couple of days.

We've arrived to a good understanding of what needs to be done, and we
have no issues left to discuss.  What's left is to implement the
thing.  Volunteers are welcome.

> Unrelated with this I wanted to ask you if you think we should continue
> with the indentation highlight implementation... because that discussion
> never ended. Or you think it does not worth the effort.

I'm not sure.  A few people said that what we wanted to leave out
cannot be left out.

> I have seen that elpy already have something like what we want to
> implement in the lisp level. So they will actually switch for sure to
> ours if available. 

Where can I see what they have?

Re: Question about display engine

[Ergus]

On Tue, Aug 20, 2019 at 05:09:00PM +0300, Eli Zaretskii wrote:
>
>We've arrived to a good understanding of what needs to be done, and we
>have no issues left to discuss.  What's left is to implement the
>thing.  Volunteers are welcome.
>
Hi Eli I could be interested in this but I don't even understand clearly what
you agreed; so it may take me a long time without some extra
hints. (When things are related with the lisp part I just get lost.)

>> I have seen that elpy already have something like what we want to
>> implement in the lisp level. So they will actually switch for sure to
>> ours if available.
>
>Where can I see what they have?
>

They just rely on: 

https://github.com/antonj/Highlight-Indentation-for-Emacs/blob/master/highlight-indentation.el

with some modifications.

Re: Question about display engine

[Eli Zaretskii]

> Date: Sun, 25 Aug 2019 12:22:05 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> On Tue, Aug 20, 2019 at 05:09:00PM +0300, Eli Zaretskii wrote:
> >
> >We've arrived to a good understanding of what needs to be done, and we
> >have no issues left to discuss.  What's left is to implement the
> >thing.  Volunteers are welcome.
> >
> Hi Eli I could be interested in this but I don't even understand clearly what
> you agreed; so it may take me a long time without some extra
> hints. (When things are related with the lisp part I just get lost.)

Feel free to ask questions.  (I don't think anything in the summary
was related to Lisp, but maybe I'm wrong.)

> >> I have seen that elpy already have something like what we want to
> >> implement in the lisp level. So they will actually switch for sure to
> >> ours if available.
> >
> >Where can I see what they have?
> >
> 
> They just rely on: 
> 
> https://github.com/antonj/Highlight-Indentation-for-Emacs/blob/master/highlight-indentation.el
> 
> with some modifications.

The question is will many/enough users agree to such a limited
implementation.  I was under the impression that quite a few
responders said it would not be good enough.

Re: Question about display engine

[Ergus]

On Sun, Aug 25, 2019 at 01:44:04PM +0300, Eli Zaretskii wrote:

>
>Feel free to ask questions.  (I don't think anything in the summary
>was related to Lisp, but maybe I'm wrong.)
>

Hi:

Could you please finally explain what are all the points needed in the
implementation? And more or less the changes needed/expected?

>The question is will many/enough users agree to such a limited
>implementation.  I was under the impression that quite a few
>responders said it would not be good enough.
>

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 26 Aug 2019 06:31:45 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> On Sun, Aug 25, 2019 at 01:44:04PM +0300, Eli Zaretskii wrote:
> 
> >
> >Feel free to ask questions.  (I don't think anything in the summary
> >was related to Lisp, but maybe I'm wrong.)
> >
> 
> Hi:
> 
> Could you please finally explain what are all the points needed in the
> implementation? And more or less the changes needed/expected?

Regarding face extension, or regarding highlight-indentation?  These
were 2 separate discussions, but you somehow conflated them together
in your last email.

Re: Question about display engine

[Ergus]

On Mon, Aug 26, 2019 at 10:45:11AM +0300, Eli Zaretskii wrote:
>> Date: Mon, 26 Aug 2019 06:31:45 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> On Sun, Aug 25, 2019 at 01:44:04PM +0300, Eli Zaretskii wrote:
>>
>> >
>> >Feel free to ask questions.  (I don't think anything in the summary
>> >was related to Lisp, but maybe I'm wrong.)
>> >
>>
>> Hi:
>>
>> Could you please finally explain what are all the points needed in the
>> implementation? And more or less the changes needed/expected?
>
>Regarding face extension, or regarding highlight-indentation?  These
>were 2 separate discussions, but you somehow conflated them together
>in your last email.

Regarding face extension

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 26 Aug 2019 10:18:19 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >> Could you please finally explain what are all the points needed in the
> >> implementation? And more or less the changes needed/expected?
> >
> >Regarding face extension, or regarding highlight-indentation?  These
> >were 2 separate discussions, but you somehow conflated them together
> >in your last email.
> 
> Regarding face extension

The summary was posted by Martin here:

  https://lists.gnu.org/archive/html/emacs-devel/2019-08/msg00324.html

If something there is unclear, please ask specific questions.

Thanks.

Re: Question about display engine

[Ergus]

On Mon, Aug 26, 2019 at 12:49:27PM +0300, Eli Zaretskii wrote:
>> Date: Mon, 26 Aug 2019 10:18:19 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >> Could you please finally explain what are all the points needed in the
>> >> implementation? And more or less the changes needed/expected?
>> >
>> >Regarding face extension, or regarding highlight-indentation?  These
>> >were 2 separate discussions, but you somehow conflated them together
>> >in your last email.
>>
>> Regarding face extension
>
>The summary was posted by Martin here:
>
>  https://lists.gnu.org/archive/html/emacs-devel/2019-08/msg00324.html
>
>If something there is unclear, please ask specific questions.
>
>Thanks.
>

Hi Eli:

Starting with this I just need a couple of details to clarify if I
understand everything:

(1) Provide an :extend face attribute with the semantics to extend, if
    set, any "background-related" attributes like :background,
    :underline, :box ... specified by this face.

The extend attribute here must be just a boolean to extend everything
set in this class or something to specify what to extend from within
this face?

(2) When merging faces, set an extend-background, extend-underline,
    extend-box ... bit for all background-related attributes whenever
    the face merged in has both the :extend attribute non-nil and the
    corresponding background-related attribute set.

(3) When the display engine encounters a newline character and the
    current face has one of the extend-* bits set, either reuse or
    create a new realized face based on the current face, removing
    from a new realized face any background-related information for
    which the current face that has the corresponding extend-* bit
    set.  For example, if the current face specifies a background,
    remove that in the new face if the extend-background bit is unset.

(4) Use the face found or made in (3) for glyphs on the rest of the
    current line.


1- Are the bits stored in the class or they are supposed to be like a
display engine state variable?

Check this true table please:

merge_faces (struct window *w, Lisp_Object face_name, int face_id,
             int base_face_id)

struct face = {extend_flag; bg, extend_bg_flag}

base_face -> face = merged

{true, nil, false} -> {false, y, false} -> {false, y, true}    // ???? 

{false, x, false} -> {true, nil, false} -> {true, x, true}     // Is this fine?

When in 3 it says "remove", what does it means? set it to x, to default or
to nil?

In case we don't extend, the extra space we add after the line should
have the default, face or the last in the line??

Re: Question about display engine

[martin rudalics]

 > (1) Provide an :extend face attribute with the semantics to extend, if
 >     set, any "background-related" attributes like :background,
 >     :underline, :box ... specified by this face.
 >
 > The extend attribute here must be just a boolean to extend everything
 > set in this class or something to specify what to extend from within
 > this face?

The former, conceptually.  Only if we detect a corner case which tells
us that inevitably "more" is needed, it might become necessary to
implement the latter.  So it might be a good idea to have the values
unspecified, t, nil and a list where the semantics of a list would be
currently undefined.

 > (2) When merging faces, set an extend-background, extend-underline,
 >     extend-box ... bit for all background-related attributes whenever
 >     the face merged in has both the :extend attribute non-nil and the
 >     corresponding background-related attribute set.
 >
 > (3) When the display engine encounters a newline character and the
 >     current face has one of the extend-* bits set, either reuse or
 >     create a new realized face based on the current face, removing
 >     from a new realized face any background-related information for
 >     which the current face that has the corresponding extend-* bit
 >     set.  For example, if the current face specifies a background,
 >     remove that in the new face if the extend-background bit is unset.
 >
 > (4) Use the face found or made in (3) for glyphs on the rest of the
 >     current line.
 >
 >
 > 1- Are the bits stored in the class or they are supposed to be like a
 > display engine state variable?

In the class, IIUC.  That is, when merge_face_vectors merges in "an
attribute we care about" it sets, for example, the extend_background
bit of the TO face when both the :extend and :background values of the
FROM vector are set.

 > Check this true table please:
 >
 > merge_faces (struct window *w, Lisp_Object face_name, int face_id,
 >              int base_face_id)
 >
 > struct face = {extend_flag; bg, extend_bg_flag}
 >
 > base_face -> face = merged
 >
 > {true, nil, false} -> {false, y, false} -> {false, y, true}    // ????
 > {false, x, false} -> {true, nil, false} -> {true, x, true}     // Is this fine?

I'm too silly to understand these, please elaborate.

 > When in 3 it says "remove", what does it means? set it to x, to default or
 > to nil?
 >
 > In case we don't extend, the extra space we add after the line should
 > have the default, face or the last in the line??

The term "remove" was meant wrt the current face used by the iterator.
If, for example, the current face of the iterator is the one specified
by the region face, then it probably specifies some background used
for "normal" text.  When the display engine arrives at the newline
character it checks whether the current face has the extend_background
bit set and either reuses an existing or realizes a new face based on
the background of the current face and that bit.

What realizing has to do when extend_background is false is not
entirely clear to me either.  Suppose we have a newline character that
is contained both in the region and a multiline comment and the region
face results in an extend_background false bit while the comment face
results in an extend_background true bit for the corresponding
realized faces.

Conceptually, this means that the spaces at the end of the line should
have the comment background but unless we expand the extend_background
bits into full pointers to other realized faces (and have face merging
set up these pointers and the display engine resolve them) I see no
way how this could be realized.  So the rest of the line would be
shown with the default face's background instead (or do whatever we do
now when we don't expand) which obviously won't look good.

Personally, I consider a setting like the above a misspecification: A
"lower priority extend" face should never result in being covered by a
"higher priority no-extend" face but others might obviously disagree.

Maybe the display engine could consult, with the stop position
privilege triggered by the newline character in mind, whether the
extend_background false setting of the current face could result in
applying another, lower-priority face specified by font-locking and
consult the extend_background bit of the corresponding realized face.
But I doubt that this would be easy to code.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Wed, 28 Aug 2019 10:35:11 +0200
> 
>  > When in 3 it says "remove", what does it means? set it to x, to default or
>  > to nil?
>  >
>  > In case we don't extend, the extra space we add after the line should
>  > have the default, face or the last in the line??
> 
> The term "remove" was meant wrt the current face used by the iterator.
> If, for example, the current face of the iterator is the one specified
> by the region face, then it probably specifies some background used
> for "normal" text.  When the display engine arrives at the newline
> character it checks whether the current face has the extend_background
> bit set and either reuses an existing or realizes a new face based on
> the background of the current face and that bit.

I think we should simply not merge the background color of the region
face when its extend bit is reset.  Then the merged face will not have
that background color.

> What realizing has to do when extend_background is false is not
> entirely clear to me either.  Suppose we have a newline character that
> is contained both in the region and a multiline comment and the region
> face results in an extend_background false bit while the comment face
> results in an extend_background true bit for the corresponding
> realized faces.
> 
> Conceptually, this means that the spaces at the end of the line should
> have the comment background but unless we expand the extend_background
> bits into full pointers to other realized faces (and have face merging
> set up these pointers and the display engine resolve them) I see no
> way how this could be realized.  So the rest of the line would be
> shown with the default face's background instead (or do whatever we do
> now when we don't expand) which obviously won't look good.

I don't see a problem here.  A user who doesn't want the region face's
background extend to the end of line wants only text (as opposed to
whitespace after the newline) to have the region's background, and
that's true both to regions that cross line boundaries and regions that
end at a newline.

> Maybe the display engine could consult, with the stop position
> privilege triggered by the newline character in mind, whether the
> extend_background false setting of the current face could result in
> applying another, lower-priority face specified by font-locking and
> consult the extend_background bit of the corresponding realized face.

I don't think I understand this proposal.

Re: Question about display engine

[martin rudalics]

 > I think we should simply not merge the background color of the region
 > face when its extend bit is reset.  Then the merged face will not have
 > that background color.

Then which background color would we use?  That of the comment was
lost when setting up the current face for the iterator.

 > I don't see a problem here.  A user who doesn't want the region face's
 > background extend to the end of line wants only text (as opposed to
 > whitespace after the newline) to have the region's background, and
 > that's true both to regions that cross line boundaries and regions that
 > end at a newline.

I agree that we don't want to extend the region's background.  But the
question I raised above still stands.

We could make :extend sticky in the sense that once an :extend for the
background has been defined, it will apply to all higher priority faces
as well.  This would make specifying a nil :extend value idempotent to
not specifying a value at all as you (IIRC) proposed earlier.  But the
mechanism then becomes considerably less powerful.

 >> Maybe the display engine could consult, with the stop position
 >> privilege triggered by the newline character in mind, whether the
 >> extend_background false setting of the current face could result in
 >> applying another, lower-priority face specified by font-locking and
 >> consult the extend_background bit of the corresponding realized face.
 >
 > I don't think I understand this proposal.

Let's talk about it when we both see a problem here.

martin

Re: Question about display engine

[Ergus]

On Wed, Aug 28, 2019 at 02:19:03PM +0200, martin rudalics wrote:
>> I think we should simply not merge the background color of the region
>> face when its extend bit is reset.  Then the merged face will not have
>> that background color.
>
>Then which background color would we use?  That of the comment was
>lost when setting up the current face for the iterator.
>
>> I don't see a problem here.  A user who doesn't want the region face's
>> background extend to the end of line wants only text (as opposed to
>> whitespace after the newline) to have the region's background, and
>> that's true both to regions that cross line boundaries and regions that
>> end at a newline.
>
>I agree that we don't want to extend the region's background.  But the
>question I raised above still stands.
>
>We could make :extend sticky in the sense that once an :extend for the
>background has been defined, it will apply to all higher priority faces
>as well.  This would make specifying a nil :extend value idempotent to
>not specifying a value at all as you (IIRC) proposed earlier.  But the
>mechanism then becomes considerably less powerful.
>

Any way my question comes from 2 frequent use cases I don't know what's
the expected behavior:

1) Base face sets background and extend; and face sets only background.

2) Base face sets extend but not background; and face sets both.

in what condition the :extend attribute goes to the merged face? Always?
when in base_face? When in face?

in what condition the extend_element bits are reset after a merge? 

>>> Maybe the display engine could consult, with the stop position
>>> privilege triggered by the newline character in mind, whether the
>>> extend_background false setting of the current face could result in
>>> applying another, lower-priority face specified by font-locking and
>>> consult the extend_background bit of the corresponding realized face.
>>
>> I don't think I understand this proposal.
>
>Let's talk about it when we both see a problem here.
>
>martin

Re: Question about display engine

[Eli Zaretskii]

> Date: Wed, 28 Aug 2019 18:31:42 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Any way my question comes from 2 frequent use cases I don't know what's
> the expected behavior:
> 
> 1) Base face sets background and extend; and face sets only background.

The background of the base face will be extended.

> 2) Base face sets extend but not background; and face sets both.

The background of the face will be extended.

> in what condition the :extend attribute goes to the merged face? Always?
> when in base_face? When in face?

The :extend attribute determines whether the background color of the
face with that attribute does or doesn't get merged into the "merged
face" to be used by the iterator.

Re: Question about display engine

[Ergus]

On Wed, Aug 28, 2019 at 08:24:26PM +0300, Eli Zaretskii wrote:
>> Date: Wed, 28 Aug 2019 18:31:42 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>>
>> Any way my question comes from 2 frequent use cases I don't know what's
>> the expected behavior:
>>
>> 1) Base face sets background and extend; and face sets only background.
>
>The background of the base face will be extended.
>

But AFAIU the actual merging rules now will create a new face (if not
there already) that will have the new background; so this means that
then we need a way to remember the last background color where
background and extend where set at the same time; so the extend flag
will be not a bool flag, but a pointer (or a copy) to a previous point
(value) in the faces stack??

Then at the last step of the merge we could potentially have N exact
equal faces that only have difference in those points.

>> 2) Base face sets extend but not background; and face sets both.
>
>The background of the face will be extended.
>
>> in what condition the :extend attribute goes to the merged face? Always?
>> when in base_face? When in face?
>
>The :extend attribute determines whether the background color of the
>face with that attribute does or doesn't get merged into the "merged
>face" to be used by the iterator.
>

Re: Question about display engine

[Eli Zaretskii]

> Date: Wed, 28 Aug 2019 20:19:58 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >> 1) Base face sets background and extend; and face sets only background.
> >
> >The background of the base face will be extended.
> >
> 
> But AFAIU the actual merging rules now will create a new face (if not
> there already) that will have the new background

No.  At the point which we are discussing, only background of faces
which have the :extend bit set get merged.  So the background of a
face without that bit will simply not get merged.

You will probably need to write a variant of merge_face_ref (or add an
extra argument to the existing function) such that background color
attribute of a face is not merged unless the :extend bit of the face
is set; and similar for underline and other relevant attributes we
discussed.

Re: Question about display engine

[martin rudalics]

 >> But AFAIU the actual merging rules now will create a new face (if not
 >> there already) that will have the new background
 >
 > No.  At the point which we are discussing, only background of faces
 > which have the :extend bit set get merged.  So the background of a
 > face without that bit will simply not get merged.

It will have to get merged for text that appears _before_ the newline
character.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Fri, 30 Aug 2019 09:02:54 +0200
> 
>  >> But AFAIU the actual merging rules now will create a new face (if not
>  >> there already) that will have the new background
>  >
>  > No.  At the point which we are discussing, only background of faces
>  > which have the :extend bit set get merged.  So the background of a
>  > face without that bit will simply not get merged.
> 
> It will have to get merged for text that appears _before_ the newline
> character.

For the text before the newline, the :extend bit is ignored when
merging faces.

Re: Question about display engine

[Ergus]

Sorry Eli I still don't see anything clear. It seems that only you have
enough knowledge to understand how this solution may work. Because I am
missing a lot of details.

I will actually stop trying, because I can't implement something I don't
understand how it works and I am wasting a precious amount of time.


On Thu, Aug 29, 2019 at 09:28:55PM +0300, Eli Zaretskii wrote:
>> Date: Wed, 28 Aug 2019 20:19:58 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >> 1) Base face sets background and extend; and face sets only background.
>> >
>> >The background of the base face will be extended.
>> >
>>
>> But AFAIU the actual merging rules now will create a new face (if not
>> there already) that will have the new background
>
>No.  At the point which we are discussing, only background of faces
>which have the :extend bit set get merged.  So the background of a
>face without that bit will simply not get merged.
>
>You will probably need to write a variant of merge_face_ref (or add an
>extra argument to the existing function) such that background color
>attribute of a face is not merged unless the :extend bit of the face
>is set; and similar for underline and other relevant attributes we
>discussed.

Re: Question about display engine

[martin rudalics]

 > Any way my question comes from 2 frequent use cases I don't know what's
 > the expected behavior:
 >
 > 1) Base face sets background and extend; and face sets only background.
 >
 > 2) Base face sets extend but not background; and face sets both.
 >
 > in what condition the :extend attribute goes to the merged face? Always?
 > when in base_face? When in face?

Assuming that "base face" is merged in before "face", both should
conceptually expand face.  The problematic case is

Base face sets background and extend to t; and face sets background to
t and extend to nil.

In this case merge should provide to show normal text with face but
extend the background of the basic face.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Wed, 28 Aug 2019 14:19:03 +0200
> 
>  > I think we should simply not merge the background color of the region
>  > face when its extend bit is reset.  Then the merged face will not have
>  > that background color.
> 
> Then which background color would we use?  That of the comment was
> lost when setting up the current face for the iterator.

The one that was there before the region was activated.  Which one is
that will be determined by the order in which the merging process
merges the faces, and by the faces themselves -- whether they do or
don't define a background color, and whether they do or don't have the
:extend bit set.

>  > I don't see a problem here.  A user who doesn't want the region face's
>  > background extend to the end of line wants only text (as opposed to
>  > whitespace after the newline) to have the region's background, and
>  > that's true both to regions that cross line boundaries and regions that
>  > end at a newline.
> 
> I agree that we don't want to extend the region's background.  But the
> question I raised above still stands.

Did I answer it now?

> We could make :extend sticky in the sense that once an :extend for the
> background has been defined, it will apply to all higher priority faces
> as well.  This would make specifying a nil :extend value idempotent to
> not specifying a value at all as you (IIRC) proposed earlier.  But the
> mechanism then becomes considerably less powerful.

I don't think this will be needed, or even desirable.

Re: Question about display engine

[martin rudalics]

 >>   > I think we should simply not merge the background color of the region
 >>   > face when its extend bit is reset.  Then the merged face will not have
 >>   > that background color.
 >>
 >> Then which background color would we use?  That of the comment was
 >> lost when setting up the current face for the iterator.
 >
 > The one that was there before the region was activated.

But where do we get that from?  Consider a three lines C buffer with
code on the first line and a two-line comment covering the remaining
lines with the region covering the entire buffer.  Hence, the spaces
following the comment on the second line should be drawn with the
background of the comment face.  But when the display engine gets
there, the respective face was not even realized.

 > Which one is
 > that will be determined by the order in which the merging process
 > merges the faces, and by the faces themselves -- whether they do or
 > don't define a background color, and whether they do or don't have the
 > :extend bit set.

The merging process will kick in iff we treat the newline character at
the end of line 2 as a stop position.  Since no text property changes
there, I don't see how this could happen.  The only stop position in
this buffer (apart from its beginning and end) is where the comment
starts on line 2.

 >> I agree that we don't want to extend the region's background.  But the
 >> question I raised above still stands.
 >
 > Did I answer it now?

If so, I didn't understand the answer yet.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Thu, 29 Aug 2019 09:45:50 +0200
> 
>  >>   > I think we should simply not merge the background color of the region
>  >>   > face when its extend bit is reset.  Then the merged face will not have
>  >>   > that background color.
>  >>
>  >> Then which background color would we use?  That of the comment was
>  >> lost when setting up the current face for the iterator.
>  >
>  > The one that was there before the region was activated.
> 
> But where do we get that from?

It will come out automatically from the face merging process.

> Consider a three lines C buffer with
> code on the first line and a two-line comment covering the remaining
> lines with the region covering the entire buffer.  Hence, the spaces
> following the comment on the second line should be drawn with the
> background of the comment face.

The comment face doesn't have a background, btw.  And for a good
reason.  But I digress.  Maybe.

> But when the display engine gets there, the respective face was not
> even realized.

It will be realized when the display engine comes to that place.
That's exactly what we have been talking about here, right?  Or maybe
I don't understand what difficulty you are seeing there.

>  > Which one is
>  > that will be determined by the order in which the merging process
>  > merges the faces, and by the faces themselves -- whether they do or
>  > don't define a background color, and whether they do or don't have the
>  > :extend bit set.
> 
> The merging process will kick in iff we treat the newline character at
> the end of line 2 as a stop position.  Since no text property changes
> there, I don't see how this could happen.

It will happen because we call append_space_for_newline and
extend_face_to_end_of_line, and those functions _know_ they are at end
of a line.

>  >> I agree that we don't want to extend the region's background.  But the
>  >> question I raised above still stands.
>  >
>  > Did I answer it now?
> 
> If so, I didn't understand the answer yet.

Maybe I don't understand the question, because I see no problems
here.  Just coding.

Re: Question about display engine

[martin rudalics]

 >> But where do we get that from?
 >
 > It will come out automatically from the face merging process.

You told me before and I'm still afraid that no such automatism
exists.

 >> But when the display engine gets there, the respective face was not
 >> even realized.
 >
 > It will be realized when the display engine comes to that place.
 > That's exactly what we have been talking about here, right?  Or maybe
 > I don't understand what difficulty you are seeing there.

But then "realizing" the face for the spaces at the end of line
becomes much harder: The display engine has to handle _every newline_
in every buffer as _if it were a stop position_ in order to merge all
faces at that position and find out which background to use.

 >> The merging process will kick in iff we treat the newline character at
 >> the end of line 2 as a stop position.  Since no text property changes
 >> there, I don't see how this could happen.
 >
 > It will happen because we call append_space_for_newline and
 > extend_face_to_end_of_line, and those functions _know_ they are at end
 > of a line.

But they would still have to fully realize the face to use by merging
all faces at each newline.  This means we can do away with all those
precalculated extend_background, extend_... bits because they cease to
contribute anything to the solution.

The only practical solution I see is to, instead of extend_background
bits, use extend_background colors.  In my example, this would mean
that when, at the stop position prescribed by the comment start text
property change, the face merger sets the background value of the face
to realize to that of the region and the extend_background value of
the face to realize to that of the comment.  The display engine would
then realize the face for writing the remainder of the line right from
extend_background instead of from background + extend_background.

 > Maybe I don't understand the question, because I see no problems
 > here.  Just coding.

Maybe you're right.  But I still don't see the light at the end of
this tunnel.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Fri, 30 Aug 2019 09:03:06 +0200
> 
>  >> But where do we get that from?
>  >
>  > It will come out automatically from the face merging process.
> 
> You told me before and I'm still afraid that no such automatism
> exists.

It happens as part of face merging.  The attribute that doesn't get
merged causes the same attribute of another face to become the
attribute of the merged face.

>  >> But when the display engine gets there, the respective face was not
>  >> even realized.
>  >
>  > It will be realized when the display engine comes to that place.
>  > That's exactly what we have been talking about here, right?  Or maybe
>  > I don't understand what difficulty you are seeing there.
> 
> But then "realizing" the face for the spaces at the end of line
> becomes much harder: The display engine has to handle _every newline_
> in every buffer as _if it were a stop position_ in order to merge all
> faces at that position and find out which background to use.

I thought we already agreed that there's no other way of having this
feature than to realize a face at each EOL.

>  >> The merging process will kick in iff we treat the newline character at
>  >> the end of line 2 as a stop position.  Since no text property changes
>  >> there, I don't see how this could happen.
>  >
>  > It will happen because we call append_space_for_newline and
>  > extend_face_to_end_of_line, and those functions _know_ they are at end
>  > of a line.
> 
> But they would still have to fully realize the face to use by merging
> all faces at each newline.  This means we can do away with all those
> precalculated extend_background, extend_... bits because they cease to
> contribute anything to the solution.

I don't think I understand what the "precalculated" part refers to.

> The only practical solution I see is to, instead of extend_background
> bits, use extend_background colors.  In my example, this would mean
> that when, at the stop position prescribed by the comment start text
> property change, the face merger sets the background value of the face
> to realize to that of the region and the extend_background value of
> the face to realize to that of the comment.  The display engine would
> then realize the face for writing the remainder of the line right from
> extend_background instead of from background + extend_background.

I don't see how this will help anything.  To remind you: the display
engine manipulates face IDs, it doesn't know anything about the faces
themselves, and in particular cannot magically exchange a face's
background color for some other color.

>  > Maybe I don't understand the question, because I see no problems
>  > here.  Just coding.
> 
> Maybe you're right.  But I still don't see the light at the end of
> this tunnel.

I don't think I see the tunnel.  I thought the issue was resolved when
you posted your summary up-thread.  What am I missing?

Re: Question about display engine

[martin rudalics]

 > I thought we already agreed that there's no other way of having this
 > feature than to realize a face at each EOL.

Then that's where we have been miscommunicating.  The whole idea
behind the extend_background, extend_underline, ... bits was to avoid
the full handle_stop rigmarole at EOL, like finding the overlays and
text properties and processing them in the right order.  All this was
already done at the last "real" stop position encountered by the
display engine and has not changed since then.

The only missing step is to process the :extend attribute of any face
merged.  For the normal background, this is the value of the
:background attribute of the last face that specified that attribute
and was merged in.  For the extended background this is the value of
the :background attribute of the last face that specified that
attribute, had the :extend attribute set and was merged in.

To put this into praxis, the face merger would maintain a shadow copy
of the background value.  That shadow value would not get overwritten
when merging in the :background attribute of a face that does not have
the :extend attribute set.  Eventually, we would wind up with two,
possibly distinct values of the background to realize - one for the
normal and the shadow value for the extended background.

If we wanted to realize both faces - the one to use for the current
face and the one to use for extension - immediately (eagerly as I
coined it earlier), we could do that right after merging terminates.
We probably would set up a pointer to the realized face to use for
extension, so the iterator, when arriving at EOL, would find it right
away and make it current.  That's all.

But we probably don't want to realize the face to use for extension
immediately and do it - as we agreed earlier - lazily when the
iterator arrives at EOL.  In my region/comment example, the comment
could have terminated before or at the EOL where it started, resulting
in a new stop position and we would have needlessly realized an
extended face.  Initially, that's where I wanted the extend_background
etc. bits to kick in, just that I wrongly thought that filling the end
of the line with the background of the default face would be
sufficient ...

Note: We could also try to find out whether there _is_ another stop
position before the next EOL after merging faces and, if there's none,
realize the extended face eagerly, but I'm not sure whether this idea
can be incorporated easily.

 >> But they would still have to fully realize the face to use by merging
 >> all faces at each newline.  This means we can do away with all those
 >> precalculated extend_background, extend_... bits because they cease to
 >> contribute anything to the solution.
 >
 > I don't think I understand what the "precalculated" part refers to.

Is it clear from what I wrote above?  The extend_background, ... bits
would remember the precalculated differences between the attributes
used for normal text and those for the extensions.

 >> The only practical solution I see is to, instead of extend_background
 >> bits, use extend_background colors.  In my example, this would mean
 >> that when, at the stop position prescribed by the comment start text
 >> property change, the face merger sets the background value of the face
 >> to realize to that of the region and the extend_background value of
 >> the face to realize to that of the comment.  The display engine would
 >> then realize the face for writing the remainder of the line right from
 >> extend_background instead of from background + extend_background.
 >
 > I don't see how this will help anything.  To remind you: the display
 > engine manipulates face IDs, it doesn't know anything about the faces
 > themselves, and in particular cannot magically exchange a face's
 > background color for some other color.

But it knows where the background of a realized face is stored and
could easily realize a new face which is the same but for a different
background swapped in.  Hardly rocket science for the display engine.

So what we could do is to simply maintain a vector of the values for
background, underline, etc. calculated at the last real stop position
and whenever the display engine encounters a newline, realize a face
with the values of that vector replacing the current values, use that
face for the spaces at the rest of the line, and restore the old face
for the normal text on the next line (unless we have several newlines
in a row and similar optimizations).

 > I don't think I see the tunnel.  I thought the issue was resolved when
 > you posted your summary up-thread.  What am I missing?

When you saw my proposal to use a number of extend_background,
extend_underline, ... bits, your crystal ball should have complained
that multiple bits are useless because we would have to realize a new
face anyway at EOL.  A single flag (if at all) could have been set by
the face merger to detect the case where an attribute without the
:extend attribute set overwrote an attribute with the :extend
attribute set.  In my region/comment example this would have happened
where the no-extend :background of the region face overwrote the
extend :background of the comment face.  The display engine, whenever
it encountered the flag set at EOL, would have triggered a new face
merge (and avoided it otherwise).

That single flag approach wouldn't penalize the display engine if
attributes were extended by default.  In that case, the re-merging step
at EOL would be needed only in cases as used in my region/comment
example.  But if, as Ergus requested earlier, we did _not_ want to
extend attributes by default, the re-merge penalty would practically
happen at each EOL.

BTW: One problem with Ergus' proposal is that hacks like the one
proposed for Bug#15934 won't work any more.  For that bug, we could
obviously set the :extend attribute of the respective highlight line
face but all instances in the wild using the same hack already would
be affected by our change.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Sat, 31 Aug 2019 09:29:13 +0200
> 
>  > I thought we already agreed that there's no other way of having this
>  > feature than to realize a face at each EOL.
> 
> Then that's where we have been miscommunicating.  The whole idea
> behind the extend_background, extend_underline, ... bits was to avoid
> the full handle_stop rigmarole at EOL, like finding the overlays and
> text properties and processing them in the right order.

We don't need everything from handle_stop, we only need the parts that
affect the face.

> The only missing step is to process the :extend attribute of any face
> merged.  For the normal background, this is the value of the
> :background attribute of the last face that specified that attribute
> and was merged in.  For the extended background this is the value of
> the :background attribute of the last face that specified that
> attribute, had the :extend attribute set and was merged in.
> 
> To put this into praxis, the face merger would maintain a shadow copy
> of the background value.  That shadow value would not get overwritten
> when merging in the :background attribute of a face that does not have
> the :extend attribute set.  Eventually, we would wind up with two,
> possibly distinct values of the background to realize - one for the
> normal and the shadow value for the extended background.

The face merger doesn't maintain any state, so I don't think this is
easily done.

> If we wanted to realize both faces - the one to use for the current
> face and the one to use for extension - immediately (eagerly as I
> coined it earlier), we could do that right after merging terminates.
> We probably would set up a pointer to the realized face to use for
> extension, so the iterator, when arriving at EOL, would find it right
> away and make it current.  That's all.
> 
> But we probably don't want to realize the face to use for extension
> immediately and do it - as we agreed earlier - lazily when the
> iterator arrives at EOL.

If lazy realization is hard to implement, there's nothing wrong with
realizing both faces immediately, at each stop position.

> Note: We could also try to find out whether there _is_ another stop
> position before the next EOL after merging faces and, if there's none,
> realize the extended face eagerly, but I'm not sure whether this idea
> can be incorporated easily.

Right.

>  > I don't see how this will help anything.  To remind you: the display
>  > engine manipulates face IDs, it doesn't know anything about the faces
>  > themselves, and in particular cannot magically exchange a face's
>  > background color for some other color.
> 
> But it knows where the background of a realized face is stored and
> could easily realize a new face which is the same but for a different
> background swapped in.  Hardly rocket science for the display engine.

If we extend faces to keep the information about how to do that, then
yes, this can be done.

> So what we could do is to simply maintain a vector of the values for
> background, underline, etc. calculated at the last real stop position
> and whenever the display engine encounters a newline, realize a face
> with the values of that vector replacing the current values, use that
> face for the spaces at the rest of the line, and restore the old face
> for the normal text on the next line (unless we have several newlines
> in a row and similar optimizations).

That's possible, assuming faces are extended as mentioned above.  But
I'm not sure realizing both faces immediately ("eagerly") will not be
an easier solution.

> BTW: One problem with Ergus' proposal is that hacks like the one
> proposed for Bug#15934 won't work any more.  For that bug, we could
> obviously set the :extend attribute of the respective highlight line
> face but all instances in the wild using the same hack already would
> be affected by our change.

Yes, of course.

Re: Question about display engine

[martin rudalics]

 >> To put this into praxis, the face merger would maintain a shadow copy
 >> of the background value.  That shadow value would not get overwritten
 >> when merging in the :background attribute of a face that does not have
 >> the :extend attribute set.  Eventually, we would wind up with two,
 >> possibly distinct values of the background to realize - one for the
 >> normal and the shadow value for the extended background.
 >
 > The face merger doesn't maintain any state, so I don't think this is
 > easily done.

The background would be stored in an extend_background slot of the
face vector (or maybe even a separate extend_vector) which would
contain the part of the state responsible for handling the background.

 >> Note: We could also try to find out whether there _is_ another stop
 >> position before the next EOL after merging faces and, if there's none,
 >> realize the extended face eagerly, but I'm not sure whether this idea
 >> can be incorporated easily.
 >
 > Right.

IIUC next_interval is newline agnostic.  The 'auto-composition-mode'
check does look for a newline within the next 1000 characters (around
line 3761 of xdisp.c) so we maybe could use that.

 >> BTW: One problem with Ergus' proposal is that hacks like the one
 >> proposed for Bug#15934 won't work any more.  For that bug, we could
 >> obviously set the :extend attribute of the respective highlight line
 >> face but all instances in the wild using the same hack already would
 >> be affected by our change.
 >
 > Yes, of course.

So we should probably provide a :no-extend attribute, extend face
attributes by default and explicitly not extend some attributes like
underline and box.

martin

Re: Question about display engine

[Ergus]

Hi Marting and Eli:

I have been thinking about this a little bit more and it seems like we
are trying to replicate a behavior that is already there.

If we had an extra face pointer within the face struct initialized to
null (or default face), and then when we merge and the extend attribute is set we
initialize it to the current merged face and we transmit that pointer
from merge to merge until we have a new face with :extend, so we merge
then those two... I think when we reach the end of the line we will
have there already all the information we need.

Lets say we have a sequence of consecutive merges (a,b,c,d,e); but only
b and d have :extend.

At the end we will have in the merges:

(a, a+b, a+b+c, a+b+c+d, a+b+c+d+e)

but with this pointers in them:

(nul, b,     b,   b + d,     b + d)

Which for me it seems to be what we expect to have right?

Does it makes sense?

On Sun, Sep 01, 2019 at 10:14:06AM +0200, martin rudalics wrote:
>>> To put this into praxis, the face merger would maintain a shadow copy
>>> of the background value.  That shadow value would not get overwritten
>>> when merging in the :background attribute of a face that does not have
>>> the :extend attribute set.  Eventually, we would wind up with two,
>>> possibly distinct values of the background to realize - one for the
>>> normal and the shadow value for the extended background.
>>
>> The face merger doesn't maintain any state, so I don't think this is
>> easily done.
>
>The background would be stored in an extend_background slot of the
>face vector (or maybe even a separate extend_vector) which would
>contain the part of the state responsible for handling the background.
>
>>> Note: We could also try to find out whether there _is_ another stop
>>> position before the next EOL after merging faces and, if there's none,
>>> realize the extended face eagerly, but I'm not sure whether this idea
>>> can be incorporated easily.
>>
>> Right.
>
>IIUC next_interval is newline agnostic.  The 'auto-composition-mode'
>check does look for a newline within the next 1000 characters (around
>line 3761 of xdisp.c) so we maybe could use that.
>
>>> BTW: One problem with Ergus' proposal is that hacks like the one
>>> proposed for Bug#15934 won't work any more.  For that bug, we could
>>> obviously set the :extend attribute of the respective highlight line
>>> face but all instances in the wild using the same hack already would
>>> be affected by our change.
>>
>> Yes, of course.
>
>So we should probably provide a :no-extend attribute, extend face
>attributes by default and explicitly not extend some attributes like
>underline and box.
>
>martin
>

Re: Question about display engine

[martin rudalics]

 > I have been thinking about this a little bit more and it seems like we
 > are trying to replicate a behavior that is already there.
 >
 > If we had an extra face pointer within the face struct initialized to
 > null (or default face), and then when we merge and the extend attribute is set we
 > initialize it to the current merged face and we transmit that pointer
 > from merge to merge until we have a new face with :extend, so we merge
 > then those two... I think when we reach the end of the line we will
 > have there already all the information we need.
 >
 > Lets say we have a sequence of consecutive merges (a,b,c,d,e); but only
 > b and d have :extend.
 >
 > At the end we will have in the merges:
 >
 > (a, a+b, a+b+c, a+b+c+d, a+b+c+d+e)
 >
 > but with this pointers in them:
 >
 > (nul, b,     b,   b + d,     b + d)
 >
 > Which for me it seems to be what we expect to have right?
 >
 > Does it makes sense?

I think so.  Now the realized face for normal text is a+b+c+d+e and
the display engine would use it right away.  When the display engine
encounters a newline character it has to "switch" from a+b+c+d+e to
b+d.  But it has to assure that b+d _is a realized face_ because a
reference to that face is the output expected from the display engine.
Any additional pointers like yours would be ignored after that.

Do you agree so far?  If so, then the question remains _when_ to
realize b+d.  Eagerly, when the face merger has done its work, or
lazily when the display engine encounters the newline.  Eagerly has
the advantage that the display engine has all the faces already
realized.  Its disadvantage is that when a new stop position is found
before the EOL, the b+d face was realized needlessly.  Do you still
agree?

If so then we might consider the following optimization: The extra
face pointer in each face is no more needed after the display engine
has processed the face.  So we could build a "shadow" face for b+d,
keep it in a separate, static face structure and realize a face from
that structure whenever we want to (eagerly or lazily).  The display
engine, when it finds a pointer to such an extra face at EOL, uses it
(maybe realizing it on the fly).

Also, the merger could nullify the extra face if it's the same as the
normal one, that is no single merge step had an :extend false value
override a former :extend true value.  So the display engine would
know beforehand that it does not have to change the current face.

Last but not least, the display engine has to, after processing the
spaces at the EOL from b+d, restore the a+b+c+d+e face as its current
face.  So we have two static pointers to keep around: One for the b+d
face structure (or its already realized face) while the display engine
processes normal text and one for the a+b+c+d+e realized face while
the display engine processes the spaces at EOL.  Can you still agree?
Can you imagine any difficulties with implementing such an approach?

And we would have happily put to rest all those crazy extend_... bits
I proposed earlier.

martin

Re: Question about display engine

[Ergus]

On Mon, Sep 02, 2019 at 10:36:34AM +0200, martin rudalics wrote:
>> I have been thinking about this a little bit more and it seems like we
>> are trying to replicate a behavior that is already there.
>>
>> If we had an extra face pointer within the face struct initialized to
>> null (or default face), and then when we merge and the extend attribute is set we
>> initialize it to the current merged face and we transmit that pointer
>> from merge to merge until we have a new face with :extend, so we merge
>> then those two... I think when we reach the end of the line we will
>> have there already all the information we need.
>>
>> Lets say we have a sequence of consecutive merges (a,b,c,d,e); but only
>> b and d have :extend.
>>
>> At the end we will have in the merges:
>>
>> (a, a+b, a+b+c, a+b+c+d, a+b+c+d+e)
>>
>> but with this pointers in them:
>>
>> (nul, b,     b,   b + d,     b + d)
>>
>> Which for me it seems to be what we expect to have right?
>>
>> Does it makes sense?
>
>I think so.  Now the realized face for normal text is a+b+c+d+e and
>the display engine would use it right away.  When the display engine
>encounters a newline character it has to "switch" from a+b+c+d+e to
>b+d.  But it has to assure that b+d _is a realized face_ because a
>reference to that face is the output expected from the display engine.
>Any additional pointers like yours would be ignored after that.
>
>Do you agree so far?  If so, then the question remains _when_ to
>realize b+d.  Eagerly, when the face merger has done its work, or
>lazily when the display engine encounters the newline.  Eagerly has
>the advantage that the display engine has all the faces already
>realized.  Its disadvantage is that when a new stop position is found
>before the EOL, the b+d face was realized needlessly.  Do you still
>agree?
>
I think any of them is still a good trade-off because it is very
use-case specific. When coding and with the actual wide screens; in most
of the cases the lines are always extended, So I don't think there will
be a significant difference.

In any case we could implement any of them more or less with the same
complexity...
>
>If so then we might consider the following optimization: The extra
>face pointer in each face is no more needed after the display engine
>has processed the face.  So we could build a "shadow" face for b+d,
>keep it in a separate, static face structure and realize a face from
>that structure whenever we want to (eagerly or lazily).  The display
>engine, when it finds a pointer to such an extra face at EOL, uses it
>(maybe realizing it on the fly).
>
>Also, the merger could nullify the extra face if it's the same as the
>normal one, that is no single merge step had an :extend false value
>override a former :extend true value.  So the display engine would
>know beforehand that it does not have to change the current face.
>
Yes; in my initial proposed approach the local pointer will be null in
that case.

>Last but not least, the display engine has to, after processing the
>spaces at the EOL from b+d, restore the a+b+c+d+e face as its current
>face.  So we have two static pointers to keep around: One for the b+d
>face structure (or its already realized face) while the display engine
>processes normal text and one for the a+b+c+d+e realized face while
>the display engine processes the spaces at EOL.  Can you still agree?

In the display engine we do this very
frequently. As extend_face_to_end_of_line is very localized we just need
to save a pointer to a+b+c+d on the beginning of the function and
restore it at the end.

It is a bit more complex than that because the code for gui and TUI is
in different portions of an if condition, but this part is almost the
same I made in the first patch I proposed.

>Can you imagine any difficulties with implementing such an approach?
>
I only see small details like that in some cases we need to
reinitialize the extend face and the default-face value maybe is not the
right choice in all the cases (I need to look into it a bit more).

And that in X there is some extra code (somewhere) to extend the
background color using the color in the last inserted glyph (it is
something happening by default without calling even
extend_face_to_end_of_line). That code should be removed after this
change; but I don't know where is it. But for sure Eli will tell.

>And we would have happily put to rest all those crazy extend_... bits
>I proposed earlier.
>
>martin
>

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 2 Sep 2019 13:05:04 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> >If so then we might consider the following optimization: The extra
> >face pointer in each face is no more needed after the display engine
> >has processed the face.  So we could build a "shadow" face for b+d,
> >keep it in a separate, static face structure and realize a face from
> >that structure whenever we want to (eagerly or lazily).  The display
> >engine, when it finds a pointer to such an extra face at EOL, uses it
> >(maybe realizing it on the fly).
> >
> >Also, the merger could nullify the extra face if it's the same as the
> >normal one, that is no single merge step had an :extend false value
> >override a former :extend true value.  So the display engine would
> >know beforehand that it does not have to change the current face.
> >
> Yes; in my initial proposed approach the local pointer will be null in
> that case.

I don't think I understand why you are talking about face pointers.
The iterator doesn't keep any face pointers, it keeps face IDs (which
allow to obtain the face pointer, when needed, by using FACE_FROM_ID).

So all you need is to have another face ID member in the iterator, to
be used for extending past EOL; depending on how the :extend bits are
set, that face ID may or may not be identical to the "normal" face ID,
the one we have now and use for buffer text.

> >Last but not least, the display engine has to, after processing the
> >spaces at the EOL from b+d, restore the a+b+c+d+e face as its current
> >face.  So we have two static pointers to keep around: One for the b+d
> >face structure (or its already realized face) while the display engine
> >processes normal text and one for the a+b+c+d+e realized face while
> >the display engine processes the spaces at EOL.  Can you still agree?
> 
> In the display engine we do this very
> frequently. As extend_face_to_end_of_line is very localized we just need
> to save a pointer to a+b+c+d on the beginning of the function and
> restore it at the end.

Again, not a pointer: a face ID.  And yes, we have saved_face_id
member of the iterator for this purpose.

> And that in X there is some extra code (somewhere) to extend the
> background color using the color in the last inserted glyph (it is
> something happening by default without calling even
> extend_face_to_end_of_line). That code should be removed after this
> change; but I don't know where is it. But for sure Eli will tell.

It's just that we clear to EOL with the last background color, and
avoid doing that if the background color is identical to the frame's
background.  I don't think anything will have to be changed there, but
we will see (I hope).

Re: Question about display engine

[Ergus]

Hi Eli and martin.

Here I attach a patch minimal proof of concept for the last solution I
proposed.

At the moment it is not showing the color extension to the end of the
line, but I cannot make it extend because it seems like the merge
function I added is never executed (which makes no sense but it does not
print the messages I added with printf.)

Could any of you give a look to the patch to detect what is failing at
least to triger the merge and extend?
Probably the initialization. (which btw the lisp glue code may be buggy
for sure.)

Please, any help is very welcome.

On Mon, Sep 02, 2019 at 07:18:19PM +0300, Eli Zaretskii wrote:
>> Date: Mon, 2 Sep 2019 13:05:04 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>>
>> >If so then we might consider the following optimization: The extra
>> >face pointer in each face is no more needed after the display engine
>> >has processed the face.  So we could build a "shadow" face for b+d,
>> >keep it in a separate, static face structure and realize a face from
>> >that structure whenever we want to (eagerly or lazily).  The display
>> >engine, when it finds a pointer to such an extra face at EOL, uses it
>> >(maybe realizing it on the fly).
>> >
>> >Also, the merger could nullify the extra face if it's the same as the
>> >normal one, that is no single merge step had an :extend false value
>> >override a former :extend true value.  So the display engine would
>> >know beforehand that it does not have to change the current face.
>> >
>> Yes; in my initial proposed approach the local pointer will be null in
>> that case.
>
>I don't think I understand why you are talking about face pointers.
>The iterator doesn't keep any face pointers, it keeps face IDs (which
>allow to obtain the face pointer, when needed, by using FACE_FROM_ID).
>
>So all you need is to have another face ID member in the iterator, to
>be used for extending past EOL; depending on how the :extend bits are
>set, that face ID may or may not be identical to the "normal" face ID,
>the one we have now and use for buffer text.
>
>> >Last but not least, the display engine has to, after processing the
>> >spaces at the EOL from b+d, restore the a+b+c+d+e face as its current
>> >face.  So we have two static pointers to keep around: One for the b+d
>> >face structure (or its already realized face) while the display engine
>> >processes normal text and one for the a+b+c+d+e realized face while
>> >the display engine processes the spaces at EOL.  Can you still agree?
>>
>> In the display engine we do this very
>> frequently. As extend_face_to_end_of_line is very localized we just need
>> to save a pointer to a+b+c+d on the beginning of the function and
>> restore it at the end.
>
>Again, not a pointer: a face ID.  And yes, we have saved_face_id
>member of the iterator for this purpose.
>
>> And that in X there is some extra code (somewhere) to extend the
>> background color using the color in the last inserted glyph (it is
>> something happening by default without calling even
>> extend_face_to_end_of_line). That code should be removed after this
>> change; but I don't know where is it. But for sure Eli will tell.
>
>It's just that we clear to EOL with the last background color, and
>avoid doing that if the background color is identical to the frame's
>background.  I don't think anything will have to be changed there, but
>we will see (I hope).
>

Re: Question about display engine

[martin rudalics]

 > Could any of you give a look to the patch

I cannot build Emacs on Windows with the patch due to


   CC       w32select.o
../../src/w32term.c: In function 'w32_draw_glyph_string':
../../src/w32term.c:2482:20: error: 'struct face' has no member named 'underline_p'; did you mean 'underline'?
        if (s->face->underline_p)
                     ^~~~~~~~~~~
                     underline
../../src/w32term.c:2484:24: error: 'struct face' has no member named 'underline_type'; did you mean 'underline_color'?
            if (s->face->underline_type == FACE_UNDER_WAVE)
                         ^~~~~~~~~~~~~~
                         underline_color
../../src/w32term.c:2495:29: error: 'struct face' has no member named 'underline_type'; did you mean 'underline_color'?
            else if (s->face->underline_type == FACE_UNDER_LINE)
                              ^~~~~~~~~~~~~~
                              underline_color
../../src/w32term.c:2500:45: error: 'struct face' has no member named 'underline_p'; did you mean 'underline'?
                if (s->prev && s->prev->face->underline_p
                                              ^~~~~~~~~~~
                                              underline
../../src/w32term.c:2501:23: error: 'struct face' has no member named 'underline_type'; did you mean 'underline_color'?
      && s->prev->face->underline_type == FACE_UNDER_LINE)
                        ^~~~~~~~~~~~~~
                        underline_color
make[1]: *** [Makefile:402: w32term.o] Fehler 1
make[1]: *** Es wird auf noch nicht beendete Prozesse gewartet....
make[1]: Verzeichnis „/c/emacs/trunk/non-64/src“ wird verlassen
make: *** [Makefile:424: src] Fehler 2


and since I don't understand the underline_p/_type rationale I have no
good idea how to proceed.

 > to detect what is failing at
 > least to triger the merge and extend?
 > Probably the initialization. (which btw the lisp glue code may be buggy
 > for sure.)

I'd run Emacs under gdb to find out whether merge_extend_glyph_face
gets called in the first place.  And if it doesn't get called, I would
continue investigating the places where it should get called.

martin

Re: Question about display engine

[Ergus]

On Tue, Sep 03, 2019 at 10:45:46AM +0200, martin rudalics wrote:
>> Could any of you give a look to the patch
>
>I cannot build Emacs on Windows with the patch due to
>
>
>  CC       w32select.o
>../../src/w32term.c: In function 'w32_draw_glyph_string':
>../../src/w32term.c:2482:20: error: 'struct face' has no member named 'underline_p'; did you mean 'underline'?
>       if (s->face->underline_p)
>                    ^~~~~~~~~~~
>                    underline
>../../src/w32term.c:2484:24: error: 'struct face' has no member named 'underline_type'; did you mean 'underline_color'?
>           if (s->face->underline_type == FACE_UNDER_WAVE)
>                        ^~~~~~~~~~~~~~
>                        underline_color
>../../src/w32term.c:2495:29: error: 'struct face' has no member named 'underline_type'; did you mean 'underline_color'?
>           else if (s->face->underline_type == FACE_UNDER_LINE)
>                             ^~~~~~~~~~~~~~
>                             underline_color
>../../src/w32term.c:2500:45: error: 'struct face' has no member named 'underline_p'; did you mean 'underline'?
>               if (s->prev && s->prev->face->underline_p
>                                             ^~~~~~~~~~~
>                                             underline
>../../src/w32term.c:2501:23: error: 'struct face' has no member named 'underline_type'; did you mean 'underline_color'?
>     && s->prev->face->underline_type == FACE_UNDER_LINE)
>                       ^~~~~~~~~~~~~~
>                       underline_color
>make[1]: *** [Makefile:402: w32term.o] Fehler 1
>make[1]: *** Es wird auf noch nicht beendete Prozesse gewartet....
>make[1]: Verzeichnis ???/c/emacs/trunk/non-64/src??? wird verlassen
>make: *** [Makefile:424: src] Fehler 2
>
>
>and since I don't understand the underline_p/_type rationale I have no
>good idea how to proceed.
>
Ohh sorry. My bad I made a code simplify and removed this member. I will
fix it now.

>> to detect what is failing at
>> least to triger the merge and extend?
>> Probably the initialization. (which btw the lisp glue code may be buggy
>> for sure.)
>
>I'd run Emacs under gdb to find out whether merge_extend_glyph_face
>gets called in the first place.  And if it doesn't get called, I would
>continue investigating the places where it should get called.
>
>martin
>
>
After a while I made it to be called, so, forget the patch I will email
a different one in a while.

I found also that we where very concerned about what happen when
merging, but there was not any comment about face_id changes in other
cases (like reassign).

For example, when we select a word in the middle of a line. The
extend_face will be set to the region face when the iterator iterates
throw the word, but then it shouldn't be extended at EOL.

So we are dealing with a 2D problem here. Iterate on X (the line) vs
merging on Y (the face).

So I am not sure we have a criteria about when to propagate the extend
in both dimensions; because as I see in most of the cases the face_id in
X changes due to reassign associated to text properties. So we have been
talking all the time about the Y problem?

maybe we should test only the last face in the line before EOL to check
for the extend flag; but then the extend_face_id will be again an
intrinsic parameter of that face.

I feel I am missing something for sure.

Re: Question about display engine

[martin rudalics]

 > I found also that we where very concerned about what happen when
 > merging, but there was not any comment about face_id changes in other
 > cases (like reassign).

I don't know what reassign does but don't we have to redo all windows
in that case?

 > For example, when we select a word in the middle of a line. The
 > extend_face will be set to the region face when the iterator iterates
 > throw the word, but then it shouldn't be extended at EOL.

Why would that be a problem?  The last stop position we care about
here is the end of the word (because that's where the region text
property terminates) and the region face should be long forgotten at
the end of the line.  Or what am I missing?

 > So we are dealing with a 2D problem here. Iterate on X (the line) vs
 > merging on Y (the face).

I don't see this as two-dimensional.  The iterator linearly iterates
through the buffer, re-contemplating faces at each stop position.

 > So I am not sure we have a criteria about when to propagate the extend
 > in both dimensions; because as I see in most of the cases the face_id in
 > X changes due to reassign associated to text properties. So we have been
 > talking all the time about the Y problem?
 >
 > maybe we should test only the last face in the line before EOL to check
 > for the extend flag; but then the extend_face_id will be again an
 > intrinsic parameter of that face.

The extend face to consider at any EOL is the extend face calculated
at the last stop position before that EOL (which sometimes might be
the beginning of the buffer).

martin

Re: Question about display engine

[Eli Zaretskii]

> Date: Tue, 3 Sep 2019 13:23:37 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> I found also that we where very concerned about what happen when
> merging, but there was not any comment about face_id changes in other
> cases (like reassign).
> 
> For example, when we select a word in the middle of a line. The
> extend_face will be set to the region face when the iterator iterates
> throw the word, but then it shouldn't be extended at EOL.

We need to recalculate extend_face_id at the same places where we
recalculate face_id.

> maybe we should test only the last face in the line before EOL to check
> for the extend flag; but then the extend_face_id will be again an
> intrinsic parameter of that face.

How do you know it's "the last face"?

Re: Question about display engine

[Ergus via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 2920 bytes --]

PATCH: (relative to  dd162a3f2264940e3e329d0bf)

On Mon, Sep 02, 2019 at 07:18:19PM +0300, Eli Zaretskii wrote:
>> Date: Mon, 2 Sep 2019 13:05:04 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>>
>> >If so then we might consider the following optimization: The extra
>> >face pointer in each face is no more needed after the display engine
>> >has processed the face.  So we could build a "shadow" face for b+d,
>> >keep it in a separate, static face structure and realize a face from
>> >that structure whenever we want to (eagerly or lazily).  The display
>> >engine, when it finds a pointer to such an extra face at EOL, uses it
>> >(maybe realizing it on the fly).
>> >
>> >Also, the merger could nullify the extra face if it's the same as the
>> >normal one, that is no single merge step had an :extend false value
>> >override a former :extend true value.  So the display engine would
>> >know beforehand that it does not have to change the current face.
>> >
>> Yes; in my initial proposed approach the local pointer will be null in
>> that case.
>
>I don't think I understand why you are talking about face pointers.
>The iterator doesn't keep any face pointers, it keeps face IDs (which
>allow to obtain the face pointer, when needed, by using FACE_FROM_ID).
>
>So all you need is to have another face ID member in the iterator, to
>be used for extending past EOL; depending on how the :extend bits are
>set, that face ID may or may not be identical to the "normal" face ID,
>the one we have now and use for buffer text.
>
>> >Last but not least, the display engine has to, after processing the
>> >spaces at the EOL from b+d, restore the a+b+c+d+e face as its current
>> >face.  So we have two static pointers to keep around: One for the b+d
>> >face structure (or its already realized face) while the display engine
>> >processes normal text and one for the a+b+c+d+e realized face while
>> >the display engine processes the spaces at EOL.  Can you still agree?
>>
>> In the display engine we do this very
>> frequently. As extend_face_to_end_of_line is very localized we just need
>> to save a pointer to a+b+c+d on the beginning of the function and
>> restore it at the end.
>
>Again, not a pointer: a face ID.  And yes, we have saved_face_id
>member of the iterator for this purpose.
>
>> And that in X there is some extra code (somewhere) to extend the
>> background color using the color in the last inserted glyph (it is
>> something happening by default without calling even
>> extend_face_to_end_of_line). That code should be removed after this
>> change; but I don't know where is it. But for sure Eli will tell.
>
>It's just that we clear to EOL with the last background color, and
>avoid doing that if the background color is identical to the frame's
>background.  I don't think anything will have to be changed there, but
>we will see (I hope).
>

[-- Attachment #2: extend.patch --]
[-- Type: text/plain, Size: 31218 bytes --]

diff --git a/lisp/cus-face.el b/lisp/cus-face.el
index d73bce42c3..5a49a81043 100644
--- a/lisp/cus-face.el
+++ b/lisp/cus-face.el
@@ -233,7 +233,11 @@ custom-face-attributes
 	     (file :tag "File"
 		   :help-echo "Name of bitmap file."
 		   :must-match t)))
-
+    (:extend
+     (choice :tag "Extend"
+	     :help-echo "Control whether attributes should be extended after EOL."
+	     (const :tag "Off" nil)
+	     (const :tag "On" t)))
     (:inherit
      (repeat :tag "Inherit"
 	     :help-echo "List of faces to inherit attributes from."
diff --git a/lisp/faces.el b/lisp/faces.el
index 5193c216d0..11e42c5380 100644
--- a/lisp/faces.el
+++ b/lisp/faces.el
@@ -342,6 +342,7 @@ face-x-resources
     (:box (".attributeBox" . "Face.AttributeBox"))
     (:underline (".attributeUnderline" . "Face.AttributeUnderline"))
     (:inverse-video (".attributeInverse" . "Face.AttributeInverse"))
+    (:extend (".attributeExtend" . "Face.AttributeExtend"))
     (:stipple
      (".attributeStipple" . "Face.AttributeStipple")
      (".attributeBackgroundPixmap" . "Face.AttributeBackgroundPixmap"))
@@ -594,6 +595,13 @@ face-italic-p
   (let ((italic (face-attribute face :slant frame inherit)))
     (memq italic '(italic oblique))))
 
+(defun face-extend-p (face &optional frame inherit)
+ "Return non-nil if FACE specifies a non-nil extend.
+If the optional argument FRAME is given, report on face FACE in that frame.
+If FRAME is t, report on the defaults for face FACE (for new frames).
+If FRAME is omitted or nil, use the selected frame.
+Optional argument INHERIT is passed to `face-attribute'."
+ (eq (face-attribute face :extend frame inherit) t))
 
 \f
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -760,6 +768,11 @@ set-face-attribute
 `:height', `:weight', and `:slant' may also be set in one step
 from an X font name:
 
+`:extend'
+
+VALUE specifies whether the FACE should be extended after EOL.
+VALUE must be one of t or nil.
+
 `:font'
 
 Set font-related face attributes from VALUE.
@@ -979,6 +992,18 @@ set-face-italic
 
 (define-obsolete-function-alias 'set-face-italic-p 'set-face-italic "24.4")
 
+(defun set-face-extend (face extend-p &optional frame)
+  "Specify whether face FACE should be extended.
+EXTEND-P nil means FACE explicitly doesn't extend after EOL.
+EXTEND-P t means FACE extends after EOL.
+
+FRAME nil or not specified means change face on all frames.
+Use `set-face-attribute' to \"unspecify\" underlining."
+  (interactive
+   (let ((list (read-face-and-attribute :extend)))
+     (list (car list) (if (cadr list) t))))
+  (set-face-attribute face frame :extend extend-p))
+
 
 (defalias 'set-face-background-pixmap 'set-face-stipple)
 
@@ -1102,7 +1127,7 @@ face-valid-attribute-values
 	   (:slant
 	    (mapcar #'(lambda (x) (cons (symbol-name (aref x 1)) (aref x 1)))
 		    font-slant-table))
-	   (:inverse-video
+	   ((or :inverse-video :extend)
 	    (mapcar #'(lambda (x) (cons (symbol-name x) x))
 		    (internal-lisp-face-attribute-values attribute)))
            ((or :underline :overline :strike-through :box)
@@ -1147,12 +1172,13 @@ face-attribute-name-alist
     (:slant . "slant")
     (:underline . "underline")
     (:overline . "overline")
+    (:extend . "extend")
     (:strike-through . "strike-through")
     (:box . "box")
     (:inverse-video . "inverse-video display")
     (:foreground . "foreground color")
     (:background . "background color")
-    (:stipple . "background stipple")
+    (:background . "background color")
     (:inherit . "inheritance"))
   "An alist of descriptive names for face attributes.
 Each element has the form (ATTRIBUTE-NAME . DESCRIPTION) where
@@ -2432,24 +2458,24 @@ highlight
 ;; if background is light.
 (defface region
   '((((class color) (min-colors 88) (background dark))
-     :background "blue3")
+     :background "blue3" :extend t)
     (((class color) (min-colors 88) (background light) (type gtk))
      :distant-foreground "gtk_selection_fg_color"
-     :background "gtk_selection_bg_color")
+     :background "gtk_selection_bg_color" :extend t)
     (((class color) (min-colors 88) (background light) (type ns))
      :distant-foreground "ns_selection_fg_color"
-     :background "ns_selection_bg_color")
+     :background "ns_selection_bg_color" :extend t)
     (((class color) (min-colors 88) (background light))
-     :background "lightgoldenrod2")
+     :background "lightgoldenrod2" :extend t)
     (((class color) (min-colors 16) (background dark))
-     :background "blue3")
+     :background "blue3" :extend t)
     (((class color) (min-colors 16) (background light))
-     :background "lightgoldenrod2")
+     :background "lightgoldenrod2" :extend t)
     (((class color) (min-colors 8))
-     :background "blue" :foreground "white")
+     :background "blue" :foreground "white" :extend t)
     (((type tty) (class mono))
      :inverse-video t)
-    (t :background "gray"))
+    (t :background "gray" :extend t))
   "Basic face for highlighting the region."
   :version "21.1"
   :group 'basic-faces)
diff --git a/src/dispextern.h b/src/dispextern.h
index 05f199ff35..e09d36944a 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -1564,6 +1564,7 @@ #define FONT_TOO_HIGH(ft)						\
   LFACE_INHERIT_INDEX,
   LFACE_FONTSET_INDEX,
   LFACE_DISTANT_FOREGROUND_INDEX,
+  LFACE_EXTEND_INDEX,
   LFACE_VECTOR_SIZE
 };
 
@@ -1589,6 +1590,7 @@ #define FONT_TOO_HIGH(ft)						\
 
 enum face_underline_type
 {
+  FACE_NO_UNDERLINE = 0,
   FACE_UNDER_LINE,
   FACE_UNDER_WAVE
 };
@@ -1632,11 +1634,9 @@ #define FONT_TOO_HIGH(ft)						\
   /* Pixel value or color index of background color.  */
   unsigned long background;
 
-  /* Pixel value or color index of underline color.  */
+  /* Pixel value or color index of underline, overlined,
+     strike-through, or box color.  */
   unsigned long underline_color;
-
-  /* Pixel value or color index of overlined, strike-through, or box
-     color.  */
   unsigned long overline_color;
   unsigned long strike_through_color;
   unsigned long box_color;
@@ -1663,7 +1663,7 @@ #define FONT_TOO_HIGH(ft)						\
   ENUM_BF (face_box_type) box : 2;
 
   /* Style of underlining. */
-  ENUM_BF (face_underline_type) underline_type : 1;
+  ENUM_BF (face_underline_type) underline : 2;
 
   /* If `box' above specifies a 3D type, true means use box_color for
      drawing shadows.  */
@@ -1671,7 +1671,6 @@ #define FONT_TOO_HIGH(ft)						\
 
   /* Non-zero if text in this face should be underlined, overlined,
      strike-through or have a box drawn around it.  */
-  bool_bf underline_p : 1;
   bool_bf overline_p : 1;
   bool_bf strike_through_p : 1;
 
@@ -1681,14 +1680,10 @@ #define FONT_TOO_HIGH(ft)						\
   bool_bf foreground_defaulted_p : 1;
   bool_bf background_defaulted_p : 1;
 
-  /* True means that either no color is specified for underlining or that
-     the specified color couldn't be loaded.  Use the foreground
-     color when drawing in that case. */
-  bool_bf underline_defaulted_p : 1;
-
   /* True means that either no color is specified for the corresponding
      attribute or that the specified color couldn't be loaded.
      Use the foreground color when drawing in that case. */
+  bool_bf underline_defaulted_p : 1;
   bool_bf overline_color_defaulted_p : 1;
   bool_bf strike_through_color_defaulted_p : 1;
   bool_bf box_color_defaulted_p : 1;
@@ -2448,6 +2443,9 @@ #define OVERLAY_STRING_CHUNK_SIZE 16
   /* Face to use.  */
   int face_id;
 
+  /* Face to extend at EOL/  */
+  int extend_face_id;
+
   /* Setting of buffer-local variable selective-display-ellipses.  */
   bool_bf selective_display_ellipsis_p : 1;
 
diff --git a/src/xdisp.c b/src/xdisp.c
index 75bc536cb9..015a80af42 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -7141,18 +7141,50 @@ lookup_glyphless_char_display (int c, struct it *it)
 }
 
 /* Merge escape glyph face and cache the result.  */
+static struct frame *last_extend_glyph_frame = NULL;
+static int last_extend_glyph_face_id = (1 << FACE_ID_BITS);
+static int last_extend_glyph_merged_face_id = 0;
 
+static int
+merge_extend_glyph_face (struct it *it)
+{
+  int extend_face_id = it->extend_face_id;
+
+  struct face *face = FACE_FROM_ID (it->f, it->extend_face_id);
+  fprintf(stderr, "extending1!!!\n");
+
+  if (!NILP (face->lface[LFACE_EXTEND_INDEX]))
+    {
+      fprintf(stderr,"extending2!!!\n");
+
+      if (it->f == last_extend_glyph_frame
+          && it->extend_face_id == last_extend_glyph_face_id)
+	return last_extend_glyph_merged_face_id;
+
+      /* Merge the `glyphless-char' face into the current face.  */
+      extend_face_id = merge_faces (it->w, Qt, it->face_id, extend_face_id);
+      last_extend_glyph_frame = it->f;
+      last_extend_glyph_face_id = it->extend_face_id;
+      last_extend_glyph_merged_face_id = extend_face_id;
+    }
+  return extend_face_id;
+}
+
+
+/* Merge escape glyph face and cache the result.  */
 static struct frame *last_escape_glyph_frame = NULL;
 static int last_escape_glyph_face_id = (1 << FACE_ID_BITS);
 static int last_escape_glyph_merged_face_id = 0;
 
 static int
-merge_escape_glyph_face (struct it *it)
+merge_escape_glyph_face (struct it *it, int lface_id)
 {
   int face_id;
 
-  if (it->f == last_escape_glyph_frame
-      && it->face_id == last_escape_glyph_face_id)
+  if (lface_id)
+    face_id = merge_faces (it->w, Qt, lface_id, it->face_id);
+  else if (it->f == last_escape_glyph_frame
+           && it->face_id == last_escape_glyph_face_id)
     face_id = last_escape_glyph_merged_face_id;
   else
     {
@@ -7162,6 +7194,8 @@ merge_escape_glyph_face (struct it *it)
       last_escape_glyph_face_id = it->face_id;
       last_escape_glyph_merged_face_id = face_id;
     }
+
+  merge_extend_glyph_face (it);
   return face_id;
 }
 
@@ -7187,9 +7221,12 @@ merge_glyphless_glyph_face (struct it *it)
       last_glyphless_glyph_face_id = it->face_id;
       last_glyphless_glyph_merged_face_id = face_id;
     }
+
+  merge_extend_glyph_face (it);
   return face_id;
 }
 
+
 /* Forget the `escape-glyph' and `glyphless-char' faces.  This should
    be called before redisplaying windows, and when the frame's face
    cache is freed.  */
@@ -7355,9 +7392,7 @@ get_next_display_element (struct it *it)
 		      lface_id = GLYPH_CODE_FACE (gc);
 		    }
 
-		  face_id = (lface_id
-			     ? merge_faces (it->w, Qt, lface_id, it->face_id)
-			     : merge_escape_glyph_face (it));
+		  face_id = merge_escape_glyph_face (it, lface_id);
 
 		  XSETINT (it->ctl_chars[0], g);
 		  XSETINT (it->ctl_chars[1], c ^ 0100);
@@ -7373,6 +7408,8 @@ get_next_display_element (struct it *it)
 		  /* Merge `nobreak-space' into the current face.  */
 		  face_id = merge_faces (it->w, Qnobreak_space, 0,
 					 it->face_id);
+
+		  merge_extend_glyph_face (it);
 		  XSETINT (it->ctl_chars[0], ' ');
 		  ctl_len = 1;
 		  goto display_control;
@@ -7385,7 +7422,10 @@ get_next_display_element (struct it *it)
 		{
 		  /* Merge `nobreak-space' into the current face.  */
 		  face_id = merge_faces (it->w, Qnobreak_hyphen, 0,
-					 it->face_id);
+		                         it->face_id);
+
+		  merge_extend_glyph_face (it);
+
 		  XSETINT (it->ctl_chars[0], '-');
 		  ctl_len = 1;
 		  goto display_control;
@@ -7403,12 +7443,10 @@ get_next_display_element (struct it *it)
 		  lface_id = GLYPH_CODE_FACE (gc);
 		}
 
-	      face_id = (lface_id
-			 ? merge_faces (it->w, Qt, lface_id, it->face_id)
-			 : merge_escape_glyph_face (it));
+	      face_id = merge_escape_glyph_face (it, lface_id);
+	      
 
 	      /* Draw non-ASCII space/hyphen with escape glyph: */
-
 	      if (nonascii_space_p || nonascii_hyphen_p)
 		{
 		  XSETINT (it->ctl_chars[0], escape_glyph);
@@ -8032,8 +8070,11 @@ next_element_from_display_vector (struct it *it)
 	{
 	  int lface_id = GLYPH_CODE_FACE (gc);
 	  if (lface_id > 0)
-	    it->face_id = merge_faces (it->w, Qt, lface_id,
-				       it->saved_face_id);
+	    {
+	      it->face_id = merge_faces (it->w, Qt, lface_id,
+	                                 it->saved_face_id);
+	      merge_extend_glyph_face (it);
+	    }
 	}
 
       /* Glyphs in the display vector could have the box face, so we
@@ -8061,8 +8102,11 @@ next_element_from_display_vector (struct it *it)
 		GLYPH_CODE_FACE (it->dpvec[it->current.dpvec_index + 1]);
 
 	      if (lface_id > 0)
-		next_face_id = merge_faces (it->w, Qt, lface_id,
-					    it->saved_face_id);
+		{
+		  next_face_id = merge_faces (it->w, Qt, lface_id,
+		                              it->saved_face_id);
+		  merge_extend_glyph_face (it);
+		}
 	    }
 	}
       next_face = FACE_FROM_ID_OR_NULL (it->f, next_face_id);
@@ -20324,7 +20368,7 @@ append_space_for_newline (struct it *it, bool default_face_p)
 		     = XFIXNAT (Vdisplay_fill_column_indicator_character);
 	           it->face_id
 		     = merge_faces (it->w, Qfill_column_indicator,
-				    0, saved_face_id);
+		                    0, it->extend_face_id);
 	           face = FACE_FROM_ID (it->f, it->face_id);
 	           goto produce_glyphs;
 	         }
@@ -20471,33 +20515,33 @@ extend_face_to_end_of_line (struct it *it)
      1-``pixel'' wide, so they hit the equality too early.  This grace
      is needed only for R2L rows that are not continued, to produce
      one extra blank where we could display the cursor.  */
-  if ((it->current_x >= it->last_visible_x
-       + (!FRAME_WINDOW_P (f)
-	  && it->glyph_row->reversed_p
-	  && !it->glyph_row->continued_p))
-      /* If the window has display margins, we will need to extend
-	 their face even if the text area is filled.  */
-      && !(WINDOW_LEFT_MARGIN_WIDTH (it->w) > 0
-	   || WINDOW_RIGHT_MARGIN_WIDTH (it->w) > 0))
-    return;
-
-  /* Face extension extends the background and box of IT->face_id
-     to the end of the line.  If the background equals the background
-     of the frame, we don't have to do anything.  */
+/*   if ((it->current_x >= it->last_visible_x */
+/*        + (!FRAME_WINDOW_P (f) */
+/*           && it->glyph_row->reversed_p */
+/*           && !it->glyph_row->continued_p)) */
+/*       /\* If the window has display margins, we will need to extend */
+/* 	 their face even if the text area is filled.  *\/ */
+/*       && !(WINDOW_LEFT_MARGIN_WIDTH (it->w) > 0 */
+/* 	   || WINDOW_RIGHT_MARGIN_WIDTH (it->w) > 0)) */
+/*     return; */
+
+/*   /\* Face extension extends the background and box of IT->face_id */
+/*      to the end of the line.  If the background equals the background */
+/*      of the frame, we don't have to do anything.  *\/ */
   face = FACE_FROM_ID (f, (it->face_before_selective_p
 			   ? it->saved_face_id
-			   : it->face_id));
-
-  if (FRAME_WINDOW_P (f)
-      && MATRIX_ROW_DISPLAYS_TEXT_P (it->glyph_row)
-      && face->box == FACE_NO_BOX
-      && FACE_COLOR_TO_PIXEL (face->background, f) == FRAME_BACKGROUND_PIXEL (f)
-#ifdef HAVE_WINDOW_SYSTEM
-      && !face->stipple
-#endif
-      && !it->glyph_row->reversed_p
-      && !Vdisplay_fill_column_indicator)
-    return;
+			   : it->extend_face_id));
+
+/*   if (FRAME_WINDOW_P (f) */
+/*       && MATRIX_ROW_DISPLAYS_TEXT_P (it->glyph_row) */
+/*       && face->box == FACE_NO_BOX */
+/*       && FACE_COLOR_TO_PIXEL (face->background, f) == FRAME_BACKGROUND_PIXEL (f) */
+/* #ifdef HAVE_WINDOW_SYSTEM */
+/*       && !face->stipple */
+/* #endif */
+/*       && !it->glyph_row->reversed_p */
+/*       && !Vdisplay_fill_column_indicator) */
+/*     return; */
 
   /* Set the glyph row flag indicating that the face of the last glyph
      in the text area has to be drawn to the end of the text area.  */
@@ -20560,79 +20604,88 @@ extend_face_to_end_of_line (struct it *it)
 	  /* Display fill column indicator if not in modeline or
 	     toolbar and display fill column indicator mode is
 	     active.  */
+	  const char saved_char = it->char_to_display;
+	  const struct text_pos saved_pos = it->position;
+	  const bool saved_avoid_cursor = it->avoid_cursor_p;
+	  const int saved_face_id = it->face_id;
+	  const bool saved_box_start = it->start_of_box_run_p;
+	  Lisp_Object save_object = it->object;
+
+	  it->avoid_cursor_p = true;
+	  it->object = Qnil;
+	  memset (&it->position, 0, sizeof it->position);
+
 	  int indicator_column = (it->w->pseudo_window_p == 0
 				  ? fill_column_indicator_column (it)
 				  : -1);
-	  if (indicator_column >= 0)
-            {
-	      struct font *font = (default_face->font
+
+	  struct font *font = (default_face->font
 				   ? default_face->font
 				   : FRAME_FONT (f));
-	      const int char_width = (font->average_width
-				      ? font->average_width
-				      : font->space_width);
-	      int column_x;
-
-	      if (!INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
-		  && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
-		  && column_x >= it->current_x
-		  && column_x <= it->last_visible_x)
-	        {
-	          const char saved_char = it->char_to_display;
-	          const struct text_pos saved_pos = it->position;
-	          const bool saved_avoid_cursor = it->avoid_cursor_p;
-	          const int saved_face_id = it->face_id;
-	          const bool saved_box_start = it->start_of_box_run_p;
-	          Lisp_Object save_object = it->object;
-
-	          /* The stretch width needs to considet the latter
-	             added glyph.  */
-	          const int stretch_width
-		    = column_x - it->current_x - char_width;
-
-	          memset (&it->position, 0, sizeof it->position);
-	          it->avoid_cursor_p = true;
-	          it->object = Qnil;
-
-	          /* Only generate a stretch glyph if there is distance
-	             between current_x and and the indicator position.  */
-	          if (stretch_width > 0)
-		    {
-		      int stretch_ascent = (((it->ascent + it->descent)
-		                             * FONT_BASE (font)) / FONT_HEIGHT (font));
-		      append_stretch_glyph (it, Qnil, stretch_width,
-		                            it->ascent + it->descent,
-		                            stretch_ascent);
-		    }
+	  const int char_width = (font->average_width
+	                          ? font->average_width
+	                          : font->space_width);
+	  int column_x;
+
+	  if (indicator_column >= 0
+	      && !INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
+	      && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
+	      && column_x >= it->current_x
+	      && column_x <= it->last_visible_x)
+	    {
+
+	      /* The stretch width needs to considet the latter
+		 added glyph.  */
+	      const int stretch_width
+		= column_x - it->current_x - char_width;
+
+	      /* Only generate a stretch glyph if there is distance
+		 between current_x and and the indicator position.  */
+	      if (stretch_width > 0)
+		{
+		  it->face_id = it->extend_face_id;
 
-	          /* Generate the glyph indicator only if
-	             append_space_for_newline didn't already.  */
-	          if (it->current_x < column_x)
-	            {
-		      it->char_to_display
-			= XFIXNAT (Vdisplay_fill_column_indicator_character);
-	              it->face_id
-			= merge_faces (it->w, Qfill_column_indicator,
-				       0, saved_face_id);
-	              PRODUCE_GLYPHS (it);
-	            }
-
-	          /* Restore the face after the indicator was generated.  */
-	          it->face_id = saved_face_id;
-
-	          /* If there is space after the indicator generate an
-	             extra empty glyph to restore the face.  Issue was
-	             observed in X systems.  */
-	          it->char_to_display = ' ';
-	          PRODUCE_GLYPHS (it);
-
-	          it->char_to_display = saved_char;
-	          it->position = saved_pos;
-	          it->avoid_cursor_p = saved_avoid_cursor;
-	          it->start_of_box_run_p = saved_box_start;
-	          it->object = save_object;
-	        }
-            }
+		  int stretch_ascent = (((it->ascent + it->descent)
+		                         * FONT_BASE (font)) / FONT_HEIGHT (font));
+		  append_stretch_glyph (it, Qnil, stretch_width,
+		                        it->ascent + it->descent,
+		                        stretch_ascent);
+		}
+
+	      /* Generate the glyph indicator only if
+		 append_space_for_newline didn't already.  */
+	      if (it->current_x < column_x)
+		{
+		  it->char_to_display
+		    = XFIXNAT (Vdisplay_fill_column_indicator_character);
+		  it->face_id
+		    = merge_faces (it->w, Qfill_column_indicator,
+		                   0, it->extend_face_id);
+		  PRODUCE_GLYPHS (it);
+		  it->char_to_display = saved_char;
+		}
+
+	    }
+
+	  const int stretch_width = it->last_visible_x - it->current_x;
+
+	  if (stretch_width > 0)
+	    {
+	      it->face_id = it->extend_face_id;
+
+	      int stretch_ascent = (((it->ascent + it->descent)
+	                             * FONT_BASE (font)) / FONT_HEIGHT (font));
+	      append_stretch_glyph (it, Qnil, stretch_width,
+	                            it->ascent + it->descent,
+	                            stretch_ascent);
+	    }
+
+	  it->char_to_display = saved_char;
+	  it->position = saved_pos;
+	  it->avoid_cursor_p = saved_avoid_cursor;
+	  it->face_id = saved_face_id;
+	  it->start_of_box_run_p = saved_box_start;
+	  it->object = save_object;
 	}
       if (it->glyph_row->reversed_p)
 	{
@@ -20718,7 +20771,7 @@ extend_face_to_end_of_line (struct it *it)
       it->len = 1;
 
       if (WINDOW_LEFT_MARGIN_WIDTH (it->w) > 0
-	  && (it->glyph_row->used[LEFT_MARGIN_AREA]
+          && (it->glyph_row->used[LEFT_MARGIN_AREA]
 	      < WINDOW_LEFT_MARGIN_WIDTH (it->w))
 	  && !it->glyph_row->mode_line_p
 	  && FACE_COLOR_TO_PIXEL (face->background, f) != FRAME_BACKGROUND_PIXEL (f))
@@ -20762,24 +20815,25 @@ extend_face_to_end_of_line (struct it *it)
 	indicator_column = -1;
       do
 	{
-	  int saved_face_id;
-	  bool indicate = it->current_x == indicator_column;
-	  if (indicate)
+	  if (it->current_x == indicator_column)
 	    {
-	      saved_face_id = it->face_id;
+	      int saved_face_id = it->face_id;
+
 	      it->face_id
-		= merge_faces (it->w, Qfill_column_indicator, 0, saved_face_id);
+		= merge_faces (it->w, Qfill_column_indicator, 0,
+		               it->extend_face_id);
 	      it->c = it->char_to_display
 		= XFIXNAT (Vdisplay_fill_column_indicator_character);
-	    }
 
-	  PRODUCE_GLYPHS (it);
+	      PRODUCE_GLYPHS (it);
 
-	  if (indicate)
-	    {
 	      it->face_id = saved_face_id;
 	      it->c = it->char_to_display = ' ';
 	    }
+	  else
+	    {
+	      PRODUCE_GLYPHS (it);
+	    }
 	}
       while (it->current_x <= it->last_visible_x);
 
@@ -21760,6 +21814,7 @@ display_line (struct it *it, int cursor_vpos)
   it->starts_in_middle_of_char_p = false;
   it->tab_offset = 0;
   it->line_number_produced_p = false;
+  it->extend_face_id = DEFAULT_FACE_ID;
 
   /* Arrange the overlays nicely for our purposes.  Usually, we call
      display_line on only one line at a time, in which case this
@@ -27354,7 +27409,7 @@ font_for_underline_metrics (struct glyph_string *s)
   for (g = s->first_glyph - 1; g >= g0; g--)
     {
       struct face *prev_face = FACE_FROM_ID (s->f, g->face_id);
-      if (!(prev_face && prev_face->underline_p))
+      if (!(prev_face && prev_face->underline != FACE_NO_UNDERLINE))
 	break;
     }
 
diff --git a/src/xfaces.c b/src/xfaces.c
index c3cae7e2a6..092e110de5 100644
--- a/src/xfaces.c
+++ b/src/xfaces.c
@@ -1209,7 +1209,7 @@ free_face_colors (struct frame *f, struct face *face)
       IF_DEBUG (--ncolors_allocated);
     }
 
-  if (face->underline_p
+  if (face->underline
       && !face->underline_defaulted_p)
     {
       x_free_colors (f, &face->underline_color, 1);
@@ -1590,6 +1590,7 @@ #define LFACE_BOX(LFACE)	    AREF ((LFACE), LFACE_BOX_INDEX)
 #define LFACE_FONT(LFACE)	    AREF ((LFACE), LFACE_FONT_INDEX)
 #define LFACE_INHERIT(LFACE)	    AREF ((LFACE), LFACE_INHERIT_INDEX)
 #define LFACE_FONTSET(LFACE)	    AREF ((LFACE), LFACE_FONTSET_INDEX)
+#define LFACE_EXTEND(LFACE)	    AREF ((LFACE), LFACE_EXTEND_INDEX)
 #define LFACE_DISTANT_FOREGROUND(LFACE) \
   AREF ((LFACE), LFACE_DISTANT_FOREGROUND_INDEX)
 
@@ -2512,6 +2513,13 @@ merge_face_ref (struct window *w,
 					err_msgs, named_merge_points))
 		    err = true;
 		}
+	      else if (EQ (keyword, QCextend))
+		{
+		  if (EQ (value, Qt) || NILP (value))
+		    to[LFACE_EXTEND_INDEX] = value;
+		  else
+		    err = true;
+		}
 	      else
 		err = true;
 
@@ -3030,6 +3038,17 @@ DEFUN ("internal-set-lisp-face-attribute", Finternal_set_lisp_face_attribute,
       old_value = LFACE_INVERSE (lface);
       ASET (lface, LFACE_INVERSE_INDEX, value);
     }
+  else if (EQ (attr, QCextend))
+    {
+      if (!UNSPECIFIEDP (value) && !IGNORE_DEFFACE_P (value))
+	{
+	  CHECK_SYMBOL (value);
+	  if (!EQ (value, Qt) && !NILP (value))
+	    signal_error ("Invalid inverse-video face attribute value", value);
+	}
+      old_value = LFACE_EXTEND (lface);
+      ASET (lface, LFACE_EXTEND_INDEX, value);
+    }
   else if (EQ (attr, QCforeground))
     {
       /* Compatibility with 20.x.  */
@@ -3203,6 +3222,14 @@ DEFUN ("internal-set-lisp-face-attribute", Finternal_set_lisp_face_attribute,
       ASET (lface, LFACE_SLANT_INDEX, NILP (value) ? Qnormal : Qitalic);
       prop_index = FONT_SLANT_INDEX;
     }
+  else if (EQ (attr, QCitalic))
+    {
+      attr = QCslant;
+      old_value = LFACE_SLANT (lface);
+      ASET (lface, LFACE_SLANT_INDEX, NILP (value) ? Qnormal : Qitalic);
+      prop_index = FONT_SLANT_INDEX;
+    }
+
   else
     signal_error ("Invalid face attribute name", attr);
 
@@ -3503,7 +3530,9 @@ DEFUN ("internal-set-lisp-face-attribute-from-resource",
     value = face_boolean_x_resource_value (value, true);
   else if (EQ (attr, QCweight) || EQ (attr, QCslant) || EQ (attr, QCwidth))
     value = intern (SSDATA (value));
-  else if (EQ (attr, QCreverse_video) || EQ (attr, QCinverse_video))
+  else if (EQ (attr, QCreverse_video)
+           || EQ (attr, QCinverse_video)
+           || EQ (attr, QCextend))
     value = face_boolean_x_resource_value (value, true);
   else if (EQ (attr, QCunderline)
 	   || EQ (attr, QCoverline)
@@ -3727,6 +3756,8 @@ DEFUN ("internal-get-lisp-face-attribute", Finternal_get_lisp_face_attribute,
     value = LFACE_SWIDTH (lface);
   else if (EQ (keyword, QCinherit))
     value = LFACE_INHERIT (lface);
+  else if (EQ (keyword, QCextend))
+    value = LFACE_EXTEND (lface);
   else if (EQ (keyword, QCfont))
     value = LFACE_FONT (lface);
   else if (EQ (keyword, QCfontset))
@@ -5694,16 +5725,14 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
   if (EQ (underline, Qt))
     {
       /* Use default color (same as foreground color).  */
-      face->underline_p = true;
-      face->underline_type = FACE_UNDER_LINE;
+      face->underline = FACE_UNDER_LINE;
       face->underline_defaulted_p = true;
       face->underline_color = 0;
     }
   else if (STRINGP (underline))
     {
       /* Use specified color.  */
-      face->underline_p = true;
-      face->underline_type = FACE_UNDER_LINE;
+      face->underline = FACE_UNDER_LINE;
       face->underline_defaulted_p = false;
       face->underline_color
 	= load_color (f, face, underline,
@@ -5711,7 +5740,7 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
     }
   else if (NILP (underline))
     {
-      face->underline_p = false;
+      face->underline = FACE_NO_UNDERLINE;
       face->underline_defaulted_p = false;
       face->underline_color = 0;
     }
@@ -5719,10 +5748,9 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
     {
       /* `(:color COLOR :style STYLE)'.
          STYLE being one of `line' or `wave'. */
-      face->underline_p = true;
+      face->underline = FACE_UNDER_LINE;
       face->underline_color = 0;
       face->underline_defaulted_p = true;
-      face->underline_type = FACE_UNDER_LINE;
 
       /* FIXME?  This is also not robust about checking the precise form.
          See comments in Finternal_set_lisp_face_attribute.  */
@@ -5755,9 +5783,9 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
           else if (EQ (keyword, QCstyle))
             {
               if (EQ (value, Qline))
-                face->underline_type = FACE_UNDER_LINE;
+                face->underline = FACE_UNDER_LINE;
               else if (EQ (value, Qwave))
-                face->underline_type = FACE_UNDER_WAVE;
+                face->underline = FACE_UNDER_WAVE;
             }
         }
     }
@@ -6292,9 +6320,8 @@ merge_faces (struct window *w, Lisp_Object face_name, int face_id,
 {
   struct frame *f = WINDOW_XFRAME (w);
   Lisp_Object attrs[LFACE_VECTOR_SIZE];
-  struct face *base_face;
+  struct face *base_face = FACE_FROM_ID_OR_NULL (f, base_face_id);
 
-  base_face = FACE_FROM_ID_OR_NULL (f, base_face_id);
   if (!base_face)
     return base_face_id;
 
@@ -6319,12 +6346,14 @@ merge_faces (struct window *w, Lisp_Object face_name, int face_id,
     }
   else
     {
-      struct face *face;
       if (face_id < 0)
 	return base_face_id;
-      face = FACE_FROM_ID_OR_NULL (f, face_id);
+
+      struct face *face = FACE_FROM_ID_OR_NULL (f, face_id);
+
       if (!face)
 	return base_face_id;
+
       merge_face_vectors (w, f, face->lface, attrs, 0);
     }
 
@@ -6412,7 +6441,7 @@ dump_realized_face (struct face *face)
 #endif
   fprintf (stderr, "fontset: %d\n", face->fontset);
   fprintf (stderr, "underline: %d (%s)\n",
-	   face->underline_p,
+	   face->underline,
 	   SDATA (Fsymbol_name (face->lface[LFACE_UNDERLINE_INDEX])));
   fprintf (stderr, "hash: %" PRIuPTR "\n", face->hash);
 }
@@ -6537,6 +6566,7 @@ syms_of_xfaces (void)
   DEFSYM (QCstrike_through, ":strike-through");
   DEFSYM (QCbox, ":box");
   DEFSYM (QCinherit, ":inherit");
+  DEFSYM (QCextend, ":extend");
 
   /* Symbols used for Lisp face attribute values.  */
   DEFSYM (QCcolor, ":color");
diff --git a/src/xterm.c b/src/xterm.c
index b761eaf4d1..b8f8db56a7 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -3798,9 +3798,9 @@ x_draw_glyph_string (struct glyph_string *s)
   if (!s->for_overlaps)
     {
       /* Draw underline.  */
-      if (s->face->underline_p)
+      if (s->face->underline)
         {
-          if (s->face->underline_type == FACE_UNDER_WAVE)
+          if (s->face->underline == FACE_UNDER_WAVE)
             {
               if (s->face->underline_defaulted_p)
                 x_draw_underwave (s);
@@ -3814,13 +3814,13 @@ x_draw_glyph_string (struct glyph_string *s)
                   XSetForeground (display, s->gc, xgcv.foreground);
                 }
             }
-          else if (s->face->underline_type == FACE_UNDER_LINE)
+          else if (s->face->underline == FACE_UNDER_LINE)
             {
               unsigned long thickness, position;
               int y;
 
-              if (s->prev && s->prev->face->underline_p
-		  && s->prev->face->underline_type == FACE_UNDER_LINE)
+              if (s->prev &&
+	          s->prev->face->underline == FACE_UNDER_LINE)
                 {
                   /* We use the same underline style as the previous one.  */
                   thickness = s->prev->underline_thickness;

Re: Question about display engine

[martin rudalics]

 > I don't think I understand why you are talking about face pointers.
 > The iterator doesn't keep any face pointers, it keeps face IDs (which
 > allow to obtain the face pointer, when needed, by using FACE_FROM_ID).
 >
 > So all you need is to have another face ID member in the iterator, to
 > be used for extending past EOL; depending on how the :extend bits are
 > set, that face ID may or may not be identical to the "normal" face ID,
 > the one we have now and use for buffer text.

But a face ID is only available after realizing a face.  With a lazy
approach there might be no realized face yet and the iterator would
realize that face on the fly.

 >> In the display engine we do this very
 >> frequently. As extend_face_to_end_of_line is very localized we just need
 >> to save a pointer to a+b+c+d on the beginning of the function and
 >> restore it at the end.
 >
 > Again, not a pointer: a face ID.  And yes, we have saved_face_id
 > member of the iterator for this purpose.

In that case we would have a face ID for sure.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Tue, 3 Sep 2019 10:45:18 +0200
> 
>  > I don't think I understand why you are talking about face pointers.
>  > The iterator doesn't keep any face pointers, it keeps face IDs (which
>  > allow to obtain the face pointer, when needed, by using FACE_FROM_ID).
>  >
>  > So all you need is to have another face ID member in the iterator, to
>  > be used for extending past EOL; depending on how the :extend bits are
>  > set, that face ID may or may not be identical to the "normal" face ID,
>  > the one we have now and use for buffer text.
> 
> But a face ID is only available after realizing a face.  With a lazy
> approach there might be no realized face yet and the iterator would
> realize that face on the fly.

What do you call "realize a face", exactly, and why do you think it is
so expensive as to justify doing that lazily?

Re: Question about display engine

[martin rudalics]

 > What do you call "realize a face", exactly,

Whatever realize_face does with its attrs argument.

 > and why do you think it is
 > so expensive as to justify doing that lazily?

Due to the overhead to calculate, cache and eventually free the
realized face.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: spacibba@aol.com, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Tue, 3 Sep 2019 18:41:40 +0200
> 
>  > What do you call "realize a face", exactly,
> 
> Whatever realize_face does with its attrs argument.
> 
>  > and why do you think it is
>  > so expensive as to justify doing that lazily?
> 
> Due to the overhead to calculate, cache and eventually free the
> realized face.

I don't think it's expensive enough to justify such premature
optimization.

Re: Question about display engine

[martin rudalics]

 > I don't think it's expensive enough to justify such premature
 > optimization.

Earlier you preferred a lazy variant because

 > it scales better, because the
 > display code is frequently invoked on short portions of the text, so
 > there's no guarantee that it will actually get to producing glyphs
 > with the "extension" variant of the face, so realizing that face in
 > advance might well be waste of unneeded effort, because the additional
 > face will never be used.

Either way, let's see what Ergus comes up with.

martin

Re: Question about display engine

[Ergus]

Hi:

I have just uploaded some changes but the functionality is still not
working.

I separated the changes in 3 commits and in the last one are only the
ones I made in the xdisp.c (the ones that need to be checked, because
the rest is only infrastructure.)

see the master branch in: https://github.com/Ergus/Emacs

But at this point I will need some extra comments about what I am doing
wrong there. Ir if I am doing anything right.

The extend now is working like always (testing for the region).

In principle the region is the only face I have added :extend t

But when I set it to nil it should be not extended and it is; so some
extra condition is still missing..

The filter if condition (to merge or not) is a new macro
FACE_EXTENSIBLE_P (face). So please any hint is very welcome now.



On Tue, Sep 03, 2019 at 08:59:33PM +0200, martin rudalics wrote:
>> I don't think it's expensive enough to justify such premature
>> optimization.
>
>Earlier you preferred a lazy variant because
>
>> it scales better, because the
>> display code is frequently invoked on short portions of the text, so
>> there's no guarantee that it will actually get to producing glyphs
>> with the "extension" variant of the face, so realizing that face in
>> advance might well be waste of unneeded effort, because the additional
>> face will never be used.
>
>Either way, let's see what Ergus comes up with.
>
>martin
>

Re: Question about display engine

[martin rudalics]

 > I have just uploaded some changes but the functionality is still not
 > working.
 >
 > I separated the changes in 3 commits and in the last one are only the
 > ones I made in the xdisp.c (the ones that need to be checked, because
 > the rest is only infrastructure.)
 >
 > see the master branch in: https://github.com/Ergus/Emacs

Please send a patch for current master.  Otherwise I have no idea how
to compare your changes.

Thanks, martin

Re: Question about display engine

[Ergus via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]


Here is the diff of the latest commit. (Patch attached anyway).

https://github.com/Ergus/Emacs/commit/4943087027acd3f2c7a54a092b64bc839ef8850e


On Wed, Sep 04, 2019 at 10:04:16PM +0200, martin rudalics wrote:
>> I have just uploaded some changes but the functionality is still not
>> working.
>>
>> I separated the changes in 3 commits and in the last one are only the
>> ones I made in the xdisp.c (the ones that need to be checked, because
>> the rest is only infrastructure.)
>>
>> see the master branch in: https://github.com/Ergus/Emacs
>
>Please send a patch for current master.  Otherwise I have no idea how
>to compare your changes.
>
>Thanks, martin

[-- Attachment #2: patch.patch --]
[-- Type: text/plain, Size: 42425 bytes --]

diff --git a/lisp/cus-face.el b/lisp/cus-face.el
index d73bce42c3..5a49a81043 100644
--- a/lisp/cus-face.el
+++ b/lisp/cus-face.el
@@ -233,7 +233,11 @@ custom-face-attributes
 	     (file :tag "File"
 		   :help-echo "Name of bitmap file."
 		   :must-match t)))
-
+    (:extend
+     (choice :tag "Extend"
+	     :help-echo "Control whether attributes should be extended after EOL."
+	     (const :tag "Off" nil)
+	     (const :tag "On" t)))
     (:inherit
      (repeat :tag "Inherit"
 	     :help-echo "List of faces to inherit attributes from."
diff --git a/lisp/faces.el b/lisp/faces.el
index 5193c216d0..6b6fcf0d2d 100644
--- a/lisp/faces.el
+++ b/lisp/faces.el
@@ -342,6 +342,7 @@ face-x-resources
     (:box (".attributeBox" . "Face.AttributeBox"))
     (:underline (".attributeUnderline" . "Face.AttributeUnderline"))
     (:inverse-video (".attributeInverse" . "Face.AttributeInverse"))
+    (:extend (".attributeExtend" . "Face.AttributeExtend"))
     (:stipple
      (".attributeStipple" . "Face.AttributeStipple")
      (".attributeBackgroundPixmap" . "Face.AttributeBackgroundPixmap"))
@@ -594,6 +595,13 @@ face-italic-p
   (let ((italic (face-attribute face :slant frame inherit)))
     (memq italic '(italic oblique))))
 
+(defun face-extend-p (face &optional frame inherit)
+ "Return non-nil if FACE specifies a non-nil extend.
+If the optional argument FRAME is given, report on face FACE in that frame.
+If FRAME is t, report on the defaults for face FACE (for new frames).
+If FRAME is omitted or nil, use the selected frame.
+Optional argument INHERIT is passed to `face-attribute'."
+ (eq (face-attribute face :extend frame inherit) t))
 
 \f
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -760,6 +768,11 @@ set-face-attribute
 `:height', `:weight', and `:slant' may also be set in one step
 from an X font name:
 
+`:extend'
+
+VALUE specifies whether the FACE should be extended after EOL.
+VALUE must be one of t or nil.
+
 `:font'
 
 Set font-related face attributes from VALUE.
@@ -979,6 +992,18 @@ set-face-italic
 
 (define-obsolete-function-alias 'set-face-italic-p 'set-face-italic "24.4")
 
+(defun set-face-extend (face extend-p &optional frame)
+  "Specify whether face FACE should be extended.
+EXTEND-P nil means FACE explicitly doesn't extend after EOL.
+EXTEND-P t means FACE extends after EOL.
+
+FRAME nil or not specified means change face on all frames.
+Use `set-face-attribute' to \"unspecify\" underlining."
+  (interactive
+   (let ((list (read-face-and-attribute :extend)))
+     (list (car list) (if (cadr list) t))))
+  (set-face-attribute face frame :extend extend-p))
+
 
 (defalias 'set-face-background-pixmap 'set-face-stipple)
 
@@ -1102,7 +1127,7 @@ face-valid-attribute-values
 	   (:slant
 	    (mapcar #'(lambda (x) (cons (symbol-name (aref x 1)) (aref x 1)))
 		    font-slant-table))
-	   (:inverse-video
+	   ((or :inverse-video :extend)
 	    (mapcar #'(lambda (x) (cons (symbol-name x) x))
 		    (internal-lisp-face-attribute-values attribute)))
            ((or :underline :overline :strike-through :box)
@@ -1147,6 +1172,7 @@ face-attribute-name-alist
     (:slant . "slant")
     (:underline . "underline")
     (:overline . "overline")
+    (:extend . "extend")
     (:strike-through . "strike-through")
     (:box . "box")
     (:inverse-video . "inverse-video display")
@@ -2432,24 +2458,24 @@ highlight
 ;; if background is light.
 (defface region
   '((((class color) (min-colors 88) (background dark))
-     :background "blue3")
+     :background "blue3" :extend t)
     (((class color) (min-colors 88) (background light) (type gtk))
      :distant-foreground "gtk_selection_fg_color"
-     :background "gtk_selection_bg_color")
+     :background "gtk_selection_bg_color" :extend t)
     (((class color) (min-colors 88) (background light) (type ns))
      :distant-foreground "ns_selection_fg_color"
-     :background "ns_selection_bg_color")
+     :background "ns_selection_bg_color" :extend t)
     (((class color) (min-colors 88) (background light))
-     :background "lightgoldenrod2")
+     :background "lightgoldenrod2" :extend t)
     (((class color) (min-colors 16) (background dark))
-     :background "blue3")
+     :background "blue3" :extend t)
     (((class color) (min-colors 16) (background light))
-     :background "lightgoldenrod2")
+     :background "lightgoldenrod2" :extend t)
     (((class color) (min-colors 8))
-     :background "blue" :foreground "white")
+     :background "blue" :foreground "white" :extend t)
     (((type tty) (class mono))
      :inverse-video t)
-    (t :background "gray"))
+    (t :background "gray" :extend t))
   "Basic face for highlighting the region."
   :version "21.1"
   :group 'basic-faces)
diff --git a/src/dispextern.h b/src/dispextern.h
index 05f199ff35..c11a3a7b54 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -1564,6 +1564,7 @@ #define FONT_TOO_HIGH(ft)						\
   LFACE_INHERIT_INDEX,
   LFACE_FONTSET_INDEX,
   LFACE_DISTANT_FOREGROUND_INDEX,
+  LFACE_EXTEND_INDEX,
   LFACE_VECTOR_SIZE
 };
 
@@ -1589,6 +1590,7 @@ #define FONT_TOO_HIGH(ft)						\
 
 enum face_underline_type
 {
+  FACE_NO_UNDERLINE = 0,
   FACE_UNDER_LINE,
   FACE_UNDER_WAVE
 };
@@ -1632,11 +1634,9 @@ #define FONT_TOO_HIGH(ft)						\
   /* Pixel value or color index of background color.  */
   unsigned long background;
 
-  /* Pixel value or color index of underline color.  */
+  /* Pixel value or color index of underline, overlined,
+     strike-through, or box color.  */
   unsigned long underline_color;
-
-  /* Pixel value or color index of overlined, strike-through, or box
-     color.  */
   unsigned long overline_color;
   unsigned long strike_through_color;
   unsigned long box_color;
@@ -1663,7 +1663,7 @@ #define FONT_TOO_HIGH(ft)						\
   ENUM_BF (face_box_type) box : 2;
 
   /* Style of underlining. */
-  ENUM_BF (face_underline_type) underline_type : 1;
+  ENUM_BF (face_underline_type) underline : 2;
 
   /* If `box' above specifies a 3D type, true means use box_color for
      drawing shadows.  */
@@ -1671,7 +1671,6 @@ #define FONT_TOO_HIGH(ft)						\
 
   /* Non-zero if text in this face should be underlined, overlined,
      strike-through or have a box drawn around it.  */
-  bool_bf underline_p : 1;
   bool_bf overline_p : 1;
   bool_bf strike_through_p : 1;
 
@@ -1681,14 +1680,10 @@ #define FONT_TOO_HIGH(ft)						\
   bool_bf foreground_defaulted_p : 1;
   bool_bf background_defaulted_p : 1;
 
-  /* True means that either no color is specified for underlining or that
-     the specified color couldn't be loaded.  Use the foreground
-     color when drawing in that case. */
-  bool_bf underline_defaulted_p : 1;
-
   /* True means that either no color is specified for the corresponding
      attribute or that the specified color couldn't be loaded.
      Use the foreground color when drawing in that case. */
+  bool_bf underline_defaulted_p : 1;
   bool_bf overline_color_defaulted_p : 1;
   bool_bf strike_through_color_defaulted_p : 1;
   bool_bf box_color_defaulted_p : 1;
@@ -1822,6 +1817,9 @@ #define FACE_FROM_ID_OR_NULL(F, ID)			\
    ? FRAME_FACE_CACHE (F)->faces_by_id[ID]		\
    : NULL)
 
+#define FACE_EXTENSIBLE_P(F)			\
+  (!NILP (F->lface[LFACE_EXTEND_INDEX]))
+
 /* True if FACE is suitable for displaying ASCII characters.  */
 INLINE bool
 FACE_SUITABLE_FOR_ASCII_CHAR_P (struct face *face)
@@ -2328,7 +2326,7 @@ #define IT_STACK_SIZE 5
   /* Face id of the iterator saved in case a glyph from dpvec contains
      a face.  The face is restored when all glyphs from dpvec have
      been delivered.  */
-  int saved_face_id;
+  int saved_face_id, saved_extend_face_id;
 
   /* Vector of glyphs for control character translation.  The pointer
      dpvec is set to ctl_chars when a control character is translated.
@@ -2390,7 +2388,7 @@ #define OVERLAY_STRING_CHUNK_SIZE 16
     ptrdiff_t prev_stop;
     ptrdiff_t base_level_stop;
     struct composition_it cmp_it;
-    int face_id;
+    int face_id, extend_face_id;
 
     /* Save values specific to a given method.  */
     union {
@@ -2448,6 +2446,9 @@ #define OVERLAY_STRING_CHUNK_SIZE 16
   /* Face to use.  */
   int face_id;
 
+  /* Face to extend at EOL/  */
+  int extend_face_id;
+
   /* Setting of buffer-local variable selective-display-ellipses.  */
   bool_bf selective_display_ellipsis_p : 1;
 
diff --git a/src/nsterm.m b/src/nsterm.m
index 42ef4dd010..99b621533a 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -3404,9 +3404,9 @@ Note that CURSOR_WIDTH is meaningful only for (h)bar cursors.
     return;
 
   /* Do underline.  */
-  if (face->underline_p)
+  if (face->underline)
     {
-      if (s->face->underline_type == FACE_UNDER_WAVE)
+      if (s->face->underline == FACE_UNDER_WAVE)
         {
           if (face->underline_defaulted_p)
             [defaultCol set];
@@ -3415,15 +3415,15 @@ Note that CURSOR_WIDTH is meaningful only for (h)bar cursors.
 
           ns_draw_underwave (s, width, x);
         }
-      else if (s->face->underline_type == FACE_UNDER_LINE)
+      else if (s->face->underline == FACE_UNDER_LINE)
         {
 
           NSRect r;
           unsigned long thickness, position;
 
           /* If the prev was underlined, match its appearance.  */
-          if (s->prev && s->prev->face->underline_p
-	      && s->prev->face->underline_type == FACE_UNDER_LINE
+          if (s->prev
+	      && s->prev->face->underline == FACE_UNDER_LINE
               && s->prev->underline_thickness > 0)
             {
               thickness = s->prev->underline_thickness;
diff --git a/src/w32term.c b/src/w32term.c
index e5874f2d36..99a1db5784 100644
--- a/src/w32term.c
+++ b/src/w32term.c
@@ -2479,9 +2479,9 @@ w32_draw_glyph_string (struct glyph_string *s)
   if (!s->for_overlaps)
     {
       /* Draw underline.  */
-      if (s->face->underline_p)
+      if (s->face->underline)
         {
-          if (s->face->underline_type == FACE_UNDER_WAVE)
+          if (s->face->underline == FACE_UNDER_WAVE)
             {
               COLORREF color;
 
@@ -2492,13 +2492,13 @@ w32_draw_glyph_string (struct glyph_string *s)
 
               w32_draw_underwave (s, color);
             }
-          else if (s->face->underline_type == FACE_UNDER_LINE)
+          else if (s->face->underline == FACE_UNDER_LINE)
             {
               unsigned long thickness, position;
               int y;
 
-              if (s->prev && s->prev->face->underline_p
-		  && s->prev->face->underline_type == FACE_UNDER_LINE)
+              if (s->prev
+	          && s->prev->face->underline == FACE_UNDER_LINE)
                 {
                   /* We use the same underline style as the previous one.  */
                   thickness = s->prev->underline_thickness;
@@ -2512,7 +2512,7 @@ w32_draw_glyph_string (struct glyph_string *s)
 		  BOOL use_underline_position_properties;
 		  Lisp_Object val
 		    = buffer_local_value (Qunderline_minimum_offset,
-					s->w->contents);
+		                          s->w->contents);
 		  if (FIXNUMP (val))
 		    minimum_offset = max (0, XFIXNUM (val));
 		  else
diff --git a/src/xdisp.c b/src/xdisp.c
index 94f969f37c..db7f1e8d34 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -3120,6 +3120,7 @@ init_iterator (struct it *it, struct window *w,
       struct face *face;
 
       it->face_id = remapped_base_face_id;
+      it->extend_face_id = remapped_base_face_id;
 
       /* If we have a boxed mode line, make the first character appear
 	 with a left box line.  */
@@ -3141,6 +3142,8 @@ init_iterator (struct it *it, struct window *w,
       /* We will rely on `reseat' to set this up properly, via
 	 handle_face_prop.  */
       it->face_id = it->base_face_id;
+      it->extend_face_id = it->base_face_id;  // ERGUS: FIX_THIS
+
 
       it->start = it->current;
       /* Do we need to reorder bidirectional text?  Not if this is a
@@ -3536,7 +3539,10 @@ handle_stop (struct it *it)
 
   /* Use face of preceding text for ellipsis (if invisible) */
   if (it->selective_display_ellipsis_p)
-    it->saved_face_id = it->face_id;
+    {
+      it->saved_face_id = it->face_id;
+      it->saved_extend_face_id = it->extend_face_id;
+    }
 
   /* Here's the description of the semantics of, and the logic behind,
      the various HANDLED_* statuses:
@@ -4154,7 +4160,15 @@ handle_face_prop (struct it *it)
 	  it->start_of_box_run_p = (new_face->box != FACE_NO_BOX
 				    && (old_face == NULL || !old_face->box));
 	  it->face_box_p = new_face->box != FACE_NO_BOX;
+
+	  /* Update the faces id and extend.  */
+	  it->face_id = new_face_id;
+
+	  if (FACE_EXTENSIBLE_P (new_face))
+	    it->extend_face_id = new_face_id;
+
 	}
+
     }
   else
     {
@@ -4253,10 +4267,16 @@ handle_face_prop (struct it *it)
 	  it->start_of_box_run_p
 	    = new_face->box && (old_face == NULL || !old_face->box);
 	  it->face_box_p = new_face->box != FACE_NO_BOX;
+
+	  /* Update the faces id and extend.  */
+	  it->face_id = new_face_id;
+
+	  if (FACE_EXTENSIBLE_P (new_face))
+	    it->extend_face_id = new_face_id;
+
 	}
     }
 
-  it->face_id = new_face_id;
   return HANDLED_NORMALLY;
 }
 
@@ -4854,6 +4874,9 @@ setup_for_ellipsis (struct it *it, int len)
   if (it->saved_face_id >= 0)
     it->face_id = it->saved_face_id;
 
+  if (it->saved_extend_face_id >= 0)
+    it->extend_face_id = it->saved_extend_face_id;
+
   /* If the ellipsis represents buffer text, it means we advanced in
      the buffer, so we should no longer ignore overlay strings.  */
   if (it->method == GET_FROM_BUFFER)
@@ -5063,7 +5086,8 @@ display_prop_end (struct it *it, Lisp_Object object, struct text_pos start_pos)
    of buffer or string text.  */
 
 static int
-handle_single_display_spec (struct it *it, Lisp_Object spec, Lisp_Object object,
+handle_single_display_spec (struct it *it, Lisp_Object spec,
+                            Lisp_Object object,
 			    Lisp_Object overlay, struct text_pos *position,
 			    ptrdiff_t bufpos, int display_replaced,
 			    bool frame_window_p, bool enable_eval_p)
@@ -5137,7 +5161,11 @@ handle_single_display_spec (struct it *it, Lisp_Object spec, Lisp_Object object,
 		  int steps = XFIXNUM (XCAR (XCDR (it->font_height)));
 		  if (EQ (XCAR (it->font_height), Qplus))
 		    steps = - steps;
+
 		  it->face_id = smaller_face (it->f, it->face_id, steps);
+		  it->extend_face_id
+		    = smaller_face (it->f, it->extend_face_id, steps);
+
 		}
 	      else if (FUNCTIONP (it->font_height) && enable_eval_p)
 		{
@@ -5180,7 +5208,12 @@ handle_single_display_spec (struct it *it, Lisp_Object spec, Lisp_Object object,
 		}
 
 	      if (new_height > 0)
-		it->face_id = face_with_height (it->f, it->face_id, new_height);
+		{
+		  it->face_id
+		    = face_with_height (it->f, it->face_id, new_height);
+		  it->extend_face_id
+		    = face_with_height (it->f, it->extend_face_id, new_height);
+		}
 	    }
 	}
 
@@ -5370,6 +5403,7 @@ handle_single_display_spec (struct it *it, Lisp_Object spec, Lisp_Object object,
 	  it->method = GET_FROM_IMAGE;
 	  it->from_overlay = Qnil;
 	  it->face_id = face_id;
+	  it->extend_face_id = face_id; // ERGUS: FIX_THIS
 	  it->from_disp_prop_p = true;
 
 	  /* Say that we haven't consumed the characters with
@@ -6263,6 +6297,7 @@ push_it (struct it *it, struct text_pos *position)
   p->cmp_it = it->cmp_it;
   eassert (it->face_id >= 0);
   p->face_id = it->face_id;
+  p->extend_face_id = it->extend_face_id;
   p->string = it->string;
   p->method = it->method;
   p->from_overlay = it->from_overlay;
@@ -6366,6 +6401,7 @@ pop_it (struct it *it)
   it->base_level_stop = p->base_level_stop;
   it->cmp_it = p->cmp_it;
   it->face_id = p->face_id;
+  it->extend_face_id = p->extend_face_id;
   it->current = p->current;
   it->position = p->position;
   it->string = p->string;
@@ -7142,17 +7178,32 @@ lookup_glyphless_char_display (int c, struct it *it)
 
 /* Merge escape glyph face and cache the result.  */
 
+static int
+merge_extend_glyph_face (struct it *it, int lface_id)
+{
+
+  struct face *lface = FACE_FROM_ID (it->f, lface_id);
+
+  if (lface && FACE_EXTENSIBLE_P (lface))
+    return merge_faces (it->w, Qt, lface_id, it->extend_face_id);
+
+  return it->extend_face_id;
+}
+
+/* Merge escape glyph face and cache the result.  */
 static struct frame *last_escape_glyph_frame = NULL;
 static int last_escape_glyph_face_id = (1 << FACE_ID_BITS);
 static int last_escape_glyph_merged_face_id = 0;
 
 static int
-merge_escape_glyph_face (struct it *it)
+merge_escape_glyph_face (struct it *it, int lface_id)
 {
   int face_id;
 
-  if (it->f == last_escape_glyph_frame
-      && it->face_id == last_escape_glyph_face_id)
+  if (lface_id)
+    face_id = merge_faces (it->w, Qt, lface_id, it->face_id);
+  else if (it->f == last_escape_glyph_frame
+           && it->face_id == last_escape_glyph_face_id)
     face_id = last_escape_glyph_merged_face_id;
   else
     {
@@ -7162,6 +7213,7 @@ merge_escape_glyph_face (struct it *it)
       last_escape_glyph_face_id = it->face_id;
       last_escape_glyph_merged_face_id = face_id;
     }
+
   return face_id;
 }
 
@@ -7187,9 +7239,11 @@ merge_glyphless_glyph_face (struct it *it)
       last_glyphless_glyph_face_id = it->face_id;
       last_glyphless_glyph_merged_face_id = face_id;
     }
+
   return face_id;
 }
 
+
 /* Forget the `escape-glyph' and `glyphless-char' faces.  This should
    be called before redisplaying windows, and when the frame's face
    cache is freed.  */
@@ -7275,6 +7329,7 @@ get_next_display_element (struct it *it)
 		  it->current.dpvec_index = 0;
 		  it->dpvec_face_id = -1;
 		  it->saved_face_id = it->face_id;
+		  it->saved_extend_face_id = it->extend_face_id;
 		  it->method = GET_FROM_DISPLAY_VECTOR;
 		  it->ellipsis_p = false;
 		}
@@ -7355,9 +7410,7 @@ get_next_display_element (struct it *it)
 		      lface_id = GLYPH_CODE_FACE (gc);
 		    }
 
-		  face_id = (lface_id
-			     ? merge_faces (it->w, Qt, lface_id, it->face_id)
-			     : merge_escape_glyph_face (it));
+		  face_id = merge_escape_glyph_face (it, lface_id);
 
 		  XSETINT (it->ctl_chars[0], g);
 		  XSETINT (it->ctl_chars[1], c ^ 0100);
@@ -7373,6 +7426,7 @@ get_next_display_element (struct it *it)
 		  /* Merge `nobreak-space' into the current face.  */
 		  face_id = merge_faces (it->w, Qnobreak_space, 0,
 					 it->face_id);
+
 		  XSETINT (it->ctl_chars[0], ' ');
 		  ctl_len = 1;
 		  goto display_control;
@@ -7385,7 +7439,8 @@ get_next_display_element (struct it *it)
 		{
 		  /* Merge `nobreak-space' into the current face.  */
 		  face_id = merge_faces (it->w, Qnobreak_hyphen, 0,
-					 it->face_id);
+		                         it->face_id);
+
 		  XSETINT (it->ctl_chars[0], '-');
 		  ctl_len = 1;
 		  goto display_control;
@@ -7403,12 +7458,9 @@ get_next_display_element (struct it *it)
 		  lface_id = GLYPH_CODE_FACE (gc);
 		}
 
-	      face_id = (lface_id
-			 ? merge_faces (it->w, Qt, lface_id, it->face_id)
-			 : merge_escape_glyph_face (it));
+	      face_id = merge_escape_glyph_face (it, lface_id);
 
 	      /* Draw non-ASCII space/hyphen with escape glyph: */
-
 	      if (nonascii_space_p || nonascii_hyphen_p)
 		{
 		  XSETINT (it->ctl_chars[0], escape_glyph);
@@ -7443,6 +7495,7 @@ get_next_display_element (struct it *it)
 	      it->current.dpvec_index = 0;
 	      it->dpvec_face_id = face_id;
 	      it->saved_face_id = it->face_id;
+	      it->saved_extend_face_id = it->extend_face_id;
 	      it->method = GET_FROM_DISPLAY_VECTOR;
 	      it->ellipsis_p = false;
 	      goto get_next;
@@ -7778,6 +7831,7 @@ set_iterator_to_next (struct it *it, bool reseat_p)
       /* Restore face of the iterator to what they were before the
          display vector entry (these entries may contain faces).  */
       it->face_id = it->saved_face_id;
+      it->extend_face_id = it->saved_extend_face_id;
 
       if (it->dpvec + it->current.dpvec_index >= it->dpend)
 	{
@@ -8012,6 +8066,7 @@ next_element_from_display_vector (struct it *it)
   eassert (it->dpvec && it->current.dpvec_index >= 0);
 
   it->face_id = it->saved_face_id;
+  it->extend_face_id = it->saved_extend_face_id;
 
   /* KFS: This code used to check ip->dpvec[0] instead of the current element.
      That seemed totally bogus - so I changed it...  */
@@ -8027,13 +8082,24 @@ next_element_from_display_vector (struct it *it)
 	 the id of a Lisp face, not a realized face.  A face id of
 	 zero means no face is specified.  */
       if (it->dpvec_face_id >= 0)
-	it->face_id = it->dpvec_face_id;
+	{
+	  it->face_id = it->dpvec_face_id;
+	  //it->extend_face_id = it->dpvec_face_id; // ERGUS: FIX_THIS
+	}
       else
 	{
 	  int lface_id = GLYPH_CODE_FACE (gc);
 	  if (lface_id > 0)
-	    it->face_id = merge_faces (it->w, Qt, lface_id,
-				       it->saved_face_id);
+	    {
+	      it->face_id = merge_faces (it->w, Qt, lface_id,
+	                                 it->saved_face_id);
+
+	      struct face *lface = FACE_FROM_ID (it->f, lface_id);
+
+	      if (FACE_EXTENSIBLE_P (lface))
+		it->extend_face_id = merge_faces (it->w, Qt, lface_id,
+		                                  it->saved_extend_face_id);;
+	    }
 	}
 
       /* Glyphs in the display vector could have the box face, so we
@@ -8061,8 +8127,12 @@ next_element_from_display_vector (struct it *it)
 		GLYPH_CODE_FACE (it->dpvec[it->current.dpvec_index + 1]);
 
 	      if (lface_id > 0)
-		next_face_id = merge_faces (it->w, Qt, lface_id,
-					    it->saved_face_id);
+		{
+		  next_face_id = merge_faces (it->w, Qt, lface_id,
+		                              it->saved_face_id);
+		  it->extend_face_id =
+		    merge_extend_glyph_face (it, lface_id);
+		}
 	    }
 	}
       next_face = FACE_FROM_ID_OR_NULL (it->f, next_face_id);
@@ -8411,6 +8481,7 @@ next_element_from_ellipsis (struct it *it)
 	 was in IT->saved_face_id, and signal that it's there by
 	 setting face_before_selective_p.  */
       it->saved_face_id = it->face_id;
+      it->saved_extend_face_id = it->extend_face_id;
       it->method = GET_FROM_BUFFER;
       it->object = it->w->contents;
       reseat_at_next_visible_line_start (it, true);
@@ -12848,7 +12919,10 @@ display_tool_bar_line (struct it *it, int height)
      use the tool-bar face for the border too.  */
   if (!MATRIX_ROW_DISPLAYS_TEXT_P (row)
       && !EQ (Vauto_resize_tool_bars, Qgrow_only))
-    it->face_id = DEFAULT_FACE_ID;
+    {
+      it->face_id = DEFAULT_FACE_ID;
+      it->extend_face_id = DEFAULT_FACE_ID;
+    }
 
   extend_face_to_end_of_line (it);
   last = row->glyphs[TEXT_AREA] + row->used[TEXT_AREA] - 1;
@@ -20325,7 +20399,7 @@ append_space_for_newline (struct it *it, bool default_face_p)
 		     = XFIXNAT (Vdisplay_fill_column_indicator_character);
 	           it->face_id
 		     = merge_faces (it->w, Qfill_column_indicator,
-				    0, saved_face_id);
+		                    0, it->extend_face_id);
 	           face = FACE_FROM_ID (it->f, it->face_id);
 	           goto produce_glyphs;
 	         }
@@ -20472,33 +20546,33 @@ extend_face_to_end_of_line (struct it *it)
      1-``pixel'' wide, so they hit the equality too early.  This grace
      is needed only for R2L rows that are not continued, to produce
      one extra blank where we could display the cursor.  */
-  if ((it->current_x >= it->last_visible_x
-       + (!FRAME_WINDOW_P (f)
-	  && it->glyph_row->reversed_p
-	  && !it->glyph_row->continued_p))
-      /* If the window has display margins, we will need to extend
-	 their face even if the text area is filled.  */
-      && !(WINDOW_LEFT_MARGIN_WIDTH (it->w) > 0
-	   || WINDOW_RIGHT_MARGIN_WIDTH (it->w) > 0))
-    return;
-
-  /* Face extension extends the background and box of IT->face_id
-     to the end of the line.  If the background equals the background
-     of the frame, we don't have to do anything.  */
+/*   if ((it->current_x >= it->last_visible_x */
+/*        + (!FRAME_WINDOW_P (f) */
+/*           && it->glyph_row->reversed_p */
+/*           && !it->glyph_row->continued_p)) */
+/*       /\* If the window has display margins, we will need to extend */
+/* 	 their face even if the text area is filled.  *\/ */
+/*       && !(WINDOW_LEFT_MARGIN_WIDTH (it->w) > 0 */
+/* 	   || WINDOW_RIGHT_MARGIN_WIDTH (it->w) > 0)) */
+/*     return; */
+
+/*   /\* Face extension extends the background and box of IT->face_id */
+/*      to the end of the line.  If the background equals the background */
+/*      of the frame, we don't have to do anything.  *\/ */
   face = FACE_FROM_ID (f, (it->face_before_selective_p
-			   ? it->saved_face_id
-			   : it->face_id));
-
-  if (FRAME_WINDOW_P (f)
-      && MATRIX_ROW_DISPLAYS_TEXT_P (it->glyph_row)
-      && face->box == FACE_NO_BOX
-      && FACE_COLOR_TO_PIXEL (face->background, f) == FRAME_BACKGROUND_PIXEL (f)
-#ifdef HAVE_WINDOW_SYSTEM
-      && !face->stipple
-#endif
-      && !it->glyph_row->reversed_p
-      && !Vdisplay_fill_column_indicator)
-    return;
+			   ? it->saved_extend_face_id
+			   : it->extend_face_id));
+
+/*   if (FRAME_WINDOW_P (f) */
+/*       && MATRIX_ROW_DISPLAYS_TEXT_P (it->glyph_row) */
+/*       && face->box == FACE_NO_BOX */
+/*       && FACE_COLOR_TO_PIXEL (face->background, f) == FRAME_BACKGROUND_PIXEL (f) */
+/* #ifdef HAVE_WINDOW_SYSTEM */
+/*       && !face->stipple */
+/* #endif */
+/*       && !it->glyph_row->reversed_p */
+/*       && !Vdisplay_fill_column_indicator) */
+/*     return; */
 
   /* Set the glyph row flag indicating that the face of the last glyph
      in the text area has to be drawn to the end of the text area.  */
@@ -20561,79 +20635,88 @@ extend_face_to_end_of_line (struct it *it)
 	  /* Display fill column indicator if not in modeline or
 	     toolbar and display fill column indicator mode is
 	     active.  */
+	  const char saved_char = it->char_to_display;
+	  const struct text_pos saved_pos = it->position;
+	  const bool saved_avoid_cursor = it->avoid_cursor_p;
+	  const int saved_face_id = it->face_id;
+	  const bool saved_box_start = it->start_of_box_run_p;
+	  Lisp_Object save_object = it->object;
+
+	  it->avoid_cursor_p = true;
+	  it->object = Qnil;
+	  memset (&it->position, 0, sizeof it->position);
+
 	  int indicator_column = (it->w->pseudo_window_p == 0
 				  ? fill_column_indicator_column (it)
 				  : -1);
-	  if (indicator_column >= 0)
-            {
-	      struct font *font = (default_face->font
+
+	  struct font *font = (default_face->font
 				   ? default_face->font
 				   : FRAME_FONT (f));
-	      const int char_width = (font->average_width
-				      ? font->average_width
-				      : font->space_width);
-	      int column_x;
-
-	      if (!INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
-		  && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
-		  && column_x >= it->current_x
-		  && column_x <= it->last_visible_x)
-	        {
-	          const char saved_char = it->char_to_display;
-	          const struct text_pos saved_pos = it->position;
-	          const bool saved_avoid_cursor = it->avoid_cursor_p;
-	          const int saved_face_id = it->face_id;
-	          const bool saved_box_start = it->start_of_box_run_p;
-	          Lisp_Object save_object = it->object;
-
-	          /* The stretch width needs to considet the latter
-	             added glyph.  */
-	          const int stretch_width
-		    = column_x - it->current_x - char_width;
-
-	          memset (&it->position, 0, sizeof it->position);
-	          it->avoid_cursor_p = true;
-	          it->object = Qnil;
-
-	          /* Only generate a stretch glyph if there is distance
-	             between current_x and and the indicator position.  */
-	          if (stretch_width > 0)
-		    {
-		      int stretch_ascent = (((it->ascent + it->descent)
-		                             * FONT_BASE (font)) / FONT_HEIGHT (font));
-		      append_stretch_glyph (it, Qnil, stretch_width,
-		                            it->ascent + it->descent,
-		                            stretch_ascent);
-		    }
+	  const int char_width = (font->average_width
+	                          ? font->average_width
+	                          : font->space_width);
+	  int column_x;
+
+	  if (indicator_column >= 0
+	      && !INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
+	      && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
+	      && column_x >= it->current_x
+	      && column_x <= it->last_visible_x)
+	    {
+
+	      /* The stretch width needs to considet the latter
+		 added glyph.  */
+	      const int stretch_width
+		= column_x - it->current_x - char_width;
+
+	      /* Only generate a stretch glyph if there is distance
+		 between current_x and and the indicator position.  */
+	      if (stretch_width > 0)
+		{
+		  it->face_id = it->extend_face_id;
 
-	          /* Generate the glyph indicator only if
-	             append_space_for_newline didn't already.  */
-	          if (it->current_x < column_x)
-	            {
-		      it->char_to_display
-			= XFIXNAT (Vdisplay_fill_column_indicator_character);
-	              it->face_id
-			= merge_faces (it->w, Qfill_column_indicator,
-				       0, saved_face_id);
-	              PRODUCE_GLYPHS (it);
-	            }
-
-	          /* Restore the face after the indicator was generated.  */
-	          it->face_id = saved_face_id;
-
-	          /* If there is space after the indicator generate an
-	             extra empty glyph to restore the face.  Issue was
-	             observed in X systems.  */
-	          it->char_to_display = ' ';
-	          PRODUCE_GLYPHS (it);
-
-	          it->char_to_display = saved_char;
-	          it->position = saved_pos;
-	          it->avoid_cursor_p = saved_avoid_cursor;
-	          it->start_of_box_run_p = saved_box_start;
-	          it->object = save_object;
-	        }
-            }
+		  int stretch_ascent = (((it->ascent + it->descent)
+		                         * FONT_BASE (font)) / FONT_HEIGHT (font));
+		  append_stretch_glyph (it, Qnil, stretch_width,
+		                        it->ascent + it->descent,
+		                        stretch_ascent);
+		}
+
+	      /* Generate the glyph indicator only if
+		 append_space_for_newline didn't already.  */
+	      if (it->current_x < column_x)
+		{
+		  it->char_to_display
+		    = XFIXNAT (Vdisplay_fill_column_indicator_character);
+		  it->face_id
+		    = merge_faces (it->w, Qfill_column_indicator,
+		                   0, it->extend_face_id);
+		  PRODUCE_GLYPHS (it);
+		  it->char_to_display = saved_char;
+		}
+
+	    }
+
+	  const int stretch_width = it->last_visible_x - it->current_x;
+
+	  if (stretch_width > 0)
+	    {
+	      it->face_id = it->extend_face_id;
+
+	      int stretch_ascent = (((it->ascent + it->descent)
+	                             * FONT_BASE (font)) / FONT_HEIGHT (font));
+	      append_stretch_glyph (it, Qnil, stretch_width,
+	                            it->ascent + it->descent,
+	                            stretch_ascent);
+	    }
+
+	  it->char_to_display = saved_char;
+	  it->position = saved_pos;
+	  it->avoid_cursor_p = saved_avoid_cursor;
+	  it->face_id = saved_face_id;
+	  it->start_of_box_run_p = saved_box_start;
+	  it->object = save_object;
 	}
       if (it->glyph_row->reversed_p)
 	{
@@ -20679,10 +20762,9 @@ extend_face_to_end_of_line (struct it *it)
 	      /* The last row's stretch glyph should get the default
 		 face, to avoid painting the rest of the window with
 		 the region face, if the region ends at ZV.  */
-	      if (it->glyph_row->ends_at_zv_p)
-		it->face_id = default_face->id;
-	      else
-		it->face_id = face->id;
+	      it->face_id = (it->glyph_row->ends_at_zv_p ?
+	                     default_face->id : face->id);
+
 	      it->start_of_box_run_p = false;
 	      append_stretch_glyph (it, Qnil, stretch_width,
 				    it->ascent + it->descent, stretch_ascent);
@@ -20719,7 +20801,7 @@ extend_face_to_end_of_line (struct it *it)
       it->len = 1;
 
       if (WINDOW_LEFT_MARGIN_WIDTH (it->w) > 0
-	  && (it->glyph_row->used[LEFT_MARGIN_AREA]
+          && (it->glyph_row->used[LEFT_MARGIN_AREA]
 	      < WINDOW_LEFT_MARGIN_WIDTH (it->w))
 	  && !it->glyph_row->mode_line_p
 	  && FACE_COLOR_TO_PIXEL (face->background, f) != FRAME_BACKGROUND_PIXEL (f))
@@ -20750,10 +20832,8 @@ extend_face_to_end_of_line (struct it *it)
       /* The last row's blank glyphs should get the default face, to
 	 avoid painting the rest of the window with the region face,
 	 if the region ends at ZV.  */
-      if (it->glyph_row->ends_at_zv_p)
-	it->face_id = default_face->id;
-      else
-	it->face_id = face->id;
+      it->face_id = (it->glyph_row->ends_at_zv_p ?
+                     default_face->id : face->id);
 
       /* Display fill-column indicator if needed.  */
       int indicator_column = fill_column_indicator_column (it);
@@ -20763,24 +20843,25 @@ extend_face_to_end_of_line (struct it *it)
 	indicator_column = -1;
       do
 	{
-	  int saved_face_id;
-	  bool indicate = it->current_x == indicator_column;
-	  if (indicate)
+	  if (it->current_x == indicator_column)
 	    {
-	      saved_face_id = it->face_id;
+	      int saved_face_id = it->face_id;
+
 	      it->face_id
-		= merge_faces (it->w, Qfill_column_indicator, 0, saved_face_id);
+		= merge_faces (it->w, Qfill_column_indicator, 0,
+		               it->extend_face_id);
 	      it->c = it->char_to_display
 		= XFIXNAT (Vdisplay_fill_column_indicator_character);
-	    }
 
-	  PRODUCE_GLYPHS (it);
+	      PRODUCE_GLYPHS (it);
 
-	  if (indicate)
-	    {
 	      it->face_id = saved_face_id;
 	      it->c = it->char_to_display = ' ';
 	    }
+	  else
+	    {
+	      PRODUCE_GLYPHS (it);
+	    }
 	}
       while (it->current_x <= it->last_visible_x);
 
@@ -25423,7 +25504,8 @@ display_count_lines (ptrdiff_t start_byte,
    Value is the number of columns displayed.  */
 
 static int
-display_string (const char *string, Lisp_Object lisp_string, Lisp_Object face_string,
+display_string (const char *string, Lisp_Object lisp_string,
+                Lisp_Object face_string,
 		ptrdiff_t face_string_pos, ptrdiff_t start, struct it *it,
 		int field_width, int precision, int max_x, int multibyte)
 {
@@ -25446,12 +25528,13 @@ display_string (const char *string, Lisp_Object lisp_string, Lisp_Object face_st
   if (STRINGP (face_string))
     {
       ptrdiff_t endptr;
-      struct face *face;
 
       it->face_id
 	= face_at_string_position (it->w, face_string, face_string_pos,
 				   0, &endptr, it->base_face_id, false);
-      face = FACE_FROM_ID (it->f, it->face_id);
+
+      struct face *face = FACE_FROM_ID (it->f, it->face_id);
+
       it->face_box_p = face->box != FACE_NO_BOX;
     }
 
@@ -27355,7 +27438,7 @@ font_for_underline_metrics (struct glyph_string *s)
   for (g = s->first_glyph - 1; g >= g0; g--)
     {
       struct face *prev_face = FACE_FROM_ID (s->f, g->face_id);
-      if (!(prev_face && prev_face->underline_p))
+      if (!(prev_face && prev_face->underline != FACE_NO_UNDERLINE))
 	break;
     }
 
diff --git a/src/xfaces.c b/src/xfaces.c
index c3cae7e2a6..7987681ce9 100644
--- a/src/xfaces.c
+++ b/src/xfaces.c
@@ -1209,7 +1209,7 @@ free_face_colors (struct frame *f, struct face *face)
       IF_DEBUG (--ncolors_allocated);
     }
 
-  if (face->underline_p
+  if (face->underline
       && !face->underline_defaulted_p)
     {
       x_free_colors (f, &face->underline_color, 1);
@@ -1590,6 +1590,7 @@ #define LFACE_BOX(LFACE)	    AREF ((LFACE), LFACE_BOX_INDEX)
 #define LFACE_FONT(LFACE)	    AREF ((LFACE), LFACE_FONT_INDEX)
 #define LFACE_INHERIT(LFACE)	    AREF ((LFACE), LFACE_INHERIT_INDEX)
 #define LFACE_FONTSET(LFACE)	    AREF ((LFACE), LFACE_FONTSET_INDEX)
+#define LFACE_EXTEND(LFACE)	    AREF ((LFACE), LFACE_EXTEND_INDEX)
 #define LFACE_DISTANT_FOREGROUND(LFACE) \
   AREF ((LFACE), LFACE_DISTANT_FOREGROUND_INDEX)
 
@@ -2512,6 +2513,13 @@ merge_face_ref (struct window *w,
 					err_msgs, named_merge_points))
 		    err = true;
 		}
+	      else if (EQ (keyword, QCextend))
+		{
+		  if (EQ (value, Qt) || NILP (value))
+		    to[LFACE_EXTEND_INDEX] = value;
+		  else
+		    err = true;
+		}
 	      else
 		err = true;
 
@@ -3030,6 +3038,17 @@ DEFUN ("internal-set-lisp-face-attribute", Finternal_set_lisp_face_attribute,
       old_value = LFACE_INVERSE (lface);
       ASET (lface, LFACE_INVERSE_INDEX, value);
     }
+  else if (EQ (attr, QCextend))
+    {
+      if (!UNSPECIFIEDP (value) && !IGNORE_DEFFACE_P (value))
+	{
+	  CHECK_SYMBOL (value);
+	  if (!EQ (value, Qt) && !NILP (value))
+	    signal_error ("Invalid inverse-video face attribute value", value);
+	}
+      old_value = LFACE_EXTEND (lface);
+      ASET (lface, LFACE_EXTEND_INDEX, value);
+    }
   else if (EQ (attr, QCforeground))
     {
       /* Compatibility with 20.x.  */
@@ -3503,7 +3522,9 @@ DEFUN ("internal-set-lisp-face-attribute-from-resource",
     value = face_boolean_x_resource_value (value, true);
   else if (EQ (attr, QCweight) || EQ (attr, QCslant) || EQ (attr, QCwidth))
     value = intern (SSDATA (value));
-  else if (EQ (attr, QCreverse_video) || EQ (attr, QCinverse_video))
+  else if (EQ (attr, QCreverse_video)
+           || EQ (attr, QCinverse_video)
+           || EQ (attr, QCextend))
     value = face_boolean_x_resource_value (value, true);
   else if (EQ (attr, QCunderline)
 	   || EQ (attr, QCoverline)
@@ -3727,6 +3748,8 @@ DEFUN ("internal-get-lisp-face-attribute", Finternal_get_lisp_face_attribute,
     value = LFACE_SWIDTH (lface);
   else if (EQ (keyword, QCinherit))
     value = LFACE_INHERIT (lface);
+  else if (EQ (keyword, QCextend))
+    value = LFACE_EXTEND (lface);
   else if (EQ (keyword, QCfont))
     value = LFACE_FONT (lface);
   else if (EQ (keyword, QCfontset))
@@ -5694,16 +5717,14 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
   if (EQ (underline, Qt))
     {
       /* Use default color (same as foreground color).  */
-      face->underline_p = true;
-      face->underline_type = FACE_UNDER_LINE;
+      face->underline = FACE_UNDER_LINE;
       face->underline_defaulted_p = true;
       face->underline_color = 0;
     }
   else if (STRINGP (underline))
     {
       /* Use specified color.  */
-      face->underline_p = true;
-      face->underline_type = FACE_UNDER_LINE;
+      face->underline = FACE_UNDER_LINE;
       face->underline_defaulted_p = false;
       face->underline_color
 	= load_color (f, face, underline,
@@ -5711,7 +5732,7 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
     }
   else if (NILP (underline))
     {
-      face->underline_p = false;
+      face->underline = FACE_NO_UNDERLINE;
       face->underline_defaulted_p = false;
       face->underline_color = 0;
     }
@@ -5719,10 +5740,9 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
     {
       /* `(:color COLOR :style STYLE)'.
          STYLE being one of `line' or `wave'. */
-      face->underline_p = true;
+      face->underline = FACE_UNDER_LINE;
       face->underline_color = 0;
       face->underline_defaulted_p = true;
-      face->underline_type = FACE_UNDER_LINE;
 
       /* FIXME?  This is also not robust about checking the precise form.
          See comments in Finternal_set_lisp_face_attribute.  */
@@ -5755,9 +5775,9 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
           else if (EQ (keyword, QCstyle))
             {
               if (EQ (value, Qline))
-                face->underline_type = FACE_UNDER_LINE;
+                face->underline = FACE_UNDER_LINE;
               else if (EQ (value, Qwave))
-                face->underline_type = FACE_UNDER_WAVE;
+                face->underline = FACE_UNDER_WAVE;
             }
         }
     }
@@ -6292,9 +6312,8 @@ merge_faces (struct window *w, Lisp_Object face_name, int face_id,
 {
   struct frame *f = WINDOW_XFRAME (w);
   Lisp_Object attrs[LFACE_VECTOR_SIZE];
-  struct face *base_face;
+  struct face *base_face = FACE_FROM_ID_OR_NULL (f, base_face_id);
 
-  base_face = FACE_FROM_ID_OR_NULL (f, base_face_id);
   if (!base_face)
     return base_face_id;
 
@@ -6319,12 +6338,14 @@ merge_faces (struct window *w, Lisp_Object face_name, int face_id,
     }
   else
     {
-      struct face *face;
       if (face_id < 0)
 	return base_face_id;
-      face = FACE_FROM_ID_OR_NULL (f, face_id);
+
+      struct face *face = FACE_FROM_ID_OR_NULL (f, face_id);
+
       if (!face)
 	return base_face_id;
+
       merge_face_vectors (w, f, face->lface, attrs, 0);
     }
 
@@ -6412,7 +6433,7 @@ dump_realized_face (struct face *face)
 #endif
   fprintf (stderr, "fontset: %d\n", face->fontset);
   fprintf (stderr, "underline: %d (%s)\n",
-	   face->underline_p,
+	   face->underline,
 	   SDATA (Fsymbol_name (face->lface[LFACE_UNDERLINE_INDEX])));
   fprintf (stderr, "hash: %" PRIuPTR "\n", face->hash);
 }
@@ -6537,6 +6558,7 @@ syms_of_xfaces (void)
   DEFSYM (QCstrike_through, ":strike-through");
   DEFSYM (QCbox, ":box");
   DEFSYM (QCinherit, ":inherit");
+  DEFSYM (QCextend, ":extend");
 
   /* Symbols used for Lisp face attribute values.  */
   DEFSYM (QCcolor, ":color");
diff --git a/src/xterm.c b/src/xterm.c
index b761eaf4d1..b8f8db56a7 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -3798,9 +3798,9 @@ x_draw_glyph_string (struct glyph_string *s)
   if (!s->for_overlaps)
     {
       /* Draw underline.  */
-      if (s->face->underline_p)
+      if (s->face->underline)
         {
-          if (s->face->underline_type == FACE_UNDER_WAVE)
+          if (s->face->underline == FACE_UNDER_WAVE)
             {
               if (s->face->underline_defaulted_p)
                 x_draw_underwave (s);
@@ -3814,13 +3814,13 @@ x_draw_glyph_string (struct glyph_string *s)
                   XSetForeground (display, s->gc, xgcv.foreground);
                 }
             }
-          else if (s->face->underline_type == FACE_UNDER_LINE)
+          else if (s->face->underline == FACE_UNDER_LINE)
             {
               unsigned long thickness, position;
               int y;
 
-              if (s->prev && s->prev->face->underline_p
-		  && s->prev->face->underline_type == FACE_UNDER_LINE)
+              if (s->prev &&
+	          s->prev->face->underline == FACE_UNDER_LINE)
                 {
                   /* We use the same underline style as the previous one.  */
                   thickness = s->prev->underline_thickness;

Re: Question about display engine

[martin rudalics]

 > Here is the diff of the latest commit. (Patch attached anyway).

Thanks.  I tried with the patch.patch you attached.  When trying a gtk
build I get:

../../src/xfaces.c:5434: Emacs fatal error: assertion failed: lface_fully_specified_p (XVECTOR (lface)->contents)
Fatal error 6: Aborted
Backtrace:
../src/bootstrap-emacs[0x67e32f]
../src/bootstrap-emacs[0x6344c8]
../src/bootstrap-emacs[0x74fb6f]
../src/bootstrap-emacs[0x5a8e43]
../src/bootstrap-emacs[0x5a83f8]
../src/bootstrap-emacs[0x59bccf]
../src/bootstrap-emacs[0x436ad1]
../src/bootstrap-emacs[0x4fd0ad]
../src/bootstrap-emacs[0x76376c]
../src/bootstrap-emacs[0x63530c]
../src/bootstrap-emacs[0x635860]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f68217952e1]
../src/bootstrap-emacs[0x4151aa]
/bin/bash: Zeile 2: 12088 Abgebrochen             EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../../lisp/cus-face.el
Makefile:280: die Regel für Ziel „../../lisp/cus-face.elc“ scheiterte
make[2]: *** [../../lisp/cus-face.elc] Fehler 134
Makefile:784: die Regel für Ziel „../../lisp/cus-face.elc“ scheiterte
make[1]: *** [../../lisp/cus-face.elc] Fehler 2
make[1]: *** Es wird auf noch nicht beendete Prozesse gewartet...

../../src/xfaces.c:5434: Emacs fatal error: assertion failed: lface_fully_specified_p (XVECTOR (lface)->contents)
Backtrace:
../src/bootstrap-emacs[0x67e32f]
../src/bootstrap-emacs[0x6344c8]
../src/bootstrap-emacs[0x74fb6f]
../src/bootstrap-emacs[0x5a8e43]
../src/bootstrap-emacs[0x5a83f8]
../src/bootstrap-emacs[0x59bccf]
../src/bootstrap-emacs[0x436ad1]
../src/bootstrap-emacs[0x4fd0ad]
../src/bootstrap-emacs[0x76376c]
../src/bootstrap-emacs[0x63530c]
../src/bootstrap-emacs[0x635860]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f0e8f3a32e1]
../src/bootstrap-emacs[0x4151aa]
/bin/bash: Zeile 2: 12090 Abgebrochen             EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../../lisp/faces.el
Makefile:280: die Regel für Ziel „../../lisp/faces.elc“ scheiterte
make[2]: *** [../../lisp/faces.elc] Fehler 134
Makefile:784: die Regel für Ziel „../../lisp/faces.elc“ scheiterte
make[1]: *** [../../lisp/faces.elc] Fehler 2
make[1]: Verzeichnis „/home/martin/emacs-git/trunk/obj-gtk/src“ wird verlassen
Makefile:424: die Regel für Ziel „src“ scheiterte
make: *** [src] Fehler 2

and a similar crash on Windows.  Before proceeding to dig into this
I'd like to hear your ideas.

 > https://github.com/Ergus/Emacs/commit/4943087027acd3f2c7a54a092b64bc839ef8850e

Is there any way to get the diffs wrt current master on that site?  I
never use github for such a thing and my browser settings are quite
restrictive.

Thanks, martin

Re: Question about display engine

[Ergus]

[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]

For some reason I was not facing this; but it was actually a bug I just fixed. I'll send a patch in a while because this exposed an issue somewhere else.


-----Original Message-----
From: martin rudalics <rudalics@gmx.at>
To: Ergus <spacibba@aol.com>
Cc: Eli Zaretskii <eliz@gnu.org>; emacs-devel <emacs-devel@gnu.org>
Sent: Thu, Sep 5, 2019 11:24 am
Subject: Re: Question about display engine

> Here is the diff of the latest commit. (Patch attached anyway).

Thanks.  I tried with the patch.patch you attached.  When trying a gtk
build I get:

../../src/xfaces.c:5434: Emacs fatal error: assertion failed: lface_fully_specified_p (XVECTOR (lface)->contents)
Fatal error 6: Aborted
Backtrace:
../src/bootstrap-emacs[0x67e32f]
../src/bootstrap-emacs[0x6344c8]
../src/bootstrap-emacs[0x74fb6f]
../src/bootstrap-emacs[0x5a8e43]
../src/bootstrap-emacs[0x5a83f8]
../src/bootstrap-emacs[0x59bccf]
../src/bootstrap-emacs[0x436ad1]
../src/bootstrap-emacs[0x4fd0ad]
../src/bootstrap-emacs[0x76376c]
../src/bootstrap-emacs[0x63530c]
../src/bootstrap-emacs[0x635860]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f68217952e1]
../src/bootstrap-emacs[0x4151aa]
/bin/bash: Zeile 2: 12088 Abgebrochen            EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../../lisp/cus-face.el
Makefile:280: die Regel für Ziel „../../lisp/cus-face.elc“ scheiterte
make[2]: *** [../../lisp/cus-face.elc] Fehler 134
Makefile:784: die Regel für Ziel „../../lisp/cus-face.elc“ scheiterte
make[1]: *** [../../lisp/cus-face.elc] Fehler 2
make[1]: *** Es wird auf noch nicht beendete Prozesse gewartet...

../../src/xfaces.c:5434: Emacs fatal error: assertion failed: lface_fully_specified_p (XVECTOR (lface)->contents)
Backtrace:
../src/bootstrap-emacs[0x67e32f]
../src/bootstrap-emacs[0x6344c8]
../src/bootstrap-emacs[0x74fb6f]
../src/bootstrap-emacs[0x5a8e43]
../src/bootstrap-emacs[0x5a83f8]
../src/bootstrap-emacs[0x59bccf]
../src/bootstrap-emacs[0x436ad1]
../src/bootstrap-emacs[0x4fd0ad]
../src/bootstrap-emacs[0x76376c]
../src/bootstrap-emacs[0x63530c]
../src/bootstrap-emacs[0x635860]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f0e8f3a32e1]
../src/bootstrap-emacs[0x4151aa]
/bin/bash: Zeile 2: 12090 Abgebrochen            EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../../lisp/faces.el
Makefile:280: die Regel für Ziel „../../lisp/faces.elc“ scheiterte
make[2]: *** [../../lisp/faces.elc] Fehler 134
Makefile:784: die Regel für Ziel „../../lisp/faces.elc“ scheiterte
make[1]: *** [../../lisp/faces.elc] Fehler 2
make[1]: Verzeichnis „/home/martin/emacs-git/trunk/obj-gtk/src“ wird verlassen
Makefile:424: die Regel für Ziel „src“ scheiterte
make: *** [src] Fehler 2

and a similar crash on Windows.  Before proceeding to dig into this
I'd like to hear your ideas.

 > https://github.com/Ergus/Emacs/commit/4943087027acd3f2c7a54a092b64bc839ef8850e

Is there any way to get the diffs wrt current master on that site?  I
never use github for such a thing and my browser settings are quite
restrictive.

Thanks, martin

[-- Attachment #2: Type: text/html, Size: 4921 bytes --]

Re: Question about display engine

[Ergus]

[-- Attachment #1: Type: text/plain, Size: 3757 bytes --]

BTW: in the code are come comments starting with // ERGUS: that is code not clear for me what to do there.  


-----Original Message-----
From: Ergus <spacibba@aol.com>
To: rudalics <rudalics@gmx.at>
Cc: eliz <eliz@gnu.org>; emacs-devel <emacs-devel@gnu.org>
Sent: Thu, Sep 5, 2019 3:55 pm
Subject: Re: Question about display engine

For some reason I was not facing this; but it was actually a bug I just fixed. I'll send a patch in a while because this exposed an issue somewhere else.


-----Original Message-----
From: martin rudalics <rudalics@gmx.at>
To: Ergus <spacibba@aol.com>
Cc: Eli Zaretskii <eliz@gnu.org>; emacs-devel <emacs-devel@gnu.org>
Sent: Thu, Sep 5, 2019 11:24 am
Subject: Re: Question about display engine

> Here is the diff of the latest commit. (Patch attached anyway).

Thanks.  I tried with the patch.patch you attached.  When trying a gtk
build I get:

../../src/xfaces.c:5434: Emacs fatal error: assertion failed: lface_fully_specified_p (XVECTOR (lface)->contents)
Fatal error 6: Aborted
Backtrace:
../src/bootstrap-emacs[0x67e32f]
../src/bootstrap-emacs[0x6344c8]
../src/bootstrap-emacs[0x74fb6f]
../src/bootstrap-emacs[0x5a8e43]
../src/bootstrap-emacs[0x5a83f8]
../src/bootstrap-emacs[0x59bccf]
../src/bootstrap-emacs[0x436ad1]
../src/bootstrap-emacs[0x4fd0ad]
../src/bootstrap-emacs[0x76376c]
../src/bootstrap-emacs[0x63530c]
../src/bootstrap-emacs[0x635860]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f68217952e1]
../src/bootstrap-emacs[0x4151aa]
/bin/bash: Zeile 2: 12088 Abgebrochen            EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../../lisp/cus-face.el
Makefile:280: die Regel für Ziel „../../lisp/cus-face.elc“ scheiterte
make[2]: *** [../../lisp/cus-face.elc] Fehler 134
Makefile:784: die Regel für Ziel „../../lisp/cus-face.elc“ scheiterte
make[1]: *** [../../lisp/cus-face.elc] Fehler 2
make[1]: *** Es wird auf noch nicht beendete Prozesse gewartet...

../../src/xfaces.c:5434: Emacs fatal error: assertion failed: lface_fully_specified_p (XVECTOR (lface)->contents)
Backtrace:
../src/bootstrap-emacs[0x67e32f]
../src/bootstrap-emacs[0x6344c8]
../src/bootstrap-emacs[0x74fb6f]
../src/bootstrap-emacs[0x5a8e43]
../src/bootstrap-emacs[0x5a83f8]
../src/bootstrap-emacs[0x59bccf]
../src/bootstrap-emacs[0x436ad1]
../src/bootstrap-emacs[0x4fd0ad]
../src/bootstrap-emacs[0x76376c]
../src/bootstrap-emacs[0x63530c]
../src/bootstrap-emacs[0x635860]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f0e8f3a32e1]
../src/bootstrap-emacs[0x4151aa]
/bin/bash: Zeile 2: 12090 Abgebrochen            EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../../lisp/faces.el
Makefile:280: die Regel für Ziel „../../lisp/faces.elc“ scheiterte
make[2]: *** [../../lisp/faces.elc] Fehler 134
Makefile:784: die Regel für Ziel „../../lisp/faces.elc“ scheiterte
make[1]: *** [../../lisp/faces.elc] Fehler 2
make[1]: Verzeichnis „/home/martin/emacs-git/trunk/obj-gtk/src“ wird verlassen
Makefile:424: die Regel für Ziel „src“ scheiterte
make: *** [src] Fehler 2

and a similar crash on Windows.  Before proceeding to dig into this
I'd like to hear your ideas.

 > https://github.com/Ergus/Emacs/commit/4943087027acd3f2c7a54a092b64bc839ef8850e

Is there any way to get the diffs wrt current master on that site?  I
never use github for such a thing and my browser settings are quite
restrictive.

Thanks, martin

[-- Attachment #2: Type: text/html, Size: 5809 bytes --]

Re: Question about display engine

[Ergus]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]


Hi again:
I was just checking the code once again and it is actually working. Theproblem before was in the lisp part. the interactive functions:
(set-face-extend 'region t)
does not change the face value when called interactively. (We haveobserved similar issues before with customize-variable and maybe itneeds to be fix.) They seem to be changing the values of the variableslocally in the minibuffer instead of the "caller" buffer. But it workswhen executed throw C-x C-e; so maybe some lisper can give a look tothis please.
As now it works at least for the region and fixes these issues.
1) Tui and gui extension is consistent (still needs some work but shouldbe a minor issues to fix)
2) the region extension can be customized (which could be considered anew feature).
3) The interaction with dfci reported in the bug that started this isnot broken anymore.
At the end I implemented it lazily because it appeared to be the easiestalternative for me; but I am completely open to any comment/suggestion(please ignore code style for now.)
Finally I have a explicit question:
when we set :extend nil for the region face do you consider correct thatthe extra space we always add must have the region color (instead of thedefault) in order to seea colored space in the empty lines?



-----Original Message-----
From: martin rudalics <rudalics@gmx.at>
To: Ergus <spacibba@aol.com>
Cc: Eli Zaretskii <eliz@gnu.org>; emacs-devel <emacs-devel@gnu.org>
Sent: Wed, Sep 4, 2019 10:34 pm
Subject: Re: Question about display engine

> I have just uploaded some changes but the functionality is still not
 > working.
 >
 > I separated the changes in 3 commits and in the last one are only the
 > ones I made in the xdisp.c (the ones that need to be checked, because
 > the rest is only infrastructure.)
 >
 > see the master branch in: https://github.com/Ergus/Emacs

Please send a patch for current master.  Otherwise I have no idea how
to compare your changes.

Thanks, martin


[-- Attachment #2: Type: text/html, Size: 3886 bytes --]

Re: Question about display engine

[martin rudalics]

 > Extending the underline|overline to the right of the line in the region
 > is a fancy detail, but I don't find it useful at all (underlining empty
 > long spaces is conceptually an error), and in the actual gui it is not
 > happening (not even extend_face_to_end_of_line in gui is executed as the
 > background color is extended automatically somewhere else) and there
 > haven't been complains either, so there is not people using that, so why
 > do we provide a complex solution implementation for a problem nobody
 > really cares and that potentially will produce overheads, code
 > complexity and more issues?
 >
 > I am wondering about over-specifications and over-engineering for such a
 > detail, when most of the users only need to extend the background color.

Agreed.  But what would we do when in the foreseeable future somebody
asks us to provide such a feature?  No one here has Eli's experience
when it comes to guessing at what Emacs users might eventually want.

martin

Re: Question about display engine

[Eli Zaretskii]

> Date: Thu, 8 Aug 2019 10:38:04 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> >I now think it would make more sense to add an 'extend-to-window-edge'
> >attribute in the face definition itself.
> 
> I like that option as a concept, but it adds a new flag to a general
> struct like the face for something that only affects the region face
> now.

As mentioned up-thread, this affects not only the region face.  So
perhaps a face attribute is indeed a good solution.

But I still feel that this is not about faces, this is about face
attributes, so the customization should on the attribute level, not on
the face level.

Re: Question about display engine

[martin rudalics]

 > As mentioned up-thread, this affects not only the region face.  So
 > perhaps a face attribute is indeed a good solution.
 >
 > But I still feel that this is not about faces, this is about face
 > attributes, so the customization should on the attribute level, not on
 > the face level.

You're right.  But then the earlier 'face-extend-to-window-edge' is
not appropriate either.  Maybe we could start with a simple
'extend-face-background' option (and add others if needed)?  If we
agree that "most" backgrounds should not extend, otherwise we should
probably add an 'inhibit-extend-face-background' option.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Fri, 9 Aug 2019 14:46:03 +0200
> 
>  > But I still feel that this is not about faces, this is about face
>  > attributes, so the customization should on the attribute level, not on
>  > the face level.
> 
> You're right.  But then the earlier 'face-extend-to-window-edge' is
> not appropriate either.

Right, that idea eats dust.

> Maybe we could start with a simple 'extend-face-background' option
> (and add others if needed)?

What about users who change the region or hl-line faces to use
underlining?

IOW, the problem with the attribute-level idea is that it will affect
those attributes regardless of the face from which they came.

So I now tend to think that if we consider some faces not eligible for
extension, we should not extend any face attributes at all.  Too bad
no one else except the 3 of us is talking in this thread; we need more
opinions.

Re: Question about display engine

[Stefan Monnier]

> IOW, the problem with the attribute-level idea is that it will affect
> those attributes regardless of the face from which they came.

Per-face indeed sounds like a better option than per-attribute.
Furthermore, it should be much easier to add it in a clean way to the
current system.

Of course per-face per-attribute would be even more flexible, but I'm
not completely convinced it's worth the trouble.


        Stefan

Re: Question about display engine

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 10 Aug 2019 19:04:13 -0400
> Cc: martin rudalics <rudalics@gmx.at>, spacibba@aol.com, emacs-devel@gnu.org
> 
> > IOW, the problem with the attribute-level idea is that it will affect
> > those attributes regardless of the face from which they came.
> 
> Per-face indeed sounds like a better option than per-attribute.
> Furthermore, it should be much easier to add it in a clean way to the
> current system.
> 
> Of course per-face per-attribute would be even more flexible, but I'm
> not completely convinced it's worth the trouble.

I later came to the conclusion that even per-face is not really
workable, due to face merging.  WDYT?

Re: Question about display engine

[martin rudalics]

 >> Per-face indeed sounds like a better option than per-attribute.
 >> Furthermore, it should be much easier to add it in a clean way to the
 >> current system.
 >>
 >> Of course per-face per-attribute would be even more flexible, but I'm
 >> not completely convinced it's worth the trouble.
 >
 > I later came to the conclusion that even per-face is not really
 > workable, due to face merging.  WDYT?

I think both would be feasible.  Just that with a face attribute the
face merger would have to specify whether a certain property should
extend to the end of line while with an attribute property the (face
agnostic) display engine could handle face extension alone.

Suppose we want to display a newline character that is both
highlighted and part of the region and assume that highlighting is
realized via :underline and the region via :background.

With the face attribute based approach, the face merger would set a
'background-extend' flag according to the :extend attribute of the
region face and an 'underline-extend' flag according to the :extend
attribute of the highlight face.  The face agnostic display engine
would use these flags to decide whether the corresponding properties
should extend to the edge of the window or not

With the attribute based approach, the display engine would extend the
background directly from the newline character corresponding to
whether the background attribute's extend property was set (for
example, because 'background' is explicitly listed in an
'attribute-extend' variable) and do the same for the newline's
underline property (if we wanted to allow that).

The face-based approach will "misfire" when a user decides to show the
region via underline and sets the region's extend property.  I'd
consider that a pilot error.  The attribute-based approach will
misfire when the user makes the underline attribute have the extend
property (if we allowed to do that).  Or am I missing something?

martin

Re: Question about display engine

[martin rudalics]

 >> Maybe we could start with a simple 'extend-face-background' option
 >> (and add others if needed)?
 >
 > What about users who change the region or hl-line faces to use
 > underlining?

Any underlining behavior would remain unaffected by setting
'extend-face-background'.  If both, underlining and background are
specified, the display engine would do what it does now.

 > IOW, the problem with the attribute-level idea is that it will affect
 > those attributes regardless of the face from which they came.

If the 'extend-face-background' is a simple boolean, yes.  If it is a
list of faces, it is largely idempotent to the face-level concept: A
user could say (by including or excluding) a face from that list
whether it should have any impact on extending the background to the
end of the window edge.  Just that the face merger would have to
process that variable if it is a list of faces while handling it in
the display engine would be sufficient for a boolean.

 > So I now tend to think that if we consider some faces not eligible for

Didn't you mean to say "some face attributes not eligible" here ...

 > extension, we should not extend any face attributes at all.  Too bad
 > no one else except the 3 of us is talking in this thread; we need more
 > opinions.

... because otherwise I don't see how we possibly could do that.  IIUC
the face merger would then have to decide whether "that" occurred.

martin

Re: Question about display engine

[Eli Zaretskii]

> Date: Thu, 8 Aug 2019 10:38:04 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Looking at the merge_face function could anyone please explain better
> what means: realized face and lface_id and please direct me to where is
> documented how emacs uses the faces internally; the functionalities
> available and specially the merge rules?

You need to read xfaces.c, the code is all there.  Feel free to ask
questions if the code doesn't speak for itself.

Re: Question about display engine

[Ergus]

On Wed, Aug 07, 2019 at 07:41:25PM +0300, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: martin rudalics <rudalics@gmx.at>
>> Date: Wed, 7 Aug 2019 18:25:37 +0200
>>
>>  > And, of course, this leaves the more general problem I described in my
>>  > message: what is the right behavior for extending the face that
>>  > crosses a newline.  (Note that the display engine in general doesn't
>>  > know whether a face that doesn't end before a newline will or won't
>>  > continue on the next screen line.)
>>
>> Recalling your proposal
>>
>> (defcustom face-extend-to-window-edge t
>>     "Non-nil means extend face of last character on line to window edge.
>
>Thanks for the reminder.
>
>> Certain face attributes, if present in the face of the last character
>> of a line and different from those of the default face, cause the
>> empty space following the end of text on the line to be drawn with
>> those attributes, to give the empty space appearance similar to that
>> of the preceding text.  These attributes are those which affect the
>> background of a face: `:background', `:stipple', `:box', `:underline',
>> `:overline', and `:strike-through'.  By default, if the face of a
>> line's last character has any of these attributes, and the value is
>> different from that of the default face, the empty space following the
>> line's text will be drawn in the face of the last character.
>
>This is inaccurate, I think: on GUI frames only :background and :box
>are extended all the way, the rest only "infect" the glyph we add for
>displaying the newline.
>
>> This variable allows fine-tuning which attributes trigger the face
>> extension.  The default value of t means any of the mentioned
>> attributes will cause face extension.  The value of nil means face
>> extension is turned off.  A value that is a list of attributes will
>> extend the face only if any of the attributes from the list are
>> present in the last character's face.  Note that only attributes from
>> the above list are meaningful in list values of this variable.")
>>
>> in the discussion of Bug#23574.
>
>And I think we need to consider the case of the region using one of
>the attributes that are not in the list.
>
In gui our actual behavior seems to be very reasonable and simple (And
we haven't too many complains about this.). It extends the region
colors, but not the underline (and similes). For me this makes sense
because underline empty spaces after \n, until the page border, does not
really means anything as conceptually there is nothing that should be
underlined there.

The "surprise" is that there is no code for this effect in the display
engine and it has to do with some X||gtk feature in use. (That's why I
needed to ad an extra glyph after the indicator.)

Comparing around:

VIM, when toggle to visual mode it only highlights the real
text. The rest of the line stays unchanged. This is useful to detect
trailing whitespaces when no set by default (I am not telling that we
should adopt that approach, but it looks very reasonable and simple and
it is the default in most gui libraries; so also for the users)

Geany, QtCreator, Sublime, Web Browsers (Firefox, chromium), MS Word, all of
them do the same.

Eclipse, on the other hand, extends the color to the end of the line as
we do, but it only allows to set background and foreground to what we
call region. So no underlined region to worry about.

I think that these are the most extended approaches around (so we don't
need to reinvent the wheel) and Eli should make the design choice so
we/he can start implementing that; otherwise it will be forgotten after
hours and hours of arguments and at the end there will be always
somebody unhappy.

Re: Question about display engine

[martin rudalics]

 > VIM, when toggle to visual mode it only highlights the real
 > text. The rest of the line stays unchanged. This is useful to detect
 > trailing whitespaces when no set by default

... at the expense of not being able to tell where the region ends or
starts within a bunch of consecutive newlines ...

 > (I am not telling that we
 > should adopt that approach, but it looks very reasonable and simple and
 > it is the default in most gui libraries; so also for the users)

 > I think that these are the most extended approaches around (so we don't
 > need to reinvent the wheel) and Eli should make the design choice so
 > we/he can start implementing that; otherwise it will be forgotten after
 > hours and hours of arguments and at the end there will be always
 > somebody unhappy.

Indeed.

martin

Re: Question about display engine

[Ergus]

On August 8, 2019 10:45:49 AM GMT+02:00, martin rudalics <rudalics@gmx.at> wrote:
> > VIM, when toggle to visual mode it only highlights the real
> > text. The rest of the line stays unchanged. This is useful to detect
> > trailing whitespaces when no set by default
>
>... at the expense of not being able to tell where the region ends or
>starts within a bunch of consecutive newlines ...
>

In our case in gui we have a space for new line that can be highlited even in empty lines at the end of the line. In vim this space only exists in insert mode.

> > (I am not telling that we
>> should adopt that approach, but it looks very reasonable and simple
>and
> > it is the default in most gui libraries; so also for the users)
>
>> I think that these are the most extended approaches around (so we
>don't
> > need to reinvent the wheel) and Eli should make the design choice so
>> we/he can start implementing that; otherwise it will be forgotten
>after
> > hours and hours of arguments and at the end there will be always
> > somebody unhappy.
>
>Indeed.
>
>martin

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Question about display engine

[Eli Zaretskii]

> Date: Thu, 8 Aug 2019 10:15:53 +0200
> From: Ergus <spacibba@aol.com>
> Cc: martin rudalics <rudalics@gmx.at>, emacs-devel@gnu.org
> 
> In gui our actual behavior seems to be very reasonable and simple (And
> we haven't too many complains about this.). It extends the region
> colors, but not the underline (and similes). For me this makes sense
> because underline empty spaces after \n, until the page border, does not
> really means anything as conceptually there is nothing that should be
> underlined there.
> 
> The "surprise" is that there is no code for this effect in the display
> engine and it has to do with some X||gtk feature in use. (That's why I
> needed to ad an extra glyph after the indicator.)

We simply clear-to-end-of-line using the background color.

> I think that these are the most extended approaches around (so we don't
> need to reinvent the wheel) and Eli should make the design choice so
> we/he can start implementing that; otherwise it will be forgotten after
> hours and hours of arguments and at the end there will be always
> somebody unhappy.

I don't think this is just my decision, I still hope others will chime
in with opinions.

Re: Question about display engine

[Juri Linkov]

> I think that these are the most extended approaches around (so we don't
> need to reinvent the wheel) and Eli should make the design choice so
> we/he can start implementing that; otherwise it will be forgotten after
> hours and hours of arguments and at the end there will be always
> somebody unhappy.

While designing please consider supporting at least 3 options:
- extend highlighting to the window edge as most faces do now;
- highlight only to the end of lines as underline does now;
- limit highlighting to a fixed customizable column like
  display-fill-column-indicator defaulting to fill-column.

Re: Question about display engine

[Ergus]

On Thu, Aug 08, 2019 at 11:37:32PM +0300, Juri Linkov wrote:
>> I think that these are the most extended approaches around (so we don't
>> need to reinvent the wheel) and Eli should make the design choice so
>> we/he can start implementing that; otherwise it will be forgotten after
>> hours and hours of arguments and at the end there will be always
>> somebody unhappy.
>
>While designing please consider supporting at least 3 options:
>- extend highlighting to the window edge as most faces do now;
>- highlight only to the end of lines as underline does now;
>- limit highlighting to a fixed customizable column like
>  display-fill-column-indicator defaulting to fill-column.


Hi Juri:

Up to now we are only dealing with the first two cases... The third one
will add even more complexity (and corner cases when dealing with dfci)
for a very low gain (in my opinion) because I don't find yet any use
case for it. I am not telling it is impossible, just unneeded.

If you have a good use case for it, please share it.

Re: Question about display engine

[Eli Zaretskii]

> Date: Fri, 9 Aug 2019 00:24:04 +0200
> From: Ergus <spacibba@aol.com>
> Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org
> 
> If you have a good use case for it, please share it.

Seconded.

Re: Question about display engine

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 867 bytes --]

>>- extend highlighting to the window edge as most faces do now;
>>- highlight only to the end of lines as underline does now;
>>- limit highlighting to a fixed customizable column like
>>  display-fill-column-indicator defaulting to fill-column.
>
> Up to now we are only dealing with the first two cases... The third one
> will add even more complexity (and corner cases when dealing with dfci)
> for a very low gain (in my opinion) because I don't find yet any use
> case for it. I am not telling it is impossible, just unneeded.
>
> If you have a good use case for it, please share it.

The third one is for purely aesthetic reasons: a single window on a very
wide frame usually is full of ribbons (like on the first screenshot)
whereas for example, the second screenshot (that was edited) shows
how limiting face scope by fill-column is more visually pleasant.


[-- Attachment #2: 1_window-edge-faces.png --]
[-- Type: image/png, Size: 129102 bytes --]

[-- Attachment #3: 2_fill-column-faces.png --]
[-- Type: image/png, Size: 129138 bytes --]

Re: Question about display engine

[Keith David Bershatsky]

Inasmuch as Ergus has recently expressed a desire to work on face extensions, I thought it would be helpful to bring to everyone's attention a couple of recent threads where a user is attempting to workaround (using Lisp) a background face that gets extended when folding source code blocks in org-mode:

https://www.reddit.com/r/emacs/comments/cw0499/prevent_folded_headings_from_bleeding_out/?ref=share&ref_source=link

https://emacs.stackexchange.com/q/52324/2287

Thanks,

Keith

Re: Question about display engine

[Ergus]

[-- Attachment #1.1: Type: text/plain, Size: 5169 bytes --]

Hi Eli and Martin:

I attach here a new patch with all the changes I have just made. After fixing the latest Martin's issue there was exposed a new issue maybe related with the initialization per line. (picture attached) 
The issue is actually related with the fact that extend_face_id is never restarted to face_id when going back from an extend to a non_extend face between lines (for example when mark is active and the iterator crosses the pointer to outside the selected region like in the  picture). 
I made all the  updates of the extend_face_id in the same places where face_id was updated (even when uneeded for now.) But I don't see any special function that is called before display_line. I can set extend_face_id = face_id at the beginning of display_line... but I am not sure if this is consistent or the right to do. 
In some face_id merges I ignored the merge for extend_face_id because Qnobreak_space, Qglyphless_char or Qescape_glyph I don't think are expected to be :extend t in any case. 
So the conditional merge is only called in next_element_from_display_vector and the conditionals there seems to be the key for this. 
The rest of the times the extend_face_id is only saved and restored so as there were no merges I just asign to convenient values (caches, saved or face_id). But maybe this last could be also the problem.
I am not convinced that I am doing this right. maybe some experts eyes could help. Lets expect this time marting can execute it ;p 
Thanks in advance,Ergus.


-----Original Message-----
From: Ergus <spacibba@aol.com>
To: rudalics <rudalics@gmx.at>
Cc: eliz <eliz@gnu.org>; emacs-devel <emacs-devel@gnu.org>
Sent: Thu, Sep 5, 2019 3:55 pm
Subject: Re: Question about display engine

For some reason I was not facing this; but it was actually a bug I just fixed. I'll send a patch in a while because this exposed an issue somewhere else.


-----Original Message-----
From: martin rudalics <rudalics@gmx.at>
To: Ergus <spacibba@aol.com>
Cc: Eli Zaretskii <eliz@gnu.org>; emacs-devel <emacs-devel@gnu.org>
Sent: Thu, Sep 5, 2019 11:24 am
Subject: Re: Question about display engine

> Here is the diff of the latest commit. (Patch attached anyway).

Thanks.  I tried with the patch.patch you attached.  When trying a gtk
build I get:

../../src/xfaces.c:5434: Emacs fatal error: assertion failed: lface_fully_specified_p (XVECTOR (lface)->contents)
Fatal error 6: Aborted
Backtrace:
../src/bootstrap-emacs[0x67e32f]
../src/bootstrap-emacs[0x6344c8]
../src/bootstrap-emacs[0x74fb6f]
../src/bootstrap-emacs[0x5a8e43]
../src/bootstrap-emacs[0x5a83f8]
../src/bootstrap-emacs[0x59bccf]
../src/bootstrap-emacs[0x436ad1]
../src/bootstrap-emacs[0x4fd0ad]
../src/bootstrap-emacs[0x76376c]
../src/bootstrap-emacs[0x63530c]
../src/bootstrap-emacs[0x635860]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f68217952e1]
../src/bootstrap-emacs[0x4151aa]
/bin/bash: Zeile 2: 12088 Abgebrochen            EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../../lisp/cus-face.el
Makefile:280: die Regel für Ziel „../../lisp/cus-face.elc“ scheiterte
make[2]: *** [../../lisp/cus-face.elc] Fehler 134
Makefile:784: die Regel für Ziel „../../lisp/cus-face.elc“ scheiterte
make[1]: *** [../../lisp/cus-face.elc] Fehler 2
make[1]: *** Es wird auf noch nicht beendete Prozesse gewartet...

../../src/xfaces.c:5434: Emacs fatal error: assertion failed: lface_fully_specified_p (XVECTOR (lface)->contents)
Backtrace:
../src/bootstrap-emacs[0x67e32f]
../src/bootstrap-emacs[0x6344c8]
../src/bootstrap-emacs[0x74fb6f]
../src/bootstrap-emacs[0x5a8e43]
../src/bootstrap-emacs[0x5a83f8]
../src/bootstrap-emacs[0x59bccf]
../src/bootstrap-emacs[0x436ad1]
../src/bootstrap-emacs[0x4fd0ad]
../src/bootstrap-emacs[0x76376c]
../src/bootstrap-emacs[0x63530c]
../src/bootstrap-emacs[0x635860]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f0e8f3a32e1]
../src/bootstrap-emacs[0x4151aa]
/bin/bash: Zeile 2: 12090 Abgebrochen            EMACSLOADPATH= '../src/bootstrap-emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../../lisp/faces.el
Makefile:280: die Regel für Ziel „../../lisp/faces.elc“ scheiterte
make[2]: *** [../../lisp/faces.elc] Fehler 134
Makefile:784: die Regel für Ziel „../../lisp/faces.elc“ scheiterte
make[1]: *** [../../lisp/faces.elc] Fehler 2
make[1]: Verzeichnis „/home/martin/emacs-git/trunk/obj-gtk/src“ wird verlassen
Makefile:424: die Regel für Ziel „src“ scheiterte
make: *** [src] Fehler 2

and a similar crash on Windows.  Before proceeding to dig into this
I'd like to hear your ideas.

 > https://github.com/Ergus/Emacs/commit/4943087027acd3f2c7a54a092b64bc839ef8850e

Is there any way to get the diffs wrt current master on that site?  I
never use github for such a thing and my browser settings are quite
restrictive.

Thanks, martin

[-- Attachment #1.2: Type: text/html, Size: 7578 bytes --]

[-- Attachment #2: Screenshot_2019-09-05_20-49-41.png --]
[-- Type: image/png, Size: 29280 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: patch.patch --]
[-- Type: text/x-patch, Size: 45844 bytes --]

diff --git a/lisp/cus-face.el b/lisp/cus-face.el
index d73bce42c3..5a49a81043 100644
--- a/lisp/cus-face.el
+++ b/lisp/cus-face.el
@@ -233,7 +233,11 @@ custom-face-attributes
 	     (file :tag "File"
 		   :help-echo "Name of bitmap file."
 		   :must-match t)))
-
+    (:extend
+     (choice :tag "Extend"
+	     :help-echo "Control whether attributes should be extended after EOL."
+	     (const :tag "Off" nil)
+	     (const :tag "On" t)))
     (:inherit
      (repeat :tag "Inherit"
 	     :help-echo "List of faces to inherit attributes from."
diff --git a/lisp/faces.el b/lisp/faces.el
index 5193c216d0..814a4b2c9a 100644
--- a/lisp/faces.el
+++ b/lisp/faces.el
@@ -342,6 +342,7 @@ face-x-resources
     (:box (".attributeBox" . "Face.AttributeBox"))
     (:underline (".attributeUnderline" . "Face.AttributeUnderline"))
     (:inverse-video (".attributeInverse" . "Face.AttributeInverse"))
+    (:extend (".attributeExtend" . "Face.AttributeExtend"))
     (:stipple
      (".attributeStipple" . "Face.AttributeStipple")
      (".attributeBackgroundPixmap" . "Face.AttributeBackgroundPixmap"))
@@ -594,6 +595,13 @@ face-italic-p
   (let ((italic (face-attribute face :slant frame inherit)))
     (memq italic '(italic oblique))))
 
+(defun face-extend-p (face &optional frame inherit)
+ "Return non-nil if FACE specifies a non-nil extend.
+If the optional argument FRAME is given, report on face FACE in that frame.
+If FRAME is t, report on the defaults for face FACE (for new frames).
+If FRAME is omitted or nil, use the selected frame.
+Optional argument INHERIT is passed to `face-attribute'."
+ (eq (face-attribute face :extend frame inherit) t))
 
 \f
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -760,6 +768,11 @@ set-face-attribute
 `:height', `:weight', and `:slant' may also be set in one step
 from an X font name:
 
+`:extend'
+
+VALUE specifies whether the FACE should be extended after EOL.
+VALUE must be one of t or nil.
+
 `:font'
 
 Set font-related face attributes from VALUE.
@@ -979,6 +992,18 @@ set-face-italic
 
 (define-obsolete-function-alias 'set-face-italic-p 'set-face-italic "24.4")
 
+(defun set-face-extend (face extend-p &optional frame)
+  "Specify whether face FACE should be extended.
+EXTEND-P nil means FACE explicitly doesn't extend after EOL.
+EXTEND-P t means FACE extends after EOL.
+
+FRAME nil or not specified means change face on all frames.
+Use `set-face-attribute' to \"unspecify\" underlining."
+  (interactive
+   (let ((list (read-face-and-attribute :extend)))
+     (list (car list) (if (cadr list) t))))
+  (set-face-attribute face frame :extend extend-p))
+
 
 (defalias 'set-face-background-pixmap 'set-face-stipple)
 
@@ -1102,7 +1127,7 @@ face-valid-attribute-values
 	   (:slant
 	    (mapcar #'(lambda (x) (cons (symbol-name (aref x 1)) (aref x 1)))
 		    font-slant-table))
-	   (:inverse-video
+	   ((or :inverse-video :extend)
 	    (mapcar #'(lambda (x) (cons (symbol-name x) x))
 		    (internal-lisp-face-attribute-values attribute)))
            ((or :underline :overline :strike-through :box)
@@ -1147,6 +1172,7 @@ face-attribute-name-alist
     (:slant . "slant")
     (:underline . "underline")
     (:overline . "overline")
+    (:extend . "extend")
     (:strike-through . "strike-through")
     (:box . "box")
     (:inverse-video . "inverse-video display")
@@ -1448,6 +1474,7 @@ describe-face
 		  (:stipple . "Stipple")
 		  (:font . "Font")
 		  (:fontset . "Fontset")
+                  (:extend . "Extend")
 		  (:inherit . "Inherit")))
 	(max-width (apply #'max (mapcar #'(lambda (x) (length (cdr x)))
 					attrs))))
@@ -1667,7 +1694,8 @@ face-spec-reset-face
 	     ;; (see also realize_default_face in xfaces.c).
 	     (append
 	      '(:underline nil :overline nil :strike-through nil
-		:box nil :inverse-video nil :stipple nil :inherit nil)
+		:box nil :inverse-video nil :stipple nil :inherit nil
+                :extend nil)
 	      ;; `display-graphic-p' is unavailable when running
 	      ;; temacs, prior to loading frame.el.
 	      (when (fboundp 'display-graphic-p)
@@ -2432,24 +2460,24 @@ highlight
 ;; if background is light.
 (defface region
   '((((class color) (min-colors 88) (background dark))
-     :background "blue3")
+     :background "blue3" :extend t)
     (((class color) (min-colors 88) (background light) (type gtk))
      :distant-foreground "gtk_selection_fg_color"
-     :background "gtk_selection_bg_color")
+     :background "gtk_selection_bg_color" :extend t)
     (((class color) (min-colors 88) (background light) (type ns))
      :distant-foreground "ns_selection_fg_color"
-     :background "ns_selection_bg_color")
+     :background "ns_selection_bg_color" :extend t)
     (((class color) (min-colors 88) (background light))
-     :background "lightgoldenrod2")
+     :background "lightgoldenrod2" :extend t)
     (((class color) (min-colors 16) (background dark))
-     :background "blue3")
+     :background "blue3" :extend t)
     (((class color) (min-colors 16) (background light))
-     :background "lightgoldenrod2")
+     :background "lightgoldenrod2" :extend t)
     (((class color) (min-colors 8))
-     :background "blue" :foreground "white")
+     :background "blue" :foreground "white" :extend t)
     (((type tty) (class mono))
      :inverse-video t)
-    (t :background "gray"))
+    (t :background "gray" :extend t))
   "Basic face for highlighting the region."
   :version "21.1"
   :group 'basic-faces)
diff --git a/src/dispextern.h b/src/dispextern.h
index 05f199ff35..c11a3a7b54 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -1564,6 +1564,7 @@ #define FONT_TOO_HIGH(ft)						\
   LFACE_INHERIT_INDEX,
   LFACE_FONTSET_INDEX,
   LFACE_DISTANT_FOREGROUND_INDEX,
+  LFACE_EXTEND_INDEX,
   LFACE_VECTOR_SIZE
 };
 
@@ -1589,6 +1590,7 @@ #define FONT_TOO_HIGH(ft)						\
 
 enum face_underline_type
 {
+  FACE_NO_UNDERLINE = 0,
   FACE_UNDER_LINE,
   FACE_UNDER_WAVE
 };
@@ -1632,11 +1634,9 @@ #define FONT_TOO_HIGH(ft)						\
   /* Pixel value or color index of background color.  */
   unsigned long background;
 
-  /* Pixel value or color index of underline color.  */
+  /* Pixel value or color index of underline, overlined,
+     strike-through, or box color.  */
   unsigned long underline_color;
-
-  /* Pixel value or color index of overlined, strike-through, or box
-     color.  */
   unsigned long overline_color;
   unsigned long strike_through_color;
   unsigned long box_color;
@@ -1663,7 +1663,7 @@ #define FONT_TOO_HIGH(ft)						\
   ENUM_BF (face_box_type) box : 2;
 
   /* Style of underlining. */
-  ENUM_BF (face_underline_type) underline_type : 1;
+  ENUM_BF (face_underline_type) underline : 2;
 
   /* If `box' above specifies a 3D type, true means use box_color for
      drawing shadows.  */
@@ -1671,7 +1671,6 @@ #define FONT_TOO_HIGH(ft)						\
 
   /* Non-zero if text in this face should be underlined, overlined,
      strike-through or have a box drawn around it.  */
-  bool_bf underline_p : 1;
   bool_bf overline_p : 1;
   bool_bf strike_through_p : 1;
 
@@ -1681,14 +1680,10 @@ #define FONT_TOO_HIGH(ft)						\
   bool_bf foreground_defaulted_p : 1;
   bool_bf background_defaulted_p : 1;
 
-  /* True means that either no color is specified for underlining or that
-     the specified color couldn't be loaded.  Use the foreground
-     color when drawing in that case. */
-  bool_bf underline_defaulted_p : 1;
-
   /* True means that either no color is specified for the corresponding
      attribute or that the specified color couldn't be loaded.
      Use the foreground color when drawing in that case. */
+  bool_bf underline_defaulted_p : 1;
   bool_bf overline_color_defaulted_p : 1;
   bool_bf strike_through_color_defaulted_p : 1;
   bool_bf box_color_defaulted_p : 1;
@@ -1822,6 +1817,9 @@ #define FACE_FROM_ID_OR_NULL(F, ID)			\
    ? FRAME_FACE_CACHE (F)->faces_by_id[ID]		\
    : NULL)
 
+#define FACE_EXTENSIBLE_P(F)			\
+  (!NILP (F->lface[LFACE_EXTEND_INDEX]))
+
 /* True if FACE is suitable for displaying ASCII characters.  */
 INLINE bool
 FACE_SUITABLE_FOR_ASCII_CHAR_P (struct face *face)
@@ -2328,7 +2326,7 @@ #define IT_STACK_SIZE 5
   /* Face id of the iterator saved in case a glyph from dpvec contains
      a face.  The face is restored when all glyphs from dpvec have
      been delivered.  */
-  int saved_face_id;
+  int saved_face_id, saved_extend_face_id;
 
   /* Vector of glyphs for control character translation.  The pointer
      dpvec is set to ctl_chars when a control character is translated.
@@ -2390,7 +2388,7 @@ #define OVERLAY_STRING_CHUNK_SIZE 16
     ptrdiff_t prev_stop;
     ptrdiff_t base_level_stop;
     struct composition_it cmp_it;
-    int face_id;
+    int face_id, extend_face_id;
 
     /* Save values specific to a given method.  */
     union {
@@ -2448,6 +2446,9 @@ #define OVERLAY_STRING_CHUNK_SIZE 16
   /* Face to use.  */
   int face_id;
 
+  /* Face to extend at EOL/  */
+  int extend_face_id;
+
   /* Setting of buffer-local variable selective-display-ellipses.  */
   bool_bf selective_display_ellipsis_p : 1;
 
diff --git a/src/nsterm.m b/src/nsterm.m
index 42ef4dd010..99b621533a 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -3404,9 +3404,9 @@ Note that CURSOR_WIDTH is meaningful only for (h)bar cursors.
     return;
 
   /* Do underline.  */
-  if (face->underline_p)
+  if (face->underline)
     {
-      if (s->face->underline_type == FACE_UNDER_WAVE)
+      if (s->face->underline == FACE_UNDER_WAVE)
         {
           if (face->underline_defaulted_p)
             [defaultCol set];
@@ -3415,15 +3415,15 @@ Note that CURSOR_WIDTH is meaningful only for (h)bar cursors.
 
           ns_draw_underwave (s, width, x);
         }
-      else if (s->face->underline_type == FACE_UNDER_LINE)
+      else if (s->face->underline == FACE_UNDER_LINE)
         {
 
           NSRect r;
           unsigned long thickness, position;
 
           /* If the prev was underlined, match its appearance.  */
-          if (s->prev && s->prev->face->underline_p
-	      && s->prev->face->underline_type == FACE_UNDER_LINE
+          if (s->prev
+	      && s->prev->face->underline == FACE_UNDER_LINE
               && s->prev->underline_thickness > 0)
             {
               thickness = s->prev->underline_thickness;
diff --git a/src/w32term.c b/src/w32term.c
index e5874f2d36..99a1db5784 100644
--- a/src/w32term.c
+++ b/src/w32term.c
@@ -2479,9 +2479,9 @@ w32_draw_glyph_string (struct glyph_string *s)
   if (!s->for_overlaps)
     {
       /* Draw underline.  */
-      if (s->face->underline_p)
+      if (s->face->underline)
         {
-          if (s->face->underline_type == FACE_UNDER_WAVE)
+          if (s->face->underline == FACE_UNDER_WAVE)
             {
               COLORREF color;
 
@@ -2492,13 +2492,13 @@ w32_draw_glyph_string (struct glyph_string *s)
 
               w32_draw_underwave (s, color);
             }
-          else if (s->face->underline_type == FACE_UNDER_LINE)
+          else if (s->face->underline == FACE_UNDER_LINE)
             {
               unsigned long thickness, position;
               int y;
 
-              if (s->prev && s->prev->face->underline_p
-		  && s->prev->face->underline_type == FACE_UNDER_LINE)
+              if (s->prev
+	          && s->prev->face->underline == FACE_UNDER_LINE)
                 {
                   /* We use the same underline style as the previous one.  */
                   thickness = s->prev->underline_thickness;
@@ -2512,7 +2512,7 @@ w32_draw_glyph_string (struct glyph_string *s)
 		  BOOL use_underline_position_properties;
 		  Lisp_Object val
 		    = buffer_local_value (Qunderline_minimum_offset,
-					s->w->contents);
+		                          s->w->contents);
 		  if (FIXNUMP (val))
 		    minimum_offset = max (0, XFIXNUM (val));
 		  else
diff --git a/src/xdisp.c b/src/xdisp.c
index 94f969f37c..62730903fd 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -3120,6 +3120,7 @@ init_iterator (struct it *it, struct window *w,
       struct face *face;
 
       it->face_id = remapped_base_face_id;
+      it->extend_face_id = remapped_base_face_id;
 
       /* If we have a boxed mode line, make the first character appear
 	 with a left box line.  */
@@ -3141,6 +3142,8 @@ init_iterator (struct it *it, struct window *w,
       /* We will rely on `reseat' to set this up properly, via
 	 handle_face_prop.  */
       it->face_id = it->base_face_id;
+      it->extend_face_id = it->base_face_id;  // ERGUS: FIX_THIS
+
 
       it->start = it->current;
       /* Do we need to reorder bidirectional text?  Not if this is a
@@ -3536,7 +3539,10 @@ handle_stop (struct it *it)
 
   /* Use face of preceding text for ellipsis (if invisible) */
   if (it->selective_display_ellipsis_p)
-    it->saved_face_id = it->face_id;
+    {
+      it->saved_face_id = it->face_id;
+      it->saved_extend_face_id = it->extend_face_id;
+    }
 
   /* Here's the description of the semantics of, and the logic behind,
      the various HANDLED_* statuses:
@@ -4154,7 +4160,15 @@ handle_face_prop (struct it *it)
 	  it->start_of_box_run_p = (new_face->box != FACE_NO_BOX
 				    && (old_face == NULL || !old_face->box));
 	  it->face_box_p = new_face->box != FACE_NO_BOX;
+
+	  /* Update the faces id and extend.  */
+	  it->face_id = new_face_id;
+
+	  if (FACE_EXTENSIBLE_P (new_face))
+	    it->extend_face_id = new_face_id;
+
 	}
+
     }
   else
     {
@@ -4250,13 +4264,19 @@ handle_face_prop (struct it *it)
 	  /* If new face has a box but old face hasn't, this is the
 	     start of a run of characters with box, i.e. it has a
 	     shadow on the left side.  */
-	  it->start_of_box_run_p
-	    = new_face->box && (old_face == NULL || !old_face->box);
+	  it->start_of_box_run_p = (new_face->box != FACE_NO_BOX
+	                            && (old_face == NULL || !old_face->box));
 	  it->face_box_p = new_face->box != FACE_NO_BOX;
+
+	  /* Update the faces id and extend.  */
+	  it->face_id = new_face_id;
+
+	  if (FACE_EXTENSIBLE_P (new_face))
+	    it->extend_face_id = new_face_id;
+
 	}
     }
 
-  it->face_id = new_face_id;
   return HANDLED_NORMALLY;
 }
 
@@ -4854,6 +4874,9 @@ setup_for_ellipsis (struct it *it, int len)
   if (it->saved_face_id >= 0)
     it->face_id = it->saved_face_id;
 
+  if (it->saved_extend_face_id >= 0)
+    it->extend_face_id = it->saved_extend_face_id;
+
   /* If the ellipsis represents buffer text, it means we advanced in
      the buffer, so we should no longer ignore overlay strings.  */
   if (it->method == GET_FROM_BUFFER)
@@ -5063,7 +5086,8 @@ display_prop_end (struct it *it, Lisp_Object object, struct text_pos start_pos)
    of buffer or string text.  */
 
 static int
-handle_single_display_spec (struct it *it, Lisp_Object spec, Lisp_Object object,
+handle_single_display_spec (struct it *it, Lisp_Object spec,
+                            Lisp_Object object,
 			    Lisp_Object overlay, struct text_pos *position,
 			    ptrdiff_t bufpos, int display_replaced,
 			    bool frame_window_p, bool enable_eval_p)
@@ -5137,7 +5161,11 @@ handle_single_display_spec (struct it *it, Lisp_Object spec, Lisp_Object object,
 		  int steps = XFIXNUM (XCAR (XCDR (it->font_height)));
 		  if (EQ (XCAR (it->font_height), Qplus))
 		    steps = - steps;
+
 		  it->face_id = smaller_face (it->f, it->face_id, steps);
+		  it->extend_face_id
+		    = smaller_face (it->f, it->extend_face_id, steps);
+
 		}
 	      else if (FUNCTIONP (it->font_height) && enable_eval_p)
 		{
@@ -5180,7 +5208,12 @@ handle_single_display_spec (struct it *it, Lisp_Object spec, Lisp_Object object,
 		}
 
 	      if (new_height > 0)
-		it->face_id = face_with_height (it->f, it->face_id, new_height);
+		{
+		  it->face_id
+		    = face_with_height (it->f, it->face_id, new_height);
+		  it->extend_face_id
+		    = face_with_height (it->f, it->extend_face_id, new_height);
+		}
 	    }
 	}
 
@@ -5370,6 +5403,7 @@ handle_single_display_spec (struct it *it, Lisp_Object spec, Lisp_Object object,
 	  it->method = GET_FROM_IMAGE;
 	  it->from_overlay = Qnil;
 	  it->face_id = face_id;
+	  it->extend_face_id = face_id; // ERGUS: FIX_THIS
 	  it->from_disp_prop_p = true;
 
 	  /* Say that we haven't consumed the characters with
@@ -6263,6 +6297,7 @@ push_it (struct it *it, struct text_pos *position)
   p->cmp_it = it->cmp_it;
   eassert (it->face_id >= 0);
   p->face_id = it->face_id;
+  p->extend_face_id = it->extend_face_id;
   p->string = it->string;
   p->method = it->method;
   p->from_overlay = it->from_overlay;
@@ -6366,6 +6401,7 @@ pop_it (struct it *it)
   it->base_level_stop = p->base_level_stop;
   it->cmp_it = p->cmp_it;
   it->face_id = p->face_id;
+  it->extend_face_id = p->extend_face_id;
   it->current = p->current;
   it->position = p->position;
   it->string = p->string;
@@ -7142,17 +7178,32 @@ lookup_glyphless_char_display (int c, struct it *it)
 
 /* Merge escape glyph face and cache the result.  */
 
+static int
+merge_extend_glyph_face (struct it *it, int lface_id)
+{
+
+  struct face *lface = FACE_FROM_ID (it->f, lface_id);
+
+  if (lface && FACE_EXTENSIBLE_P (lface))
+    return merge_faces (it->w, Qt, lface_id, it->extend_face_id);
+
+  return it->extend_face_id;
+}
+
+/* Merge escape glyph face and cache the result.  */
 static struct frame *last_escape_glyph_frame = NULL;
 static int last_escape_glyph_face_id = (1 << FACE_ID_BITS);
 static int last_escape_glyph_merged_face_id = 0;
 
 static int
-merge_escape_glyph_face (struct it *it)
+merge_escape_glyph_face (struct it *it, int lface_id)
 {
   int face_id;
 
-  if (it->f == last_escape_glyph_frame
-      && it->face_id == last_escape_glyph_face_id)
+  if (lface_id)
+    face_id = merge_faces (it->w, Qt, lface_id, it->face_id);
+  else if (it->f == last_escape_glyph_frame
+           && it->face_id == last_escape_glyph_face_id)
     face_id = last_escape_glyph_merged_face_id;
   else
     {
@@ -7162,6 +7213,7 @@ merge_escape_glyph_face (struct it *it)
       last_escape_glyph_face_id = it->face_id;
       last_escape_glyph_merged_face_id = face_id;
     }
+
   return face_id;
 }
 
@@ -7190,6 +7242,7 @@ merge_glyphless_glyph_face (struct it *it)
   return face_id;
 }
 
+
 /* Forget the `escape-glyph' and `glyphless-char' faces.  This should
    be called before redisplaying windows, and when the frame's face
    cache is freed.  */
@@ -7275,6 +7328,7 @@ get_next_display_element (struct it *it)
 		  it->current.dpvec_index = 0;
 		  it->dpvec_face_id = -1;
 		  it->saved_face_id = it->face_id;
+		  it->saved_extend_face_id = it->extend_face_id;
 		  it->method = GET_FROM_DISPLAY_VECTOR;
 		  it->ellipsis_p = false;
 		}
@@ -7355,9 +7409,7 @@ get_next_display_element (struct it *it)
 		      lface_id = GLYPH_CODE_FACE (gc);
 		    }
 
-		  face_id = (lface_id
-			     ? merge_faces (it->w, Qt, lface_id, it->face_id)
-			     : merge_escape_glyph_face (it));
+		  face_id = merge_escape_glyph_face (it, lface_id);
 
 		  XSETINT (it->ctl_chars[0], g);
 		  XSETINT (it->ctl_chars[1], c ^ 0100);
@@ -7373,6 +7425,7 @@ get_next_display_element (struct it *it)
 		  /* Merge `nobreak-space' into the current face.  */
 		  face_id = merge_faces (it->w, Qnobreak_space, 0,
 					 it->face_id);
+
 		  XSETINT (it->ctl_chars[0], ' ');
 		  ctl_len = 1;
 		  goto display_control;
@@ -7385,7 +7438,8 @@ get_next_display_element (struct it *it)
 		{
 		  /* Merge `nobreak-space' into the current face.  */
 		  face_id = merge_faces (it->w, Qnobreak_hyphen, 0,
-					 it->face_id);
+		                         it->face_id);
+
 		  XSETINT (it->ctl_chars[0], '-');
 		  ctl_len = 1;
 		  goto display_control;
@@ -7403,12 +7457,9 @@ get_next_display_element (struct it *it)
 		  lface_id = GLYPH_CODE_FACE (gc);
 		}
 
-	      face_id = (lface_id
-			 ? merge_faces (it->w, Qt, lface_id, it->face_id)
-			 : merge_escape_glyph_face (it));
+	      face_id = merge_escape_glyph_face (it, lface_id);
 
 	      /* Draw non-ASCII space/hyphen with escape glyph: */
-
 	      if (nonascii_space_p || nonascii_hyphen_p)
 		{
 		  XSETINT (it->ctl_chars[0], escape_glyph);
@@ -7443,6 +7494,7 @@ get_next_display_element (struct it *it)
 	      it->current.dpvec_index = 0;
 	      it->dpvec_face_id = face_id;
 	      it->saved_face_id = it->face_id;
+	      it->saved_extend_face_id = it->extend_face_id;
 	      it->method = GET_FROM_DISPLAY_VECTOR;
 	      it->ellipsis_p = false;
 	      goto get_next;
@@ -7778,6 +7830,7 @@ set_iterator_to_next (struct it *it, bool reseat_p)
       /* Restore face of the iterator to what they were before the
          display vector entry (these entries may contain faces).  */
       it->face_id = it->saved_face_id;
+      it->extend_face_id = it->saved_extend_face_id;
 
       if (it->dpvec + it->current.dpvec_index >= it->dpend)
 	{
@@ -8012,6 +8065,7 @@ next_element_from_display_vector (struct it *it)
   eassert (it->dpvec && it->current.dpvec_index >= 0);
 
   it->face_id = it->saved_face_id;
+  it->extend_face_id = it->saved_extend_face_id;
 
   /* KFS: This code used to check ip->dpvec[0] instead of the current element.
      That seemed totally bogus - so I changed it...  */
@@ -8027,13 +8081,21 @@ next_element_from_display_vector (struct it *it)
 	 the id of a Lisp face, not a realized face.  A face id of
 	 zero means no face is specified.  */
       if (it->dpvec_face_id >= 0)
-	it->face_id = it->dpvec_face_id;
+	{
+	  it->face_id = it->dpvec_face_id;
+	  it->extend_face_id = it->dpvec_face_id; // ERGUS: FIX_THIS
+	}
       else
 	{
 	  int lface_id = GLYPH_CODE_FACE (gc);
 	  if (lface_id > 0)
-	    it->face_id = merge_faces (it->w, Qt, lface_id,
-				       it->saved_face_id);
+	    {
+	      it->face_id = merge_faces (it->w, Qt, lface_id,
+	                                 it->saved_face_id);
+
+	      it->extend_face_id =
+		merge_extend_glyph_face (it, lface_id);
+	    }
 	}
 
       /* Glyphs in the display vector could have the box face, so we
@@ -8061,8 +8123,12 @@ next_element_from_display_vector (struct it *it)
 		GLYPH_CODE_FACE (it->dpvec[it->current.dpvec_index + 1]);
 
 	      if (lface_id > 0)
-		next_face_id = merge_faces (it->w, Qt, lface_id,
-					    it->saved_face_id);
+		{
+		  next_face_id = merge_faces (it->w, Qt, lface_id,
+		                              it->saved_face_id);
+		  it->extend_face_id =
+		    merge_extend_glyph_face (it, lface_id);
+		}
 	    }
 	}
       next_face = FACE_FROM_ID_OR_NULL (it->f, next_face_id);
@@ -8411,6 +8477,7 @@ next_element_from_ellipsis (struct it *it)
 	 was in IT->saved_face_id, and signal that it's there by
 	 setting face_before_selective_p.  */
       it->saved_face_id = it->face_id;
+      it->saved_extend_face_id = it->extend_face_id;
       it->method = GET_FROM_BUFFER;
       it->object = it->w->contents;
       reseat_at_next_visible_line_start (it, true);
@@ -12848,7 +12915,10 @@ display_tool_bar_line (struct it *it, int height)
      use the tool-bar face for the border too.  */
   if (!MATRIX_ROW_DISPLAYS_TEXT_P (row)
       && !EQ (Vauto_resize_tool_bars, Qgrow_only))
-    it->face_id = DEFAULT_FACE_ID;
+    {
+      it->face_id = DEFAULT_FACE_ID;
+      it->extend_face_id = DEFAULT_FACE_ID;
+    }
 
   extend_face_to_end_of_line (it);
   last = row->glyphs[TEXT_AREA] + row->used[TEXT_AREA] - 1;
@@ -20301,7 +20371,7 @@ append_space_for_newline (struct it *it, bool default_face_p)
 
 	  /* Corner case for when display-fill-column-indicator-mode
 	     is active and the extra character should be added in the
-	     same place than the line.  */
+	     same place than the space.  */
 	  int indicator_column = (it->w->pseudo_window_p == 0
 				  ? fill_column_indicator_column (it)
 				  : -1);
@@ -20325,7 +20395,7 @@ append_space_for_newline (struct it *it, bool default_face_p)
 		     = XFIXNAT (Vdisplay_fill_column_indicator_character);
 	           it->face_id
 		     = merge_faces (it->w, Qfill_column_indicator,
-				    0, saved_face_id);
+		                    0, it->face_id);
 	           face = FACE_FROM_ID (it->f, it->face_id);
 	           goto produce_glyphs;
 	         }
@@ -20337,7 +20407,7 @@ append_space_for_newline (struct it *it, bool default_face_p)
 	  if (default_face_p)
 	    it->face_id = local_default_face_id;
 	  else if (it->face_before_selective_p)
-	    it->face_id = it->saved_face_id;
+	    it->face_id = it->face_id;
 
 	  face = FACE_FROM_ID (it->f, it->face_id);
 	  it->face_id = FACE_FOR_CHAR (it->f, face, 0, -1, Qnil);
@@ -20472,33 +20542,35 @@ extend_face_to_end_of_line (struct it *it)
      1-``pixel'' wide, so they hit the equality too early.  This grace
      is needed only for R2L rows that are not continued, to produce
      one extra blank where we could display the cursor.  */
-  if ((it->current_x >= it->last_visible_x
-       + (!FRAME_WINDOW_P (f)
-	  && it->glyph_row->reversed_p
-	  && !it->glyph_row->continued_p))
-      /* If the window has display margins, we will need to extend
-	 their face even if the text area is filled.  */
-      && !(WINDOW_LEFT_MARGIN_WIDTH (it->w) > 0
-	   || WINDOW_RIGHT_MARGIN_WIDTH (it->w) > 0))
-    return;
-
-  /* Face extension extends the background and box of IT->face_id
-     to the end of the line.  If the background equals the background
-     of the frame, we don't have to do anything.  */
-  face = FACE_FROM_ID (f, (it->face_before_selective_p
-			   ? it->saved_face_id
-			   : it->face_id));
-
-  if (FRAME_WINDOW_P (f)
-      && MATRIX_ROW_DISPLAYS_TEXT_P (it->glyph_row)
-      && face->box == FACE_NO_BOX
-      && FACE_COLOR_TO_PIXEL (face->background, f) == FRAME_BACKGROUND_PIXEL (f)
-#ifdef HAVE_WINDOW_SYSTEM
-      && !face->stipple
-#endif
-      && !it->glyph_row->reversed_p
-      && !Vdisplay_fill_column_indicator)
-    return;
+/*   if ((it->current_x >= it->last_visible_x */
+/*        + (!FRAME_WINDOW_P (f) */
+/*           && it->glyph_row->reversed_p */
+/*           && !it->glyph_row->continued_p)) */
+/*       /\* If the window has display margins, we will need to extend */
+/* 	 their face even if the text area is filled.  *\/ */
+/*       && !(WINDOW_LEFT_MARGIN_WIDTH (it->w) > 0 */
+/* 	   || WINDOW_RIGHT_MARGIN_WIDTH (it->w) > 0)) */
+/*     return; */
+
+/*   /\* Face extension extends the background and box of IT->face_id */
+/*      to the end of the line.  If the background equals the background */
+/*      of the frame, we don't have to do anything.  *\/ */
+  /* face = FACE_FROM_ID (f, (it->face_before_selective_p */
+  /* 			   ? it->saved_extend_face_id */
+  /* 			   : it->extend_face_id)); */
+
+  face = FACE_FROM_ID (f, it->extend_face_id);
+
+/*   if (FRAME_WINDOW_P (f) */
+/*       && MATRIX_ROW_DISPLAYS_TEXT_P (it->glyph_row) */
+/*       && face->box == FACE_NO_BOX */
+/*       && FACE_COLOR_TO_PIXEL (face->background, f) == FRAME_BACKGROUND_PIXEL (f) */
+/* #ifdef HAVE_WINDOW_SYSTEM */
+/*       && !face->stipple */
+/* #endif */
+/*       && !it->glyph_row->reversed_p */
+/*       && !Vdisplay_fill_column_indicator) */
+/*     return; */
 
   /* Set the glyph row flag indicating that the face of the last glyph
      in the text area has to be drawn to the end of the text area.  */
@@ -20561,79 +20633,88 @@ extend_face_to_end_of_line (struct it *it)
 	  /* Display fill column indicator if not in modeline or
 	     toolbar and display fill column indicator mode is
 	     active.  */
+	  const char saved_char = it->char_to_display;
+	  const struct text_pos saved_pos = it->position;
+	  const bool saved_avoid_cursor = it->avoid_cursor_p;
+	  const int saved_face_id = it->face_id;
+	  const bool saved_box_start = it->start_of_box_run_p;
+	  Lisp_Object save_object = it->object;
+
+	  it->avoid_cursor_p = true;
+	  it->object = Qnil;
+	  memset (&it->position, 0, sizeof it->position);
+
 	  int indicator_column = (it->w->pseudo_window_p == 0
 				  ? fill_column_indicator_column (it)
 				  : -1);
-	  if (indicator_column >= 0)
-            {
-	      struct font *font = (default_face->font
+
+	  struct font *font = (default_face->font
 				   ? default_face->font
 				   : FRAME_FONT (f));
-	      const int char_width = (font->average_width
-				      ? font->average_width
-				      : font->space_width);
-	      int column_x;
-
-	      if (!INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
-		  && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
-		  && column_x >= it->current_x
-		  && column_x <= it->last_visible_x)
-	        {
-	          const char saved_char = it->char_to_display;
-	          const struct text_pos saved_pos = it->position;
-	          const bool saved_avoid_cursor = it->avoid_cursor_p;
-	          const int saved_face_id = it->face_id;
-	          const bool saved_box_start = it->start_of_box_run_p;
-	          Lisp_Object save_object = it->object;
-
-	          /* The stretch width needs to considet the latter
-	             added glyph.  */
-	          const int stretch_width
-		    = column_x - it->current_x - char_width;
-
-	          memset (&it->position, 0, sizeof it->position);
-	          it->avoid_cursor_p = true;
-	          it->object = Qnil;
-
-	          /* Only generate a stretch glyph if there is distance
-	             between current_x and and the indicator position.  */
-	          if (stretch_width > 0)
-		    {
-		      int stretch_ascent = (((it->ascent + it->descent)
-		                             * FONT_BASE (font)) / FONT_HEIGHT (font));
-		      append_stretch_glyph (it, Qnil, stretch_width,
-		                            it->ascent + it->descent,
-		                            stretch_ascent);
-		    }
+	  const int char_width = (font->average_width
+	                          ? font->average_width
+	                          : font->space_width);
+	  int column_x;
+
+	  if (indicator_column >= 0
+	      && !INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
+	      && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
+	      && column_x >= it->current_x
+	      && column_x <= it->last_visible_x)
+	    {
+
+	      /* The stretch width needs to considet the latter
+		 added glyph.  */
+	      const int stretch_width
+		= column_x - it->current_x - char_width;
+
+	      /* Only generate a stretch glyph if there is distance
+		 between current_x and and the indicator position.  */
+	      if (stretch_width > 0)
+		{
+		  it->face_id = it->extend_face_id;
 
-	          /* Generate the glyph indicator only if
-	             append_space_for_newline didn't already.  */
-	          if (it->current_x < column_x)
-	            {
-		      it->char_to_display
-			= XFIXNAT (Vdisplay_fill_column_indicator_character);
-	              it->face_id
-			= merge_faces (it->w, Qfill_column_indicator,
-				       0, saved_face_id);
-	              PRODUCE_GLYPHS (it);
-	            }
-
-	          /* Restore the face after the indicator was generated.  */
-	          it->face_id = saved_face_id;
-
-	          /* If there is space after the indicator generate an
-	             extra empty glyph to restore the face.  Issue was
-	             observed in X systems.  */
-	          it->char_to_display = ' ';
-	          PRODUCE_GLYPHS (it);
-
-	          it->char_to_display = saved_char;
-	          it->position = saved_pos;
-	          it->avoid_cursor_p = saved_avoid_cursor;
-	          it->start_of_box_run_p = saved_box_start;
-	          it->object = save_object;
-	        }
-            }
+		  int stretch_ascent = (((it->ascent + it->descent)
+		                         * FONT_BASE (font)) / FONT_HEIGHT (font));
+		  append_stretch_glyph (it, Qnil, stretch_width,
+		                        it->ascent + it->descent,
+		                        stretch_ascent);
+		}
+
+	      /* Generate the glyph indicator only if
+		 append_space_for_newline didn't already.  */
+	      if (it->current_x < column_x)
+		{
+		  it->char_to_display
+		    = XFIXNAT (Vdisplay_fill_column_indicator_character);
+		  it->face_id
+		    = merge_faces (it->w, Qfill_column_indicator,
+		                   0, it->extend_face_id);
+		  PRODUCE_GLYPHS (it);
+		  it->char_to_display = saved_char;
+		}
+
+	    }
+
+	  const int stretch_width = it->last_visible_x - it->current_x;
+
+	  if (stretch_width > 0)
+	    {
+	      it->face_id = it->extend_face_id;
+
+	      int stretch_ascent = (((it->ascent + it->descent)
+	                             * FONT_BASE (font)) / FONT_HEIGHT (font));
+	      append_stretch_glyph (it, Qnil, stretch_width,
+	                            it->ascent + it->descent,
+	                            stretch_ascent);
+	    }
+
+	  it->char_to_display = saved_char;
+	  it->position = saved_pos;
+	  it->avoid_cursor_p = saved_avoid_cursor;
+	  it->face_id = saved_face_id;
+	  it->start_of_box_run_p = saved_box_start;
+	  it->object = save_object;
 	}
       if (it->glyph_row->reversed_p)
 	{
@@ -20679,10 +20760,9 @@ extend_face_to_end_of_line (struct it *it)
 	      /* The last row's stretch glyph should get the default
 		 face, to avoid painting the rest of the window with
 		 the region face, if the region ends at ZV.  */
-	      if (it->glyph_row->ends_at_zv_p)
-		it->face_id = default_face->id;
-	      else
-		it->face_id = face->id;
+	      it->face_id = (it->glyph_row->ends_at_zv_p ?
+	                     default_face->id : face->id);
+
 	      it->start_of_box_run_p = false;
 	      append_stretch_glyph (it, Qnil, stretch_width,
 				    it->ascent + it->descent, stretch_ascent);
@@ -20719,7 +20799,7 @@ extend_face_to_end_of_line (struct it *it)
       it->len = 1;
 
       if (WINDOW_LEFT_MARGIN_WIDTH (it->w) > 0
-	  && (it->glyph_row->used[LEFT_MARGIN_AREA]
+          && (it->glyph_row->used[LEFT_MARGIN_AREA]
 	      < WINDOW_LEFT_MARGIN_WIDTH (it->w))
 	  && !it->glyph_row->mode_line_p
 	  && FACE_COLOR_TO_PIXEL (face->background, f) != FRAME_BACKGROUND_PIXEL (f))
@@ -20750,10 +20830,8 @@ extend_face_to_end_of_line (struct it *it)
       /* The last row's blank glyphs should get the default face, to
 	 avoid painting the rest of the window with the region face,
 	 if the region ends at ZV.  */
-      if (it->glyph_row->ends_at_zv_p)
-	it->face_id = default_face->id;
-      else
-	it->face_id = face->id;
+      it->face_id = (it->glyph_row->ends_at_zv_p ?
+                     default_face->id : face->id);
 
       /* Display fill-column indicator if needed.  */
       int indicator_column = fill_column_indicator_column (it);
@@ -20763,24 +20841,25 @@ extend_face_to_end_of_line (struct it *it)
 	indicator_column = -1;
       do
 	{
-	  int saved_face_id;
-	  bool indicate = it->current_x == indicator_column;
-	  if (indicate)
+	  if (it->current_x == indicator_column)
 	    {
-	      saved_face_id = it->face_id;
+	      int saved_face_id = it->face_id;
+
 	      it->face_id
-		= merge_faces (it->w, Qfill_column_indicator, 0, saved_face_id);
+		= merge_faces (it->w, Qfill_column_indicator, 0,
+		               it->extend_face_id);
 	      it->c = it->char_to_display
 		= XFIXNAT (Vdisplay_fill_column_indicator_character);
-	    }
 
-	  PRODUCE_GLYPHS (it);
+	      PRODUCE_GLYPHS (it);
 
-	  if (indicate)
-	    {
 	      it->face_id = saved_face_id;
 	      it->c = it->char_to_display = ' ';
 	    }
+	  else
+	    {
+	      PRODUCE_GLYPHS (it);
+	    }
 	}
       while (it->current_x <= it->last_visible_x);
 
@@ -25423,7 +25502,8 @@ display_count_lines (ptrdiff_t start_byte,
    Value is the number of columns displayed.  */
 
 static int
-display_string (const char *string, Lisp_Object lisp_string, Lisp_Object face_string,
+display_string (const char *string, Lisp_Object lisp_string,
+                Lisp_Object face_string,
 		ptrdiff_t face_string_pos, ptrdiff_t start, struct it *it,
 		int field_width, int precision, int max_x, int multibyte)
 {
@@ -25446,12 +25526,13 @@ display_string (const char *string, Lisp_Object lisp_string, Lisp_Object face_st
   if (STRINGP (face_string))
     {
       ptrdiff_t endptr;
-      struct face *face;
 
       it->face_id
 	= face_at_string_position (it->w, face_string, face_string_pos,
 				   0, &endptr, it->base_face_id, false);
-      face = FACE_FROM_ID (it->f, it->face_id);
+
+      struct face *face = FACE_FROM_ID (it->f, it->face_id);
+
       it->face_box_p = face->box != FACE_NO_BOX;
     }
 
@@ -27355,7 +27436,7 @@ font_for_underline_metrics (struct glyph_string *s)
   for (g = s->first_glyph - 1; g >= g0; g--)
     {
       struct face *prev_face = FACE_FROM_ID (s->f, g->face_id);
-      if (!(prev_face && prev_face->underline_p))
+      if (!(prev_face && prev_face->underline != FACE_NO_UNDERLINE))
 	break;
     }
 
diff --git a/src/xfaces.c b/src/xfaces.c
index c3cae7e2a6..9c58e3e51a 100644
--- a/src/xfaces.c
+++ b/src/xfaces.c
@@ -1209,7 +1209,7 @@ free_face_colors (struct frame *f, struct face *face)
       IF_DEBUG (--ncolors_allocated);
     }
 
-  if (face->underline_p
+  if (face->underline
       && !face->underline_defaulted_p)
     {
       x_free_colors (f, &face->underline_color, 1);
@@ -1590,6 +1590,7 @@ #define LFACE_BOX(LFACE)	    AREF ((LFACE), LFACE_BOX_INDEX)
 #define LFACE_FONT(LFACE)	    AREF ((LFACE), LFACE_FONT_INDEX)
 #define LFACE_INHERIT(LFACE)	    AREF ((LFACE), LFACE_INHERIT_INDEX)
 #define LFACE_FONTSET(LFACE)	    AREF ((LFACE), LFACE_FONTSET_INDEX)
+#define LFACE_EXTEND(LFACE)	    AREF ((LFACE), LFACE_EXTEND_INDEX)
 #define LFACE_DISTANT_FOREGROUND(LFACE) \
   AREF ((LFACE), LFACE_DISTANT_FOREGROUND_INDEX)
 
@@ -1633,6 +1634,10 @@ check_lface_attrs (Lisp_Object attrs[LFACE_VECTOR_SIZE])
 	   || SYMBOLP (attrs[LFACE_UNDERLINE_INDEX])
 	   || STRINGP (attrs[LFACE_UNDERLINE_INDEX])
 	   || CONSP (attrs[LFACE_UNDERLINE_INDEX]));
+  eassert (UNSPECIFIEDP (attrs[LFACE_EXTEND_INDEX])
+	   || IGNORE_DEFFACE_P (attrs[LFACE_EXTEND_INDEX])
+	   || SYMBOLP (attrs[LFACE_EXTEND_INDEX])
+	   || STRINGP (attrs[LFACE_EXTEND_INDEX]));
   eassert (UNSPECIFIEDP (attrs[LFACE_OVERLINE_INDEX])
 	   || IGNORE_DEFFACE_P (attrs[LFACE_OVERLINE_INDEX])
 	   || SYMBOLP (attrs[LFACE_OVERLINE_INDEX])
@@ -2512,6 +2517,13 @@ merge_face_ref (struct window *w,
 					err_msgs, named_merge_points))
 		    err = true;
 		}
+	      else if (EQ (keyword, QCextend))
+		{
+		  if (EQ (value, Qt) || NILP (value))
+		    to[LFACE_EXTEND_INDEX] = value;
+		  else
+		    err = true;
+		}
 	      else
 		err = true;
 
@@ -3030,6 +3042,17 @@ DEFUN ("internal-set-lisp-face-attribute", Finternal_set_lisp_face_attribute,
       old_value = LFACE_INVERSE (lface);
       ASET (lface, LFACE_INVERSE_INDEX, value);
     }
+  else if (EQ (attr, QCextend))
+    {
+      if (!UNSPECIFIEDP (value) && !IGNORE_DEFFACE_P (value))
+	{
+	  CHECK_SYMBOL (value);
+	  if (!EQ (value, Qt) && !NILP (value))
+	    signal_error ("Invalid extend face attribute value", value);
+	}
+      old_value = LFACE_EXTEND (lface);
+      ASET (lface, LFACE_EXTEND_INDEX, value);
+    }
   else if (EQ (attr, QCforeground))
     {
       /* Compatibility with 20.x.  */
@@ -3503,7 +3526,9 @@ DEFUN ("internal-set-lisp-face-attribute-from-resource",
     value = face_boolean_x_resource_value (value, true);
   else if (EQ (attr, QCweight) || EQ (attr, QCslant) || EQ (attr, QCwidth))
     value = intern (SSDATA (value));
-  else if (EQ (attr, QCreverse_video) || EQ (attr, QCinverse_video))
+  else if (EQ (attr, QCreverse_video)
+           || EQ (attr, QCinverse_video)
+           || EQ (attr, QCextend))
     value = face_boolean_x_resource_value (value, true);
   else if (EQ (attr, QCunderline)
 	   || EQ (attr, QCoverline)
@@ -3727,6 +3752,8 @@ DEFUN ("internal-get-lisp-face-attribute", Finternal_get_lisp_face_attribute,
     value = LFACE_SWIDTH (lface);
   else if (EQ (keyword, QCinherit))
     value = LFACE_INHERIT (lface);
+  else if (EQ (keyword, QCextend))
+    value = LFACE_EXTEND (lface);
   else if (EQ (keyword, QCfont))
     value = LFACE_FONT (lface);
   else if (EQ (keyword, QCfontset))
@@ -3754,7 +3781,9 @@ DEFUN ("internal-lisp-face-attribute-values",
 
   if (EQ (attr, QCunderline) || EQ (attr, QCoverline)
       || EQ (attr, QCstrike_through)
-      || EQ (attr, QCinverse_video) || EQ (attr, QCreverse_video))
+      || EQ (attr, QCinverse_video)
+      || EQ (attr, QCreverse_video)
+      || EQ (attr, QCextend))
     result = list2 (Qt, Qnil);
 
   return result;
@@ -4782,6 +4811,9 @@ gui_supports_face_attributes_p (struct frame *f,
       || (!UNSPECIFIEDP (attrs[LFACE_INVERSE_INDEX])
 	  && face_attr_equal_p (attrs[LFACE_INVERSE_INDEX],
 				def_attrs[LFACE_INVERSE_INDEX]))
+      || (!UNSPECIFIEDP (attrs[LFACE_EXTEND_INDEX])
+	  && face_attr_equal_p (attrs[LFACE_EXTEND_INDEX],
+				def_attrs[LFACE_EXTEND_INDEX]))
       || (!UNSPECIFIEDP (attrs[LFACE_FOREGROUND_INDEX])
 	  && face_attr_equal_p (attrs[LFACE_FOREGROUND_INDEX],
 				def_attrs[LFACE_FOREGROUND_INDEX]))
@@ -5358,6 +5390,9 @@ realize_default_face (struct frame *f)
 	ASET (lface, LFACE_FONTSET_INDEX, Qnil);
     }
 
+  if (UNSPECIFIEDP (LFACE_EXTEND (lface)))
+    ASET (lface, LFACE_EXTEND_INDEX, Qnil);
+
   if (UNSPECIFIEDP (LFACE_UNDERLINE (lface)))
     ASET (lface, LFACE_UNDERLINE_INDEX, Qnil);
 
@@ -5694,16 +5729,14 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
   if (EQ (underline, Qt))
     {
       /* Use default color (same as foreground color).  */
-      face->underline_p = true;
-      face->underline_type = FACE_UNDER_LINE;
+      face->underline = FACE_UNDER_LINE;
       face->underline_defaulted_p = true;
       face->underline_color = 0;
     }
   else if (STRINGP (underline))
     {
       /* Use specified color.  */
-      face->underline_p = true;
-      face->underline_type = FACE_UNDER_LINE;
+      face->underline = FACE_UNDER_LINE;
       face->underline_defaulted_p = false;
       face->underline_color
 	= load_color (f, face, underline,
@@ -5711,7 +5744,7 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
     }
   else if (NILP (underline))
     {
-      face->underline_p = false;
+      face->underline = FACE_NO_UNDERLINE;
       face->underline_defaulted_p = false;
       face->underline_color = 0;
     }
@@ -5719,10 +5752,9 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
     {
       /* `(:color COLOR :style STYLE)'.
          STYLE being one of `line' or `wave'. */
-      face->underline_p = true;
+      face->underline = FACE_UNDER_LINE;
       face->underline_color = 0;
       face->underline_defaulted_p = true;
-      face->underline_type = FACE_UNDER_LINE;
 
       /* FIXME?  This is also not robust about checking the precise form.
          See comments in Finternal_set_lisp_face_attribute.  */
@@ -5755,9 +5787,9 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
           else if (EQ (keyword, QCstyle))
             {
               if (EQ (value, Qline))
-                face->underline_type = FACE_UNDER_LINE;
+                face->underline = FACE_UNDER_LINE;
               else if (EQ (value, Qwave))
-                face->underline_type = FACE_UNDER_WAVE;
+                face->underline = FACE_UNDER_WAVE;
             }
         }
     }
@@ -6292,9 +6324,8 @@ merge_faces (struct window *w, Lisp_Object face_name, int face_id,
 {
   struct frame *f = WINDOW_XFRAME (w);
   Lisp_Object attrs[LFACE_VECTOR_SIZE];
-  struct face *base_face;
+  struct face *base_face = FACE_FROM_ID_OR_NULL (f, base_face_id);
 
-  base_face = FACE_FROM_ID_OR_NULL (f, base_face_id);
   if (!base_face)
     return base_face_id;
 
@@ -6319,12 +6350,14 @@ merge_faces (struct window *w, Lisp_Object face_name, int face_id,
     }
   else
     {
-      struct face *face;
       if (face_id < 0)
 	return base_face_id;
-      face = FACE_FROM_ID_OR_NULL (f, face_id);
+
+      struct face *face = FACE_FROM_ID_OR_NULL (f, face_id);
+
       if (!face)
 	return base_face_id;
+
       merge_face_vectors (w, f, face->lface, attrs, 0);
     }
 
@@ -6412,7 +6445,7 @@ dump_realized_face (struct face *face)
 #endif
   fprintf (stderr, "fontset: %d\n", face->fontset);
   fprintf (stderr, "underline: %d (%s)\n",
-	   face->underline_p,
+	   face->underline,
 	   SDATA (Fsymbol_name (face->lface[LFACE_UNDERLINE_INDEX])));
   fprintf (stderr, "hash: %" PRIuPTR "\n", face->hash);
 }
@@ -6537,6 +6570,7 @@ syms_of_xfaces (void)
   DEFSYM (QCstrike_through, ":strike-through");
   DEFSYM (QCbox, ":box");
   DEFSYM (QCinherit, ":inherit");
+  DEFSYM (QCextend, ":extend");
 
   /* Symbols used for Lisp face attribute values.  */
   DEFSYM (QCcolor, ":color");
diff --git a/src/xterm.c b/src/xterm.c
index b761eaf4d1..b8f8db56a7 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -3798,9 +3798,9 @@ x_draw_glyph_string (struct glyph_string *s)
   if (!s->for_overlaps)
     {
       /* Draw underline.  */
-      if (s->face->underline_p)
+      if (s->face->underline)
         {
-          if (s->face->underline_type == FACE_UNDER_WAVE)
+          if (s->face->underline == FACE_UNDER_WAVE)
             {
               if (s->face->underline_defaulted_p)
                 x_draw_underwave (s);
@@ -3814,13 +3814,13 @@ x_draw_glyph_string (struct glyph_string *s)
                   XSetForeground (display, s->gc, xgcv.foreground);
                 }
             }
-          else if (s->face->underline_type == FACE_UNDER_LINE)
+          else if (s->face->underline == FACE_UNDER_LINE)
             {
               unsigned long thickness, position;
               int y;
 
-              if (s->prev && s->prev->face->underline_p
-		  && s->prev->face->underline_type == FACE_UNDER_LINE)
+              if (s->prev &&
+	          s->prev->face->underline == FACE_UNDER_LINE)
                 {
                   /* We use the same underline style as the previous one.  */
                   thickness = s->prev->underline_thickness;

Re: Question about display engine

[martin rudalics]

Thanks for the new patch.  It applies and Emacs builds here on Windows
without problems.

 > The issue is actually related with the fact that
 > extend_face_id is never restarted to face_id when going back from an
 > extend to a non_extend face between lines (for example when mark is
 > active and the iterator crosses the pointer to outside the selected
 > region like in the picture).

The end of the region is a stop position for the iterator.  It can
occur in the middle of a line, so handling this problem at the
beginning of a display line would be neither sufficient nor useful.
When the stop position coinciding with the end of the region is
processed, the extend_face_id has to be reset to some new face_id at
that position.

I'm yet too silly to understand your patch so I cannot figure out
where this should happen.  But the following parts appear somehow
suspicious:

	  /* Update the faces id and extend.  */
	  it->face_id = new_face_id;

	  if (FACE_EXTENSIBLE_P (new_face))
	    it->extend_face_id = new_face_id;

At the end of the region this may get you a new_face that is not
extensible.  But that means that you do not reset it->extend_face_id
although you should (IMHO).

Just to make sure that we see the same things: Is my interpretation
correct that in your screenshot you use blue for the region and black
as default background and the regions starts at line 9 of your window?

martin

Re: Question about display engine

[Ergus]

On Fri, Sep 06, 2019 at 10:22:48AM +0200, martin rudalics wrote:
>Thanks for the new patch.  It applies and Emacs builds here on Windows
>without problems.
>
>> The issue is actually related with the fact that
>> extend_face_id is never restarted to face_id when going back from an
>> extend to a non_extend face between lines (for example when mark is
>> active and the iterator crosses the pointer to outside the selected
>> region like in the picture).
>
>The end of the region is a stop position for the iterator.  It can
>occur in the middle of a line, so handling this problem at the
>beginning of a display line would be neither sufficient nor useful.
>When the stop position coinciding with the end of the region is
>processed, the extend_face_id has to be reset to some new face_id at
>that position.
>
>I'm yet too silly to understand your patch so I cannot figure out
>where this should happen.  But the following parts appear somehow
>suspicious:
>
>	  /* Update the faces id and extend.  */
>	  it->face_id = new_face_id;
>
>	  if (FACE_EXTENSIBLE_P (new_face))
>	    it->extend_face_id = new_face_id;
>
>At the end of the region this may get you a new_face that is not
>extensible.  But that means that you do not reset it->extend_face_id
>although you should (IMHO).
>
>Just to make sure that we see the same things: Is my interpretation
>correct that in your screenshot you use blue for the region and black
>as default background and the regions starts at line 9 of your window?
>
Yes that's it.

>martin

Re: Question about display engine

[martin rudalics]

 >> Just to make sure that we see the same things: Is my interpretation
 >> correct that in your screenshot you use blue for the region and black
 >> as default background and the regions starts at line 9 of your window?
 >>
 > Yes that's it.

When I set the :extend attribute of the default face, everything seems
to work as expected.  The problem is that if, in the earlier mentioned,

       /* Update the faces id and extend.  */
       it->face_id = new_face_id;

       if (FACE_EXTENSIBLE_P (new_face))
         it->extend_face_id = new_face_id;

no extensible new face is found, the assignment fails and the old
region background stays in place for the rest of the window.  So it
seems that you have to make sure that _some_ default background is
always assigned here in case no face applied has the :extend attribute
set.

martin

Re: Question about display engine

[Eli Zaretskii]

> Cc: eliz@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Sat, 7 Sep 2019 08:52:56 +0200
> 
> When I set the :extend attribute of the default face, everything seems
> to work as expected.  The problem is that if, in the earlier mentioned,
> 
>        /* Update the faces id and extend.  */
>        it->face_id = new_face_id;
> 
>        if (FACE_EXTENSIBLE_P (new_face))
>          it->extend_face_id = new_face_id;
> 
> no extensible new face is found, the assignment fails and the old
> region background stays in place for the rest of the window.  So it
> seems that you have to make sure that _some_ default background is
> always assigned here in case no face applied has the :extend attribute
> set.

The default face should always be merged into the face used for line
extension, that goes without saying.

Re: Question about display engine

[Eli Zaretskii]

> Date: Sat, 07 Sep 2019 10:37:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: spacibba@aol.com, emacs-devel@gnu.org
> 
> The default face should always be merged into the face used for line
> extension, that goes without saying.

IOW, the default face should always be treated as having the :extend
attribute by default.

Re: Question about display engine

[Ergus via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 2064 bytes --]


Hi Eli and Martin:

Please give a look to the attached patch and check if it is working fine
for you. (I added a new branch "extend_face_id" in savannah too)

I tried to make it as optimized and less changes as possible from the
beginning.

In general I added a parameter to handle_face_prop renamed to
handle_face_prop_general (keeping the original name as a wrapper with
the same signature.)

And the extra parameter is an enum lface_attribute_index. The parameter
is passed from function to function until merge_named_face which checks:

(attr_filter == 0 || (!NILP (from[attr_filter])
		     && !UNSPECIFIEDP (from[attr_filter])))

So if we pass zero as the parameter then the merge is unconditional;
else the attribute needs to be non-nil and specified to merge.

I made it in this way because it is general enough and with low overhead
in case we want to condition the merge for different conditions in
the future.

Right now only region is extensible by default. So you can try with
region and hl-line-mode for example.

I added also the commands to change extensibility set-face-extend and
the internal lisp commands to get/set extend as usual.

Now the only annoying thing is that when extend is disabled for the
region, the extra space after eol has the same face than the text before
(which for me is fine) but in the terminal it is not added... so I
should ask if you consider correct to add the space in terminal or
remove the extra "colored" space in gui? I vote for the first... but you
say.

(I made some changes in the extend_face_to_end_of_line code related with
fill column indicator to reuse part of the code and avoid unneeded
duplication.



On Sat, Sep 07, 2019 at 10:55:14AM +0300, Eli Zaretskii wrote:
>> Date: Sat, 07 Sep 2019 10:37:07 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: spacibba@aol.com, emacs-devel@gnu.org
>>
>> The default face should always be merged into the face used for line
>> extension, that goes without saying.
>
>IOW, the default face should always be treated as having the :extend
>attribute by default.
>

[-- Attachment #2: test.patch --]
[-- Type: text/plain, Size: 46270 bytes --]

diff --git a/lisp/cus-face.el b/lisp/cus-face.el
index d73bce42c3..5a49a81043 100644
--- a/lisp/cus-face.el
+++ b/lisp/cus-face.el
@@ -233,7 +233,11 @@ custom-face-attributes
 	     (file :tag "File"
 		   :help-echo "Name of bitmap file."
 		   :must-match t)))
-
+    (:extend
+     (choice :tag "Extend"
+	     :help-echo "Control whether attributes should be extended after EOL."
+	     (const :tag "Off" nil)
+	     (const :tag "On" t)))
     (:inherit
      (repeat :tag "Inherit"
 	     :help-echo "List of faces to inherit attributes from."
diff --git a/lisp/faces.el b/lisp/faces.el
index 5193c216d0..814a4b2c9a 100644
--- a/lisp/faces.el
+++ b/lisp/faces.el
@@ -342,6 +342,7 @@ face-x-resources
     (:box (".attributeBox" . "Face.AttributeBox"))
     (:underline (".attributeUnderline" . "Face.AttributeUnderline"))
     (:inverse-video (".attributeInverse" . "Face.AttributeInverse"))
+    (:extend (".attributeExtend" . "Face.AttributeExtend"))
     (:stipple
      (".attributeStipple" . "Face.AttributeStipple")
      (".attributeBackgroundPixmap" . "Face.AttributeBackgroundPixmap"))
@@ -594,6 +595,13 @@ face-italic-p
   (let ((italic (face-attribute face :slant frame inherit)))
     (memq italic '(italic oblique))))
 
+(defun face-extend-p (face &optional frame inherit)
+ "Return non-nil if FACE specifies a non-nil extend.
+If the optional argument FRAME is given, report on face FACE in that frame.
+If FRAME is t, report on the defaults for face FACE (for new frames).
+If FRAME is omitted or nil, use the selected frame.
+Optional argument INHERIT is passed to `face-attribute'."
+ (eq (face-attribute face :extend frame inherit) t))
 
 \f
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -760,6 +768,11 @@ set-face-attribute
 `:height', `:weight', and `:slant' may also be set in one step
 from an X font name:
 
+`:extend'
+
+VALUE specifies whether the FACE should be extended after EOL.
+VALUE must be one of t or nil.
+
 `:font'
 
 Set font-related face attributes from VALUE.
@@ -979,6 +992,18 @@ set-face-italic
 
 (define-obsolete-function-alias 'set-face-italic-p 'set-face-italic "24.4")
 
+(defun set-face-extend (face extend-p &optional frame)
+  "Specify whether face FACE should be extended.
+EXTEND-P nil means FACE explicitly doesn't extend after EOL.
+EXTEND-P t means FACE extends after EOL.
+
+FRAME nil or not specified means change face on all frames.
+Use `set-face-attribute' to \"unspecify\" underlining."
+  (interactive
+   (let ((list (read-face-and-attribute :extend)))
+     (list (car list) (if (cadr list) t))))
+  (set-face-attribute face frame :extend extend-p))
+
 
 (defalias 'set-face-background-pixmap 'set-face-stipple)
 
@@ -1102,7 +1127,7 @@ face-valid-attribute-values
 	   (:slant
 	    (mapcar #'(lambda (x) (cons (symbol-name (aref x 1)) (aref x 1)))
 		    font-slant-table))
-	   (:inverse-video
+	   ((or :inverse-video :extend)
 	    (mapcar #'(lambda (x) (cons (symbol-name x) x))
 		    (internal-lisp-face-attribute-values attribute)))
            ((or :underline :overline :strike-through :box)
@@ -1147,6 +1172,7 @@ face-attribute-name-alist
     (:slant . "slant")
     (:underline . "underline")
     (:overline . "overline")
+    (:extend . "extend")
     (:strike-through . "strike-through")
     (:box . "box")
     (:inverse-video . "inverse-video display")
@@ -1448,6 +1474,7 @@ describe-face
 		  (:stipple . "Stipple")
 		  (:font . "Font")
 		  (:fontset . "Fontset")
+                  (:extend . "Extend")
 		  (:inherit . "Inherit")))
 	(max-width (apply #'max (mapcar #'(lambda (x) (length (cdr x)))
 					attrs))))
@@ -1667,7 +1694,8 @@ face-spec-reset-face
 	     ;; (see also realize_default_face in xfaces.c).
 	     (append
 	      '(:underline nil :overline nil :strike-through nil
-		:box nil :inverse-video nil :stipple nil :inherit nil)
+		:box nil :inverse-video nil :stipple nil :inherit nil
+                :extend nil)
 	      ;; `display-graphic-p' is unavailable when running
 	      ;; temacs, prior to loading frame.el.
 	      (when (fboundp 'display-graphic-p)
@@ -2432,24 +2460,24 @@ highlight
 ;; if background is light.
 (defface region
   '((((class color) (min-colors 88) (background dark))
-     :background "blue3")
+     :background "blue3" :extend t)
     (((class color) (min-colors 88) (background light) (type gtk))
      :distant-foreground "gtk_selection_fg_color"
-     :background "gtk_selection_bg_color")
+     :background "gtk_selection_bg_color" :extend t)
     (((class color) (min-colors 88) (background light) (type ns))
      :distant-foreground "ns_selection_fg_color"
-     :background "ns_selection_bg_color")
+     :background "ns_selection_bg_color" :extend t)
     (((class color) (min-colors 88) (background light))
-     :background "lightgoldenrod2")
+     :background "lightgoldenrod2" :extend t)
     (((class color) (min-colors 16) (background dark))
-     :background "blue3")
+     :background "blue3" :extend t)
     (((class color) (min-colors 16) (background light))
-     :background "lightgoldenrod2")
+     :background "lightgoldenrod2" :extend t)
     (((class color) (min-colors 8))
-     :background "blue" :foreground "white")
+     :background "blue" :foreground "white" :extend t)
     (((type tty) (class mono))
      :inverse-video t)
-    (t :background "gray"))
+    (t :background "gray" :extend t))
   "Basic face for highlighting the region."
   :version "21.1"
   :group 'basic-faces)
diff --git a/src/dispextern.h b/src/dispextern.h
index 05f199ff35..de438e069e 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -1564,6 +1564,7 @@ #define FONT_TOO_HIGH(ft)						\
   LFACE_INHERIT_INDEX,
   LFACE_FONTSET_INDEX,
   LFACE_DISTANT_FOREGROUND_INDEX,
+  LFACE_EXTEND_INDEX,
   LFACE_VECTOR_SIZE
 };
 
@@ -1589,6 +1590,7 @@ #define FONT_TOO_HIGH(ft)						\
 
 enum face_underline_type
 {
+  FACE_NO_UNDERLINE = 0,
   FACE_UNDER_LINE,
   FACE_UNDER_WAVE
 };
@@ -1632,11 +1634,9 @@ #define FONT_TOO_HIGH(ft)						\
   /* Pixel value or color index of background color.  */
   unsigned long background;
 
-  /* Pixel value or color index of underline color.  */
+  /* Pixel value or color index of underline, overlined,
+     strike-through, or box color.  */
   unsigned long underline_color;
-
-  /* Pixel value or color index of overlined, strike-through, or box
-     color.  */
   unsigned long overline_color;
   unsigned long strike_through_color;
   unsigned long box_color;
@@ -1663,7 +1663,7 @@ #define FONT_TOO_HIGH(ft)						\
   ENUM_BF (face_box_type) box : 2;
 
   /* Style of underlining. */
-  ENUM_BF (face_underline_type) underline_type : 1;
+  ENUM_BF (face_underline_type) underline : 2;
 
   /* If `box' above specifies a 3D type, true means use box_color for
      drawing shadows.  */
@@ -1671,7 +1671,6 @@ #define FONT_TOO_HIGH(ft)						\
 
   /* Non-zero if text in this face should be underlined, overlined,
      strike-through or have a box drawn around it.  */
-  bool_bf underline_p : 1;
   bool_bf overline_p : 1;
   bool_bf strike_through_p : 1;
 
@@ -1681,14 +1680,10 @@ #define FONT_TOO_HIGH(ft)						\
   bool_bf foreground_defaulted_p : 1;
   bool_bf background_defaulted_p : 1;
 
-  /* True means that either no color is specified for underlining or that
-     the specified color couldn't be loaded.  Use the foreground
-     color when drawing in that case. */
-  bool_bf underline_defaulted_p : 1;
-
   /* True means that either no color is specified for the corresponding
      attribute or that the specified color couldn't be loaded.
      Use the foreground color when drawing in that case. */
+  bool_bf underline_defaulted_p : 1;
   bool_bf overline_color_defaulted_p : 1;
   bool_bf strike_through_color_defaulted_p : 1;
   bool_bf box_color_defaulted_p : 1;
@@ -1822,6 +1817,9 @@ #define FACE_FROM_ID_OR_NULL(F, ID)			\
    ? FRAME_FACE_CACHE (F)->faces_by_id[ID]		\
    : NULL)
 
+#define FACE_EXTENSIBLE_P(F)			\
+  (!NILP (F->lface[LFACE_EXTEND_INDEX]))
+
 /* True if FACE is suitable for displaying ASCII characters.  */
 INLINE bool
 FACE_SUITABLE_FOR_ASCII_CHAR_P (struct face *face)
@@ -2328,7 +2326,7 @@ #define IT_STACK_SIZE 5
   /* Face id of the iterator saved in case a glyph from dpvec contains
      a face.  The face is restored when all glyphs from dpvec have
      been delivered.  */
-  int saved_face_id;
+  int saved_face_id, saved_extend_face_id;
 
   /* Vector of glyphs for control character translation.  The pointer
      dpvec is set to ctl_chars when a control character is translated.
@@ -2390,7 +2388,7 @@ #define OVERLAY_STRING_CHUNK_SIZE 16
     ptrdiff_t prev_stop;
     ptrdiff_t base_level_stop;
     struct composition_it cmp_it;
-    int face_id;
+    int face_id, extend_face_id;
 
     /* Save values specific to a given method.  */
     union {
@@ -2448,6 +2446,9 @@ #define OVERLAY_STRING_CHUNK_SIZE 16
   /* Face to use.  */
   int face_id;
 
+  /* Face to extend at EOL/  */
+  int extend_face_id;
+
   /* Setting of buffer-local variable selective-display-ellipses.  */
   bool_bf selective_display_ellipsis_p : 1;
 
@@ -3458,8 +3459,8 @@ #define RGB_PIXEL_COLOR COLORREF
 void init_frame_faces (struct frame *);
 void free_frame_faces (struct frame *);
 void recompute_basic_faces (struct frame *);
-int face_at_buffer_position (struct window *, ptrdiff_t, ptrdiff_t *, ptrdiff_t,
-                             bool, int);
+int face_at_buffer_position (struct window *, ptrdiff_t, ptrdiff_t *,
+                             ptrdiff_t, bool, int, enum lface_attribute_index);
 int face_for_overlay_string (struct window *, ptrdiff_t, ptrdiff_t *, ptrdiff_t,
                              bool, Lisp_Object);
 int face_at_string_position (struct window *, Lisp_Object, ptrdiff_t, ptrdiff_t,
diff --git a/src/font.c b/src/font.c
index 935dd64e64..2b16839bba 100644
--- a/src/font.c
+++ b/src/font.c
@@ -3781,10 +3781,10 @@ font_at (int c, ptrdiff_t pos, struct face *face, struct window *w,
 
       if (STRINGP (string))
 	face_id = face_at_string_position (w, string, pos, 0, &endptr,
-					   DEFAULT_FACE_ID, false);
+					   DEFAULT_FACE_ID, 0);
       else
 	face_id = face_at_buffer_position (w, pos, &endptr,
-					   pos + 100, false, -1);
+	                                   pos + 100, false, -1, 0);
       face = FACE_FROM_ID (f, face_id);
     }
   if (multibyte)
@@ -3828,7 +3828,7 @@ font_range (ptrdiff_t pos, ptrdiff_t pos_byte, ptrdiff_t *limit,
 
       if (NILP (string))
 	  face_id = face_at_buffer_position (w, pos, &ignore, *limit,
-					     false, -1);
+	                                     false, -1, 0);
       else
 	{
 	  face_id =
@@ -4614,7 +4614,7 @@ DEFUN ("internal-char-font", Finternal_char_font, Sinternal_char_font, 1, 2, 0,
       w = XWINDOW (window);
       f = XFRAME (w->frame);
       face_id = face_at_buffer_position (w, pos, &dummy,
-					 pos + 100, false, -1);
+                                         pos + 100, false, -1, 0);
     }
   if (! CHAR_VALID_P (c))
     return Qnil;
diff --git a/src/nsterm.m b/src/nsterm.m
index 42ef4dd010..99b621533a 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -3404,9 +3404,9 @@ Note that CURSOR_WIDTH is meaningful only for (h)bar cursors.
     return;
 
   /* Do underline.  */
-  if (face->underline_p)
+  if (face->underline)
     {
-      if (s->face->underline_type == FACE_UNDER_WAVE)
+      if (s->face->underline == FACE_UNDER_WAVE)
         {
           if (face->underline_defaulted_p)
             [defaultCol set];
@@ -3415,15 +3415,15 @@ Note that CURSOR_WIDTH is meaningful only for (h)bar cursors.
 
           ns_draw_underwave (s, width, x);
         }
-      else if (s->face->underline_type == FACE_UNDER_LINE)
+      else if (s->face->underline == FACE_UNDER_LINE)
         {
 
           NSRect r;
           unsigned long thickness, position;
 
           /* If the prev was underlined, match its appearance.  */
-          if (s->prev && s->prev->face->underline_p
-	      && s->prev->face->underline_type == FACE_UNDER_LINE
+          if (s->prev
+	      && s->prev->face->underline == FACE_UNDER_LINE
               && s->prev->underline_thickness > 0)
             {
               thickness = s->prev->underline_thickness;
diff --git a/src/w32term.c b/src/w32term.c
index e5874f2d36..99a1db5784 100644
--- a/src/w32term.c
+++ b/src/w32term.c
@@ -2479,9 +2479,9 @@ w32_draw_glyph_string (struct glyph_string *s)
   if (!s->for_overlaps)
     {
       /* Draw underline.  */
-      if (s->face->underline_p)
+      if (s->face->underline)
         {
-          if (s->face->underline_type == FACE_UNDER_WAVE)
+          if (s->face->underline == FACE_UNDER_WAVE)
             {
               COLORREF color;
 
@@ -2492,13 +2492,13 @@ w32_draw_glyph_string (struct glyph_string *s)
 
               w32_draw_underwave (s, color);
             }
-          else if (s->face->underline_type == FACE_UNDER_LINE)
+          else if (s->face->underline == FACE_UNDER_LINE)
             {
               unsigned long thickness, position;
               int y;
 
-              if (s->prev && s->prev->face->underline_p
-		  && s->prev->face->underline_type == FACE_UNDER_LINE)
+              if (s->prev
+	          && s->prev->face->underline == FACE_UNDER_LINE)
                 {
                   /* We use the same underline style as the previous one.  */
                   thickness = s->prev->underline_thickness;
@@ -2512,7 +2512,7 @@ w32_draw_glyph_string (struct glyph_string *s)
 		  BOOL use_underline_position_properties;
 		  Lisp_Object val
 		    = buffer_local_value (Qunderline_minimum_offset,
-					s->w->contents);
+		                          s->w->contents);
 		  if (FIXNUMP (val))
 		    minimum_offset = max (0, XFIXNUM (val));
 		  else
diff --git a/src/xdisp.c b/src/xdisp.c
index 94f969f37c..9fb4f6be58 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4105,15 +4105,18 @@ handle_fontified_prop (struct it *it)
 				Faces
  ***********************************************************************/
 
-/* Set up iterator IT from face properties at its current position.
-   Called from handle_stop.  */
-
 static enum prop_handled
-handle_face_prop (struct it *it)
+handle_face_prop_general (struct it *it,
+                          enum lface_attribute_index attr_filter)
 {
-  int new_face_id;
+  int new_face_id, *face_id_ptr;
   ptrdiff_t next_stop;
 
+  if (attr_filter == LFACE_EXTEND_INDEX)
+    face_id_ptr = &(it->extend_face_id);
+  else
+    face_id_ptr = &(it->face_id);
+
   if (!STRINGP (it->string))
     {
       new_face_id
@@ -4122,7 +4125,7 @@ handle_face_prop (struct it *it)
 				   &next_stop,
 				   (IT_CHARPOS (*it)
 				    + TEXT_PROP_DISTANCE_LIMIT),
-				   false, it->base_face_id);
+	                           false, it->base_face_id, attr_filter);
 
       /* Is this a start of a run of characters with box face?
 	 Caveat: this can be called for a freshly initialized
@@ -4130,13 +4133,13 @@ handle_face_prop (struct it *it)
 	 face will not change until limit, i.e. if the new face has a
 	 box, all characters up to limit will have one.  But, as
 	 usual, we don't know whether limit is really the end.  */
-      if (new_face_id != it->face_id)
+      if (new_face_id != *face_id_ptr)
 	{
 	  struct face *new_face = FACE_FROM_ID (it->f, new_face_id);
 	  /* If it->face_id is -1, old_face below will be NULL, see
 	     the definition of FACE_FROM_ID_OR_NULL.  This will happen
 	     if this is the initial call that gets the face.  */
-	  struct face *old_face = FACE_FROM_ID_OR_NULL (it->f, it->face_id);
+	  struct face *old_face = FACE_FROM_ID_OR_NULL (it->f, *face_id_ptr);
 
 	  /* If the value of face_id of the iterator is -1, we have to
 	     look in front of IT's position and see whether there is a
@@ -4242,10 +4245,10 @@ handle_face_prop (struct it *it)
 	 box, all characters up to that position will have a
 	 box.  But, as usual, we don't know whether that position
 	 is really the end.  */
-      if (new_face_id != it->face_id)
+      if (new_face_id != *face_id_ptr)
 	{
 	  struct face *new_face = FACE_FROM_ID (it->f, new_face_id);
-	  struct face *old_face = FACE_FROM_ID_OR_NULL (it->f, it->face_id);
+	  struct face *old_face = FACE_FROM_ID_OR_NULL (it->f, *face_id_ptr);
 
 	  /* If new face has a box but old face hasn't, this is the
 	     start of a run of characters with box, i.e. it has a
@@ -4256,11 +4259,21 @@ handle_face_prop (struct it *it)
 	}
     }
 
-  it->face_id = new_face_id;
+  *face_id_ptr = new_face_id;
   return HANDLED_NORMALLY;
 }
 
 
+/* Set up iterator IT from face properties at its current position.
+   Called from handle_stop.  */
+
+static enum prop_handled
+handle_face_prop (struct it *it)
+{
+  return handle_face_prop_general (it, 0);
+}
+
+
 /* Return the ID of the face ``underlying'' IT's current position,
    which is in a string.  If the iterator is associated with a
    buffer, return the face at IT's current buffer position.
@@ -4474,7 +4487,7 @@ face_before_or_after_it_pos (struct it *it, bool before_p)
       face_id = face_at_buffer_position (it->w,
 					 CHARPOS (pos),
 					 &next_check_charpos,
-					 limit, false, -1);
+                                         limit, false, -1, 0);
 
       /* Correct the face for charsets different from ASCII.  Do it
 	 for the multibyte case only.  The face returned above is
@@ -7596,10 +7609,11 @@ get_next_display_element (struct it *it)
 		  else
 		    {
 		      next_face_id =
-			face_at_buffer_position (it->w, CHARPOS (pos), &ignore,
+			face_at_buffer_position (it->w, CHARPOS (pos),
+			                         &ignore,
 						 CHARPOS (pos)
 						 + TEXT_PROP_DISTANCE_LIMIT,
-						 false, -1);
+			                         false, -1, 0);
 		      it->end_of_box_run_p
 			= (FACE_FROM_ID (it->f, next_face_id)->box
 			   == FACE_NO_BOX);
@@ -20482,12 +20496,14 @@ extend_face_to_end_of_line (struct it *it)
 	   || WINDOW_RIGHT_MARGIN_WIDTH (it->w) > 0))
     return;
 
-  /* Face extension extends the background and box of IT->face_id
+  handle_face_prop_general (it, LFACE_EXTEND_INDEX);
+
+  /* Face extension extends the background and box of IT->extend_face_id
      to the end of the line.  If the background equals the background
      of the frame, we don't have to do anything.  */
   face = FACE_FROM_ID (f, (it->face_before_selective_p
 			   ? it->saved_face_id
-			   : it->face_id));
+			   : it->extend_face_id));
 
   if (FRAME_WINDOW_P (f)
       && MATRIX_ROW_DISPLAYS_TEXT_P (it->glyph_row)
@@ -20510,9 +20526,7 @@ extend_face_to_end_of_line (struct it *it)
      that the character will always be single byte in unibyte
      text.  */
   if (!ASCII_CHAR_P (it->c))
-    {
       it->face_id = FACE_FOR_CHAR (f, face, 0, -1, Qnil);
-    }
 
   /* The default face, possibly remapped. */
   struct face *default_face =
@@ -20561,79 +20575,86 @@ extend_face_to_end_of_line (struct it *it)
 	  /* Display fill column indicator if not in modeline or
 	     toolbar and display fill column indicator mode is
 	     active.  */
-	  int indicator_column = (it->w->pseudo_window_p == 0
+	  const int indicator_column = (it->w->pseudo_window_p == 0
 				  ? fill_column_indicator_column (it)
 				  : -1);
-	  if (indicator_column >= 0)
+
+	  struct font *font = (default_face->font
+	                       ? default_face->font
+	                       : FRAME_FONT (f));
+
+	  const int char_width = (font->average_width
+	                          ? font->average_width
+	                          : font->space_width);
+	  int column_x;
+
+	  const char saved_char = it->char_to_display;
+	  const struct text_pos saved_pos = it->position;
+	  const bool saved_avoid_cursor = it->avoid_cursor_p;
+	  const bool saved_box_start = it->start_of_box_run_p;
+	  Lisp_Object save_object = it->object;
+	  const int saved_face_id = it->face_id;
+
+	  it->face_id = it->extend_face_id;
+
+	  if (indicator_column >= 0
+	      && !INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
+	      && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
+	      && column_x >= it->current_x
+	      && column_x <= it->last_visible_x)
             {
-	      struct font *font = (default_face->font
-				   ? default_face->font
-				   : FRAME_FONT (f));
-	      const int char_width = (font->average_width
-				      ? font->average_width
-				      : font->space_width);
-	      int column_x;
-
-	      if (!INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
-		  && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
-		  && column_x >= it->current_x
-		  && column_x <= it->last_visible_x)
-	        {
-	          const char saved_char = it->char_to_display;
-	          const struct text_pos saved_pos = it->position;
-	          const bool saved_avoid_cursor = it->avoid_cursor_p;
-	          const int saved_face_id = it->face_id;
-	          const bool saved_box_start = it->start_of_box_run_p;
-	          Lisp_Object save_object = it->object;
-
-	          /* The stretch width needs to considet the latter
-	             added glyph.  */
-	          const int stretch_width
-		    = column_x - it->current_x - char_width;
-
-	          memset (&it->position, 0, sizeof it->position);
-	          it->avoid_cursor_p = true;
-	          it->object = Qnil;
-
-	          /* Only generate a stretch glyph if there is distance
-	             between current_x and and the indicator position.  */
-	          if (stretch_width > 0)
-		    {
-		      int stretch_ascent = (((it->ascent + it->descent)
-		                             * FONT_BASE (font)) / FONT_HEIGHT (font));
-		      append_stretch_glyph (it, Qnil, stretch_width,
-		                            it->ascent + it->descent,
-		                            stretch_ascent);
-		    }
 
-	          /* Generate the glyph indicator only if
-	             append_space_for_newline didn't already.  */
-	          if (it->current_x < column_x)
-	            {
-		      it->char_to_display
-			= XFIXNAT (Vdisplay_fill_column_indicator_character);
-	              it->face_id
-			= merge_faces (it->w, Qfill_column_indicator,
-				       0, saved_face_id);
-	              PRODUCE_GLYPHS (it);
-	            }
-
-	          /* Restore the face after the indicator was generated.  */
-	          it->face_id = saved_face_id;
-
-	          /* If there is space after the indicator generate an
-	             extra empty glyph to restore the face.  Issue was
-	             observed in X systems.  */
-	          it->char_to_display = ' ';
-	          PRODUCE_GLYPHS (it);
-
-	          it->char_to_display = saved_char;
-	          it->position = saved_pos;
-	          it->avoid_cursor_p = saved_avoid_cursor;
-	          it->start_of_box_run_p = saved_box_start;
-	          it->object = save_object;
-	        }
+	      /* The stretch width needs to considet the latter
+		 added glyph.  */
+	      const int stretch_width
+		= column_x - it->current_x - char_width;
+
+	      memset (&it->position, 0, sizeof it->position);
+	      it->avoid_cursor_p = true;
+	      it->object = Qnil;
+
+	      /* Only generate a stretch glyph if there is distance
+		 between current_x and and the indicator position.  */
+	      if (stretch_width > 0)
+		{
+		  int stretch_ascent = (((it->ascent + it->descent)
+		                         * FONT_BASE (font)) / FONT_HEIGHT (font));
+		  append_stretch_glyph (it, Qnil, stretch_width,
+		                        it->ascent + it->descent,
+		                        stretch_ascent);
+		}
+
+	      /* Generate the glyph indicator only if
+		 append_space_for_newline didn't already.  */
+	      if (it->current_x < column_x)
+		{
+		  const int save_face_id = it->face_id;
+		  it->char_to_display
+		    = XFIXNAT (Vdisplay_fill_column_indicator_character);
+		  it->face_id
+		    = merge_faces (it->w, Qfill_column_indicator,
+		                   0, it->extend_face_id);
+		  PRODUCE_GLYPHS (it);
+		  it->face_id = save_face_id;
+
+		}
             }
+
+	  /* Restore the face after the indicator was generated.  */
+
+	  /* If there is space after the indicator generate an
+	     extra empty glyph to restore the face.  Issue was
+	     observed in X systems.  */
+	  it->char_to_display = ' ';
+	  PRODUCE_GLYPHS (it);
+
+	  it->char_to_display = saved_char;
+	  it->position = saved_pos;
+	  it->avoid_cursor_p = saved_avoid_cursor;
+	  it->start_of_box_run_p = saved_box_start;
+	  it->object = save_object;
+	  it->face_id = saved_face_id;
+
 	}
       if (it->glyph_row->reversed_p)
 	{
@@ -20679,10 +20700,9 @@ extend_face_to_end_of_line (struct it *it)
 	      /* The last row's stretch glyph should get the default
 		 face, to avoid painting the rest of the window with
 		 the region face, if the region ends at ZV.  */
-	      if (it->glyph_row->ends_at_zv_p)
-		it->face_id = default_face->id;
-	      else
-		it->face_id = face->id;
+	      it->face_id = (it->glyph_row->ends_at_zv_p ?
+	                     default_face->id : face->id);
+
 	      it->start_of_box_run_p = false;
 	      append_stretch_glyph (it, Qnil, stretch_width,
 				    it->ascent + it->descent, stretch_ascent);
@@ -20704,14 +20724,11 @@ extend_face_to_end_of_line (struct it *it)
     {
       /* Save some values that must not be changed.  */
       int saved_x = it->current_x;
-      struct text_pos saved_pos;
-      Lisp_Object saved_object;
+      struct text_pos saved_pos = it->position;
+      Lisp_Object saved_object = it->object;;
       enum display_element_type saved_what = it->what;
       int saved_face_id = it->face_id;
 
-      saved_object = it->object;
-      saved_pos = it->position;
-
       it->what = IT_CHARACTER;
       memset (&it->position, 0, sizeof it->position);
       it->object = Qnil;
@@ -20750,10 +20767,8 @@ extend_face_to_end_of_line (struct it *it)
       /* The last row's blank glyphs should get the default face, to
 	 avoid painting the rest of the window with the region face,
 	 if the region ends at ZV.  */
-      if (it->glyph_row->ends_at_zv_p)
-	it->face_id = default_face->id;
-      else
-	it->face_id = face->id;
+      it->face_id = (it->glyph_row->ends_at_zv_p ?
+                     default_face->id : face->id);
 
       /* Display fill-column indicator if needed.  */
       int indicator_column = fill_column_indicator_column (it);
@@ -20763,24 +20778,21 @@ extend_face_to_end_of_line (struct it *it)
 	indicator_column = -1;
       do
 	{
-	  int saved_face_id;
-	  bool indicate = it->current_x == indicator_column;
-	  if (indicate)
+	  if (it->current_x == indicator_column)
 	    {
-	      saved_face_id = it->face_id;
+	      int saved_face_id = it->face_id;
 	      it->face_id
-		= merge_faces (it->w, Qfill_column_indicator, 0, saved_face_id);
+		= merge_faces (it->w, Qfill_column_indicator, 0, it->extend_face_id);
 	      it->c = it->char_to_display
 		= XFIXNAT (Vdisplay_fill_column_indicator_character);
-	    }
 
-	  PRODUCE_GLYPHS (it);
+	      PRODUCE_GLYPHS (it);
 
-	  if (indicate)
-	    {
 	      it->face_id = saved_face_id;
 	      it->c = it->char_to_display = ' ';
 	    }
+	  else
+	    PRODUCE_GLYPHS (it);
 	}
       while (it->current_x <= it->last_visible_x);
 
@@ -27355,7 +27367,7 @@ font_for_underline_metrics (struct glyph_string *s)
   for (g = s->first_glyph - 1; g >= g0; g--)
     {
       struct face *prev_face = FACE_FROM_ID (s->f, g->face_id);
-      if (!(prev_face && prev_face->underline_p))
+      if (!(prev_face && prev_face->underline != FACE_NO_UNDERLINE))
 	break;
     }
 
@@ -30919,7 +30931,7 @@ mouse_face_from_buffer_pos (Lisp_Object window,
   hlinfo->mouse_face_face_id
     = face_at_buffer_position (w, mouse_charpos, &ignore,
 			       mouse_charpos + 1,
-			       !hlinfo->mouse_face_hidden, -1);
+                               !hlinfo->mouse_face_hidden, -1, 0);
   show_mouse_face (hlinfo, DRAW_MOUSE_FACE);
 }
 
diff --git a/src/xfaces.c b/src/xfaces.c
index c3cae7e2a6..7c5a0858bb 100644
--- a/src/xfaces.c
+++ b/src/xfaces.c
@@ -347,7 +347,8 @@ #define CLEAR_FONT_TABLE_NFONTS	10
 static void free_face_cache (struct face_cache *);
 static bool merge_face_ref (struct window *w,
                             struct frame *, Lisp_Object, Lisp_Object *,
-			    bool, struct named_merge_point *);
+                            bool, struct named_merge_point *,
+                            enum lface_attribute_index);
 static int color_distance (Emacs_Color *x, Emacs_Color *y);
 
 #ifdef HAVE_WINDOW_SYSTEM
@@ -1209,7 +1210,7 @@ free_face_colors (struct frame *f, struct face *face)
       IF_DEBUG (--ncolors_allocated);
     }
 
-  if (face->underline_p
+  if (face->underline
       && !face->underline_defaulted_p)
     {
       x_free_colors (f, &face->underline_color, 1);
@@ -1590,6 +1591,7 @@ #define LFACE_BOX(LFACE)	    AREF ((LFACE), LFACE_BOX_INDEX)
 #define LFACE_FONT(LFACE)	    AREF ((LFACE), LFACE_FONT_INDEX)
 #define LFACE_INHERIT(LFACE)	    AREF ((LFACE), LFACE_INHERIT_INDEX)
 #define LFACE_FONTSET(LFACE)	    AREF ((LFACE), LFACE_FONTSET_INDEX)
+#define LFACE_EXTEND(LFACE)	    AREF ((LFACE), LFACE_EXTEND_INDEX)
 #define LFACE_DISTANT_FOREGROUND(LFACE) \
   AREF ((LFACE), LFACE_DISTANT_FOREGROUND_INDEX)
 
@@ -1633,6 +1635,10 @@ check_lface_attrs (Lisp_Object attrs[LFACE_VECTOR_SIZE])
 	   || SYMBOLP (attrs[LFACE_UNDERLINE_INDEX])
 	   || STRINGP (attrs[LFACE_UNDERLINE_INDEX])
 	   || CONSP (attrs[LFACE_UNDERLINE_INDEX]));
+  eassert (UNSPECIFIEDP (attrs[LFACE_EXTEND_INDEX])
+	   || IGNORE_DEFFACE_P (attrs[LFACE_EXTEND_INDEX])
+	   || SYMBOLP (attrs[LFACE_EXTEND_INDEX])
+	   || STRINGP (attrs[LFACE_EXTEND_INDEX]));
   eassert (UNSPECIFIEDP (attrs[LFACE_OVERLINE_INDEX])
 	   || IGNORE_DEFFACE_P (attrs[LFACE_OVERLINE_INDEX])
 	   || SYMBOLP (attrs[LFACE_OVERLINE_INDEX])
@@ -1905,7 +1911,8 @@ get_lface_attributes (struct window *w,
 	    attrs[i] = Qunspecified;
 
 	  return merge_face_ref (w, f, XCDR (face_remapping), attrs,
-				 signal_p, named_merge_points);
+	                         signal_p, named_merge_points,
+	                         0);
 	}
     }
 
@@ -2060,7 +2067,8 @@ merge_face_vectors (struct window *w,
   if (!UNSPECIFIEDP (from[LFACE_INHERIT_INDEX])
       && !NILP (from[LFACE_INHERIT_INDEX]))
     merge_face_ref (w, f, from[LFACE_INHERIT_INDEX],
-                    to, false, named_merge_points);
+                    to, false, named_merge_points,
+		    0);
 
   if (FONT_SPEC_P (from[LFACE_FONT_INDEX]))
     {
@@ -2126,7 +2134,8 @@ merge_face_vectors (struct window *w,
 static bool
 merge_named_face (struct window *w,
                   struct frame *f, Lisp_Object face_name, Lisp_Object *to,
-		  struct named_merge_point *named_merge_points)
+                  struct named_merge_point *named_merge_points,
+                  enum lface_attribute_index attr_filter)
 {
   struct named_merge_point named_merge_point;
 
@@ -2136,9 +2145,13 @@ merge_named_face (struct window *w,
     {
       Lisp_Object from[LFACE_VECTOR_SIZE];
       bool ok = get_lface_attributes (w, f, face_name, from, false,
-				      named_merge_points);
+                                      named_merge_points);
 
-      if (ok)
+      eassert (attr_filter <  LFACE_VECTOR_SIZE);
+
+      if (ok && (attr_filter == 0
+                 || (!NILP (from[attr_filter])
+		     && !UNSPECIFIEDP (from[attr_filter]))))
 	merge_face_vectors (w, f, from, to, named_merge_points);
 
       return ok;
@@ -2269,6 +2282,11 @@ filter_face_ref (Lisp_Object face_ref,
    of ERR_MSGS).  Use NAMED_MERGE_POINTS to detect loops in face
    inheritance or list structure; it may be 0 for most callers.
 
+   attr_filter is the index of a parameter that conditions the merging
+   for named faces (case 1) to only the face_ref where
+   lface[merge_face_ref] is non-nil. To merge unconditionally set this
+   value to 0.
+
    FACE_REF may be a single face specification or a list of such
    specifications.  Each face specification can be:
 
@@ -2297,7 +2315,8 @@ filter_face_ref (Lisp_Object face_ref,
 static bool
 merge_face_ref (struct window *w,
                 struct frame *f, Lisp_Object face_ref, Lisp_Object *to,
-		bool err_msgs, struct named_merge_point *named_merge_points)
+                bool err_msgs, struct named_merge_point *named_merge_points,
+                enum lface_attribute_index attr_filter)
 {
   bool ok = true;		/* Succeed without an error? */
   Lisp_Object filtered_face_ref;
@@ -2509,7 +2528,15 @@ merge_face_ref (struct window *w,
 		  /* This is not really very useful; it's just like a
 		     normal face reference.  */
 		  if (! merge_face_ref (w, f, value, to,
-					err_msgs, named_merge_points))
+		                        err_msgs, named_merge_points,
+		                        0))
+		    err = true;
+		}
+	      else if (EQ (keyword, QCextend))
+		{
+		  if (EQ (value, Qt) || NILP (value))
+		    to[LFACE_EXTEND_INDEX] = value;
+		  else
 		    err = true;
 		}
 	      else
@@ -2532,16 +2559,19 @@ merge_face_ref (struct window *w,
 	  Lisp_Object next = XCDR (face_ref);
 
 	  if (! NILP (next))
-	    ok = merge_face_ref (w, f, next, to, err_msgs, named_merge_points);
+	    ok = merge_face_ref (w, f, next, to, err_msgs,
+	                         named_merge_points, 0);
 
-	  if (! merge_face_ref (w, f, first, to, err_msgs, named_merge_points))
+	  if (! merge_face_ref (w, f, first, to, err_msgs,
+	                        named_merge_points, 0))
 	    ok = false;
 	}
     }
   else
     {
       /* FACE_REF ought to be a face name.  */
-      ok = merge_named_face (w, f, face_ref, to, named_merge_points);
+      ok = merge_named_face (w, f, face_ref, to, named_merge_points,
+                             attr_filter);
       if (!ok && err_msgs)
 	add_to_log ("Invalid face reference: %s", face_ref);
     }
@@ -3030,6 +3060,17 @@ DEFUN ("internal-set-lisp-face-attribute", Finternal_set_lisp_face_attribute,
       old_value = LFACE_INVERSE (lface);
       ASET (lface, LFACE_INVERSE_INDEX, value);
     }
+  else if (EQ (attr, QCextend))
+    {
+      if (!UNSPECIFIEDP (value) && !IGNORE_DEFFACE_P (value))
+	{
+	  CHECK_SYMBOL (value);
+	  if (!EQ (value, Qt) && !NILP (value))
+	    signal_error ("Invalid extend face attribute value", value);
+	}
+      old_value = LFACE_EXTEND (lface);
+      ASET (lface, LFACE_EXTEND_INDEX, value);
+    }
   else if (EQ (attr, QCforeground))
     {
       /* Compatibility with 20.x.  */
@@ -3503,7 +3544,9 @@ DEFUN ("internal-set-lisp-face-attribute-from-resource",
     value = face_boolean_x_resource_value (value, true);
   else if (EQ (attr, QCweight) || EQ (attr, QCslant) || EQ (attr, QCwidth))
     value = intern (SSDATA (value));
-  else if (EQ (attr, QCreverse_video) || EQ (attr, QCinverse_video))
+  else if (EQ (attr, QCreverse_video)
+           || EQ (attr, QCinverse_video)
+           || EQ (attr, QCextend))
     value = face_boolean_x_resource_value (value, true);
   else if (EQ (attr, QCunderline)
 	   || EQ (attr, QCoverline)
@@ -3727,6 +3770,8 @@ DEFUN ("internal-get-lisp-face-attribute", Finternal_get_lisp_face_attribute,
     value = LFACE_SWIDTH (lface);
   else if (EQ (keyword, QCinherit))
     value = LFACE_INHERIT (lface);
+  else if (EQ (keyword, QCextend))
+    value = LFACE_EXTEND (lface);
   else if (EQ (keyword, QCfont))
     value = LFACE_FONT (lface);
   else if (EQ (keyword, QCfontset))
@@ -3754,7 +3799,9 @@ DEFUN ("internal-lisp-face-attribute-values",
 
   if (EQ (attr, QCunderline) || EQ (attr, QCoverline)
       || EQ (attr, QCstrike_through)
-      || EQ (attr, QCinverse_video) || EQ (attr, QCreverse_video))
+      || EQ (attr, QCinverse_video)
+      || EQ (attr, QCreverse_video)
+      || EQ (attr, QCextend))
     result = list2 (Qt, Qnil);
 
   return result;
@@ -4505,7 +4552,8 @@ lookup_face (struct frame *f, Lisp_Object *attr)
    suitable face is found, realize a new one.  */
 
 int
-face_for_font (struct frame *f, Lisp_Object font_object, struct face *base_face)
+face_for_font (struct frame *f, Lisp_Object font_object,
+               struct face *base_face)
 {
   struct face_cache *cache = FRAME_FACE_CACHE (f);
   unsigned hash;
@@ -4738,7 +4786,7 @@ DEFUN ("face-attributes-as-vector", Fface_attributes_as_vector,
   Lisp_Object lface = make_vector (LFACE_VECTOR_SIZE, Qunspecified);
   merge_face_ref (NULL, XFRAME (selected_frame),
                   plist, XVECTOR (lface)->contents,
-		  true, 0);
+                  true, NULL, 0);
   return lface;
 }
 
@@ -4782,6 +4830,9 @@ gui_supports_face_attributes_p (struct frame *f,
       || (!UNSPECIFIEDP (attrs[LFACE_INVERSE_INDEX])
 	  && face_attr_equal_p (attrs[LFACE_INVERSE_INDEX],
 				def_attrs[LFACE_INVERSE_INDEX]))
+      || (!UNSPECIFIEDP (attrs[LFACE_EXTEND_INDEX])
+	  && face_attr_equal_p (attrs[LFACE_EXTEND_INDEX],
+				def_attrs[LFACE_EXTEND_INDEX]))
       || (!UNSPECIFIEDP (attrs[LFACE_FOREGROUND_INDEX])
 	  && face_attr_equal_p (attrs[LFACE_FOREGROUND_INDEX],
 				def_attrs[LFACE_FOREGROUND_INDEX]))
@@ -5092,7 +5143,7 @@ Point (2) implies that a `:weight black' attribute will be satisfied by
 
   for (i = 0; i < LFACE_VECTOR_SIZE; i++)
     attrs[i] = Qunspecified;
-  merge_face_ref (NULL, f, attributes, attrs, true, 0);
+  merge_face_ref (NULL, f, attributes, attrs, true, NULL, 0);
 
   def_face = FACE_FROM_ID_OR_NULL (f, DEFAULT_FACE_ID);
   if (def_face == NULL)
@@ -5358,6 +5409,9 @@ realize_default_face (struct frame *f)
 	ASET (lface, LFACE_FONTSET_INDEX, Qnil);
     }
 
+  if (UNSPECIFIEDP (LFACE_EXTEND (lface)))
+    ASET (lface, LFACE_EXTEND_INDEX, Qnil);
+
   if (UNSPECIFIEDP (LFACE_UNDERLINE (lface)))
     ASET (lface, LFACE_UNDERLINE_INDEX, Qnil);
 
@@ -5694,16 +5748,14 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
   if (EQ (underline, Qt))
     {
       /* Use default color (same as foreground color).  */
-      face->underline_p = true;
-      face->underline_type = FACE_UNDER_LINE;
+      face->underline = FACE_UNDER_LINE;
       face->underline_defaulted_p = true;
       face->underline_color = 0;
     }
   else if (STRINGP (underline))
     {
       /* Use specified color.  */
-      face->underline_p = true;
-      face->underline_type = FACE_UNDER_LINE;
+      face->underline = FACE_UNDER_LINE;
       face->underline_defaulted_p = false;
       face->underline_color
 	= load_color (f, face, underline,
@@ -5711,7 +5763,7 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
     }
   else if (NILP (underline))
     {
-      face->underline_p = false;
+      face->underline = FACE_NO_UNDERLINE;
       face->underline_defaulted_p = false;
       face->underline_color = 0;
     }
@@ -5719,10 +5771,9 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
     {
       /* `(:color COLOR :style STYLE)'.
          STYLE being one of `line' or `wave'. */
-      face->underline_p = true;
+      face->underline = FACE_UNDER_LINE;
       face->underline_color = 0;
       face->underline_defaulted_p = true;
-      face->underline_type = FACE_UNDER_LINE;
 
       /* FIXME?  This is also not robust about checking the precise form.
          See comments in Finternal_set_lisp_face_attribute.  */
@@ -5755,9 +5806,9 @@ realize_gui_face (struct face_cache *cache, Lisp_Object attrs[LFACE_VECTOR_SIZE]
           else if (EQ (keyword, QCstyle))
             {
               if (EQ (value, Qline))
-                face->underline_type = FACE_UNDER_LINE;
+                face->underline = FACE_UNDER_LINE;
               else if (EQ (value, Qwave))
-                face->underline_type = FACE_UNDER_WAVE;
+                face->underline = FACE_UNDER_WAVE;
             }
         }
     }
@@ -5976,7 +6027,7 @@ compute_char_face (struct frame *f, int ch, Lisp_Object prop)
       Lisp_Object attrs[LFACE_VECTOR_SIZE];
       struct face *default_face = FACE_FROM_ID (f, DEFAULT_FACE_ID);
       memcpy (attrs, default_face->lface, sizeof attrs);
-      merge_face_ref (NULL, f, prop, attrs, true, 0);
+      merge_face_ref (NULL, f, prop, attrs, true, NULL, 0);
       face_id = lookup_face (f, attrs);
     }
 
@@ -5988,6 +6039,8 @@ compute_char_face (struct frame *f, int ch, Lisp_Object prop)
    which a different face is needed, as far as text properties and
    overlays are concerned.  W is a window displaying current_buffer.
 
+   attr_filter is passed merge_face_ref.
+
    REGION_BEG, REGION_END delimit the region, so it can be
    highlighted.
 
@@ -6007,7 +6060,8 @@ compute_char_face (struct frame *f, int ch, Lisp_Object prop)
 int
 face_at_buffer_position (struct window *w, ptrdiff_t pos,
 			 ptrdiff_t *endptr, ptrdiff_t limit,
-			 bool mouse, int base_face_id)
+                         bool mouse, int base_face_id,
+                         enum lface_attribute_index attr_filter)
 {
   struct frame *f = XFRAME (w->frame);
   Lisp_Object attrs[LFACE_VECTOR_SIZE];
@@ -6068,8 +6122,7 @@ face_at_buffer_position (struct window *w, ptrdiff_t pos,
   }
 
   /* Optimize common cases where we can use the default face.  */
-  if (noverlays == 0
-      && NILP (prop))
+  if (noverlays == 0 && NILP (prop))
     {
       SAFE_FREE ();
       return default_face->id;
@@ -6080,7 +6133,7 @@ face_at_buffer_position (struct window *w, ptrdiff_t pos,
 
   /* Merge in attributes specified via text properties.  */
   if (!NILP (prop))
-    merge_face_ref (w, f, prop, attrs, true, 0);
+    merge_face_ref (w, f, prop, attrs, true, NULL, 0);
 
   /* Now merge the overlay data.  */
   noverlays = sort_overlays (overlay_vec, noverlays, w);
@@ -6100,7 +6153,7 @@ face_at_buffer_position (struct window *w, ptrdiff_t pos,
 		 so discard the mouse-face text property, if any, and
 		 use the overlay property instead.  */
 	      memcpy (attrs, default_face->lface, sizeof attrs);
-	      merge_face_ref (w, f, prop, attrs, true, 0);
+	      merge_face_ref (w, f, prop, attrs, true, NULL, attr_filter);
 	    }
 
 	  oend = OVERLAY_END (overlay_vec[i]);
@@ -6117,8 +6170,9 @@ face_at_buffer_position (struct window *w, ptrdiff_t pos,
 	  ptrdiff_t oendpos;
 
 	  prop = Foverlay_get (overlay_vec[i], propname);
+
 	  if (!NILP (prop))
-	    merge_face_ref (w, f, prop, attrs, true, 0);
+	    merge_face_ref (w, f, prop, attrs, true, NULL, attr_filter);
 
 	  oend = OVERLAY_END (overlay_vec[i]);
 	  oendpos = OVERLAY_POSITION (oend);
@@ -6184,7 +6238,7 @@ face_for_overlay_string (struct window *w, ptrdiff_t pos,
 
   /* Merge in attributes specified via text properties.  */
   if (!NILP (prop))
-    merge_face_ref (w, f, prop, attrs, true, 0);
+    merge_face_ref (w, f, prop, attrs, true, NULL, 0);
 
   *endptr = endpos;
 
@@ -6219,7 +6273,7 @@ face_for_overlay_string (struct window *w, ptrdiff_t pos,
 face_at_string_position (struct window *w, Lisp_Object string,
 			 ptrdiff_t pos, ptrdiff_t bufpos,
 			 ptrdiff_t *endptr, enum face_id base_face_id,
-			 bool mouse_p)
+                         bool mouse_p)
 {
   Lisp_Object prop, position, end, limit;
   struct frame *f = XFRAME (WINDOW_FRAME (w));
@@ -6263,7 +6317,7 @@ face_at_string_position (struct window *w, Lisp_Object string,
 
   /* Merge in attributes specified via text properties.  */
   if (!NILP (prop))
-    merge_face_ref (w, f, prop, attrs, true, 0);
+    merge_face_ref (w, f, prop, attrs, true, NULL, 0);
 
   /* Look up a realized face with the given face attributes,
      or realize a new one for ASCII characters.  */
@@ -6292,9 +6346,8 @@ merge_faces (struct window *w, Lisp_Object face_name, int face_id,
 {
   struct frame *f = WINDOW_XFRAME (w);
   Lisp_Object attrs[LFACE_VECTOR_SIZE];
-  struct face *base_face;
+  struct face *base_face = FACE_FROM_ID_OR_NULL (f, base_face_id);
 
-  base_face = FACE_FROM_ID_OR_NULL (f, base_face_id);
   if (!base_face)
     return base_face_id;
 
@@ -6314,17 +6367,19 @@ merge_faces (struct window *w, Lisp_Object face_name, int face_id,
 
   if (!NILP (face_name))
     {
-      if (!merge_named_face (w, f, face_name, attrs, 0))
+      if (!merge_named_face (w, f, face_name, attrs, NULL, 0))
 	return base_face_id;
     }
   else
     {
-      struct face *face;
       if (face_id < 0)
 	return base_face_id;
-      face = FACE_FROM_ID_OR_NULL (f, face_id);
+
+      struct face *face = FACE_FROM_ID_OR_NULL (f, face_id);
+
       if (!face)
 	return base_face_id;
+
       merge_face_vectors (w, f, face->lface, attrs, 0);
     }
 
@@ -6412,7 +6467,7 @@ dump_realized_face (struct face *face)
 #endif
   fprintf (stderr, "fontset: %d\n", face->fontset);
   fprintf (stderr, "underline: %d (%s)\n",
-	   face->underline_p,
+	   face->underline,
 	   SDATA (Fsymbol_name (face->lface[LFACE_UNDERLINE_INDEX])));
   fprintf (stderr, "hash: %" PRIuPTR "\n", face->hash);
 }
@@ -6537,6 +6592,7 @@ syms_of_xfaces (void)
   DEFSYM (QCstrike_through, ":strike-through");
   DEFSYM (QCbox, ":box");
   DEFSYM (QCinherit, ":inherit");
+  DEFSYM (QCextend, ":extend");
 
   /* Symbols used for Lisp face attribute values.  */
   DEFSYM (QCcolor, ":color");
diff --git a/src/xterm.c b/src/xterm.c
index b761eaf4d1..b8f8db56a7 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -3798,9 +3798,9 @@ x_draw_glyph_string (struct glyph_string *s)
   if (!s->for_overlaps)
     {
       /* Draw underline.  */
-      if (s->face->underline_p)
+      if (s->face->underline)
         {
-          if (s->face->underline_type == FACE_UNDER_WAVE)
+          if (s->face->underline == FACE_UNDER_WAVE)
             {
               if (s->face->underline_defaulted_p)
                 x_draw_underwave (s);
@@ -3814,13 +3814,13 @@ x_draw_glyph_string (struct glyph_string *s)
                   XSetForeground (display, s->gc, xgcv.foreground);
                 }
             }
-          else if (s->face->underline_type == FACE_UNDER_LINE)
+          else if (s->face->underline == FACE_UNDER_LINE)
             {
               unsigned long thickness, position;
               int y;
 
-              if (s->prev && s->prev->face->underline_p
-		  && s->prev->face->underline_type == FACE_UNDER_LINE)
+              if (s->prev &&
+	          s->prev->face->underline == FACE_UNDER_LINE)
                 {
                   /* We use the same underline style as the previous one.  */
                   thickness = s->prev->underline_thickness;

Re: Question about display engine

[martin rudalics]

 > Please give a look to the attached patch and check if it is working fine
 > for you. (I added a new branch "extend_face_id" in savannah too)

Thanks.  The problem with my standard use case on your branch is that
when I customize 'font-lock-comment-face' to have a specific
:background and :extend nil, the background I specified nevertheless
extends to the end of the line.  So I wonder why it works to not
extend the region while it apparently does not work to not extend
'font-lock-comment-face'.

martin

Re: Question about display engine

[Ergus]

On Sun, Sep 08, 2019 at 10:40:42AM +0200, martin rudalics wrote:
>> Please give a look to the attached patch and check if it is working fine
>> for you. (I added a new branch "extend_face_id" in savannah too)
>
>Thanks.  The problem with my standard use case on your branch is that
>when I customize 'font-lock-comment-face' to have a specific
>:background and :extend nil, the background I specified nevertheless
>extends to the end of the line.  So I wonder why it works to not
>extend the region while it apparently does not work to not extend
>'font-lock-comment-face'.
>
>martin
>
I fixed that, try now please (it is in savannah already).

The problem was the merge I was doing in face_at_buffer_position which
was unconditionaly merged for props. I conditioned only the overlays.

It seems like font-lock-comment-face is a prop while region is an
overlay.

Right now as I modified merge_face_ref the conditional merge only works
when CONSP (face_ref) == false and I am not sure what to do to make it
work also for CONSP (face_ref) or if it even needed... 

Re: Question about display engine

[martin rudalics]

 > I fixed that, try now please (it is in savannah already).

Thank you.  Works now as expected.

I'm still not convinced that it is a good idea to unconditionally run
handle_face_prop_general from extend_face_to_end_of_line.  It will
penalize processing every line shown in a window even if no attribute
processed is affected by a nil :extend attribute.  But if you and Eli
think it's cleaner to do it this way, I will not object further.

Thanks again, martin

Re: Question about display engine

[Ergus]

On Mon, Sep 09, 2019 at 09:39:13AM +0200, martin rudalics wrote:
>> I fixed that, try now please (it is in savannah already).
>
>Thank you.  Works now as expected.
>
>I'm still not convinced that it is a good idea to unconditionally run
>handle_face_prop_general from extend_face_to_end_of_line.  It will
>penalize processing every line shown in a window even if no attribute
>processed is affected by a nil :extend attribute.  But if you and Eli
>think it's cleaner to do it this way, I will not object further.
>
>Thanks again, martin
>
Hi Martin:

I actually don't agree either... but the merge process is a destructive
operation (like a binary | ) So it means that we lost information during
the merge process and we don't have a method to know the merged faces
only with the face_id (in the general case) and mergng different faces
could produce the same result.

We can do some optimizations like condition the call if the face_id at
position is not the default face for example. Or compare with the
previous call result... but I think that only Eli can suggest which are
applicable in this case.

Re: Question about display engine

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Mon, 9 Sep 2019 09:39:13 +0200
> 
>  > I fixed that, try now please (it is in savannah already).
> 
> Thank you.  Works now as expected.
> 
> I'm still not convinced that it is a good idea to unconditionally run
> handle_face_prop_general from extend_face_to_end_of_line.  It will
> penalize processing every line shown in a window even if no attribute
> processed is affected by a nil :extend attribute.

The penalty is just a call to merge faces and look up the merged face
in the face cache (which, under most usual circumstances, will always
find an already realized face in the cache).  If you are worried about
face merging itself, then don't be: we do this in redisplay all the
time when displaying buffers, except in the most boring cases (like
buffer of all-ASCII text in Fundamental mode).

And the alternative is IMO worse: it will require tracking all the
face changes and deciding upon each change whether it does or doesn't
affect the line extension.

Re: Question about display engine

[Ergus]

On Mon, Sep 09, 2019 at 07:00:00PM +0300, Eli Zaretskii wrote:
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> From: martin rudalics <rudalics@gmx.at>
>> Date: Mon, 9 Sep 2019 09:39:13 +0200
>>
>>  > I fixed that, try now please (it is in savannah already).
>>
>> Thank you.  Works now as expected.
>>
>> I'm still not convinced that it is a good idea to unconditionally run
>> handle_face_prop_general from extend_face_to_end_of_line.  It will
>> penalize processing every line shown in a window even if no attribute
>> processed is affected by a nil :extend attribute.
>
>The penalty is just a call to merge faces and look up the merged face
>in the face cache (which, under most usual circumstances, will always
>find an already realized face in the cache).  If you are worried about
>face merging itself, then don't be: we do this in redisplay all the
>time when displaying buffers, except in the most boring cases (like
>buffer of all-ASCII text in Fundamental mode).
>
>And the alternative is IMO worse: it will require tracking all the
>face changes and deciding upon each change whether it does or doesn't
>affect the line extension.

Hi Eli:

About this; I agree that the penalty should be negligible at the end
compared to other things that happens on re-display. But after working
on that parts I recognize that the actual algorithm for merge are
extremely complicated and some parts are redundant. All the code is full
of conditions and branch divergences and the design of faces
infrastructure could be more efficient.

Code that is used very often like "PRODUCE_GLYPHS" checks 3 if conditions
every time (we should unify at least the if/else to bypass directly to
the pointer, asserting that it is always initialized in terminal or gui).

face_at_buffer_position and merge_face_ref are actually full of nested
and nested conditional calls and even (direct and indirect) recursive
calls. Some functions that could call merge_named_face I think they call
merge_face_ref instead (with the extra checks and divergences). While
other "low level" calls at the end also call merge_face_ref (probably
for safety) ex get_lface_attributes, merge_face_vectors and so on.

I understand that all this is probably negligible... but all that
inhibits many optimizations very important these days like code
inlining, branch prediction and vectorization. But also a big part of
the code is needed just because the divergence between the tui and the
gui code... if we unify the interfaces; then an important part of the
code will be simplified and reduced and easier to maintain and modify.

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 9 Sep 2019 19:08:58 +0200
> From: Ergus <spacibba@aol.com>
> Cc: martin rudalics <rudalics@gmx.at>, emacs-devel@gnu.org
> 
> But after working on that parts I recognize that the actual
> algorithm for merge are extremely complicated and some parts are
> redundant. All the code is full of conditions and branch divergences
> and the design of faces infrastructure could be more efficient.

I don't think I agree.  Given the complexity of the face-related
functionalities -- face remapping, the need to recompute all the faces
when some basic face changes, the various sources of face-related
information for each buffer/string position, etc. -- I find the design
and implementation of face code quite elegant and easy to understand,
maintain and change.

> Code that is used very often like "PRODUCE_GLYPHS" checks 3 if conditions
> every time (we should unify at least the if/else to bypass directly to
> the pointer, asserting that it is always initialized in terminal or gui).

I don't think I see the optimization opportunity you are talking about
here.  If you want to make a produce_glyphs method available as a TTY
hook, then why do you think it will make any tangible change of code
efficiency?  It's just a dereference and a call through a function
pointer vs a test and a direct call.

> face_at_buffer_position and merge_face_ref are actually full of nested
> and nested conditional calls and even (direct and indirect) recursive
> calls. Some functions that could call merge_named_face I think they call
> merge_face_ref instead (with the extra checks and divergences). While
> other "low level" calls at the end also call merge_face_ref (probably
> for safety) ex get_lface_attributes, merge_face_vectors and so on.

All of this is necessary to support the immense flexibility of face-related
functionality we have, starting from the half a dozen different ways
Lisp can specify a face.  There's nothing redundant in that code,
AFAIK, and unlike you, I don't find it too complicated at all.

> I understand that all this is probably negligible... but all that
> inhibits many optimizations very important these days like code
> inlining, branch prediction and vectorization.

What can I say? code should be correct first, and fast only after
that.  If the functionality we want to support disables some
optimizations, so be it.  And I wouldn't worry about this particular
part of display code anyway, because the most expensive parts of
redisplay are elsewhere.  It should be clear from a simple
consideration: the number of face changes in a typical window is an
order of magnitude or more smaller than the number of characters and
other display elements in that window.  So if we want to optimize
redisplay, we should look at the 90% part of the code, not at the 10%.

> But also a big part of the code is needed just because the
> divergence between the tui and the gui code...

I don't think I follow.  There's no difference between TTY and GUI
frames wrt face handling, except where TTY color translation is
involved.

> if we unify the interfaces; then an important part of the
> code will be simplified and reduced and easier to maintain and modify.

Which interfaces would you like to unify?  I don't think I understand.

Re: Question about display engine

[Ergus]

On Mon, Sep 09, 2019 at 09:08:51PM +0300, Eli Zaretskii wrote:
>> Date: Mon, 9 Sep 2019 19:08:58 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: martin rudalics <rudalics@gmx.at>, emacs-devel@gnu.org
>>
>> But after working on that parts I recognize that the actual
>> algorithm for merge are extremely complicated and some parts are
>> redundant. All the code is full of conditions and branch divergences
>> and the design of faces infrastructure could be more efficient.
>
>I don't think I agree.  Given the complexity of the face-related
>functionalities -- face remapping, the need to recompute all the faces
>when some basic face changes, the various sources of face-related
>information for each buffer/string position, etc. -- I find the design
>and implementation of face code quite elegant and easy to understand,
>maintain and change.
>
I am talking about efficiency.

>> Code that is used very often like "PRODUCE_GLYPHS" checks 3 if conditions
>> every time (we should unify at least the if/else to bypass directly to
>> the pointer, asserting that it is always initialized in terminal or gui).
>
>I don't think I see the optimization opportunity you are talking about
>here.  If you want to make a produce_glyphs method available as a TTY
>hook, then why do you think it will make any tangible change of code
>efficiency?  It's just a dereference and a call through a function
>pointer vs a test and a direct call.
>
This won't be an important optimization.

It is just not standard. If there is no difference why are there
different methods to select with if/else? The macro simply looks like a
workaround.

In any case, if the it/frame/rif structs are standard; then they should
be always initialized. the tests will go away. And we could add an assert
to test only in debug code. 

>> face_at_buffer_position and merge_face_ref are actually full of nested
>> and nested conditional calls and even (direct and indirect) recursive
>> calls. Some functions that could call merge_named_face I think they call
>> merge_face_ref instead (with the extra checks and divergences). While
>> other "low level" calls at the end also call merge_face_ref (probably
>> for safety) ex get_lface_attributes, merge_face_vectors and so on.
>
>All of this is necessary to support the immense flexibility of face-related
>functionality we have, starting from the half a dozen different ways
>Lisp can specify a face.  There's nothing redundant in that code,
>AFAIK, and unlike you, I don't find it too complicated at all.
>
To add one field to the faces it required many modifications. Most of
them with very similar changes here and there; else if and similar
changes... every time I see an if-else-if with more than 3-4 conditions
I start wondering.

>> I understand that all this is probably negligible... but all that
>> inhibits many optimizations very important these days like code
>> inlining, branch prediction and vectorization.
>
>What can I say? code should be correct first, and fast only after
>that.  If the functionality we want to support disables some
>optimizations, so be it.  And I wouldn't worry about this particular
>part of display code anyway, because the most expensive parts of
>redisplay are elsewhere.  It should be clear from a simple
>consideration: the number of face changes in a typical window is an
>order of magnitude or more smaller than the number of characters and
>other display elements in that window.  So if we want to optimize
>redisplay, we should look at the 90% part of the code, not at the 10%.
>
Agree. But then why when the GC fails the lagging is so intense? I
thought that the main part of the display engine related with GC was the
faces part.

>> But also a big part of the code is needed just because the
>> divergence between the tui and the gui code...
>
>I don't think I follow.  There's no difference between TTY and GUI
>frames wrt face handling, except where TTY color translation is
>involved.
>
>> if we unify the interfaces; then an important part of the
>> code will be simplified and reduced and easier to maintain and modify.
>
>Which interfaces would you like to unify?  I don't think I understand.
>
Most of the code conditioned with: if (FRAME_WINDOW_P (f)) could be
simplified. But I understand that this could be a lot of work and I
don't know enough about this.

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 9 Sep 2019 21:29:34 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >I don't think I agree.  Given the complexity of the face-related
> >functionalities -- face remapping, the need to recompute all the faces
> >when some basic face changes, the various sources of face-related
> >information for each buffer/string position, etc. -- I find the design
> >and implementation of face code quite elegant and easy to understand,
> >maintain and change.
> >
> I am talking about efficiency.

Efficiency is secondary to clean design and correct implementation.

> It is just not standard. If there is no difference why are there
> different methods to select with if/else?

History, I guess.

> To add one field to the faces it required many modifications.

That isn't a problem in my eyes.

> But then why when the GC fails the lagging is so intense? I
> thought that the main part of the display engine related with GC was the
> faces part.

No, faces code cannot GC.  What causes GC is the calls to Lisp, which
is JIT font-lock and other Lisp hooks.

> >Which interfaces would you like to unify?  I don't think I understand.
> >
> Most of the code conditioned with: if (FRAME_WINDOW_P (f)) could be
> simplified. But I understand that this could be a lot of work and I
> don't know enough about this.

This work is being done, and most of it has been done already.  Some
display features are only possible on GUI frames (multiple fonts, for
example), so such conditions are necessary, at least to some extent.

Re: Question about display engine

[Ergus]

Hi Eli:

Could you answer please the questions about the modifications in other
functions (face_at_string and so on)?

And could you tell me which are the faces that are supposed to be
:extend t by default.

I made a couple of changes in the interface to avoid passing the values
by reference. And other small fixed in the branch. Please give a look
and tell me what do you think.

It will be nice if you could tell me where in the manual I am supposed
to documents this as this actually enables 2 different functionalities:

1) Extend a face.
2) Change the region looking (whole screen width or "fixed to text" as a
side effect).

And fixes the issues with fill-column-indicator in org and the other
issue that martin indicated.

I will push tomorrow a fix for the extra space in term mode.

So after that; your comments and the documentations I think I will
rename the branch to a feature one in order to suggest people to test it
your recommended two weeks.

Is it fine?


On Tue, Sep 10, 2019 at 05:27:39AM +0300, Eli Zaretskii wrote:
>> Date: Mon, 9 Sep 2019 21:29:34 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >I don't think I agree.  Given the complexity of the face-related
>> >functionalities -- face remapping, the need to recompute all the faces
>> >when some basic face changes, the various sources of face-related
>> >information for each buffer/string position, etc. -- I find the design
>> >and implementation of face code quite elegant and easy to understand,
>> >maintain and change.
>> >
>> I am talking about efficiency.
>
>Efficiency is secondary to clean design and correct implementation.
>
>> It is just not standard. If there is no difference why are there
>> different methods to select with if/else?
>
>History, I guess.
>
>> To add one field to the faces it required many modifications.
>
>That isn't a problem in my eyes.
>
>> But then why when the GC fails the lagging is so intense? I
>> thought that the main part of the display engine related with GC was the
>> faces part.
>
>No, faces code cannot GC.  What causes GC is the calls to Lisp, which
>is JIT font-lock and other Lisp hooks.
>
>> >Which interfaces would you like to unify?  I don't think I understand.
>> >
>> Most of the code conditioned with: if (FRAME_WINDOW_P (f)) could be
>> simplified. But I understand that this could be a lot of work and I
>> don't know enough about this.
>
>This work is being done, and most of it has been done already.  Some
>display features are only possible on GUI frames (multiple fonts, for
>example), so such conditions are necessary, at least to some extent.

Re: Question about display engine

[Eli Zaretskii]

> Date: Thu, 12 Sep 2019 05:37:33 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> Hi Eli:
> 
> Could you answer please the questions about the modifications in other
> functions (face_at_string and so on)?

Sorry, I probably lost context, because I cannot figure out which
questions are those.  Could you please repeat them?

> And could you tell me which are the faces that are supposed to be
> :extend t by default.

I think 'default', 'region', and 'hl-line'.

> I made a couple of changes in the interface to avoid passing the values
> by reference. And other small fixed in the branch. Please give a look
> and tell me what do you think.

Will do.

> It will be nice if you could tell me where in the manual I am supposed
> to documents this as this actually enables 2 different functionalities:
> 
> 1) Extend a face.
> 2) Change the region looking (whole screen width or "fixed to text" as a
> side effect).
> 
> And fixes the issues with fill-column-indicator in org and the other
> issue that martin indicated.

I think the right place is in "Face Attributes".

> So after that; your comments and the documentations I think I will
> rename the branch to a feature one in order to suggest people to test it
> your recommended two weeks.

I don't see a reason for renaming the branch, you could leave it as
is.

Thanks.

Re: Question about display engine

[Eli Zaretskii]

> Date: Sun, 8 Sep 2019 02:51:10 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> Please give a look to the attached patch and check if it is working fine
> for you. (I added a new branch "extend_face_id" in savannah too)
> 
> I tried to make it as optimized and less changes as possible from the
> beginning.
> 
> In general I added a parameter to handle_face_prop renamed to
> handle_face_prop_general (keeping the original name as a wrapper with
> the same signature.)
> 
> And the extra parameter is an enum lface_attribute_index. The parameter
> is passed from function to function until merge_named_face which checks:
> 
> (attr_filter == 0 || (!NILP (from[attr_filter])
> 		     && !UNSPECIFIEDP (from[attr_filter])))
> 
> So if we pass zero as the parameter then the merge is unconditional;
> else the attribute needs to be non-nil and specified to merge.
> 
> I made it in this way because it is general enough and with low overhead
> in case we want to condition the merge for different conditions in
> the future.

I didn't yet have time to have a close look at the changes, but I can
already say that this handle_face_prop_general business I don't like.
Passing a pointer to a face ID, like this:

  +handle_face_prop_general (struct it *it, int *face_id_ptr,
  +                          enum lface_attribute_index attr_filter)

and then assigning to it via the pointer is gross.  Also, the
extension code doesn't need to return the HANDLED_NORMALLY value,
AFAIU.

Instead, it is much cleaner to have a new function with the guts of
handle_face_prop, which would _return_ a face ID.  Then
handle_face_prop would then plug this into it->face_id, and the
extension code will do what it needs.  Can you make this change,
please?

> Now the only annoying thing is that when extend is disabled for the
> region, the extra space after eol has the same face than the text before
> (which for me is fine) but in the terminal it is not added... so I
> should ask if you consider correct to add the space in terminal or
> remove the extra "colored" space in gui? I vote for the first... but you
> say.

Yes, the terminal should use the same face as GUI for the first blank
after end of line.

Thanks.

Re: Question about display engine

[Ergus]

Hi Eli:

I know all that, I was just concerned about functionality and the
changes I made in the merge_ref function if they are conceptually right
or if they could produce some issues in some use cases that maybe I am
ignoring. (specially the method I use to filter.)

I implemented this in the face_at_buffer but in the same function there
is a face_at_string equivalent... is it possible in some conditions to
reach that part of the code at eol? if so I will need to modify that one
too...

The filter now in merge_ref only works when !CONSP(ref_name). As it only
bypass the extra parameter to merge_named... it this right in the
general case? 

I will change that detail you mention and some others in the interface
later. But I need to be sure if the filter conditions already
implemented are enough. (With the first implementation I
learned that functional does not mean right.)


On Sun, Sep 08, 2019 at 08:51:52PM +0300, Eli Zaretskii wrote:
>> Date: Sun, 8 Sep 2019 02:51:10 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> Please give a look to the attached patch and check if it is working fine
>> for you. (I added a new branch "extend_face_id" in savannah too)
>>
>> I tried to make it as optimized and less changes as possible from the
>> beginning.
>>
>> In general I added a parameter to handle_face_prop renamed to
>> handle_face_prop_general (keeping the original name as a wrapper with
>> the same signature.)
>>
>> And the extra parameter is an enum lface_attribute_index. The parameter
>> is passed from function to function until merge_named_face which checks:
>>
>> (attr_filter == 0 || (!NILP (from[attr_filter])
>> 		     && !UNSPECIFIEDP (from[attr_filter])))
>>
>> So if we pass zero as the parameter then the merge is unconditional;
>> else the attribute needs to be non-nil and specified to merge.
>>
>> I made it in this way because it is general enough and with low overhead
>> in case we want to condition the merge for different conditions in
>> the future.
>
>I didn't yet have time to have a close look at the changes, but I can
>already say that this handle_face_prop_general business I don't like.
>Passing a pointer to a face ID, like this:
>
>  +handle_face_prop_general (struct it *it, int *face_id_ptr,
>  +                          enum lface_attribute_index attr_filter)
>
>and then assigning to it via the pointer is gross.  Also, the
>extension code doesn't need to return the HANDLED_NORMALLY value,
>AFAIU.
>
>Instead, it is much cleaner to have a new function with the guts of
>handle_face_prop, which would _return_ a face ID.  Then
>handle_face_prop would then plug this into it->face_id, and the
>extension code will do what it needs.  Can you make this change,
>please?
>
>> Now the only annoying thing is that when extend is disabled for the
>> region, the extra space after eol has the same face than the text before
>> (which for me is fine) but in the terminal it is not added... so I
>> should ask if you consider correct to add the space in terminal or
>> remove the extra "colored" space in gui? I vote for the first... but you
>> say.
>
>Yes, the terminal should use the same face as GUI for the first blank
>after end of line.
>
>Thanks.

Re: Question about display engine

[Ergus]

Hi Eli & Martin:

I just made some extra changes in the branch to fix the issues for the
extra space in terminal gui.

To test them try to disable the :extend for the region face and select a
region with empty lines see if this is a right behavior (I think
so). Now the extra space is colored like the text no with the extend
face in gui and tui interfaces..

Eli:

The questions I asked before were:

>
>I implemented this in the face_at_buffer_position but in the same
>function (handle_face_prop_general) there is a function called
>face_at_string_position equivalent to face_at_buffer_position... is it
>possible in some conditions to reach that part of the code at eol? if
>so I will need to modify that one too maybe right?
>
>The filter now in merge_ref only works when !CONSP(ref_name). As it only
>bypass the extra parameter to merge_named... it this right in the
>general case?
>

Could you suggest (if any) if there are some conditions we can use to
avoid the call to handle_face_prop_general in some cases?

Re: Question about display engine

[martin rudalics]

 > To test them try to disable the :extend for the region face and select a
 > region with empty lines see if this is a right behavior (I think
 > so). Now the extra space is colored like the text no with the extend
 > face in gui and tui interfaces..

FWIW I see no problems in this regard.  One thing that surprises me on
a TTY is that the 'region' background does not cover the background of
'widget-field' (as it does on a GUI) but apparently this seems to have
been the case ever since.

Maybe also the :extend attribute of the default face should show up as
"On" by default but I wouldn't even know where to specify that.

For the rest, I'm all for installing this ASAP so we can see what kind
of problems people have with it.  I do expect some sort of breakage in
external packages.

Thanks for all your work, martin

Re: Question about display engine

[Ergus]

On Sun, Sep 15, 2019 at 10:25:58AM +0200, martin rudalics wrote:
>
>FWIW I see no problems in this regard.  One thing that surprises me on
>a TTY is that the 'region' background does not cover the background of
>'widget-field' (as it does on a GUI) but apparently this seems to have
>been the case ever since.
>
Please. Could you give me how to reproduce this issue?? Maybe I could
fix it too. (if there is not a reason for having this as is, of course)

>Maybe also the :extend attribute of the default face should show up as
>"On" by default but I wouldn't even know where to specify that.
>
I didn't for two reasons. 1) In the merge process the default face is
used to initialize always the vector, so it is not needed to set it
:extend t. But also many other faces (specially in extern packages)
inherit from default and then set other attributes in their faces so all
of them will need to be changed... to behave correctly... so better not
to change the default face in my opinion... What do you think?

>For the rest, I'm all for installing this ASAP so we can see what kind
>of problems people have with it.  I do expect some sort of breakage in
>external packages.
>
I usually (as suggested by Eli) wait around 2 weeks until more people
have time to test the changes and report issues before adding this in
master. I need to update also the documentation (manual and NEWS) before
merging in master. If you want to propose what to put there it will be
very nice. Documentation takes me a lot of time because I am not
familiar with the texi format

Best,
Ergus

Re: Question about display engine

[martin rudalics]

 > Please. Could you give me how to reproduce this issue?? Maybe I could
 > fix it too. (if there is not a reason for having this as is, of course)

M-x customize-face RET region RET C-x h

Here there are two areas painted with yellow3 background instead of
the region's blue3.  I see no particular reason for that behavior.

 >> Maybe also the :extend attribute of the default face should show up as
 >> "On" by default but I wouldn't even know where to specify that.
 >>
 > I didn't for two reasons. 1) In the merge process the default face is
 > used to initialize always the vector, so it is not needed to set it
 > :extend t. But also many other faces (specially in extern packages)
 > inherit from default and then set other attributes in their faces so all
 > of them will need to be changed... to behave correctly... so better not
 > to change the default face in my opinion... What do you think?

I think that you are right.  Maybe we should leave a note in a
doc-string or in *info* about that fact.

martin

Re: Question about display engine

[Stefan Monnier]

> M-x customize-face RET region RET C-x h
>
> Here there are two areas painted with yellow3 background instead of
> the region's blue3.  I see no particular reason for that behavior.

I'm not sure what you're describing (the colors don't seem to match
what I see), but it sounds like normal behavior (smaller overlays
having precedence over larger ones).


        Stefan

Re: Question about display engine

[martin rudalics]

 > I'm not sure what you're describing (the colors don't seem to match
 > what I see), but it sounds like normal behavior (smaller overlays
 > having precedence over larger ones).

Not really.  This is 'redisplay-highlight-region-function' at work and
it's by design.  The idea is that when the region starts or ends
within an overlay, it is shown within that overlay.  If the region
covers the overlay completely, it is not shown within that overlay.

The effect is slightly different for GUIs and TTYs.  On TTYs only
'widget-field' is affected here, on GUIs 'custom-button' as well.
Which is probably by design too but I don't know why.

martin

Re: Question about display engine

[Eli Zaretskii]

> Date: Sat, 14 Sep 2019 22:42:07 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >I implemented this in the face_at_buffer_position but in the same
> >function (handle_face_prop_general) there is a function called
> >face_at_string_position equivalent to face_at_buffer_position... is it
> >possible in some conditions to reach that part of the code at eol? if
> >so I will need to modify that one too maybe right?

Yes, it's possible.  You need to arrange for a display string or an
overlay string with en embedded newline.

> >The filter now in merge_ref only works when !CONSP(ref_name). As it only
> >bypass the extra parameter to merge_named... it this right in the
> >general case?

I think we should support all the cases, otherwise the feature will
behave inconsistently, and we will get bug reports.

> Could you suggest (if any) if there are some conditions we can use to
> avoid the call to handle_face_prop_general in some cases?

Why, did you see any tangible slow-down of redisplay due to these
changes?

Re: Question about display engine

[Ergus]

On Sun, Sep 15, 2019 at 06:32:12PM +0300, Eli Zaretskii wrote:
>> Date: Sat, 14 Sep 2019 22:42:07 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >I implemented this in the face_at_buffer_position but in the same
>> >function (handle_face_prop_general) there is a function called
>> >face_at_string_position equivalent to face_at_buffer_position... is it
>> >possible in some conditions to reach that part of the code at eol? if
>> >so I will need to modify that one too maybe right?
>
>Yes, it's possible.  You need to arrange for a display string or an
>overlay string with en embedded newline.
>
Done, new commit uploaded
>> >The filter now in merge_ref only works when !CONSP(ref_name). As it only
>> >bypass the extra parameter to merge_named... it this right in the
>> >general case?
>
>I think we should support all the cases, otherwise the feature will
>behave inconsistently, and we will get bug reports.
>
About this; the problem is that in the general case I'm not sure what's
the "right" behavior for the other cases. When !CONSP(ref_name) it means
that the parameter is a face_name; but in the other cases it means that
we are explicitly specifying the attributes as a cons list or as
:atribute value pairs... What's expected to happen in those cases??


>> Could you suggest (if any) if there are some conditions we can use to
>> avoid the call to handle_face_prop_general in some cases?
>
>Why, did you see any tangible slow-down of redisplay due to these
>changes?

Re: Question about display engine

[Ergus]

>>>>The filter now in merge_ref only works when !CONSP(ref_name). As it only
>>>>bypass the extra parameter to merge_named... it this right in the
>>>>general case?
>>
>>I think we should support all the cases, otherwise the feature will
>>behave inconsistently, and we will get bug reports.
>>
>About this; the problem is that in the general case I'm not sure what's
>the "right" behavior for the other cases. When !CONSP(ref_name) it means
>that the parameter is a face_name; but in the other cases it means that
>we are explicitly specifying the attributes as a cons list or as
>:atribute value pairs... What's expected to happen in those cases??
>
Hi:
Any Idea about this? Could you suggest what to do when CONSP(ref_name)
is true?
>
>>>Could you suggest (if any) if there are some conditions we can use to
>>>avoid the call to handle_face_prop_general in some cases?
>>
>>Why, did you see any tangible slow-down of redisplay due to these
>>changes?
>
No actually, but I think that  there should be many cases where we could
avoid calling the new function. So it is better to put them and avoid
unneeded calls.

For example when face_id is the default_face_id maybe?

Re: Question about display engine

[Eli Zaretskii]

> Date: Tue, 17 Sep 2019 04:17:26 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >>>>The filter now in merge_ref only works when !CONSP(ref_name). As it only
> >>>>bypass the extra parameter to merge_named... it this right in the
> >>>>general case?
> >>
> >>I think we should support all the cases, otherwise the feature will
> >>behave inconsistently, and we will get bug reports.
> >>
> >About this; the problem is that in the general case I'm not sure what's
> >the "right" behavior for the other cases. When !CONSP(ref_name) it means
> >that the parameter is a face_name; but in the other cases it means that
> >we are explicitly specifying the attributes as a cons list or as
> >:atribute value pairs... What's expected to happen in those cases??
> >
> Hi:
> Any Idea about this? Could you suggest what to do when CONSP(ref_name)
> is true?

I will reply to this soon, too swamped now.

> >>>Could you suggest (if any) if there are some conditions we can use to
> >>>avoid the call to handle_face_prop_general in some cases?
> >>
> >>Why, did you see any tangible slow-down of redisplay due to these
> >>changes?
> >
> No actually, but I think that  there should be many cases where we could
> avoid calling the new function. So it is better to put them and avoid
> unneeded calls.
> 
> For example when face_id is the default_face_id maybe?

Yes, for the default face ID it might be a good idea to add some
optimization, as that will be the most frequent case.  But beware of
remapped default face, though.

Re: Question about display engine

[Eli Zaretskii]

> Date: Tue, 17 Sep 2019 12:48:02 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> > Date: Tue, 17 Sep 2019 04:17:26 +0200
> > From: Ergus <spacibba@aol.com>
> > Cc: rudalics@gmx.at, emacs-devel@gnu.org
> > 
> > >>>>The filter now in merge_ref only works when !CONSP(ref_name). As it only
> > >>>>bypass the extra parameter to merge_named... it this right in the
> > >>>>general case?
> > >>
> > >>I think we should support all the cases, otherwise the feature will
> > >>behave inconsistently, and we will get bug reports.
> > >>
> > >About this; the problem is that in the general case I'm not sure what's
> > >the "right" behavior for the other cases. When !CONSP(ref_name) it means
> > >that the parameter is a face_name; but in the other cases it means that
> > >we are explicitly specifying the attributes as a cons list or as
> > >:atribute value pairs... What's expected to happen in those cases??
> > >
> > Hi:
> > Any Idea about this? Could you suggest what to do when CONSP(ref_name)
> > is true?
> 
> I will reply to this soon, too swamped now.

Sorry for the delay.

Having looked at the code, I'm not sure I understand the problem.  Why
not simply pass the attr_filter down to the respective merge_face_ref
calls, where you currently force zero instead?  Am I missing
something?

Some other comments about your code:

 . Please rename handle_face_prop_general into something like
   face_at_pos, and make it just return the face ID, without assigning
   any field in 'struct it'.  handle_face_prop will then call
   face_at_pos and assign the face ID as needed.
 . handle_face_prop_general is supposed to be called just once with
   the last argument non-zero, so I see no reason why it should be
   also passed the initial_face_id argument.  It looks wrong to call
   that function with it->extend_face_id as the 2nd argument, and have
   it compute it->extend_face_id, because the value you pass as an
   argument is undefined: it hasn't been computed yet.  I think the
   function should use it->face_id internally instead of that
   argument.
 . I don't understand why you need new members of 'struct it', like
   extend_face_id, saved_extend_face_id, etc.
   extend_face_to_end_of_line correctly assigns the value of extend
   face ID to it->face_id, after saving it->face_id in a local
   variable, so I see no need for it->extend_face_id, certainly not
   for it->saved_extend_face_id.  You also have extend_face_id in
   other related structures, where it is never used.

Regarding documentation: if you have difficulties with the Texinfo
markup, you could write plain text, and someone else could then add
markup.  Adding markup is a mostly mechanical procedure, unlike coming
up with a useful text.

Thanks.

Re: Question about display engine

[Ergus]

On Sat, Sep 21, 2019 at 11:20:44AM +0300, Eli Zaretskii wrote:
>> Date: Tue, 17 Sep 2019 12:48:02 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> > Date: Tue, 17 Sep 2019 04:17:26 +0200
>> > From: Ergus <spacibba@aol.com>
>> > Cc: rudalics@gmx.at, emacs-devel@gnu.org
>> >
>> > >>>>The filter now in merge_ref only works when !CONSP(ref_name). As it only
>> > >>>>bypass the extra parameter to merge_named... it this right in the
>> > >>>>general case?
>> > >>
>> > >>I think we should support all the cases, otherwise the feature will
>> > >>behave inconsistently, and we will get bug reports.
>> > >>
>> > >About this; the problem is that in the general case I'm not sure what's
>> > >the "right" behavior for the other cases. When !CONSP(ref_name) it means
>> > >that the parameter is a face_name; but in the other cases it means that
>> > >we are explicitly specifying the attributes as a cons list or as
>> > >:atribute value pairs... What's expected to happen in those cases??
>> > >
>> > Hi:
>> > Any Idea about this? Could you suggest what to do when CONSP(ref_name)
>> > is true?
>>
>> I will reply to this soon, too swamped now.
>
Hi:

Thanks for the reply.

>Sorry for the delay.
>
>Having looked at the code, I'm not sure I understand the problem.  Why
>not simply pass the attr_filter down to the respective merge_face_ref
>calls, where you currently force zero instead?  Am I missing
>something?
>
That's what I do actually are you in the last changes? Probably I am the
one who is missing something otherwise.

>Some other comments about your code:
>
> . Please rename handle_face_prop_general into something like
>   face_at_pos, and make it just return the face ID, without assigning
>   any field in 'struct it'.  handle_face_prop will then call
>   face_at_pos and assign the face ID as needed.

Yes, I change that already to return the id without assigning anything.

> . handle_face_prop_general is supposed to be called just once with
>   the last argument non-zero, so I see no reason why it should be
>   also passed the initial_face_id argument.  It looks wrong to call
>   that function with it->extend_face_id as the 2nd argument, and have
>   it compute it->extend_face_id, because the value you pass as an
>   argument is undefined: it hasn't been computed yet.  I think the
>   function should use it->face_id internally instead of that
>   argument.

The initial face if has nothing to do with the last argument. It is
needed for an optimization at the end of the function. But yes, using it
internally is better.

> . I don't understand why you need new members of 'struct it', like
>   extend_face_id, saved_extend_face_id, etc.
>   extend_face_to_end_of_line correctly assigns the value of extend
>   face ID to it->face_id, after saving it->face_id in a local
>   variable, so I see no need for it->extend_face_id, certainly not
>   for it->saved_extend_face_id.  You also have extend_face_id in
>   other related structures, where it is never used.
>
This was for the idea to avoid recomputing it->extend_face_id in some
cases, and reuse the previous value when possible too. (that's why I
pass it as a parameter face_id actually.)

But I can change that if you think it is not needed.

>Regarding documentation: if you have difficulties with the Texinfo
>markup, you could write plain text, and someone else could then add
>markup.  Adding markup is a mostly mechanical procedure, unlike coming
>up with a useful text.
>
>Thanks.
Thanks to you

Re: Question about display engine

[Ergus]

On Sat, Sep 21, 2019 at 11:20:44AM +0300, Eli Zaretskii wrote:
>
>Sorry for the delay.
>
>Having looked at the code, I'm not sure I understand the problem.  Why
>not simply pass the attr_filter down to the respective merge_face_ref
>calls, where you currently force zero instead?  Am I missing
>something?
>
>Some other comments about your code:
>
> . Please rename handle_face_prop_general into something like
>   face_at_pos, and make it just return the face ID, without assigning
>   any field in 'struct it'.  handle_face_prop will then call
>   face_at_pos and assign the face ID as needed.
> . handle_face_prop_general is supposed to be called just once with
>   the last argument non-zero, so I see no reason why it should be
>   also passed the initial_face_id argument.  It looks wrong to call
>   that function with it->extend_face_id as the 2nd argument, and have
>   it compute it->extend_face_id, because the value you pass as an
>   argument is undefined: it hasn't been computed yet.  I think the
>   function should use it->face_id internally instead of that
>   argument.
> . I don't understand why you need new members of 'struct it', like
>   extend_face_id, saved_extend_face_id, etc.
>   extend_face_to_end_of_line correctly assigns the value of extend
>   face ID to it->face_id, after saving it->face_id in a local
>   variable, so I see no need for it->extend_face_id, certainly not
>   for it->saved_extend_face_id.  You also have extend_face_id in
>   other related structures, where it is never used.
>
>Regarding documentation: if you have difficulties with the Texinfo
>markup, you could write plain text, and someone else could then add
>markup.  Adding markup is a mostly mechanical procedure, unlike coming
>up with a useful text.
>
Hi:

I just added some documentation in the reference manuals please check
it.

I should add a comment in the NEWS file, in which part is the right one?


>Thanks.

Re: Question about display engine

[Ergus]

On Sat, Sep 21, 2019 at 11:55:51PM +0200, Ergus wrote:
>On Sat, Sep 21, 2019 at 11:20:44AM +0300, Eli Zaretskii wrote:

Hi Eli:

Please check the branch scratch/extend_face_id and tell me what's
missing to merge in master.

Best,
Ergus

>Hi:
>
>I just added some documentation in the reference manuals please check
>it.
>
>I should add a comment in the NEWS file, in which part is the right one?
>
>
>

Re: Question about display engine

[Eli Zaretskii]

> Date: Thu, 26 Sep 2019 18:32:04 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> Please check the branch scratch/extend_face_id and tell me what's
> missing to merge in master.

Thanks, I took a look.  I think the branch is almost ready to be
merged, once we address the following minor comments:

 . I don't understand why you changed the underlined_p and
   underline_type stuff in struct face.  What were the reasons for
   that part of the changeset?  Its sole effect seems to be to
   generate some redundant changes, or am I missing something?

 . In this hunk from append_space_for_newline, why did you change
   FACE_FROM_ID_OR_NULL to FACE_FROM_ID?

     -         int local_default_face_id =
     +      int char_width = 1;
     +
     +      if (default_face_p
     +#ifdef HAVE_WINDOW_SYSTEM
     +          || FRAME_WINDOW_P (it->f)
     +#endif
     +         )
     +       {
     +         const int local_default_face_id =
		 lookup_basic_face (it->w, it->f, DEFAULT_FACE_ID);
	       struct face* default_face =
     -           FACE_FROM_ID_OR_NULL (it->f, local_default_face_id);
     +           FACE_FROM_ID (it->f, local_default_face_id);

 . Please use comments /* in this style */, not // like this.

 . In this hunk from extend_face_to_end_of_line you lost the
   comment.  Is that comment no longer correct?

     -             int column_x;

     -             if (!INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
     -                 && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
     -                 && column_x >= it->current_x
     -                 && column_x <= it->last_visible_x)
     -               {
     +         const int indicator_column =
     +           fill_column_indicator_column (it, char_width);
     +
	       const char saved_char = it->char_to_display;
	       const struct text_pos saved_pos = it->position;
	       const bool saved_avoid_cursor = it->avoid_cursor_p;
     -                 const int saved_face_id = it->face_id;
	       const bool saved_box_start = it->start_of_box_run_p;
	       Lisp_Object save_object = it->object;
     +         const int saved_face_id = it->face_id;

     -                 /* The stretch width needs to considet the latter
     -                    added glyph.  */
     -                 const int stretch_width
     -                   = column_x - it->current_x - char_width;
     -
     -                 memset (&it->position, 0, sizeof it->position);
     +         it->face_id = extend_face_id;
	       it->avoid_cursor_p = true;
	       it->object = Qnil;

     +         if (indicator_column >= 0
     +             && indicator_column > it->current_x
     +             && indicator_column < it->last_visible_x)
     +            {
     +
     +             const int stretch_width =
     +               indicator_column - it->current_x - char_width;
     +
     +             memset (&it->position, 0, sizeof it->position);
     +

 . And here the comment was not moved with the code which it
   describes:

	       /* Restore the face after the indicator was generated.  */
     -                 it->face_id = saved_face_id;

	       /* If there is space after the indicator generate an
		  extra empty glyph to restore the face.  Issue was
     @@ -20634,8 +20633,8 @@ extend_face_to_end_of_line (struct it *it)
	       it->avoid_cursor_p = saved_avoid_cursor;
	       it->start_of_box_run_p = saved_box_start;
	       it->object = save_object;
     -               }
     -            }
     +         it->face_id = saved_face_id;

 . This hunk looks redundant to me:

     @@ -20681,10 +20680,9 @@ extend_face_to_end_of_line (struct it *it)
		   /* The last row's stretch glyph should get the default
		      face, to avoid painting the rest of the window with
		      the region face, if the region ends at ZV.  */
     -             if (it->glyph_row->ends_at_zv_p)
     -               it->face_id = default_face->id;
     -             else
     -               it->face_id = face->id;
     +             it->face_id = (it->glyph_row->ends_at_zv_p ?
     +                            default_face->id : face->id);

   (there's at least one more like it in the changeset).

 . This also looks redundant:

     @@ -25436,8 +25423,8 @@ display_string (const char *string, Lisp_Object lisp_str

	/* Initialize the iterator IT for iteration over STRING beginning
	   with index START.  */
     -  reseat_to_string (it, NILP (lisp_string) ? string : NULL, lisp_string, start,
     -                   precision, field_width, multibyte);
     +  reseat_to_string (it, NILP (lisp_string) ? string : NULL, lisp_string,
     +                    start, precision, field_width, multibyte);

 . In this commentary, please upcase attr_filter, as that is our
   convention for describing function arguments in comments:

     @@ -2269,6 +2282,11 @@ filter_face_ref (Lisp_Object face_ref,
	 of ERR_MSGS).  Use NAMED_MERGE_POINTS to detect loops in face
	 inheritance or list structure; it may be 0 for most callers.

     +   attr_filter is the index of a parameter that conditions the merging
     +   for named faces (case 1) to only the face_ref where
     +   lface[merge_face_ref] is non-nil. To merge unconditionally set this
     +   value to 0.

 . Likewise here:

     @@ -5988,6 +6039,8 @@ compute_char_face (struct frame *f, int ch, Lisp_Object pr
	 which a different face is needed, as far as text properties and
	 overlays are concerned.  W is a window displaying current_buffer.

     +   attr_filter is passed merge_face_ref.

   The sentence you added sounds incomplete here: did you mean to say
   "passed to merge_face_ref"?

 . This hunk looks redundant:

     @@ -4505,7 +4552,8 @@ lookup_face (struct frame *f, Lisp_Object *attr)
	 suitable face is found, realize a new one.  */

      int
     -face_for_font (struct frame *f, Lisp_Object font_object, struct face *base_face
     +face_for_font (struct frame *f, Lisp_Object font_object,
     +               struct face *base_face)

 . This also looks redundant:

     @@ -6068,19 +6122,18 @@ face_at_buffer_position (struct window *w, ptrdiff_t pos
	}

	/* Optimize common cases where we can use the default face.  */
     -  if (noverlays == 0
     -      && NILP (prop))
     +  if (noverlays == 0 && NILP (prop))

 . And this one:

	/* Begin with attributes from the default face.  */
     -  memcpy (attrs, default_face->lface, sizeof attrs);
     +  memcpy (attrs, default_face->lface, sizeof(attrs));

 . Finally, please write some short description for NEWS, as this
   feature definitely needs to be mentioned there.

Thanks again for working on this, and sorry for my unusually slow
responses.

Re: Question about display engine

[Ergus]

On Sat, Sep 28, 2019 at 01:35:30PM +0300, Eli Zaretskii wrote:
Hi Eli:
>
>Thanks, I took a look.  I think the branch is almost ready to be
>merged, once we address the following minor comments:
>
> . I don't understand why you changed the underlined_p and
>   underline_type stuff in struct face.  What were the reasons for
>   that part of the changeset?  Its sole effect seems to be to
>   generate some redundant changes, or am I missing something?
>
Use double variable to describe the same is an error prone
practice. This change simplifies the checks in all the code related (as
there will be only one variable to set/check), reduces one name and
member in the struct and avoids errors related to changing one value and
not the other.

>
> . In this hunk from append_space_for_newline, why did you change
>   FACE_FROM_ID_OR_NULL to FACE_FROM_ID?
>
>     -         int local_default_face_id =
>     +      int char_width = 1;
>     +
>     +      if (default_face_p
>     +#ifdef HAVE_WINDOW_SYSTEM
>     +          || FRAME_WINDOW_P (it->f)
>     +#endif
>     +         )
>     +       {
>     +         const int local_default_face_id =
>		 lookup_basic_face (it->w, it->f, DEFAULT_FACE_ID);
>	       struct face* default_face =
>     -           FACE_FROM_ID_OR_NULL (it->f, local_default_face_id);
>     +           FACE_FROM_ID (it->f, local_default_face_id);
>

The call is made after a lookup_basic_face and it's for DEFAULT_FACE_ID
is it needed the check? Normally for other faces if it is NULL then we
use the DEFAULT_FACE_ID. In this case it should be unneeded right?

> . Please use comments /* in this style */, not // like this.
>
Fixed
>
> . In this hunk from extend_face_to_end_of_line you lost the
>   comment.  Is that comment no longer correct?
>
>     -             int column_x;
>
>     -             if (!INT_MULTIPLY_WRAPV (indicator_column, char_width, &column_x)
>     -                 && !INT_ADD_WRAPV (it->lnum_pixel_width, column_x, &column_x)
>     -                 && column_x >= it->current_x
>     -                 && column_x <= it->last_visible_x)
>     -               {
>     +         const int indicator_column =
>     +           fill_column_indicator_column (it, char_width);
>     +
>	       const char saved_char = it->char_to_display;
>	       const struct text_pos saved_pos = it->position;
>	       const bool saved_avoid_cursor = it->avoid_cursor_p;
>     -                 const int saved_face_id = it->face_id;
>	       const bool saved_box_start = it->start_of_box_run_p;
>	       Lisp_Object save_object = it->object;
>     +         const int saved_face_id = it->face_id;
>
>     -                 /* The stretch width needs to considet the latter
>     -                    added glyph.  */
>     -                 const int stretch_width
>     -                   = column_x - it->current_x - char_width;
>     -
>     -                 memset (&it->position, 0, sizeof it->position);
>     +         it->face_id = extend_face_id;
>	       it->avoid_cursor_p = true;
>	       it->object = Qnil;
>
>     +         if (indicator_column >= 0
>     +             && indicator_column > it->current_x
>     +             && indicator_column < it->last_visible_x)
>     +            {
>     +
>     +             const int stretch_width =
>     +               indicator_column - it->current_x - char_width;
>     +
>     +             memset (&it->position, 0, sizeof it->position);
>     +
>
The comment is now in the function fill_column_indicator_column where
the calculation is performed now.
>
> . And here the comment was not moved with the code which it
>   describes:
>
>	       /* Restore the face after the indicator was generated.  */
>     -                 it->face_id = saved_face_id;
>
>	       /* If there is space after the indicator generate an
>		  extra empty glyph to restore the face.  Issue was
>     @@ -20634,8 +20633,8 @@ extend_face_to_end_of_line (struct it *it)
>	       it->avoid_cursor_p = saved_avoid_cursor;
>	       it->start_of_box_run_p = saved_box_start;
>	       it->object = save_object;
>     -               }
>     -            }
>     +         it->face_id = saved_face_id;
>
Fixed.

> . This hunk looks redundant to me:
>
>     @@ -20681,10 +20680,9 @@ extend_face_to_end_of_line (struct it *it)
>		   /* The last row's stretch glyph should get the default
>		      face, to avoid painting the rest of the window with
>		      the region face, if the region ends at ZV.  */
>     -             if (it->glyph_row->ends_at_zv_p)
>     -               it->face_id = default_face->id;
>     -             else
>     -               it->face_id = face->id;
>     +             it->face_id = (it->glyph_row->ends_at_zv_p ?
>     +                            default_face->id : face->id);
>
>   (there's at least one more like it in the changeset).
>
For me this looked better (shorter and clearer), but I will restore it
if you prefer that.

> . This also looks redundant:
>
>     @@ -25436,8 +25423,8 @@ display_string (const char *string, Lisp_Object lisp_str
>
>	/* Initialize the iterator IT for iteration over STRING beginning
>	   with index START.  */
>     -  reseat_to_string (it, NILP (lisp_string) ? string : NULL, lisp_string, start,
>     -                   precision, field_width, multibyte);
>     +  reseat_to_string (it, NILP (lisp_string) ? string : NULL, lisp_string,
>     +                    start, precision, field_width, multibyte);
>
The line was too long. I just tried to reduce it a bit.

> . In this commentary, please upcase attr_filter, as that is our
>   convention for describing function arguments in comments:
>
>     @@ -2269,6 +2282,11 @@ filter_face_ref (Lisp_Object face_ref,
>	 of ERR_MSGS).  Use NAMED_MERGE_POINTS to detect loops in face
>	 inheritance or list structure; it may be 0 for most callers.
>
>     +   attr_filter is the index of a parameter that conditions the merging
>     +   for named faces (case 1) to only the face_ref where
>     +   lface[merge_face_ref] is non-nil. To merge unconditionally set this
>     +   value to 0.
>
> . Likewise here:
>
>     @@ -5988,6 +6039,8 @@ compute_char_face (struct frame *f, int ch, Lisp_Object pr
>	 which a different face is needed, as far as text properties and
>	 overlays are concerned.  W is a window displaying current_buffer.
>
>     +   attr_filter is passed merge_face_ref.
>
>   The sentence you added sounds incomplete here: did you mean to say
>   "passed to merge_face_ref"?
>
Fixed both.

> . This hunk looks redundant:
>
>     @@ -4505,7 +4552,8 @@ lookup_face (struct frame *f, Lisp_Object *attr)
>	 suitable face is found, realize a new one.  */
>
>      int
>     -face_for_font (struct frame *f, Lisp_Object font_object, struct face *base_face
>     +face_for_font (struct frame *f, Lisp_Object font_object,
>     +               struct face *base_face)
>
The line was too long. 85 chars long.

> . This also looks redundant:
>
>     @@ -6068,19 +6122,18 @@ face_at_buffer_position (struct window *w, ptrdiff_t pos
>	}
>
>	/* Optimize common cases where we can use the default face.  */
>     -  if (noverlays == 0
>     -      && NILP (prop))
>     +  if (noverlays == 0 && NILP (prop))
>
> . And this one:
>
>	/* Begin with attributes from the default face.  */
>     -  memcpy (attrs, default_face->lface, sizeof attrs);
>     +  memcpy (attrs, default_face->lface, sizeof(attrs));
>
> . Finally, please write some short description for NEWS, as this
>   feature definitely needs to be mentioned there.
>
It seems unneeded two lines for this; but fixed. 

>Thanks again for working on this, and sorry for my unusually slow
>responses.

Thank to you.

Re: Question about display engine

[Eli Zaretskii]

> Date: Sun, 29 Sep 2019 12:30:34 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> > . I don't understand why you changed the underlined_p and
> >   underline_type stuff in struct face.  What were the reasons for
> >   that part of the changeset?  Its sole effect seems to be to
> >   generate some redundant changes, or am I missing something?
> >
> Use double variable to describe the same is an error prone
> practice. This change simplifies the checks in all the code related (as
> there will be only one variable to set/check), reduces one name and
> member in the struct and avoids errors related to changing one value and
> not the other.

That's your stylistic preference, and I can understand it.  But when
the only reason for a change is stylistic preferences, I generally
prefer to leave the code intact, so that the original authors' version
remains as long as it does TRT.

> > . In this hunk from append_space_for_newline, why did you change
> >   FACE_FROM_ID_OR_NULL to FACE_FROM_ID?
> >
> >     -         int local_default_face_id =
> >     +      int char_width = 1;
> >     +
> >     +      if (default_face_p
> >     +#ifdef HAVE_WINDOW_SYSTEM
> >     +          || FRAME_WINDOW_P (it->f)
> >     +#endif
> >     +         )
> >     +       {
> >     +         const int local_default_face_id =
> >		 lookup_basic_face (it->w, it->f, DEFAULT_FACE_ID);
> >	       struct face* default_face =
> >     -           FACE_FROM_ID_OR_NULL (it->f, local_default_face_id);
> >     +           FACE_FROM_ID (it->f, local_default_face_id);
> >
> 
> The call is made after a lookup_basic_face and it's for DEFAULT_FACE_ID
> is it needed the check? Normally for other faces if it is NULL then we
> use the DEFAULT_FACE_ID. In this case it should be unneeded right?

It's the other way around: FACE_FROM_ID could trigger an assertion
violation, where FACE_FROM_ID_OR_NULL will silently return a NULL
pointer.  So we should only use FACE_FROM_ID where we are 110% certain
it can never violate the assertion, or where a NULL pointer for a face
will cause worse problems than an assertion.

Thanks.

Re: Question about display engine

[Ergus]

Hi Eli:

I made some of the fixes you suggested in the last email about this
topic.

The reduction of the underline-p + underline-stile into a single
underline variable in the struct for me still seems like a good change
for readability and simplification in all the code. Any way in the
future I will try to avoid this kind of modifications even when they
seems to be an improvement... but could you let it pass this time?

The other changes that seemed to be stylistic too, were actually in
portions of code I wrote for dfci and It was there because I copied some
of it from the previous if-else section... but I can correct it now
right?... so no problem?

If this is fine may I merge in master?
Best,
Ergus.


On Sun, Sep 29, 2019 at 01:57:22PM +0300, Eli Zaretskii wrote:
>> Date: Sun, 29 Sep 2019 12:30:34 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> > . I don't understand why you changed the underlined_p and
>> >   underline_type stuff in struct face.  What were the reasons for
>> >   that part of the changeset?  Its sole effect seems to be to
>> >   generate some redundant changes, or am I missing something?
>> >
>> Use double variable to describe the same is an error prone
>> practice. This change simplifies the checks in all the code related (as
>> there will be only one variable to set/check), reduces one name and
>> member in the struct and avoids errors related to changing one value and
>> not the other.
>
>That's your stylistic preference, and I can understand it.  But when
>the only reason for a change is stylistic preferences, I generally
>prefer to leave the code intact, so that the original authors' version
>remains as long as it does TRT.
>
>> > . In this hunk from append_space_for_newline, why did you change
>> >   FACE_FROM_ID_OR_NULL to FACE_FROM_ID?
>> >
>> >     -         int local_default_face_id =
>> >     +      int char_width = 1;
>> >     +
>> >     +      if (default_face_p
>> >     +#ifdef HAVE_WINDOW_SYSTEM
>> >     +          || FRAME_WINDOW_P (it->f)
>> >     +#endif
>> >     +         )
>> >     +       {
>> >     +         const int local_default_face_id =
>> >		 lookup_basic_face (it->w, it->f, DEFAULT_FACE_ID);
>> >	       struct face* default_face =
>> >     -           FACE_FROM_ID_OR_NULL (it->f, local_default_face_id);
>> >     +           FACE_FROM_ID (it->f, local_default_face_id);
>> >
>>
>> The call is made after a lookup_basic_face and it's for DEFAULT_FACE_ID
>> is it needed the check? Normally for other faces if it is NULL then we
>> use the DEFAULT_FACE_ID. In this case it should be unneeded right?
>
>It's the other way around: FACE_FROM_ID could trigger an assertion
>violation, where FACE_FROM_ID_OR_NULL will silently return a NULL
>pointer.  So we should only use FACE_FROM_ID where we are 110% certain
>it can never violate the assertion, or where a NULL pointer for a face
>will cause worse problems than an assertion.
>
>Thanks.
>

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 7 Oct 2019 17:40:55 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> I made some of the fixes you suggested in the last email about this
> topic.

Thanks, see a couple of comments below.

> The reduction of the underline-p + underline-stile into a single
> underline variable in the struct for me still seems like a good change
> for readability and simplification in all the code. Any way in the
> future I will try to avoid this kind of modifications even when they
> seems to be an improvement... but could you let it pass this time?

I don't understand what makes this time special, but OK.

> The other changes that seemed to be stylistic too, were actually in
> portions of code I wrote for dfci and It was there because I copied some
> of it from the previous if-else section... but I can correct it now
> right?... so no problem?
> 
> If this is fine may I merge in master?

It's OK to merge, after fixing the following nits:

> ++++
> +** There is a new face attribute :extend to use the face attributes to
> +extend after the end of the line until the end of the window.  Such
> +:extend is set to nil by default in all faces except for `hl-line` and
> +`region` because those extend until the end of the window by default.

Please quote 'like this' in NEWS, not `like this`.

Also, this NEWS entry should have a header line:

  ** New face attribute ':extend' to control face extension at EOL.

> +      /* The stretch width needs to considet the latter
                                     ^^^^^^^^
A typo.

>        /* Display fill-column indicator if needed.  */
> -      int indicator_column = fill_column_indicator_column (it);
> -      if (indicator_column >= 0
> -         && INT_ADD_WRAPV (it->lnum_pixel_width, indicator_column,
> -                           &indicator_column))
> -       indicator_column = -1;
> +      const int indicator_column =
> +       fill_column_indicator_column (it, 1) - 1;

Why did you need to subtract 1 in the last line?  If this is indeed
needed, it needs a comment to explain it.

> +   ATTR_FILTER is the index of a parameter that conditions the merging
> +   for named faces (case 1) to only the face_ref where
> +   lface[merge_face_ref] is non-nil. To merge unconditionally set this
> +   value to 0.                     ^^

Two spaces between sentences, please.

Thanks.

Re: Question about display engine

[Ergus]

Hi Eli:

I haven't merged this yet because 2 days ago I found that there is a bug
in the code very difficult to locate for me and I have not understand
what's going on.

The problem is that emacs just freezes in the magit-log buffer in
tui. (M-x magit-log-all for example.) 

This seems to be a weird issue because in gui it doesn't happen only in
the terminal interface (which should be agnostic to magit). And I have
only observed it in magit-log. And if this happens for magit-log I am
wondering that there should be other possible situations that produce
the same problem.

After some tests and moving in the history of my changes I got that the
issue is the call to handle_face_prop in extend_face_to_end_of_line.

Trying the next in current master I got the same issue: 

// ===============================

diff --git a/src/xdisp.c b/src/xdisp.c
index 893ce9269c..af50dd0bcd 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -21587,6 +21587,7 @@ extend_face_to_end_of_line (struct it *it)
           || WINDOW_RIGHT_MARGIN_WIDTH (it->w) > 0))
     return;
 
+  handle_face_prop (it);
   /* Face extension extends the background and box of IT->face_id
      to the end of the line.  If the background equals the background
      of the frame, we don't have to do anything.  */

// ===============================

In gdb I saee that it goes in a very complex inf loop within the display
engine functions and emacs becomes completely unresponsive (No C-g or
ESC ESC ESC) the only solution is to kill it from outside.

Magit provides a way to execute emacs loading only magit
(magit-emacs-Q-command), so it is nothing in my config... but something
probably tricky used in magit-log that exposes the issue.

Could you give a look please?

Re: Question about display engine

[Eli Zaretskii]

> Date: Sat, 12 Oct 2019 20:16:18 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> Trying the next in current master I got the same issue: 
> 
> // ===============================
> 
> diff --git a/src/xdisp.c b/src/xdisp.c
> index 893ce9269c..af50dd0bcd 100644
> --- a/src/xdisp.c
> +++ b/src/xdisp.c
> @@ -21587,6 +21587,7 @@ extend_face_to_end_of_line (struct it *it)
>            || WINDOW_RIGHT_MARGIN_WIDTH (it->w) > 0))
>      return;
>  
> +  handle_face_prop (it);
>    /* Face extension extends the background and box of IT->face_id
>       to the end of the line.  If the background equals the background
>       of the frame, we don't have to do anything.  */
> 
> // ===============================

I don't understand why would we want to add a call to handle_face_prop
there.

> In gdb I saee that it goes in a very complex inf loop within the display
> engine functions and emacs becomes completely unresponsive (No C-g or
> ESC ESC ESC) the only solution is to kill it from outside.

Can you tell where it loops?  That is, describe the sequence of calls
and the return values for a single iteration through the loop?

> Could you give a look please?

Not sure what should I look at.  If you want me to run a GDB session,
I would need a recipe to reproduce the loop.  Note that I don't have
Magit installed, so loading it (if needed) would have be part of the
recipe.

Re: Question about display engine

[Eli Zaretskii]

> Date: Thu, 5 Sep 2019 19:26:09 +0000 (UTC)
> From: Ergus <spacibba@aol.com>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> 
> I attach here a new patch with all the changes I have just made. After fixing the latest Martin's issue there was
> exposed a new issue maybe related with the initialization per line. (picture attached) 

Thanks.  I admit that I'm confused by your patch: I don't understand
your design of calculating and applying the face used for EOL
extension.  E.g., where's the code that merges only the non-extensible
attributes of the face at buffer position and assigns that face ID of
the calculated face to it->extend_face_id?  And why did you copy all
the lines that assign to it->face_id with similar lines that assign
something similar (sometimes identical) to it->extend_face_id?  And
why do you have to save and restore extend_face_id during some
operations, like we do with it->face_id?  Etc. etc.

Can you post a description of the design and the implementation, to
help me find the light here?  In particular, I don't think I
understand the meaning of "the face should be extended after EOL", if
that face is merged with other faces to realize the face to be
actually used in display.  This semantics seems not to be explained
anywhere, so it's hard to judge whether the implementation satisfies
the requirements/expectations.

> The issue is actually related with the fact that extend_face_id is never restarted to face_id when going back
> from an extend to a non_extend face between lines (for example when mark is active and the iterator crosses
> the pointer to outside the selected region like in the  picture). 

I cannot answer this question because my mental model is the opposite:
that the code should temporarily set face_id to be equal to
extend_face_id when producing glyphs beyond EOL, then reset face_id
back to its previous value.  But your code doesn't fit this mental
model of mine, so I'm probably missing something.

> I made all the  updates of the extend_face_id in the same places where face_id was updated (even when
> uneeded for now.)

This is another place for confusion: I don't understand why
extend_face_id should be updated in all those places.  In my metal
model, extend_face_id is independent of many/most of the factors that
cause us update face_id.

> But I don't see any special function that is called before
> display_line.

face_id is initialized in init_iterator, which is always called once
before the first call to display_line.  Thereafter, any subsequent
call to display_line "inherits" the value of face_id left in the
iterator object at the end of the previous call to display_line.

Whether this fits the logic of using extend_face_id, I cannot say yet;
see the above questions that describe my confusion.

Thanks for working on this.

Re: Question about display engine

[Ergus]

On Fri, Sep 06, 2019 at 11:55:19AM +0300, Eli Zaretskii wrote:
>> Date: Thu, 5 Sep 2019 19:26:09 +0000 (UTC)
>> From: Ergus <spacibba@aol.com>
>> Cc: eliz@gnu.org, emacs-devel@gnu.org
>>
>> I attach here a new patch with all the changes I have just made. After fixing the latest Martin's issue there was
>> exposed a new issue maybe related with the initialization per line. (picture attached)
>
>Thanks.  I admit that I'm confused by your patch: I don't understand
>your design of calculating and applying the face used for EOL
>extension.  E.g., where's the code that merges only the non-extensible
>attributes of the face at buffer position and assigns that face ID of
>the calculated face to it->extend_face_id?

1) This is something confusing even for me. because I was doing that the
other way around as in my initial mental scheme the "extensible" part of
the face should be associated (somehow, contained or mapped) with every
extensible face || else 0 (DEFAULT_FACE_ID). So it will be easy to check
that and go forth and back at EOL.

>And why did you copy all
>the lines that assign to it->face_id with similar lines that assign
>something similar (sometimes identical) to it->extend_face_id?

2) This was a desperate try to check where I was caching something
wrong. In some cases it will be needed to do that because there are
points were the face_id is set and reset latter and it is faster to
cache also the extend_face_id instead of calculating it because there is
not conversion method. (explication at the end)

>And
>why do you have to save and restore extend_face_id during some
>operations, like we do with it->face_id?  Etc. etc.
>

3) Same than above. Hopefully this will be removed.

For some reason before Martin faced the issue he reported on yesterday,
the code was somehow working for me more or less as expected. So it
seems to be that something was not initialized and hiding the bigger
issues in my part.

>Can you post a description of the design and the implementation, to
>help me find the light here?  In particular, I don't think I
>understand the meaning of "the face should be extended after EOL", if
>that face is merged with other faces to realize the face to be
>actually used in display.  This semantics seems not to be explained
>anywhere, so it's hard to judge whether the implementation satisfies
>the requirements/expectations.
>
4) Sorry for that.

the face should be extended after EOL means (somehow) that the
attributes specified in the face are merged with the ones in other
extensible faces to extend after EOL. The conditional to merge (probably
wrong) is in merge_extend_glyph_face in xdisp.c.

>> The issue is actually related with the fact that extend_face_id is never restarted to face_id when going back
>> from an extend to a non_extend face between lines (for example when mark is active and the iterator crosses
>> the pointer to outside the selected region like in the  picture).
>
>I cannot answer this question because my mental model is the opposite:
>that the code should temporarily set face_id to be equal to
>extend_face_id when producing glyphs beyond EOL, then reset face_id
>back to its previous value.  But your code doesn't fit this mental
>model of mine, so I'm probably missing something.
>

5) This is actually what happens more or less. in exted_face_to_end... I
set the face_id = extend_face_id and then I reset it back at the end (as
usual).

The issue in my code is probably that I am not calculating the
extend_face_id at EOL correctly. (Mainly because I know I am missing
something crucial about where and how to do that.)

Actually extend_face_id; once it is set to an extensible face it only
merges forth and forth... so even after the region finishes it never
resets.... which is actually wrong (my bad)... but I don't know where
this happen for the face_id; that's why I was reassigning everywhere to
see if I could find it.

>> I made all the  updates of the extend_face_id in the same places where face_id was updated (even when
>> uneeded for now.)
>
>This is another place for confusion: I don't understand why
>extend_face_id should be updated in all those places.  In my metal
>model, extend_face_id is independent of many/most of the factors that
>cause us update face_id.
>
I know. I just couldn't find a condition... sorry for that.

>> But I don't see any special function that is called before
>> display_line.
>
>face_id is initialized in init_iterator, which is always called once
>before the first call to display_line.  Thereafter, any subsequent
>call to display_line "inherits" the value of face_id left in the
>iterator object at the end of the previous call to display_line.
>
I understood this later actually.
>
>
>Whether this fits the logic of using extend_face_id, I cannot say yet;
>see the above questions that describe my confusion.
>
It actually does... but when the it->face_id changes (for example the
region ends in the middle of a line) the extend_face_id should know. 

>Thanks for working on this.

I am actually rethinking the whole code... but I need to understand
better some details that are unclear for me. Like how to get the
"extensible" face_id from a non extensible mixed merged face. Lets say

e = (a + b + c + d) where only a and c were extensible. Because if I don't
have a cache/face I will need to recalculate that every time and find a
way to remember how a face was composed... (remember that e was composed
by a; b; c; d and then iterate over those ids, get_face_from_id and do a
loop that if EXTENSIBLE-P will merges in extend_face_id. This will be
sub efficient.

The other problems I see with this is that in general after several
merges the resulting face_id could be the same for different sequences
of a, b, c, d, f, g, r. So the relation face_id -> extend_face_id is not
even injective; as we lost information in the middle.

The simplest case: suppose that we have (h == b) but h is extensible and
b is not. they both will have different face_id because the vectors are
different.

Merging (a + b + c + d) == (a + h + c + d) -> same face id
but the extensible faces (a + c) != (a + h + c) -> different face_id

So I don't know how to face this if I want to do it at the EOL
only. Because of that I was somehow searching for a method that could
give me (a + h + c) or (a + c) on the fly every time... but this seems
to be wrong implemented; so I need MORE help here.

Re: Question about display engine

[Eli Zaretskii]

> Date: Fri, 6 Sep 2019 12:30:23 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> the face should be extended after EOL means (somehow) that the
> attributes specified in the face are merged with the ones in other
> extensible faces to extend after EOL.

So the face we use after EOL should be the result of merging only
those faces which have their :extend attribute set to non-nil, is that
right?

> >face_id is initialized in init_iterator, which is always called once
> >before the first call to display_line.  Thereafter, any subsequent
> >call to display_line "inherits" the value of face_id left in the
> >iterator object at the end of the previous call to display_line.
> >
> I understood this later actually.
> >
> >
> >Whether this fits the logic of using extend_face_id, I cannot say yet;
> >see the above questions that describe my confusion.
> >
> It actually does... but when the it->face_id changes (for example the
> region ends in the middle of a line) the extend_face_id should know. 

Should it?  The way I see it, we don't need to care about
extend_face_id until we actually come to EOL.

> I am actually rethinking the whole code... but I need to understand
> better some details that are unclear for me. Like how to get the
> "extensible" face_id from a non extensible mixed merged face. Lets say
> 
> e = (a + b + c + d) where only a and c were extensible. Because if I don't
> have a cache/face I will need to recalculate that every time and find a
> way to remember how a face was composed... (remember that e was composed
> by a; b; c; d and then iterate over those ids, get_face_from_id and do a
> loop that if EXTENSIBLE-P will merges in extend_face_id. This will be
> sub efficient.

I don't think you need to remember anything, because Emacs "remembers"
for you.  All of those faces (a, b, c, d) will still be in effect at
EOL (i.e. at the position of the newline character); all you need is
to merge them there while ignoring those of them whose :extend
attribute is not set.

IOW, I thing extend_face_id should only be computed at EOL, either in
extend_face_to_end_of_line or in append_space_for_newline.  Because
you don't need that face ID before you come to EOL.

> The simplest case: suppose that we have (h == b) but h is extensible and
> b is not. they both will have different face_id because the vectors are
> different.
> 
> Merging (a + b + c + d) == (a + h + c + d) -> same face id
> but the extensible faces (a + c) != (a + h + c) -> different face_id
> 
> So I don't know how to face this if I want to do it at the EOL
> only. Because of that I was somehow searching for a method that could
> give me (a + h + c) or (a + c) on the fly every time... but this seems
> to be wrong implemented; so I need MORE help here.

I think the solution should be to have a variant of the code in
handle_face_prop such that it computes the face at EOL.  It would do
that by modifying face_at_buffer_position and face_at_string_position
to accept an additional argument EOL_P, which means merge only faces
which have their :extend attribute set.  Then the face ID computed for
this specially merged face should be used as extend_face_id.

Does this make sense?

Re: Question about display engine

[Ergus]

On Fri, Sep 06, 2019 at 04:28:20PM +0300, Eli Zaretskii wrote:
>> Date: Fri, 6 Sep 2019 12:30:23 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> the face should be extended after EOL means (somehow) that the
>> attributes specified in the face are merged with the ones in other
>> extensible faces to extend after EOL.
>
>So the face we use after EOL should be the result of merging only
>those faces which have their :extend attribute set to non-nil, is that
>right?
>
Yes. 
>> >face_id is initialized in init_iterator, which is always called once
>> >before the first call to display_line.  Thereafter, any subsequent
>> >call to display_line "inherits" the value of face_id left in the
>> >iterator object at the end of the previous call to display_line.
>> >
>> I understood this later actually.
>> >
>> >
>> >Whether this fits the logic of using extend_face_id, I cannot say yet;
>> >see the above questions that describe my confusion.
>> >
>> It actually does... but when the it->face_id changes (for example the
>> region ends in the middle of a line) the extend_face_id should know.
>
>Should it?  The way I see it, we don't need to care about
>extend_face_id until we actually come to EOL.
>
You are right.

>> I am actually rethinking the whole code... but I need to understand
>> better some details that are unclear for me. Like how to get the
>> "extensible" face_id from a non extensible mixed merged face. Lets say
>>
>> e = (a + b + c + d) where only a and c were extensible. Because if I don't
>> have a cache/face I will need to recalculate that every time and find a
>> way to remember how a face was composed... (remember that e was composed
>> by a; b; c; d and then iterate over those ids, get_face_from_id and do a
>> loop that if EXTENSIBLE-P will merges in extend_face_id. This will be
>> sub efficient.
>
>I don't think you need to remember anything, because Emacs "remembers"
>for you.  All of those faces (a, b, c, d) will still be in effect at
>EOL (i.e. at the position of the newline character); all you need is
>to merge them there while ignoring those of them whose :extend
>attribute is not set.
>
>IOW, I thing extend_face_id should only be computed at EOL, either in
>extend_face_to_end_of_line or in append_space_for_newline.  Because
>you don't need that face ID before you come to EOL.
>
append_space_for_newline is not called in all the cases. and this has to
do with the yesterdays question about what face should have the extra
space (before extending).

>> The simplest case: suppose that we have (h == b) but h is extensible and
>> b is not. they both will have different face_id because the vectors are
>> different.
>>
>> Merging (a + b + c + d) == (a + h + c + d) -> same face id
>> but the extensible faces (a + c) != (a + h + c) -> different face_id
>>
>> So I don't know how to face this if I want to do it at the EOL
>> only. Because of that I was somehow searching for a method that could
>> give me (a + h + c) or (a + c) on the fly every time... but this seems
>> to be wrong implemented; so I need MORE help here.
>
>I think the solution should be to have a variant of the code in
>handle_face_prop such that it computes the face at EOL.  It would do
>that by modifying face_at_buffer_position and face_at_string_position
>to accept an additional argument EOL_P, which means merge only faces
>which have their :extend attribute set.  Then the face ID computed for
>this specially merged face should be used as extend_face_id.
>
>Does this make sense?
>
Probably yes but more questions :)

Lets say that I actually don't understand very well what
handle_face_prop does (when it is called and when not).

When you say a variant you mean another function to call directly from
extend_face_to_end_of_line? Sorry I still don't understand where is (a +
b + c + d) computed or where emacs "remembers" that, or if it is
computed all the time. But maybe the trick is actually in
face_at_buffer_position, face_for_overlay_string or
face_at_string_position?

If so; then what we really need is a variant of face_at_buffer_position
like extend_face_at_buffer_position? (or add to it a parameter to do
what we want) does it makes any sense.

handle_face_prop can't be modified as it should have a specific
prototype. But we can make it a wrapper and create a generalized or use
ITERATOR_AT_END_OF_LINE_P internally?

Re: Question about display engine

[Eli Zaretskii]

> Date: Fri, 6 Sep 2019 18:34:56 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> When you say a variant you mean another function to call directly from
> extend_face_to_end_of_line?

Yes.

> Sorry I still don't understand where is (a +
> b + c + d) computed or where emacs "remembers" that, or if it is
> computed all the time. But maybe the trick is actually in
> face_at_buffer_position, face_for_overlay_string or
> face_at_string_position?

face_at_buffer_position and face_at_string_position collect all the
faces that affect a given buffer/string position, and then they merge
these faces to produce the face ID to be used by the iterator.  By
"remember" I meant that if you call face_at_buffer_position for any
position between stop positions A and A+1, it will produce the same
face ID, because the face properties don't change between stop
positions.  The data structures maintained by Emacs that determine the
faces in effect for a particular position is what "remembers" these
faces for you.

> If so; then what we really need is a variant of face_at_buffer_position
> like extend_face_at_buffer_position? (or add to it a parameter to do
> what we want) does it makes any sense.
> 
> handle_face_prop can't be modified as it should have a specific
> prototype. But we can make it a wrapper and create a generalized or use
> ITERATOR_AT_END_OF_LINE_P internally?

What I meant is to write a function like handle_face_prop, but one
that instructs face_at_buffer/string_position to merge the relevant
faces in a special way, one that takes the :extend attribute into
account.  Whether the code will be similar enough to what
handle_face_prop does now to make them use the same code, is a
separate question; I wouldn't be bothered by that at this time.  First
let's have code that we understand and that works; we can bother about
cleaning up and optimizing later.

Re: Question about display engine

[Ergus]

On Fri, Sep 06, 2019 at 09:12:06PM +0300, Eli Zaretskii wrote:
>> Date: Fri, 6 Sep 2019 18:34:56 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> When you say a variant you mean another function to call directly from
>> extend_face_to_end_of_line?
>
>Yes.
>
>> Sorry I still don't understand where is (a +
>> b + c + d) computed or where emacs "remembers" that, or if it is
>> computed all the time. But maybe the trick is actually in
>> face_at_buffer_position, face_for_overlay_string or
>> face_at_string_position?
>
>face_at_buffer_position and face_at_string_position collect all the
>faces that affect a given buffer/string position, and then they merge
>these faces to produce the face ID to be used by the iterator.  By
>"remember" I meant that if you call face_at_buffer_position for any
>position between stop positions A and A+1, it will produce the same
>face ID, because the face properties don't change between stop
>positions.  The data structures maintained by Emacs that determine the
>faces in effect for a particular position is what "remembers" these
>faces for you.
>
>> If so; then what we really need is a variant of face_at_buffer_position
>> like extend_face_at_buffer_position? (or add to it a parameter to do
>> what we want) does it makes any sense.
>>
>> handle_face_prop can't be modified as it should have a specific
>> prototype. But we can make it a wrapper and create a generalized or use
>> ITERATOR_AT_END_OF_LINE_P internally?
>
>What I meant is to write a function like handle_face_prop, but one
>that instructs face_at_buffer/string_position to merge the relevant
>faces in a special way, one that takes the :extend attribute into
>account.  Whether the code will be similar enough to what
>handle_face_prop does now to make them use the same code, is a
>separate question; I wouldn't be bothered by that at this time.  First
>let's have code that we understand and that works; we can bother about
>cleaning up and optimizing later.
>

Hi Eli:

In your solution I am facing an issue that I am not sure how to solve:

I added a function handle_face_prop_general; that I call in
extend_face_to_end_of_line but I get an inf loop because in the call
stack I have:

extend_face_to_end_of_line
handle_face_prop_general
face_at_buffer_position
Fget_text_property
Ftext_properties_at
validate_interval_range

That has a condition:

if (!(BUF_BEGV (b) <= XFIXNUM (*begin)
     && XFIXNUM (*begin) <= XFIXNUM (*end)
     && XFIXNUM (*end) <= BUF_ZV (b)))
	args_out_of_range (*begin, *end);

And for some reason:

XFIXNUM (*end) <= BUF_ZV (b) is false; so the function args_out_of_range
emits a signal and never returns.

It is possible that in the first calls to extend_face_to_end_of_line the
BUF_ZV (b) is not initialized yet? Because it is always 1. when I print
it.

Re: Question about display engine

[Eli Zaretskii]

> Date: Sat, 7 Sep 2019 04:35:44 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> In your solution I am facing an issue that I am not sure how to solve:
> 
> I added a function handle_face_prop_general; that I call in
> extend_face_to_end_of_line but I get an inf loop because in the call
> stack I have:
> 
> extend_face_to_end_of_line
> handle_face_prop_general
> face_at_buffer_position
> Fget_text_property
> Ftext_properties_at
> validate_interval_range
> 
> That has a condition:
> 
> if (!(BUF_BEGV (b) <= XFIXNUM (*begin)
>      && XFIXNUM (*begin) <= XFIXNUM (*end)
>      && XFIXNUM (*end) <= BUF_ZV (b)))
> 	args_out_of_range (*begin, *end);
> 
> And for some reason:
> 
> XFIXNUM (*end) <= BUF_ZV (b) is false; so the function args_out_of_range
> emits a signal and never returns.

You need to understand how this happens.  What are the values of
*begin and *end in this case, and what is the position of the iterator
you pass to handle_face_prop_general?

> It is possible that in the first calls to extend_face_to_end_of_line the
> BUF_ZV (b) is not initialized yet?

No.  But please tell more about "the first call to
extend_face_to_end_of_line".  Which code calls it, and for what
buffer?

> Because it is always 1. when I print it.

If BUF_ZV is 1, it's an empty buffer.

What is it->w->contents in your call?  It should be the buffer whose
text you are displaying.

Re: Question about display engine

[Ergus]

Hi Eli:

On Sun, Oct 13, 2019 at 12:46:10PM +0300, Eli Zaretskii wrote:
>> Date: Sun, 13 Oct 2019 00:23:05 +0200
>> From: Ergus <spacibba@aol.com>
>>
>> >I don't understand why would we want to add a call to handle_face_prop
>> >there.
>> >
>>
>> The call there is to emulate the part in our branch that produced the
>> issue. And isolate the source of the problem, independently of all the
>> other changes.
>>
>> Remember we modified handle_face_prop to be called here with the extra
>> parameter to filter.
>
>But handle_face_prop modifies some members of its IT argument, whereas
>the function called from extend_face_to_end_of_line (face_at_pos)
>should not do that, it should just return the face ID to use.
>
>I now see that face_at_pos modifies it->start_of_box_run_p and
>it->face_box_p.  This is wrong, you should do that outside of
>face_at_pos, in handle_face_prop itself.  Maybe this is the reason for
>the infloop?  If extend_face_to_end_of_line needs to manipulate these
>members (does it?), it needs to save and restore the old values.
>
Fixed now, but that's not the solution for the issue.

>> >Can you tell where it loops?  That is, describe the sequence of calls
>> >and the return values for a single iteration through the loop?
>>
>> It is a very long loop it jumps here and there in the code it is very
>> difficult to explain. But basically it stays going and coming in
>> display_line for ever.
>
>You don't have to explain it, just show me one iteration through the
>loop as you step through the code in GDB.
>
In GDB I have this bt:

#0  0x000055cb1453ac98 in redisplay_windows (window=0x55cb15e3bfb5) at ../../src/xdisp.c:16126
#1  0x000055cb1453ac6d in redisplay_windows (window=0x55cb163ffc55) at ../../src/xdisp.c:16120
#2  0x000055cb1455b35d in redisplay_internal () at ../../src/xdisp.c:15596
#3  0x000055cb145fff3f in read_char (commandflag=1, map=0x55cb16838f93, prev_event=0x0, used_mouse_menu=0x7ffc1a89f4eb, end_time=0x0) at ../../src/keyboard.c:2473
#4  0x000055cb1460278a in read_key_sequence
    (keybuf=<optimized out>, prompt=0x0, dont_downcase_last=<optimized out>, can_return_switch_frame=true, fix_current_buffer=true, prevent_redisplay=<optimized out>)
    at ../../src/keyboard.c:9527
#5  0x000055cb14603e2c in command_loop_1 () at ../../src/lisp.h:1032
#6  0x000055cb1466af87 in internal_condition_case
    (bfun=bfun@entry=0x55cb14603c30 <command_loop_1>, handlers=handlers@entry=0x90, hfun=hfun@entry=0x55cb145fadc0 <cmd_error>) at ../../src/eval.c:1355
#7  0x000055cb145f5b94 in command_loop_2 (ignore=ignore@entry=0x0) at ../../src/lisp.h:1032
#8  0x000055cb1466aee1 in internal_catch (tag=tag@entry=0xd4a0, func=func@entry=0x55cb145f5b70 <command_loop_2>, arg=arg@entry=0x0) at ../../src/eval.c:1116
#9  0x000055cb145f5b3b in command_loop () at ../../src/lisp.h:1032
#10 0x000055cb145fa9d6 in recursive_edit_1 () at ../../src/keyboard.c:714
#11 0x000055cb145fad02 in Frecursive_edit () at ../../src/keyboard.c:786
#12 0x000055cb1451c957 in main (argc=18, argv=<optimized out>) at ../../src/emacs.c:2055

After some other tests I just did; I found that:

#0 seems to be the root of the loop. redisplay_windows never ends (inf
 loop) and I can't understand why. But at least this explains why in gui
 it works but not in tui, because there is the WINDOWP test.

What I can't understand is how the code can be in #1 that should always
filtered by the WINDOWP condition in tui right?

In any case the inf loop is there, but the recursions levels does not
grow... so after the first time it enters in #1, it goes in the other
branch if the if.

On the other hand I don't understand how is this related with the call
of face_at_pos in the extend_face_to_end_of_line. Any idea?

>> Maybe you look at it and you find the issue in 5 seconds, but there
>> is still too much I ignore to get it.
>
>Unlikely.  And it is not wise to lose all the information you have
>already collected about this problem, it could help me quite a lot.
>At the very least please show a backtrace from inside the infloop, and
>tell whether we are iterating over a buffer or a string, and if the
>latter, what kind of string is that (overlay string, display string?).
>
>> Magit is available in melpa. Installing and using it is trivial.
>
>I don't need to install it, I just need to load it.
>
Then the command I send before should be enough.

>> Maybe Martin have something to say about this?
>
>You didn't CC Martin on this message.

My bad

Re: Question about display engine

[Eli Zaretskii]

> Date: Sun, 13 Oct 2019 17:40:52 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> #0  0x000055cb1453ac98 in redisplay_windows (window=0x55cb15e3bfb5) at ../../src/xdisp.c:16126
> #1  0x000055cb1453ac6d in redisplay_windows (window=0x55cb163ffc55) at ../../src/xdisp.c:16120
> #2  0x000055cb1455b35d in redisplay_internal () at ../../src/xdisp.c:15596
> #3  0x000055cb145fff3f in read_char (commandflag=1, map=0x55cb16838f93, prev_event=0x0, used_mouse_menu=0x7ffc1a89f4eb, end_time=0x0) at ../../src/keyboard.c:2473
> #4  0x000055cb1460278a in read_key_sequence
>     (keybuf=<optimized out>, prompt=0x0, dont_downcase_last=<optimized out>, can_return_switch_frame=true, fix_current_buffer=true, prevent_redisplay=<optimized out>)
>     at ../../src/keyboard.c:9527
> #5  0x000055cb14603e2c in command_loop_1 () at ../../src/lisp.h:1032
> #6  0x000055cb1466af87 in internal_condition_case
>     (bfun=bfun@entry=0x55cb14603c30 <command_loop_1>, handlers=handlers@entry=0x90, hfun=hfun@entry=0x55cb145fadc0 <cmd_error>) at ../../src/eval.c:1355
> #7  0x000055cb145f5b94 in command_loop_2 (ignore=ignore@entry=0x0) at ../../src/lisp.h:1032
> #8  0x000055cb1466aee1 in internal_catch (tag=tag@entry=0xd4a0, func=func@entry=0x55cb145f5b70 <command_loop_2>, arg=arg@entry=0x0) at ../../src/eval.c:1116
> #9  0x000055cb145f5b3b in command_loop () at ../../src/lisp.h:1032
> #10 0x000055cb145fa9d6 in recursive_edit_1 () at ../../src/keyboard.c:714
> #11 0x000055cb145fad02 in Frecursive_edit () at ../../src/keyboard.c:786
> #12 0x000055cb1451c957 in main (argc=18, argv=<optimized out>) at ../../src/emacs.c:2055
> 
> After some other tests I just did; I found that:
> 
> #0 seems to be the root of the loop. redisplay_windows never ends (inf
>  loop) and I can't understand why. But at least this explains why in gui
>  it works but not in tui, because there is the WINDOWP test.

The WINDOWP test has nothing to do with GUI vs TTY, it tests whether
w->contents is a window or a buffer.

> What I can't understand is how the code can be in #1 that should always
> filtered by the WINDOWP condition in tui right?

No, see above.

This function is a simple depth-first tree traversal of the window
tree, starting from the root window of a frame.  Each leaf of the
window tree has a buffer in its w->contents pointer, while
intermediate nodes of the tree have windows in that pointer.

> In any case the inf loop is there, but the recursions levels does not
> grow... so after the first time it enters in #1, it goes in the other
> branch if the if.

If it goes to the other branch, it should descend the tree via the
w->next pointer, and eventually get to a leaf node.

Could it be that redisplay_window_0, or some function it calls,
signals an error, which is caught by internal_condition_case_1?  What
happens if you put a breakpoint in signal_or_quit, does it get called
from redisplay_window or some other function called by
redisplay_windows?

Re: Question about display engine

[Ergus]

On Sun, Oct 13, 2019 at 07:06:18PM +0300, Eli Zaretskii wrote:
>> Date: Sun, 13 Oct 2019 17:40:52 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> #0  0x000055cb1453ac98 in redisplay_windows (window=0x55cb15e3bfb5) at ../../src/xdisp.c:16126
>> #1  0x000055cb1453ac6d in redisplay_windows (window=0x55cb163ffc55) at ../../src/xdisp.c:16120
>> #2  0x000055cb1455b35d in redisplay_internal () at ../../src/xdisp.c:15596
>> #3  0x000055cb145fff3f in read_char (commandflag=1, map=0x55cb16838f93, prev_event=0x0, used_mouse_menu=0x7ffc1a89f4eb, end_time=0x0) at ../../src/keyboard.c:2473
>> #4  0x000055cb1460278a in read_key_sequence
>>     (keybuf=<optimized out>, prompt=0x0, dont_downcase_last=<optimized out>, can_return_switch_frame=true, fix_current_buffer=true, prevent_redisplay=<optimized out>)
>>     at ../../src/keyboard.c:9527
>> #5  0x000055cb14603e2c in command_loop_1 () at ../../src/lisp.h:1032
>> #6  0x000055cb1466af87 in internal_condition_case
>>     (bfun=bfun@entry=0x55cb14603c30 <command_loop_1>, handlers=handlers@entry=0x90, hfun=hfun@entry=0x55cb145fadc0 <cmd_error>) at ../../src/eval.c:1355
>> #7  0x000055cb145f5b94 in command_loop_2 (ignore=ignore@entry=0x0) at ../../src/lisp.h:1032
>> #8  0x000055cb1466aee1 in internal_catch (tag=tag@entry=0xd4a0, func=func@entry=0x55cb145f5b70 <command_loop_2>, arg=arg@entry=0x0) at ../../src/eval.c:1116
>> #9  0x000055cb145f5b3b in command_loop () at ../../src/lisp.h:1032
>> #10 0x000055cb145fa9d6 in recursive_edit_1 () at ../../src/keyboard.c:714
>> #11 0x000055cb145fad02 in Frecursive_edit () at ../../src/keyboard.c:786
>> #12 0x000055cb1451c957 in main (argc=18, argv=<optimized out>) at ../../src/emacs.c:2055
>>
>> After some other tests I just did; I found that:
>>
>> #0 seems to be the root of the loop. redisplay_windows never ends (inf
>>  loop) and I can't understand why. But at least this explains why in gui
>>  it works but not in tui, because there is the WINDOWP test.
>
>The WINDOWP test has nothing to do with GUI vs TTY, it tests whether
>w->contents is a window or a buffer.
>
>> What I can't understand is how the code can be in #1 that should always
>> filtered by the WINDOWP condition in tui right?
>
>No, see above.
>
>This function is a simple depth-first tree traversal of the window
>tree, starting from the root window of a frame.  Each leaf of the
>window tree has a buffer in its w->contents pointer, while
>intermediate nodes of the tree have windows in that pointer.
>
>> In any case the inf loop is there, but the recursions levels does not
>> grow... so after the first time it enters in #1, it goes in the other
>> branch if the if.
>
>If it goes to the other branch, it should descend the tree via the
>w->next pointer, and eventually get to a leaf node.
>
>Could it be that redisplay_window_0, or some function it calls,
>signals an error, which is caught by internal_condition_case_1?  What
>happens if you put a breakpoint in signal_or_quit, does it get called
>from redisplay_window or some other function called by
>redisplay_windows?

Yes, actually:

#0  0x00005597732a0380 in signal_or_quit (error_symbol=0x2cd0, data=0x559775ee2633, keyboard_quit=false) at ../../src/eval.c:1586
#1  0x000055977314c308 in Fsignal (error_symbol=<optimized out>, error_symbol@entry=0x2cd0, data=<optimized out>) at ../../src/eval.c:1568
#2  0x000055977314c4c9 in xsignal (data=<optimized out>, error_symbol=0x2cd0) at ../../src/lisp.h:4139
#3  0x000055977314c4c9 in xsignal2 (error_symbol=error_symbol@entry=0x2cd0, arg1=<optimized out>, arg2=<optimized out>) at ../../src/eval.c:1713
#4  0x000055977314b76e in args_out_of_range (a1=<optimized out>, a2=<optimized out>) at ../../src/lisp.h:1032
#5  0x000055977314e97b in validate_interval_range (object=0x559775cee0d5, begin=0x7fff35ec34b8, end=0x7fff35ec34b8, force=<optimized out>) at ../../src/textprop.c:158
#6  0x00005597732f4050 in Ftext_properties_at (position=<optimized out>, object=<optimized out>) at ../../src/textprop.c:572
#7  0x00005597732f40bc in Fget_text_property (position=<optimized out>, prop=0x5d30, object=<optimized out>) at ../../src/textprop.c:592
#8  0x00005597731f8eec in face_at_buffer_position
    (w=0x5597758e7620, pos=0, endptr=endptr@entry=0x7fff35ec3650, limit=100, mouse=mouse@entry=false, base_face_id=1, attr_filter=LFACE_EXTEND_INDEX)
    at ../../src/xfaces.c:6090
#9  0x000055977316ab25 in face_at_pos (it=0x7fff35ec3710, attr_filter=LFACE_EXTEND_INDEX) at ../../src/xdisp.c:4167
#10 0x000055977317083d in extend_face_to_end_of_line (it=0x7fff35ec3710) at ../../src/xdisp.c:21584
#11 0x0000559773185891 in display_mode_line (w=w@entry=0x5597758e7620, face_id=<optimized out>, format=0x7efd5d2d9ed3) at ../../src/xdisp.c:24990
#12 0x0000559773185afe in display_mode_lines (w=w@entry=0x5597758e7620) at ../../src/lisp.h:730
#13 0x000055977319ebbc in redisplay_window (window=0x5597758e7625, just_this_one_p=<optimized out>) at ../../src/xdisp.c:18803
#14 0x00005597731a327b in redisplay_window_0 (window=window@entry=0x5597758e7625) at ../../src/xdisp.c:16146
#15 0x000055977329f014 in internal_condition_case_1
    (bfun=bfun@entry=0x5597731a3250 <redisplay_window_0>, arg=arg@entry=0x5597758e7625, handlers=<optimized out>, hfun=hfun@entry=0x559773165800 <redisplay_window_error>) at ../../src/eval.c:1379
#16 0x000055977316ec98 in redisplay_windows (window=0x5597758e7625) at ../../src/xdisp.c:16126
#17 0x000055977316ec6d in redisplay_windows (window=0x5597758e7415) at ../../src/xdisp.c:16120
#18 0x000055977318f35d in redisplay_internal () at ../../src/xdisp.c:15596
#19 0x0000559773233f3f in read_char (commandflag=1, map=0x559775ee36d3, prev_event=0x0, used_mouse_menu=0x7fff35ec8cbb, end_time=0x0) at ../../src/keyboard.c:2473
#20 0x000055977323678a in read_key_sequence
    (keybuf=<optimized out>, prompt=0x0, dont_downcase_last=<optimized out>, can_return_switch_frame=true, fix_current_buffer=true, prevent_redisplay=<optimized out>)
    at ../../src/keyboard.c:9527
#21 0x0000559773237e2c in command_loop_1 () at ../../src/lisp.h:1032
#22 0x000055977329ef87 in internal_condition_case
    (bfun=bfun@entry=0x559773237c30 <command_loop_1>, handlers=handlers@entry=0x90, hfun=hfun@entry=0x55977322edc0 <cmd_error>) at ../../src/eval.c:1355
#23 0x0000559773229b94 in command_loop_2 (ignore=ignore@entry=0x0) at ../../src/lisp.h:1032
#24 0x000055977329eee1 in internal_catch (tag=tag@entry=0xd4a0, func=func@entry=0x559773229b70 <command_loop_2>, arg=arg@entry=0x0) at ../../src/eval.c:1116
#25 0x0000559773229b3b in command_loop () at ../../src/lisp.h:1032
#26 0x000055977322e9d6 in recursive_edit_1 () at ../../src/keyboard.c:714
#27 0x000055977322ed02 in Frecursive_edit () at ../../src/keyboard.c:786
#28 0x0000559773150957 in main (argc=18, argv=<optimized out>) at #../../src/emacs.c:2055

This makes more sense now.

Re: Question about display engine

[Eli Zaretskii]

> Date: Sun, 13 Oct 2019 18:44:24 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >Could it be that redisplay_window_0, or some function it calls,
> >signals an error, which is caught by internal_condition_case_1?  What
> >happens if you put a breakpoint in signal_or_quit, does it get called
> >from redisplay_window or some other function called by
> >redisplay_windows?
> 
> Yes, actually:
> 
> #0  0x00005597732a0380 in signal_or_quit (error_symbol=0x2cd0, data=0x559775ee2633, keyboard_quit=false) at ../../src/eval.c:1586
> #1  0x000055977314c308 in Fsignal (error_symbol=<optimized out>, error_symbol@entry=0x2cd0, data=<optimized out>) at ../../src/eval.c:1568
> #2  0x000055977314c4c9 in xsignal (data=<optimized out>, error_symbol=0x2cd0) at ../../src/lisp.h:4139
> #3  0x000055977314c4c9 in xsignal2 (error_symbol=error_symbol@entry=0x2cd0, arg1=<optimized out>, arg2=<optimized out>) at ../../src/eval.c:1713
> #4  0x000055977314b76e in args_out_of_range (a1=<optimized out>, a2=<optimized out>) at ../../src/lisp.h:1032
> #5  0x000055977314e97b in validate_interval_range (object=0x559775cee0d5, begin=0x7fff35ec34b8, end=0x7fff35ec34b8, force=<optimized out>) at ../../src/textprop.c:158
> #6  0x00005597732f4050 in Ftext_properties_at (position=<optimized out>, object=<optimized out>) at ../../src/textprop.c:572
> #7  0x00005597732f40bc in Fget_text_property (position=<optimized out>, prop=0x5d30, object=<optimized out>) at ../../src/textprop.c:592
> #8  0x00005597731f8eec in face_at_buffer_position
>     (w=0x5597758e7620, pos=0, endptr=endptr@entry=0x7fff35ec3650, limit=100, mouse=mouse@entry=false, base_face_id=1, attr_filter=LFACE_EXTEND_INDEX)
>     at ../../src/xfaces.c:6090

Then looking at the position that causes the error will probably tell
you what's wrong.  (Is OBJECT passed to Fget_text_property a string?
if not, position of zero is invalid.)

Re: Question about display engine

[Ergus]

On Sun, Oct 13, 2019 at 08:04:02PM +0300, Eli Zaretskii wrote:
>> Date: Sun, 13 Oct 2019 18:44:24 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >Could it be that redisplay_window_0, or some function it calls,
>> >signals an error, which is caught by internal_condition_case_1?  What
>> >happens if you put a breakpoint in signal_or_quit, does it get called
>> >from redisplay_window or some other function called by
>> >redisplay_windows?
>>
>> Yes, actually:
>>
>> #0  0x00005597732a0380 in signal_or_quit (error_symbol=0x2cd0, data=0x559775ee2633, keyboard_quit=false) at ../../src/eval.c:1586
>> #1  0x000055977314c308 in Fsignal (error_symbol=<optimized out>, error_symbol@entry=0x2cd0, data=<optimized out>) at ../../src/eval.c:1568
>> #2  0x000055977314c4c9 in xsignal (data=<optimized out>, error_symbol=0x2cd0) at ../../src/lisp.h:4139
>> #3  0x000055977314c4c9 in xsignal2 (error_symbol=error_symbol@entry=0x2cd0, arg1=<optimized out>, arg2=<optimized out>) at ../../src/eval.c:1713
>> #4  0x000055977314b76e in args_out_of_range (a1=<optimized out>, a2=<optimized out>) at ../../src/lisp.h:1032
>> #5  0x000055977314e97b in validate_interval_range (object=0x559775cee0d5, begin=0x7fff35ec34b8, end=0x7fff35ec34b8, force=<optimized out>) at ../../src/textprop.c:158
>> #6  0x00005597732f4050 in Ftext_properties_at (position=<optimized out>, object=<optimized out>) at ../../src/textprop.c:572
>> #7  0x00005597732f40bc in Fget_text_property (position=<optimized out>, prop=0x5d30, object=<optimized out>) at ../../src/textprop.c:592
>> #8  0x00005597731f8eec in face_at_buffer_position
>>     (w=0x5597758e7620, pos=0, endptr=endptr@entry=0x7fff35ec3650, limit=100, mouse=mouse@entry=false, base_face_id=1, attr_filter=LFACE_EXTEND_INDEX)
>>     at ../../src/xfaces.c:6090
>
>Then looking at the position that causes the error will probably tell
>you what's wrong.  (Is OBJECT passed to Fget_text_property a string?
>if not, position of zero is invalid.)
>

That's exactly the problem. IT_CHARPOS (*it), is returning zero in
face_at_pos and !STRINGP (it->string). Could you tell me whats going on
please. 

Re: Question about display engine

[Ergus]

On Sun, Oct 13, 2019 at 08:04:02PM +0300, Eli Zaretskii wrote:
>> Date: Sun, 13 Oct 2019 18:44:24 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >Could it be that redisplay_window_0, or some function it calls,
>> >signals an error, which is caught by internal_condition_case_1?  What
>> >happens if you put a breakpoint in signal_or_quit, does it get called
>> >from redisplay_window or some other function called by
>> >redisplay_windows?
>>
>> Yes, actually:
>>
>> #0  0x00005597732a0380 in signal_or_quit (error_symbol=0x2cd0, data=0x559775ee2633, keyboard_quit=false) at ../../src/eval.c:1586
>> #1  0x000055977314c308 in Fsignal (error_symbol=<optimized out>, error_symbol@entry=0x2cd0, data=<optimized out>) at ../../src/eval.c:1568
>> #2  0x000055977314c4c9 in xsignal (data=<optimized out>, error_symbol=0x2cd0) at ../../src/lisp.h:4139
>> #3  0x000055977314c4c9 in xsignal2 (error_symbol=error_symbol@entry=0x2cd0, arg1=<optimized out>, arg2=<optimized out>) at ../../src/eval.c:1713
>> #4  0x000055977314b76e in args_out_of_range (a1=<optimized out>, a2=<optimized out>) at ../../src/lisp.h:1032
>> #5  0x000055977314e97b in validate_interval_range (object=0x559775cee0d5, begin=0x7fff35ec34b8, end=0x7fff35ec34b8, force=<optimized out>) at ../../src/textprop.c:158
>> #6  0x00005597732f4050 in Ftext_properties_at (position=<optimized out>, object=<optimized out>) at ../../src/textprop.c:572
>> #7  0x00005597732f40bc in Fget_text_property (position=<optimized out>, prop=0x5d30, object=<optimized out>) at ../../src/textprop.c:592
>> #8  0x00005597731f8eec in face_at_buffer_position
>>     (w=0x5597758e7620, pos=0, endptr=endptr@entry=0x7fff35ec3650, limit=100, mouse=mouse@entry=false, base_face_id=1, attr_filter=LFACE_EXTEND_INDEX)
>>     at ../../src/xfaces.c:6090
>
>Then looking at the position that causes the error will probably tell
>you what's wrong.  (Is OBJECT passed to Fget_text_property a string?
>if not, position of zero is invalid.)
>

Actually conditioning the call to face_at_pos to when IT_CHARPOS (*it)
!= 0 seems to fix the issue with magit... but maybe we are just hiding
something under the carpet here. Can you imagine something more general
than that just this condition?

Or what could be doing magit to expose this and not the other packages?

Re: Question about display engine

[Eli Zaretskii]

> Date: Sun, 13 Oct 2019 20:25:42 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> >Then looking at the position that causes the error will probably tell
> >you what's wrong.  (Is OBJECT passed to Fget_text_property a string?
> >if not, position of zero is invalid.)
> >
> 
> Actually conditioning the call to face_at_pos to when IT_CHARPOS (*it)
> != 0 seems to fix the issue with magit... but maybe we are just hiding
> something under the carpet here. Can you imagine something more general
> than that just this condition?

How did it happen that IT_CHARPOS(*it) is zero?  If we are iterating
over a buffer, that cannot happen, because we begin from 1 and go
forward.

Is it->sp zero or higher?  If it's higher, then we are not iterating
over a buffer, but something else (a string, an image, or something
similar).  What are the values of it->method and it->what?  Can you
show the result of

  (gdb) pgrowx it->glyph_row

Re: Question about display engine

[Ergus]

On Sun, Oct 13, 2019 at 09:53:13PM +0300, Eli Zaretskii wrote:
>> Date: Sun, 13 Oct 2019 20:25:42 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >Then looking at the position that causes the error will probably tell
>> >you what's wrong.  (Is OBJECT passed to Fget_text_property a string?
>> >if not, position of zero is invalid.)
>> >
>>
>> Actually conditioning the call to face_at_pos to when IT_CHARPOS (*it)
>> != 0 seems to fix the issue with magit... but maybe we are just hiding
>> something under the carpet here. Can you imagine something more general
>> than that just this condition?
>
>How did it happen that IT_CHARPOS(*it) is zero?  If we are iterating
>over a buffer, that cannot happen, because we begin from 1 and go
>forward.
>
>Is it->sp zero or higher?  If it's higher, then we are not iterating
>over a buffer, but something else (a string, an image, or something
>similar).  What are the values of it->method and it->what?  Can you
>show the result of
>
>  (gdb) pgrowx it->glyph_row
>

(gdb) p it->sp
$1 = 0
(gdb) p it->method
$2 = GET_FROM_C_STRING
(gdb) p it->what
$3 = IT_CHARACTER

(gdb) pgrowx it->glyph_row                                                            
TEXT: 85 glyphs
  0    0: CHAR[-] str=0x4fab1a8f[0] blev=0,btyp=L w=1 a+d=0+0 face=1
  1    1: CHAR[U] str=0x4fab18f0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
  2    2: CHAR[U] str=0x4fab18f0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
  3    3: CHAR[U] str=0x4fab18f0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
  4    4: CHAR[:] str=0xf2787f0[0] blev=0,btyp=L w=1 a+d=0+0 face=1
  5    5: CHAR[%] str=0x4faaee54[1] blev=0,btyp=L w=1 a+d=0+0 face=1
  6    6: CHAR[%] str=0x4faaede4[1] blev=0,btyp=L w=1 a+d=0+0 face=1
  7    7: CHAR[-] str=0x4fab1721[1] blev=0,btyp=L w=1 a+d=0+0 face=1
  8    8: CHAR[-] str=0x4fa9d9a8[0] blev=0,btyp=L w=1 a+d=0+0 face=1
  9    9: CHAR[F] str=0x4fa9d9a8[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 10   10: CHAR[1] str=0x4fa9d9a8[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 11   11: CHAR[ ] str=0x4fa9d9a8[3] blev=0,btyp=L w=1 a+d=0+0 face=1
 12   12: CHAR[ ] str=0x4fa9d9a8[4] blev=0,btyp=L w=1 a+d=0+0 face=1
 13   13: CHAR[m] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 14   14: CHAR[a] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 15   15: CHAR[g] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 16   16: CHAR[i] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 17   17: CHAR[t] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 18   18: CHAR[-] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 19   19: CHAR[l] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 20   20: CHAR[o] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 21   21: CHAR[g] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 22   22: CHAR[:] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 23   23: CHAR[ ] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 24   24: CHAR[e] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 25   25: CHAR[m] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 26   26: CHAR[a] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 27   27: CHAR[c] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 28   28: CHAR[s] str=0x4fab16fc[1] blev=0,btyp=L w=1 a+d=0+0 face=20 MB
 29   29: CHAR[ ] str=0x4fab16f8[0] blev=0,btyp=L w=1 a+d=0+0 face=1
 30   30: CHAR[ ] str=0x4fab16f8[1] blev=0,btyp=L w=1 a+d=0+0 face=1
 31   31: CHAR[ ] str=0x4fab16f8[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 32   32: CHAR[T] str=0xf278800[1] blev=0,btyp=L w=1 a+d=0+0 face=1
 33   33: CHAR[o] str=0xf278800[1] blev=0,btyp=L w=1 a+d=0+0 face=1
 34   34: CHAR[p] str=0xf278800[1] blev=0,btyp=L w=1 a+d=0+0 face=1
 35   35: CHAR[ ] str=0x4fa602e6[0] blev=0,btyp=L w=1 a+d=0+0 face=1
 36   36: CHAR[L] str=0x4fa602e6[1] blev=0,btyp=L w=1 a+d=0+0 face=1
 37   37: CHAR[1] str=0x4fa602e6[3] blev=0,btyp=L w=1 a+d=0+0 face=1
 38   38: CHAR[ ] pos=-1 blev=0,btyp=B w=1 a+d=0+0 face=1
 39   39: CHAR[ ] pos=-1 blev=0,btyp=B w=1 a+d=0+0 face=1
 40   40: CHAR[ ] pos=-1 blev=0,btyp=B w=1 a+d=0+0 face=1
 41   41: CHAR[ ] str=0x4fab16f5[0] blev=0,btyp=L w=1 a+d=0+0 face=1
 42   42: CHAR[ ] str=0x4fab16f5[1] blev=0,btyp=L w=1 a+d=0+0 face=1
 43   43: CHAR[(] str=0x4fab16e0[0] blev=0,btyp=L w=1 a+d=0+0 face=1
 44   44: CHAR[M] str=0xf95dcc0[0] blev=0,btyp=L w=1 a+d=0+0 face=1
 45   45: CHAR[a] str=0xf95dcc0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
 46   46: CHAR[g] str=0xf95dcc0[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 47   47: CHAR[i] str=0xf95dcc0[3] blev=0,btyp=L w=1 a+d=0+0 face=1
 48   48: CHAR[t] str=0xf95dcc0[4] blev=0,btyp=L w=1 a+d=0+0 face=1
 49   49: CHAR[ ] str=0xf95dcc0[5] blev=0,btyp=L w=1 a+d=0+0 face=1
 50   50: CHAR[L] str=0xf95dcc0[6] blev=0,btyp=L w=1 a+d=0+0 face=1
 51   51: CHAR[o] str=0xf95dcc0[7] blev=0,btyp=L w=1 a+d=0+0 face=1
 52   52: CHAR[g] str=0xf95dcc0[8] blev=0,btyp=L w=1 a+d=0+0 face=1
 53   53: CHAR[)] str=0x4fa729d9[0] blev=0,btyp=L w=1 a+d=0+0 face=1
 54   54: CHAR[ ] str=0x4fa729ae[0] blev=0,btyp=L w=1 a+d=0+0 face=1
 55   55: CHAR[-] str=0x4fa71e84[0] blev=0,btyp=L w=1 a+d=0+0 face=1
 56   56: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 57   57: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 58   58: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 59   59: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 60   60: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 61   61: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 62   62: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 63   63: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 64   64: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 65   65: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 66   66: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 67   67: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 68   68: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 69   69: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 70   70: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 71   71: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 72   72: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 73   73: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 74   74: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 75   75: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 76   76: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 77   77: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 78   78: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 79   79: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 80   80: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 81   81: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 82   82: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 83   83: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1
 84   84: CHAR[-] str=0x4fa71e84[2] blev=0,btyp=L w=1 a+d=0+0 face=1

Re: Question about display engine

[Eli Zaretskii]

> Date: Sun, 13 Oct 2019 21:38:36 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> (gdb) p it->sp
> $1 = 0
> (gdb) p it->method
> $2 = GET_FROM_C_STRING

If we are producing glyphs from a C string, then faces should not be
used at all, because C strings cannot have faces.

So you should to condition the call to face_at_pos on something like

  it->s == NULL

because there can not be any face on any position of a C string.

> (gdb) pgrowx it->glyph_row                                                            
> TEXT: 85 glyphs
>   0    0: CHAR[-] str=0x4fab1a8f[0] blev=0,btyp=L w=1 a+d=0+0 face=1
>   1    1: CHAR[U] str=0x4fab18f0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>   2    2: CHAR[U] str=0x4fab18f0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>   3    3: CHAR[U] str=0x4fab18f0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>   4    4: CHAR[:] str=0xf2787f0[0] blev=0,btyp=L w=1 a+d=0+0 face=1
>   5    5: CHAR[%] str=0x4faaee54[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>   6    6: CHAR[%] str=0x4faaede4[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>   7    7: CHAR[-] str=0x4fab1721[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>   8    8: CHAR[-] str=0x4fa9d9a8[0] blev=0,btyp=L w=1 a+d=0+0 face=1
>   9    9: CHAR[F] str=0x4fa9d9a8[2] blev=0,btyp=L w=1 a+d=0+0 face=1
>  10   10: CHAR[1] str=0x4fa9d9a8[2] blev=0,btyp=L w=1 a+d=0+0 face=1

This is a mode line, so it figures out: extend_face_to_end_of_line was
called when the iterator was processing the final blanks of the mode
line, see display_mode_line.

Re: Question about display engine

[Ergus]

On Mon, Oct 14, 2019 at 12:01:46AM +0300, Eli Zaretskii wrote:
>> Date: Sun, 13 Oct 2019 21:38:36 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> (gdb) p it->sp
>> $1 = 0
>> (gdb) p it->method
>> $2 = GET_FROM_C_STRING
>
>If we are producing glyphs from a C string, then faces should not be
>used at all, because C strings cannot have faces.
>
>So you should to condition the call to face_at_pos on something like
>
>  it->s == NULL
>
Hi:

This have fixed the issue, I understand now what was happening. Should I
merge into master now? maybe you should close the related issues then
right?.


>because there can not be any face on any position of a C string.
>
>> (gdb) pgrowx it->glyph_row
>> TEXT: 85 glyphs
>>   0    0: CHAR[-] str=0x4fab1a8f[0] blev=0,btyp=L w=1 a+d=0+0 face=1
>>   1    1: CHAR[U] str=0x4fab18f0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>>   2    2: CHAR[U] str=0x4fab18f0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>>   3    3: CHAR[U] str=0x4fab18f0[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>>   4    4: CHAR[:] str=0xf2787f0[0] blev=0,btyp=L w=1 a+d=0+0 face=1
>>   5    5: CHAR[%] str=0x4faaee54[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>>   6    6: CHAR[%] str=0x4faaede4[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>>   7    7: CHAR[-] str=0x4fab1721[1] blev=0,btyp=L w=1 a+d=0+0 face=1
>>   8    8: CHAR[-] str=0x4fa9d9a8[0] blev=0,btyp=L w=1 a+d=0+0 face=1
>>   9    9: CHAR[F] str=0x4fa9d9a8[2] blev=0,btyp=L w=1 a+d=0+0 face=1
>>  10   10: CHAR[1] str=0x4fa9d9a8[2] blev=0,btyp=L w=1 a+d=0+0 face=1
>
>This is a mode line, so it figures out: extend_face_to_end_of_line was
>called when the iterator was processing the final blanks of the mode
>line, see display_mode_line.
>

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 14 Oct 2019 00:27:52 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> This have fixed the issue, I understand now what was happening. Should I
> merge into master now?

If there are no more outstanding issues, please go ahead and merge.

> maybe you should close the related issues then right?.

Which issues?  If you are talking about bug reports on debbugs, feel
free to close them once you merge the branch.

Thanks.

Re: Question about display engine

[Ergus]

On Mon, Oct 14, 2019 at 11:26:03AM +0300, Eli Zaretskii wrote:
>> Date: Mon, 14 Oct 2019 00:27:52 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> This have fixed the issue, I understand now what was happening. Should I
>> merge into master now?
>
>If there are no more outstanding issues, please go ahead and merge.
>
>> maybe you should close the related issues then right?.
>
>Which issues?  If you are talking about bug reports on debbugs, feel
>free to close them once you merge the branch.
>
>Thanks.

Hi:

I closed #36858, but I am not sure if the feature also fixes #23574? As
Martin referred to it during the discussion. Does it? Or it was a more
general issue not completed yet?

Re: Question about display engine

[Eli Zaretskii]

> Date: Mon, 21 Oct 2019 00:20:22 +0200
> From: Ergus <spacibba@aol.com>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> I closed #36858, but I am not sure if the feature also fixes #23574? As
> Martin referred to it during the discussion. Does it? Or it was a more
> general issue not completed yet?

I think bug#23574 should also be closed.  But if you want to be sure
110%, please ping the original submitter of the bug, asking him to try
the latest master.

Thanks.

Re: Question about display engine

[Ergus]

On Sun, Oct 13, 2019 at 09:53:13PM +0300, Eli Zaretskii wrote:
>> Date: Sun, 13 Oct 2019 20:25:42 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: rudalics@gmx.at, emacs-devel@gnu.org
>>
>> >Then looking at the position that causes the error will probably tell
>> >you what's wrong.  (Is OBJECT passed to Fget_text_property a string?
>> >if not, position of zero is invalid.)
>> >
>>
>> Actually conditioning the call to face_at_pos to when IT_CHARPOS (*it)
>> != 0 seems to fix the issue with magit... but maybe we are just hiding
>> something under the carpet here. Can you imagine something more general
>> than that just this condition?
>
>How did it happen that IT_CHARPOS(*it) is zero?  If we are iterating
>over a buffer, that cannot happen, because we begin from 1 and go
>forward.
>
If you see the bt I sent before the problem is in display_mode_line.

>Is it->sp zero or higher?  If it's higher, then we are not iterating
>over a buffer, but something else (a string, an image, or something
>similar).  What are the values of it->method and it->what?  Can you
>show the result of
>
>  (gdb) pgrowx it->glyph_row
>

Re: Question about display engine

[Eli Zaretskii]

I wrote:

> Could it be that redisplay_window_0, or some function it calls,
> signals an error, which is caught by internal_condition_case_1?  What
> happens if you put a breakpoint in signal_or_quit, does it get called
> from redisplay_window or some other function called by
> redisplay_windows?

Actually, it is better to put a breakpoint in redisplay_window_error.