bug#9771: 24.0.90; Redisplay problems with control characters

[Eli Zaretskii <eliz@gnu.org>]

> From: Johan Bockgård <bojohan@gnu.org>
> Cc: 9771@debbugs.gnu.org
> Date: Mon, 17 Oct 2011 23:04:46 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> According to the condition above, the position in column 0 before the
> >> ^ glyph (dpvec_index = 0) is not a possible stop point, but the position
> >> between ^ and @ is.
> >
> > Okay, but what is the practical problem with this?
> 
> Nothing, really. But it doesn't seem correct.

I'm not sure.  Maybe I'm missing something (move_it_in_display_line_to
is one of the trickiest functions in the display engine), so please
bear with me as I explain why I think the test is correct.

You wrote:

>     #define BUFFER_POS_REACHED_P()                                  \
>     [...]
>        && (it->method == GET_FROM_BUFFER                            \
>            || (it->method == GET_FROM_DISPLAY_VECTOR                \
>                && it->dpvec + it->current.dpvec_index + 1 >= it->dpend)))
> 
> According to the condition above, the position in column 0 before the
> ^ glyph (dpvec_index = 0) is not a possible stop point, but the position
> between ^ and @ is.

A display vector has its glyphs stored at indices 0..it->dpend-1.
IOW, the last glyph's index is it->dpend-1.  BUFFER_POS_REACHED_P is
invoked always after a call to get_next_display_element, but before
the call to set_iterator_to_next.  The former just consumes the glyph
at it->current.dpvec_index; the latter advances the index to the next
glyph of the display vector.  Therefore, after a call to
get_next_display_element, the glyph at dpvec_index was already
consumed, but the index was not yet advanced to the next position.
Thus, when the index is at dpend-1, as the condition in
BUFFER_POS_REACHED_P says, we have already consumed the last glyph.

Consuming the last glyph of a display vector means that the next call
to set_iterator_to_next will detect that the display vector is
exhausted, and will advance to the next buffer position, the one after
the position which we just passed by consuming all the glyphs from the
display vector used to display the character at that position.  Thus,
"buffer position reached".

Does this make sense?

In practice, after correcting the bug that caused the assertion
violation, I can no longer reproduce the situation where we stop in
the middle of the ^@ character.  If you can show me a recipe for
winding up in the middle of a display vector under these or similar
circumstances, I will have another look.

> > Binary nulls in a file generally cause Emacs to make the buffer
> > unibyte, where bidi reordering is disabled.
> 
> That doesn't seem to be working, then.
> 
> If I do
> 
>   emacs -Q /usr/bin/emacs
> 
> the resulting buffer is multibyte.

Indeed.  However, buffer-file-coding-system is no-conversion, which is
confusingly inconsistent with "C-x RET c no-conversion C-x C-f".
Perhaps Handa-san could shed some light on this inconsistency.

Anyway, this being so, I found a way to optimize a couple of expensive
loops inside bidi.c for the important special case of a plain L2R text
in the buffer.  With these optimizations, the bidi redisplay of the
entire window showing a buffer with 2000 binary nulls is only a few
milliseconds slower than the unidirectional one.

I will install those changes later today, after some more testing.

> Type M-> and Emacs hangs. (I waited 5 minutes and had to kill the
> process.)

It takes 1.5 seconds now (with a 44MB emacs binary ;-)