Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii <eliz@gnu.org>]

> From: Kenichi Handa <handa@m17n.org>
> Cc: stephen@xemacs.org, larsi@gnus.org, list-general@mohsen.1.banan.byname.net,
>         emacs-devel@gnu.org
> Date: Tue, 09 Aug 2011 15:11:41 +0900
> 
> In article <E1QouuP-0006aC-L8@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > >   Guidelines of [Unicode]). Paragraphs may also be
> > >   determined by higher-level protocols: for example, the
> > >   text in two different cells of a table will be in
> > >   different paragraphs.
> > > 
> > > The case of summary line belongs to the last sentence above.
> 
> > That is one possible interpretation, yes.  But it isn't the only one.
> 
> I think it is the most natural/appropriate interpretation.

I'm not convinced.  However, as I wrote, if we are okay with using
TABs to separate fields in summary buffers, the necessary
infrastructure is already implemented.  Personally, I'm not sure that
using TABs is the best way, for reasons I described in the quoted
message, but I'll leave this decision to Gnus developers and users.
Maybe making the TABs invisible (with the `invisible' text property)
will help us maintain a plausible display with predictable format, and
thus resolve the difficulties that bother me.

> > The goal is to have correct display of the summary lines, not to
> > decide whether we are dealing with cells or not.  Directional control
> > character allow to render summary lines for display correctly even
> > without "cells".
> 
> They can, but it is better to avoid using them to workaround
> the "cell" problem.

Why?  I didn't yet hear a single explanation of why people don't like
them.  They are just formatting control characters, like TAB, SPC, and
NL.

Without providing some good reasons for staying away of the
directional control characters, all this discussion is just waste of
energy and bandwidth.  Unless someone else is going to implement the
proposed text property alternative, I will need a _very_ good reason
to throw away what cost me many months of hard work and do it again
differently, because as far as I'm concerned, what we have now is very
satisfactory, to say the least, and no serious attempts were made this
far to use them to solve this issue.

> I think we should not stop finding the right solution just
> to release 24.1 on time.  Isn't it worth delaying 24.1 for
> the right solution?

How long are you prepared to wait?  How long are Emacs users prepared
to wait?

First, "the right solution" is not well defined yet.  People are
proposing to use text properties, but the proposal itself is not
detailed enough.  For example, are we going to use text properties for
any other bidi-related features, and if so, which ones?  We had better
have a good and detailed answer to this and similar questions, because
making such deep changes (see below) without a clear plan, or, worse,
just for the benefit of a single use case (lines with fields) is a bad
idea: it will complicate the heck out of the display code, which is
already a hard to maintain monstrosity, where alternatives already
exist.

I see no hope of coming up with a reasonably complete proposal from
this discussion, because most of the people arguing for the change
don't use R2L scripts and have no clear idea what is needed and how it
will affect users.  So it is again up to me, and I don't see the need
for such a change to begin with, because no one cared to tell what is
so dead wrong with the current facilities, which, btw, were not chosen
at random, but came out of long discussions and careful study of the
needs and of "prior art".

Next, implementing this suggestion is not a simple task at all.
Forcing bidi.c to check text properties will at best slow down the
display engine, which is already slower than Emacs 23.  Worse, there
are many potential pitfalls here, because code that examines text
properties will need to avoid walking the buffer, or it will cause
infinite recursion.  In the current implementation, at least handling
of the `display' and `invisible' properties does involve walking the
buffer--these parts will have to be redesigned and reimplemented.
Handlers of other properties will have to be carefully examined, to
make sure they don't run the same risk; in the current design, I never
needed such an analysis because of complete separation between
scanning the buffer and applying text properties.

Last, but not least, the elaborate logic that finds and applies text
properties when the next character in the visual order can be far away
of the current one will have to be redesigned and reimplemented.
Getting that logic right took months of trial and crash, because the
intricate details of how all this works in the display engine are not
documented anywhere.  I still bump into surprises from time to time to
this day, when I step through set_iterator_to_next and its
subroutines.  Who will have the energy and motivation to go through
all that process again?