merge emacs-bidi into the main tree

merge emacs-bidi into the main tree

[Alex Schroeder]

Some people have been waiting for a long time for a bidi-enabled Emacs
-- at least I have been waiting for more than a year.  The strange
thing is that we have a working implementation:  emacs-bidi is
available from www.m17n.org, and has been since Emacs 20.3 or earlier.

There are a few imperfections, or so I hear, but when I -- as a
non-Arabic-speaker -- saw the Arabic glyphs rendered correctly on IRC
(using ERC, the Emacs IRC client) -- I was surprised!  Yes, there may
be problems with the existing solution, but at least there *is* a
solution.

I would therefore like to see the existing emacs-bidi implementation
merged with our CVS version.

Back at in the days when Gerd was maintainer, I heard he had issues
with the emacs-bidi implementation and didn't want to use it.  Perhaps
performance was bad.  Perhaps he thought changes to the redisplay code
would be necessary, and he felt that these changes had to wait.
Perhaps you could not turn bidi off at the time.

Well, for somebody like me who just wants to learn Arabic, copy Arabic
song titles from old tapes onto his website, or read the occasional
Arabic greeting on IRC, the current solution is good enough.  And I
don't feel we should wait any longer.  We have waited long enough.

If the reasons for rejecting the current emacs-bidi are still valid,
let's list them and discuss them again.  In the mean time a few people
like Jan D and Kim Storm have had quite some exposure to the redisplay
code, so perhaps we do have the necessary know-how now to fix the
remaining issues.  That would be wonderful.  Eli has already done
some work in this area, so if prefered, people could start asking him
for some directions.

If the reasons for rejecting the current emacs-bidi are weak,
however, I think we should weight them against the long time we've
been waiting.

And last but not least, consider what I found here
(http://www.arabeyes.org/project.php?proj=VIM):

    About: VIM-6 already has a number of the required Arabization
    features developed (unicode support, a hack to support
    right-to-left (RTL) write ability, etc). What is lacking is
    shaping code as well as a firm font/keymap definition.

    Status: Completed

    Notes

    * The various common keyboard used out there (Arabic has 5
      keyboard mappings :-)

    * Shaping code ought to be modularized so that we could
      potentially release it stand-alone to other projects for
      inclusion - thoughts?

    * Bidi support is not really included (rightleft is not Bidi),
      what to do about this (long term question - not a show-stopper)?

    * Any thoughts from Bram about non-fixed width fonts?

See what I mean?  The people from arabeyes.org ("The Arabic Unix
Project") don't consider imperfect right-to-left implementation and
lacking shaping code to be a show-stopper.  Why should we?

If you are new to bidi issues, take a look at
http://www.emacswiki.org/cgi-bin/wiki.pl/CategoryBiDi
for an introduction.

Alex.
-- 
http://www.emacswiki.org/alex/
I was on holidays from 2003-07-01 to 2003-07-29
and have a lot of catching up to do.

Re: merge emacs-bidi into the main tree

[Richard Stallman]

    Some people have been waiting for a long time for a bidi-enabled Emacs
    -- at least I have been waiting for more than a year.  The strange
    thing is that we have a working implementation:  emacs-bidi is
    available from www.m17n.org, and has been since Emacs 20.3 or earlier.

I would expect that it needs merging to be able to be installed
into the current sources.  Is someone interested in doing that work?

    There are a few imperfections, or so I hear, but when I -- as a
    non-Arabic-speaker -- saw the Arabic glyphs rendered correctly on IRC
    (using ERC, the Emacs IRC client) -- I was surprised!  Yes, there may
    be problems with the existing solution, but at least there *is* a
    solution.

What do Hebrew and Arab users think of it?
Eli Z, what is your opinion?

Handa, what do you think?

Re: merge emacs-bidi into the main tree

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 07 Aug 2003 02:05:07 -0400
> 
> What do Hebrew and Arab users think of it?
> Eli Z, what is your opinion?

I feel uneasy about using something about which Gerd was most unhappy
at the time this was discussed.

I'd prefer to call for volunteers to continue the work I started,
namely, change the display engine to support bidirectional editing
without processing batches of buffer's text.  (I can elaborate if
there's enough interest.)  I have some unpolished stand-alone code
that does bidi reordering sequentially, which should fit very well
into the Emacs display engine, but someone needs to take that code and
integrate it into Emacs, since I probably won't have time any time
soon enough for this to be practical.

If anyone steps forward willing to begin hacking the display code, I
promise to help in any way I can.

Re: merge emacs-bidi into the main tree

[Oliver Scholz]

Eli Zaretskii <eliz@elta.co.il> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Date: Thu, 07 Aug 2003 02:05:07 -0400
>> 
>> What do Hebrew and Arab users think of it?
>> Eli Z, what is your opinion?
>
> I feel uneasy about using something about which Gerd was most unhappy
> at the time this was discussed.
>
> I'd prefer to call for volunteers to continue the work I started,
> namely, change the display engine to support bidirectional editing
> without processing batches of buffer's text.  (I can elaborate if
> there's enough interest.)  I have some unpolished stand-alone code
> that does bidi reordering sequentially, which should fit very well
> into the Emacs display engine, but someone needs to take that code and
> integrate it into Emacs, since I probably won't have time any time
> soon enough for this to be practical.
>
> If anyone steps forward willing to begin hacking the display code, I
> promise to help in any way I can.

Well, I am probably not yet skilled enough to be of any real help. And
I still have making fontsets customizable as the `car' of my
TODO-list, while the university does not leave much time for me.

But since I have been fascinated by m18n issues, since I started to
use Emacs. And since I'd like to become proficient in display hacking
some time in the future, I would appreciate it, if you could add me to
the emacs-bidi mailing list. Maybe I could do some minor tasks, at
least.

    Oliver
-- 
Oliver Scholz               21 Thermidor an 211 de la Révolution
Taunusstr. 25               Liberté, Egalité, Fraternité!
60329 Frankfurt a. M.       http://www.jungdemokratenhessen.de
Tel. (069) 97 40 99 42      http://www.jdjl.org

Re: merge emacs-bidi into the main tree

[Gerd Moellmann]

Eli Zaretskii <eliz@elta.co.il> writes:

> I feel uneasy about using something about which Gerd was most unhappy
> at the time this was discussed.

IIRC, I was worried that it might be a dead end, sort of a hack, which
users of r2l languages wouldn't be happy with in the long term.  But I
have to confess that I don't remember the details.

[...]

> If anyone steps forward willing to begin hacking the display code, I
> promise to help in any way I can.

And maybe I'll recover from my burn-out at some point :).

Re: Re: merge emacs-bidi into the main tree

[Kenichi Handa]

In article <ufzkd8rsj.fsf@elta.co.il>, Eli Zaretskii <eliz@elta.co.il> writes:
> I'd prefer to call for volunteers to continue the work I started,
> namely, change the display engine to support bidirectional editing
> without processing batches of buffer's text.  (I can elaborate if
> there's enough interest.)  I have some unpolished stand-alone code
> that does bidi reordering sequentially, which should fit very well
> into the Emacs display engine, but someone needs to take that code and
> integrate it into Emacs, since I probably won't have time any time
> soon enough for this to be practical.

> If anyone steps forward willing to begin hacking the display code, I
> promise to help in any way I can.

I'm now working to synchronize the emacs-unicode branch to
the latest HEAD code.  As there were many cosmetic and
restructuring changes in HEAD, it's not easy to adjust the
code of emacs-unicode.  I think it takes two or three weeks
more.

After that, I can start working on bidi.  I'd like to
incorporate your reordering routine and my display routine
of emacs-bidi into the emacs-unicode branch instead of
directly into HEAD to avoid another synching labor.

---
Ken'ichi HANDA
handa@m17n.org

Re: merge emacs-bidi into the main tree

[Kenichi Handa]

In article <ullu4msdl.fsf@ID-87814.user.dfncis.de>, Oliver Scholz <epameinondas@gmx.de> writes:
> Well, I am probably not yet skilled enough to be of any real help. And
> I still have making fontsets customizable as the `car' of my
> TODO-list, while the university does not leave much time for me.

> But since I have been fascinated by m18n issues, since I started to
> use Emacs. And since I'd like to become proficient in display hacking
> some time in the future, I would appreciate it, if you could add me to
> the emacs-bidi mailing list. Maybe I could do some minor tasks, at
> least.

I think you can subscribe to emacs-bidi mailing list from
this page:
	<http://mail.gnu.org/mailman/listinfo/emacs-bidi>

---
Ken'ichi HANDA
handa@m17n.org

Re: merge emacs-bidi into the main tree

[Eli Zaretskii]

> From: Oliver Scholz <epameinondas@gmx.de>
> Date: Fri, 08 Aug 2003 11:05:26 +0200
> 
> But since I have been fascinated by m18n issues, since I started to
> use Emacs. And since I'd like to become proficient in display hacking
> some time in the future, I would appreciate it, if you could add me to
> the emacs-bidi mailing list.

You don't need me for that: emacs-bidi is a public list, so you can
add yourself via this page:

  http:/savannah.gnu.org/projects/emacs

(follow the link to mailing lists).

And thanks for volunteering.

Re: merge emacs-bidi into the main tree

[Eli Zaretskii]

> From: gerd.moellmann@t-online.de (Gerd Moellmann)
> Date: 08 Aug 2003 13:23:28 +0200
> 
> IIRC, I was worried that it might be a dead end, sort of a hack, which
> users of r2l languages wouldn't be happy with in the long term.  But I
> have to confess that I don't remember the details.

I still have the mail we exchanged, and can dig out the details.

IIRC, you were worried about the performance hit: the code as written
in the display engine part required to disable all the optimizations
and shortcuts that exist for the left-to-right languages, and you felt
this would hamper redisplay performance even in the simplest cases,
like cursor motion and insertion of a single character.

Re: Re: merge emacs-bidi into the main tree

[Eli Zaretskii]

> Date: Fri, 8 Aug 2003 20:44:50 +0900 (JST)
> From: Kenichi Handa <handa@m17n.org>
> 
> I'd like to
> incorporate your reordering routine and my display routine
> of emacs-bidi into the emacs-unicode branch instead of
> directly into HEAD to avoid another synching labor.

That's just great.  I'll happily send you my code and provide any
other help you might need.

Thanks!

Re: merge emacs-bidi into the main tree

[Gerd Moellmann]

"Eli Zaretskii" <eliz@elta.co.il> writes:

> IIRC, you were worried about the performance hit: the code as written
> in the display engine part required to disable all the optimizations
> and shortcuts that exist for the left-to-right languages, and you felt
> this would hamper redisplay performance even in the simplest cases,
> like cursor motion and insertion of a single character.

Yes, that matches the little I remember.

Re: Re: merge emacs-bidi into the main tree

[Alex Schroeder]

Kenichi Handa <handa@m17n.org> writes:

> I'm now working to synchronize the emacs-unicode branch to
> the latest HEAD code.  As there were many cosmetic and
> restructuring changes in HEAD, it's not easy to adjust the
> code of emacs-unicode.  I think it takes two or three weeks
> more.
>
> After that, I can start working on bidi.  I'd like to
> incorporate your reordering routine and my display routine
> of emacs-bidi into the emacs-unicode branch instead of
> directly into HEAD to avoid another synching labor.

That sounds like awesome news!  :)

Alex.
-- 
http://www.emacswiki.org/alex/
I was on holidays from 2003-07-01 to 2003-07-29
and have a lot of catching up to do.

Re: merge emacs-bidi into the main tree

[Chahine M. HAMILA]

> I would expect that it needs merging to be able to be installed
> into the current sources.  Is someone interested in doing that work?

Hi, the screenshots look great, but I personally haven't used that one (and
don't know if I can right now since I'm not at my usual place and only have
a win32 machine for the few coming weeks).
Our resources on Arabeyes in terms of developers are very scarce, and my own
time resources are scarce too. But since I use Emacs a lot, I am definitely
interested in this, so if someone tells me what work is involved and where
it should be started, I could consider it if it fits within my time frame.
The problem is the code base of Emacs is huge and I certainly don't have the
necessary time to get into a fine understanding of it or even of every
detail of its architecture, but if the steps for merging the existing
patches are simpler than that I can undertake the job...

Re: merge emacs-bidi into the main tree

[Richard Stallman]

    IIRC, I was worried that it might be a dead end, sort of a hack, which
    users of r2l languages wouldn't be happy with in the long term.

That may be true--but even if that is true, but might be a good thing
to install it now.  Sure, we would like something better such as Eli
is working on, but I don't think he will have it ready soon.
If that will take another two years, we may as well have the emacs-bidi
code in the mean time, rather than nothing.

Re: merge emacs-bidi into the main tree

[Richard Stallman]

    > IIRC, you were worried about the performance hit: the code as written
    > in the display engine part required to disable all the optimizations
    > and shortcuts that exist for the left-to-right languages, 

Does it disable the optimizations always, or only when there is
right-to-left text?

								and you felt
    > this would hamper redisplay performance even in the simplest cases,
    > like cursor motion and insertion of a single character.

    Yes, that matches the little I remember.

If the optimizations are disabled only when there is right-to-left text,
and if the users of emacs-bidi think it is better than nothing, we may
as well include the feature anyway.

If the change requires disabling optimizations always, even for people
who don't use right-to-left text, that is potentially a serious
problem.  Maybe it can be solved by adding a flag that people must set
in order to use right-to-left text.  Of course, an automatic solution
would be better, but the flag would be enough to make it acceptable.

Re: merge emacs-bidi into the main tree

[Gerd Moellmann]

Richard Stallman <rms@gnu.org> writes:

>     > IIRC, you were worried about the performance hit: the code as written
>     > in the display engine part required to disable all the optimizations
>     > and shortcuts that exist for the left-to-right languages, 
> 
> Does it disable the optimizations always, or only when there is
> right-to-left text?

AFAIR, most optimizations were disabled in buffers containing
right-to-left text, so that this wouldn't be a problem for users who
don't use right-to-left text in the first place.

Re: merge emacs-bidi into the main tree

[Gerd Moellmann]

Richard Stallman <rms@gnu.org> writes:

>     IIRC, I was worried that it might be a dead end, sort of a hack, which
>     users of r2l languages wouldn't be happy with in the long term.
> 
> That may be true--but even if that is true, but might be a good thing
> to install it now.  Sure, we would like something better such as Eli
> is working on, but I don't think he will have it ready soon.
> If that will take another two years, we may as well have the emacs-bidi
> code in the mean time, rather than nothing.

That's true, no argument, if nobody steps up to finish Eli's
implementation.

I think it would be unfortunate though, and that's why I mentioned the
dead end, if users really get unhappy and redisplay is hacked to
improve performance in the right-to-left case based on a presumable
temporary solution.

Re: merge emacs-bidi into the main tree

[Nadim Shaikli]

--- Richard Stallman wrote:
>
[snip snip]
>   --- Alex Schroeder wrote:
>     There are a few imperfections, or so I hear, but when I -- as a
>     non-Arabic-speaker -- saw the Arabic glyphs rendered correctly on IRC
>     (using ERC, the Emacs IRC client) -- I was surprised!  Yes, there may
>     be problems with the existing solution, but at least there *is* a
>     solution.
> 
> What do Hebrew and Arab users think of it?

Arabic support in m17n's emacs code was very usable and almost complete.
There weren't that many issues left with it (most were minor) from a user's
prospective, our only concern was making sure that excellent effort made
its way into emacs' main trunk and I can see that there is movement in
that direction which I too applaud and highly encourage (for whatever
it's worth :-)

For those that have not received all the emails, please read the following
thread (courtesy of the emacs' mailing-lists) --

  http://mail.gnu.org/archive/html/emacs-devel/2003-08/msg00039.html

Any other Arabic user comments (ie. Arabeyes people) ?

PS: Thanks Alex again for getting the ball rolling...

Regards,

 - Nadim


__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

Re: merge emacs-bidi into the main tree

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sat, 09 Aug 2003 10:21:17 -0400
> 
>     > IIRC, you were worried about the performance hit: the code as written
>     > in the display engine part required to disable all the optimizations
>     > and shortcuts that exist for the left-to-right languages, 
> 
> Does it disable the optimizations always, or only when there is
> right-to-left text?

Handa-san should give the definitive answer, but I think it was
always, as long as Emacs was told that the buffer _could_ contain
right-to-left text.  This is because Emacs doesn't know whether there
actually is right-to-left text in the buffer, and cannot do so easily
without getting a significant performance hit (what would we do?
search the buffer for certain ranges of characters after each change
to buffer's text?).

> If the change requires disabling optimizations always, even for people
> who don't use right-to-left text, that is potentially a serious
> problem.  Maybe it can be solved by adding a flag that people must set
> in order to use right-to-left text.

Such a flag indeed existed in the implementation I saw a few years
ago, when I was in Japan.  But perhaps things have changed since then.

However, it doesn't seem right to me to have an Emacs that cannot
scroll fast enough just because I've set such a flag, assuming that
Gerd's intuition is correct.  IMHO, of course.

Re: merge emacs-bidi into the main tree

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sat, 09 Aug 2003 10:21:06 -0400
> 
> That may be true--but even if that is true, but might be a good thing
> to install it now.  Sure, we would like something better such as Eli
> is working on, but I don't think he will have it ready soon.
> If that will take another two years, we may as well have the emacs-bidi
> code in the mean time, rather than nothing.

As Handa-san says he will try to use what I did, I think this is not
an issue anymore.  If what I wrote has any value, I'm sure it will be
merged with Emacs.

Re: merge emacs-bidi into the main tree

[Gerd Moellmann]

Eli Zaretskii <eliz@elta.co.il> writes:

> However, it doesn't seem right to me to have an Emacs that cannot
> scroll fast enough just because I've set such a flag, assuming that
> Gerd's intuition is correct.

One way to see how working with disabled redisplay optimizations feels
might be to intentionally disable the optimizations in the source
code.

(Which is, BTW, where my intuition comes from, only that I didn't
disable any optimizations, but implement one after the other until the
result was "fast enough", for me anyway.  And, another BTW, redisplay
optimization was _the_ major time sink and _the_ primary source of
complexity, when rewriting redisplay, esp. under the ancillary
condition of being compatible with Emacs 19's redisplay.  I'm
mentioning that only in case that r2l redisplay optimization will
prove necessary.)

In any case, I think it might be worth the experiment.  Computers got
noticeable faster, maybe the picture is different now than it was in
the late '90s.

Re: merge emacs-bidi into the main tree

[Gerd Moellmann]

"Eli Zaretskii" <eliz@elta.co.il> writes:

> Just so we have a reference: what kind of a machine did you use at
> the time you did that?

I think I mainly used a P2 400 MHz.

Re: merge emacs-bidi into the main tree

[Stefan Monnier]

> > Just so we have a reference: what kind of a machine did you use at
> > the time you did that?
> 
> I think I mainly used a P2 400 MHz.

I.e. something faster than what I (and my girlfriend, and my officemate)
use every day.


	Stefan

Re: merge emacs-bidi into the main tree

[Eli Zaretskii]

> From: gerd.moellmann@t-online.de (Gerd Moellmann)
> Date: 10 Aug 2003 12:42:37 +0200
> 
> (Which is, BTW, where my intuition comes from, only that I didn't
> disable any optimizations, but implement one after the other until the
> result was "fast enough", for me anyway.

Just so we have a reference: what kind of a machine did you use at
the time you did that?

> In any case, I think it might be worth the experiment.  Computers got
> noticeable faster, maybe the picture is different now than it was in
> the late '90s.

True.

Btw--and this is just for completeness' sake--one of the design goals
of what I've written was to allow the reordering code to be called
from a variety of places, like if we need to encode text in visual
order so it could be sent to a printer.

Re: merge emacs-bidi into the main tree

[Richard Stallman]

    Handa-san should give the definitive answer, but I think it was
    always, as long as Emacs was told that the buffer _could_ contain
    right-to-left text.  This is because Emacs doesn't know whether there
    actually is right-to-left text in the buffer, and cannot do so easily
    without getting a significant performance hit (what would we do?
    search the buffer for certain ranges of characters after each change
    to buffer's text?).

That is acceptable, in my opinion, because it means most users
won't lose anything.

    However, it doesn't seem right to me to have an Emacs that cannot
    scroll fast enough just because I've set such a flag, assuming that
    Gerd's intuition is correct.  IMHO, of course.

It is a major, but supporting your language with slower scrolling is
better than failing to support your language at all.

    As Handa-san says he will try to use what I did, I think this is not
    an issue anymore.  If what I wrote has any value, I'm sure it will be
    merged with Emacs.

That would be very good.  Handa-san, can you tell us more about what
you plan to do with this code, and how much work you think it will be?

Re: merge emacs-bidi into the main tree

[Richard Stallman]

    > What do Hebrew and Arab users think of it?

    Arabic support in m17n's emacs code was very usable and almost complete.

We were talking about the emacs-bidi code; your response
is about m17n's Emacs code.  Are you talking about the same
emacs-bidi code with a different name, or are you talking
about a different version?

Is the m17n's Emacs code installed in the Emacs CVS repository?
If yes, is it in the trunk or in a branch?

Re: merge emacs-bidi into the main tree

[Nadim Shaikli]

--- Richard Stallman <rms@gnu.org> wrote:
>     > What do Hebrew and Arab users think of it?
> 
>     Arabic support in m17n's emacs code was very usable and almost complete.
> 
> We were talking about the emacs-bidi code; your response
> is about m17n's Emacs code.  Are you talking about the same
> emacs-bidi code with a different name, or are you talking
> about a different version?

Sorry, I was talking about the code that was developed and is
currently hosted by m17n,

  http://www.m17n.org/emacs-bidi/index.html

> Is the m17n's Emacs code installed in the Emacs CVS repository?
> If yes, is it in the trunk or in a branch?

As far as I know the code developed by m17n is _not_ in Emacs' CVS
repository (my info might be dated though).  What Alex had commented
on in his initial post, I believe, was with regard to m17n's code
(his usage with ERC, etc) and my comments were to that end as well.
m17n.org's additions are _very_ usable and accurate from a user's
prospective and they (or similar :-) ought to get folded-in.  As
there has been a plethora of posts on this topic and there is
movement in that direction, this post/opinion is non-consequential.

Sorry for any possible confusion.

 - Nadim


__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

Re: merge emacs-bidi into the main tree

[Kenichi Handa]

Sorry for lazy responses on this thread.

In article <E19mCAl-0003yH-V7@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
>     Handa-san should give the definitive answer, but I think it was
>     always, as long as Emacs was told that the buffer _could_ contain
>     right-to-left text.  This is because Emacs doesn't know whether there
>     actually is right-to-left text in the buffer, and cannot do so easily
>     without getting a significant performance hit (what would we do?
>     search the buffer for certain ranges of characters after each change
>     to buffer's text?).

> That is acceptable, in my opinion, because it means most users
> won't lose anything.

>     However, it doesn't seem right to me to have an Emacs that cannot
>     scroll fast enough just because I've set such a flag, assuming that
>     Gerd's intuition is correct.  IMHO, of course.

As to optimization, what I disabled are, as far as I remeber:

  o direct_output_xxx
  o partial update of a line.  A line is always updated fully.

The other optimizatios should be still effective even if the
buffer local variable `enable-bidi-display' or
`orientation-reversed' are non-nil.

So, for instance, just scrolling should not be delayed.

One strong objection point I remember is about caching
reordered glyphs in (struct it).

> It is a major, but supporting your language with slower scrolling is
> better than failing to support your language at all.

>     As Handa-san says he will try to use what I did, I think this is not
>     an issue anymore.  If what I wrote has any value, I'm sure it will be
>     merged with Emacs.

> That would be very good.  Handa-san, can you tell us more about what
> you plan to do with this code, and how much work you think it will be?

In brief, what I did in emacs-bidi are:

(1) change xdisp.c to call get_next_display_element_visually
  and set_iterator_to_next_visually instead of
  get_next_display_element and set_iterator_to_next.

(2) make a new file bidi.c that implements
  get_next_display_element_visually and
  set_iterator_to_next_visually.

(3) make a new file bidi.el that implements simple
  bidi-reordering function that is called from
  get_next_display_element_visually to create a cache in
  (struct it).

(4) change xterm.c to display glyphs flushing to right when
  orientation-reversed is non-nil.

My current plan is to replace (2) and (3) with Eli's code.

---
Ken'ichi HANDA
handa@m17n.org

Re: merge emacs-bidi into the main tree

[Eli Zaretskii]

> Date: Tue, 12 Aug 2003 10:49:26 +0900 (JST)
> From: Kenichi Handa <handa@m17n.org>
> 
> In brief, what I did in emacs-bidi are:
> 
> (1) change xdisp.c to call get_next_display_element_visually
>   and set_iterator_to_next_visually instead of
>   get_next_display_element and set_iterator_to_next.
> 
> (2) make a new file bidi.c that implements
>   get_next_display_element_visually and
>   set_iterator_to_next_visually.
> 
> (3) make a new file bidi.el that implements simple
>   bidi-reordering function that is called from
>   get_next_display_element_visually to create a cache in
>   (struct it).
> 
> (4) change xterm.c to display glyphs flushing to right when
>   orientation-reversed is non-nil.
> 
> My current plan is to replace (2) and (3) with Eli's code.

It's not gonna be that simple ;-)

The code I wrote is designed to be plugged into the part of the
display engine that walks the buffer and, for each character it
encounters, generates a `struct glyph' element of the glyph matrix.
Currently, ignoring overlays, text properties, etc., this buffer scan
is linear, from left to right, and going to the next buffer position
is simple:

	IT_BYTEPOS (*it) += it->len;
	IT_CHARPOS (*it) += 1;

(this is from xdisp.c:set_iterator_to_next).  That is, we simply
increment the buffer's character position by one.  (If the next
display element comes from something other than the buffer, the code
in set_iterator_to_next does similar things with the object that is
the source of characters.)

The code I wrote replaces the buffer position increment with a call to
a function.  Thus, the above two lines should roughly be replaced with
this:

	bidi_get_next_char_visually (it->bidi_it);

[IT_BYTEPOS (*it) and IT_CHARPOS (*it) get set to the right values
inside that call.]

What bidi_get_next_char_visually does is to find the next character in
the buffer in the visual order; that could involve quickly scanning
across a run of characters until we find the one we want to display.
For example, i fthe buffer's contents are

	abcdABCDefg

then after processing `d', the code will scan until `D', then go back
to `C', `B', and `A', in that order, then jump to `e', and finally
proceed to `f' and `g'.

(To save future work, some crucial information about the characters
over which we scan is cached inside the bidi_it struct, to facilitate
processing later when we need to go back to those characters and
generate the glyph matrix elements from them.  So, in the above
example, going back from 'D' to 'A' boils down to delivering
information already cached during the forward scan.)

So a linear scan becomes non-linear, but the details of that are
completely concealed from set_iterator_to_next; as far as it is
concerned, it just magically jumped over a few characters and landed
on the next character to be displayed.  The glyph matrices are layed
in the visual order.

In other words, the structure and logic of the code in xdisp.c remains
largely unchanged, the only changes being where we find the position
of the next display element.  Or so I hope, anyway...

(There still needs to be more code in the terminal-specific parts of
Emacs, that displays the glyphs either starting at the left or the
right margin of the screen/window, depending on the current
paragraph's base direction; the latter gets decided by subroutines of
bidi_get_next_char_visually and is stored in the bidi_it structure.  I
believe this code, at least for the X terminals, already largely
exists in m17n.org's emacs-bidi.)

Re: Re: merge emacs-bidi into the main tree

[Kenichi Handa]

In article <9743-Tue12Aug2003085951+0300-eliz@elta.co.il>, "Eli Zaretskii" <eliz@elta.co.il> writes:
>>  In brief, what I did in emacs-bidi are:
>>  
>>  (1) change xdisp.c to call get_next_display_element_visually
>>    and set_iterator_to_next_visually instead of
>>    get_next_display_element and set_iterator_to_next.
>>  
>>  (2) make a new file bidi.c that implements
>>    get_next_display_element_visually and
>>    set_iterator_to_next_visually.
>>  
>>  (3) make a new file bidi.el that implements simple
>>    bidi-reordering function that is called from
>>    get_next_display_element_visually to create a cache in
>>    (struct it).
>>  
>>  (4) change xterm.c to display glyphs flushing to right when
>>    orientation-reversed is non-nil.
>>  
>>  My current plan is to replace (2) and (3) with Eli's code.

> It's not gonna be that simple ;-)

[...]
> (To save future work, some crucial information about the characters
> over which we scan is cached inside the bidi_it struct, to facilitate
> processing later when we need to go back to those characters and
> generate the glyph matrix elements from them.  So, in the above
> example, going back from 'D' to 'A' boils down to delivering
> information already cached during the forward scan.)

Then, it seems that what you've done is not that different
from the current emacs-bidi.  In most cases, we anyway move
the iterator all over "abcdABCDefg".  Your code caches only
some crucial information, so get_next_display_element will
need some work at C B A e f g.  My code caches all
information given by get_next_display_element, so
get_next_display_element will work fast at C B A e f g.

In both cases, set_iterator_to_next (your code) and
set_iterator_to_next_visually (my code) moves IT_CHARPOS
(*it) non-linearly.

So, I still think incorporating your code is not that difficult.

> (There still needs to be more code in the terminal-specific parts of
> Emacs, that displays the glyphs either starting at the left or the
> right margin of the screen/window, depending on the current
> paragraph's base direction; the latter gets decided by subroutines of
> bidi_get_next_char_visually and is stored in the bidi_it structure.  I
> believe this code, at least for the X terminals, already largely
> exists in m17n.org's emacs-bidi.)

Yes.  "(4) change xterm.c" does that.

---
Ken'ichi HANDA
handa@m17n.org

Re: Re: merge emacs-bidi into the main tree

[Eli Zaretskii]

> Date: Tue, 12 Aug 2003 16:18:12 +0900 (JST)
> From: Kenichi Handa <handa@m17n.org>
> 
> Then, it seems that what you've done is not that different
> from the current emacs-bidi.  In most cases, we anyway move
> the iterator all over "abcdABCDefg".  Your code caches only
> some crucial information, so get_next_display_element will
> need some work at C B A e f g.  My code caches all
> information given by get_next_display_element, so
> get_next_display_element will work fast at C B A e f g.

Right.  IIRC, Gerd did not like the caching of glyph matrix elements.
I think he felt that producing the information stored in the glyph is
expensive, and so doing that unnecessarily would hamper performance.
The information my code caches is only relevant to the bidi properties
of the characters, which need to be computed anyway to decide what is
the next character in the visual order.

Re: merge emacs-bidi into the main tree

[Richard Stallman]

It sounds like there are three different bidi efforts for Emacs: the
emacs-bidi, the m17n, and Eli Z's code (not ready yet).  Is that
correct?

Has anyone looked at all three?

Re: merge emacs-bidi into the main tree

[Kenichi Handa]

In article <E19miSn-0004AG-IU@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
> It sounds like there are three different bidi efforts for Emacs: the
> emacs-bidi, the m17n, and Eli Z's code (not ready yet).  Is that
> correct?

No.  The emacs-bidi and the m17n are the same thing.  As
emacs-bidi is now distributed at www.m17n.org, it is
referred as "the m17n".

---
Ken'ichi HANDA
handa@m17n.org

Re: merge emacs-bidi into the main tree

[Nadim Shaikli]

--- Richard Stallman <rms@gnu.org> wrote:
> It sounds like there are three different bidi efforts for Emacs: the
> emacs-bidi, the m17n, and Eli Z's code (not ready yet).  Is that
> correct?
 
I believe there are only 2 efforts -- m17n's working code and Eli's
code now being looked into (emacs-bidi is a mailing-list ;-)

Regards,

 - Nadim


__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

Re: merge emacs-bidi into the main tree

[Richard Stallman]

      o partial update of a line.  A line is always updated fully.

It should not be to hard to make this optimization work again
when the line in question contains no rtl characters.

Re: merge emacs-bidi into the main tree

[Richard Stallman]

    > It sounds like there are three different bidi efforts for Emacs: the
    > emacs-bidi, the m17n, and Eli Z's code (not ready yet).  Is that
    > correct?

    No.  The emacs-bidi and the m17n are the same thing.  As
    emacs-bidi is now distributed at www.m17n.org, it is
    referred as "the m17n".

So it sounds like some people are saying that the emacs-bidi code
is usable as it stands and is worth installing, and nobody thinks
it is useless.  Meanwhile, it looks like you are ready to make
use of Eli's code in the short term.

If we can use Eli's code soon enough, then it's not worth installing
the Emacs-bidi code; it would just be extra work.  On the other hand,
if it looks like the difficulty of using Eli's code is enough to make
it hard, then I think we should install the Emacs-bidi code now.

Re: merge emacs-bidi into the main tree

[Kenichi Handa]

In article <E19n8T9-00031X-S0@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
> So it sounds like some people are saying that the emacs-bidi code
> is usable as it stands and is worth installing, and nobody thinks
> it is useless.  Meanwhile, it looks like you are ready to make
> use of Eli's code in the short term.

> If we can use Eli's code soon enough, then it's not worth installing
> the Emacs-bidi code; it would just be extra work.  On the other hand,
> if it looks like the difficulty of using Eli's code is enough to make
> it hard, then I think we should install the Emacs-bidi code now.

Synchronizing emacs-unicode to HEAD will take a few weeks
more.  After that, I'll take a look at Eli's code and decide
what to do.  In any cases, I will install the changes for
BIDI in the branch emacs-unicode-2 because the current codes
of emacs-bidi depends on automatic-composition facitliy to
display Arabic and Hebrew, and that exists only in
emacs-unicode (and in emacs-bidi) version.  I'd like to
avoid the extra work of installing automatic-composition for
the current HEAD.

FYI, the automatic-composition is to call various character
composition functions from the dipslay routine dynamically
so that we don't have to run XXX-compose-region on file
reading and character inputting.

---
Ken'ichi HANDA
handa@m17n.org

Re: merge emacs-bidi into the main tree

[Alex Schroeder]

Kenichi Handa <handa@m17n.org> writes:

> Synchronizing emacs-unicode to HEAD will take a few weeks
> more.  After that, I'll take a look at Eli's code and decide
> what to do.  In any cases, I will install the changes for
> BIDI in the branch emacs-unicode-2...

Well, too bad if it takes longer.  I am happy that things are finally
moving on the bidi subject.  Thanks for putting so much work into
the multilingualization of Emacs.  I really appreciate your efforts!

Alex.
-- 
http://www.emacswiki.org/alex/
There is no substitute for experience.

Re: merge emacs-bidi into the main tree

[Nadim Shaikli]

--- Alex Schroeder <alex@emacswiki.org> wrote:
> Kenichi Handa <handa@m17n.org> writes:
> 
> > Synchronizing emacs-unicode to HEAD will take a few weeks
> > more.  After that, I'll take a look at Eli's code and decide
> > what to do.  In any cases, I will install the changes for
> > BIDI in the branch emacs-unicode-2...
> 
> Well, too bad if it takes longer.  I am happy that things are finally
> moving on the bidi subject.  Thanks for putting so much work into
> the multilingualization of Emacs.  I really appreciate your efforts!

We all appreciate these outstanding efforts :-)

Keep up the great work.

 - Nadim


__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

Re: merge emacs-bidi into the main tree

[Richard Stallman]

    Synchronizing emacs-unicode to HEAD will take a few weeks
    more.  After that, I'll take a look at Eli's code and decide
    what to do.  In any cases, I will install the changes for
    BIDI in the branch emacs-unicode-2 because the current codes
    of emacs-bidi depends on automatic-composition facitliy to
    display Arabic and Hebrew, and that exists only in
    emacs-unicode (and in emacs-bidi) version.

I am not sure what this means.  What is the difference between the
branch emacs-unicode-2 and the branch emacs-unicode?

Re: merge emacs-bidi into the main tree

[Kenichi Handa]

In article <E19oBXI-0000Ro-TE@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:
>     Synchronizing emacs-unicode to HEAD will take a few weeks
>     more.  After that, I'll take a look at Eli's code and decide
>     what to do.  In any cases, I will install the changes for
>     BIDI in the branch emacs-unicode-2 because the current codes
>     of emacs-bidi depends on automatic-composition facitliy to
>     display Arabic and Hebrew, and that exists only in
>     emacs-unicode (and in emacs-bidi) version.

> I am not sure what this means.  What is the difference between the
> branch emacs-unicode-2 and the branch emacs-unicode?

To reflect the changes in HEAD to the Unicode version of
emacs, I made a branch emacs-unicode-2.  When I finish the
work, emacs-unicode branch will be an obsolete branch.

---
Ken'ichi HANDA
handa@m17n.org