bug#22786: 25.1.50; eww arabic rendering

bug#22786: 25.1.50; eww arabic rendering

[Mohamed Hibti]

Hi all,

There is a problem with the arabic composition (the letters are not
displayed isolated and not assembled in a word). 

Mohamed

In GNU Emacs 25.1.50.1 (i686-pc-linux-gnu, GTK+ Version 2.24.23)
 of 2016-01-24 built on cassiopee
Repository revision: 6d25cbeaaf93615b8d7f26024ba014104eb5d4f2
Windowing system distributor 'The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04.4 LTS

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GSETTINGS NOTIFY LIBXML2 FREETYPE
M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK2 X11

Important settings:
  value of $LC_MONETARY: fr_FR.UTF-8
  value of $LC_NUMERIC: fr_FR.UTF-8
  value of $LC_TIME: fr_FR.UTF-8
  value of $LANG: fr_FR.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: eww

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Contacting host: aljazeera.net:80
Quit
Making completion list... [2 times]

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired dired-loaddefs rfc822 mml
mml-sec epg epg-config mm-decode mm-bodies mm-encode mailabbrev
gmm-utils mailheader sendmail network-stream nsm starttls url-http tls
gnutls mail-parse rfc2231 rfc2047 rfc2045 ietf-drums url-gw url-cache
url-auth eww puny seq mm-url gnus gnus-ems nnheader mail-utils wid-edit
url-queue url url-proxy url-privacy url-expand url-methods url-history
url-cookie url-domsuf url-util url-parse auth-source cl-seq eieio
byte-opt bytecomp byte-compile cl-extra cconv eieio-core cl-macs gv
eieio-loaddefs gnus-util mm-util help-fns help-mode easymenu mail-prsvr
password-cache url-vars mailcap shr dom cl-loaddefs cl-lib subr-x pcase
browse-url format-spec time-date mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote inotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 8 196542 8585)
 (symbols 24 23712 0)
 (miscs 20 112 246)
 (strings 16 38292 8076)
 (string-bytes 1 1653802)
 (vectors 8 28616)
 (vector-slots 4 630813 16490)
 (floats 8 243 196)
 (intervals 28 7599 122)
 (buffers 520 16)
 (heap 1024 25127 830))

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Mohamed Hibti <mohamed.hibti@gmail.com> writes:

> There is a problem with the arabic composition (the letters are not
> displayed isolated and not assembled in a word). 

Do you have an example HTML file that displays this problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Mohamed Hibti]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

Thanks for this replay

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Mohamed Hibti <mohamed.hibti@gmail.com> writes:
>
>> There is a problem with the arabic composition (the letters are not
>> displayed isolated and not assembled in a word). 
>
> Do you have an example HTML file that displays this problem?


Any arabic website I have tried for instance http://www.aljazeera.net.
you can see the rendering in the attached png. The page title is however
well displayed !

Mohamed 



[-- Attachment #2: example.png --]
[-- Type: image/png, Size: 145014 bytes --]

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Mohamed Hibti <mohamed.hibti@gmail.com>
> Date: Wed, 24 Feb 2016 11:16:42 +0100
> Cc: , 22786@debbugs.gnu.org
> 
> >> There is a problem with the arabic composition (the letters are not
> >> displayed isolated and not assembled in a word). 
> >
> > Do you have an example HTML file that displays this problem?
> 
> Any arabic website I have tried for instance http://www.aljazeera.net.
> you can see the rendering in the attached png.

It doesn't happen for me, with that page, either in Emacs 25.1.50 or
25.0.91.  The Arabic shaping does work for me.

Do you have the same problem with the Arabic text in the buffer
created by "C-h H"?

The screenshot you attached indicates that complex script display is
not working for some reason.  Since your Emacs is built with libotf
and libm17n, the problem might be in old versions of these, so perhaps
rebuild with newer ones?  Or maybe try a different font?

bug#22786: 25.1.50; eww arabic rendering

[Mohamed Hibti]

Thanks Eli for this reply !
Eli Zaretskii <eliz@gnu.org> writes:

>> From: Mohamed Hibti <mohamed.hibti@gmail.com>
>> Date: Wed, 24 Feb 2016 11:16:42 +0100
>> Cc: , 22786@debbugs.gnu.org
>> 
>> >> There is a problem with the arabic composition (the letters are not
>> >> displayed isolated and not assembled in a word). 
>> >
>> > Do you have an example HTML file that displays this problem?
>> 
>> Any arabic website I have tried for instance http://www.aljazeera.net.
>> you can see the rendering in the attached png.
>
> It doesn't happen for me, with that page, either in Emacs 25.1.50 or
> 25.0.91.  The Arabic shaping does work for me.
>
> Do you have the same problem with the Arabic text in the buffer
> created by "C-h H"?

C-h H works well.
In non html buffers, it works too.

>
> The screenshot you attached indicates that complex script display is
> not working for some reason.  Since your Emacs is built with libotf
> and libm17n, the problem might be in old versions of these, so perhaps
> rebuild with newer ones?  Or maybe try a different font?

I will try your suggestions.

bug#22786: 25.1.50; eww arabic rendering

[Mohamed Hibti]

Eli Zaretskii <eliz@gnu.org> writes:

> The screenshot you attached indicates that complex script display is
> not working for some reason.  Since your Emacs is built with libotf
> and libm17n, the problem might be in old versions of these, so perhaps
> rebuild with newer ones?  Or maybe try a different font?

You are right Eli ! it worked with a different font.
For some reason, I should have typed "F" (eww-toggle-fonts). My eww
setting changed and  this prevented me from getting right complex script
display.


Thank you !!

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Mohamed Hibti <mohamed.hibti@gmail.com>
> Cc: larsi@gnus.org, 22786@debbugs.gnu.org, mohamed.hibti@gmail.com
> Date: Wed, 24 Feb 2016 19:24:17 +0100
> 
> > The screenshot you attached indicates that complex script display is
> > not working for some reason.  Since your Emacs is built with libotf
> > and libm17n, the problem might be in old versions of these, so perhaps
> > rebuild with newer ones?  Or maybe try a different font?
> 
> You are right Eli ! it worked with a different font.
> For some reason, I should have typed "F" (eww-toggle-fonts). My eww
> setting changed and  this prevented me from getting right complex script
> display.

Great, closing.  Thanks for testing.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Mohamed Hibti <mohamed.hibti@gmail.com> writes:

> Any arabic website I have tried for instance http://www.aljazeera.net.
> you can see the rendering in the attached png. The page title is however
> well displayed !

eww defaults `bidi-paragraph-direction' to `left-to-right', and depends
on web pages including the "dir" attribute to <HTML> to change that.
The Al Jazeera web site doesn't do that, so things look wrong.

Is there a simple heuristic we could use to determine that we're on a
page where we should flip the paragraph direction?  Some pages may have
a bit of RTL text without wanting to change the paragraph direction
(like most Wikipedia pages)...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Mohamed Hibti]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Mohamed Hibti <mohamed.hibti@gmail.com> writes:
>
>> Any arabic website I have tried for instance http://www.aljazeera.net.
>> you can see the rendering in the attached png. The page title is however
>> well displayed !
>
> eww defaults `bidi-paragraph-direction' to `left-to-right', and depends
> on web pages including the "dir" attribute to <HTML> to change that.
> The Al Jazeera web site doesn't do that, so things look wrong.
>
> Is there a simple heuristic we could use to determine that we're on a
> page where we should flip the paragraph direction?  Some pages may have
> a bit of RTL text without wanting to change the paragraph direction
> (like most Wikipedia pages)...


Unfortunaltely, I do not  write offten in arabic and i'm not familiar with
arabic html stuff !!  However, I tried different pages and some of them
as you said do not have specific indication to direction, some use
 (lang="ar" xml:lang="ar" dir="rtl"). 

For me toggling fonts in eww is a good solution. It worked well and have the
advantage of displaying good fonts.

Thanks for all,

Mohamed


P.S. Eli closed the bug.

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 25 Feb 2016 16:05:16 +1030
> Cc: 22786@debbugs.gnu.org
> 
> eww defaults `bidi-paragraph-direction' to `left-to-right', and depends
> on web pages including the "dir" attribute to <HTML> to change that.
> The Al Jazeera web site doesn't do that, so things look wrong.

(This, of course, is unrelated to the original problem.)

> Is there a simple heuristic we could use to determine that we're on a
> page where we should flip the paragraph direction?

The heuristic that we have happens automagically if
bidi-paragraph-direction is nil.  (It's not a heuristic, it's what the
Unicode Bidirectional Algorithm prescribes to do.)  Then each
paragraph gets its own direction computed on the fly.

> Some pages may have a bit of RTL text without wanting to change the
> paragraph direction (like most Wikipedia pages)...

Exactly.  Which is why we changed eww to its current default.

I think at this stage we should add to the eww menu an item that
allows to change the page direction, and let users override the
default if needed.  This should solve most, if not all, of the cases
where the default doesn't DTRT.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Is there a simple heuristic we could use to determine that we're on a
>> page where we should flip the paragraph direction?
>
> The heuristic that we have happens automagically if
> bidi-paragraph-direction is nil.  (It's not a heuristic, it's what the
> Unicode Bidirectional Algorithm prescribes to do.)  Then each
> paragraph gets its own direction computed on the fly.

They prescribe a heuristic.  :-)

>> Some pages may have a bit of RTL text without wanting to change the
>> paragraph direction (like most Wikipedia pages)...
>
> Exactly.  Which is why we changed eww to its current default.
>
> I think at this stage we should add to the eww menu an item that
> allows to change the page direction, and let users override the
> default if needed.  This should solve most, if not all, of the cases
> where the default doesn't DTRT.

Well, Firefox displays the web page correctly without Firefox users
having to do anything in particular, so presumably eww should be able to
do the same, I would have thought?

That is, eww should leave `bidi-paragraph-direction' to nil (which makes
the Al Jazeera web page display correctly), and then it...  shouldn't be
so eager to switch the direction of paragraphs to rtl just because a
section starts with some rtl text.  Or something.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mohamed.hibti@gmail.com,  22786@debbugs.gnu.org
> Date: Fri, 26 Feb 2016 16:15:38 +1030
> 
> > I think at this stage we should add to the eww menu an item that
> > allows to change the page direction, and let users override the
> > default if needed.  This should solve most, if not all, of the cases
> > where the default doesn't DTRT.
> 
> Well, Firefox displays the web page correctly without Firefox users
> having to do anything in particular, so presumably eww should be able to
> do the same, I would have thought?

EWW has a long way to go until it becomes a fair runner-up for
Firefox.  With all due respect.  E.g., our layout is nowhere as pretty
as Firefox, see how we display a typical Wikipedia page.

I think the catastrophic results of rendering the likes of "Other
languages" part of a Wikipedia page with Arabic the first language in
the list are much worse than an occasional need to switch the page
direction: the former affects _all_ Emacs users, while the latter
affects only those who frequently browse RTL pages.

> That is, eww should leave `bidi-paragraph-direction' to nil (which makes
> the Al Jazeera web page display correctly), and then it...  shouldn't be
> so eager to switch the direction of paragraphs to rtl just because a
> section starts with some rtl text.  Or something.

If you can come up with that magical "something", i.e. point out some
HTML tag or something else on the page that could be a hint for us not
to switch paragraph direction, then we can do that.  I will be happy
to help with the details.  But if there's no such hint, and all we
have is the text itself, then the UBA determines the paragraph
direction from the first strong directional character, and we then
keep that direction until the paragraph ends, which is indicated by an
empty line.  There's no way around this, it's how the bidirectional
display engine works.

Specifically, we need a way to distinguish between the list of
languages on a Wikipedia page from the Al Jazeera home page.  Can you
spot anything in the page sources that will allow such a distinction?

In any case, I think it will be good to add a "Switch page direction"
menu item to EWW on the emacs-25 branch.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> But if there's no such hint, and all we have is the text itself, then
> the UBA determines the paragraph direction from the first strong
> directional character, and we then keep that direction until the
> paragraph ends, which is indicated by an empty line.  There's no way
> around this, it's how the bidirectional display engine works.
>
> Specifically, we need a way to distinguish between the list of
> languages on a Wikipedia page from the Al Jazeera home page.  Can you
> spot anything in the page sources that will allow such a distinction?

Nope.  But since Firefox is able to do this, that means that their bidi
algo is doing something else than what Emacs' bidi algo is doing...

So we have demonstrated that there is a way around this.  :-)

> In any case, I think it will be good to add a "Switch page direction"
> menu item to EWW on the emacs-25 branch.

Good idea; I'll all that.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mohamed.hibti@gmail.com,  22786@debbugs.gnu.org
> Date: Sun, 28 Feb 2016 14:43:03 +1030
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But if there's no such hint, and all we have is the text itself, then
> > the UBA determines the paragraph direction from the first strong
> > directional character, and we then keep that direction until the
> > paragraph ends, which is indicated by an empty line.  There's no way
> > around this, it's how the bidirectional display engine works.
> >
> > Specifically, we need a way to distinguish between the list of
> > languages on a Wikipedia page from the Al Jazeera home page.  Can you
> > spot anything in the page sources that will allow such a distinction?
> 
> Nope.  But since Firefox is able to do this, that means that their bidi
> algo is doing something else than what Emacs' bidi algo is doing...
> 
> So we have demonstrated that there is a way around this.  :-)

The way around it is to support frames (or "side bars").  Then each
frame could have its own base paragraph direction.  AFAICT, eww
doesn't support that yet.

> > In any case, I think it will be good to add a "Switch page direction"
> > menu item to EWW on the emacs-25 branch.
> 
> Good idea; I'll all that.

Thanks.

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> Date: Sun, 28 Feb 2016 17:41:13 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 22786@debbugs.gnu.org, mohamed.hibti@gmail.com
> 
> > > In any case, I think it will be good to add a "Switch page direction"
> > > menu item to EWW on the emacs-25 branch.
> > 
> > Good idea; I'll all that.
> 
> Thanks.

Except that I think it shouldn't be a toggle, but a tristate: nil is
also a useful value for bidi-paragraph-direction in this context.

TIA

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Except that I think it shouldn't be a toggle, but a tristate: nil is
> also a useful value for bidi-paragraph-direction in this context.

But as an interactive function the nil value doesn't seem very useful.
eww defaults explicitly to left-to-right, so the user just needs
something to switch to right-to-left...  (And back again, if needed.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22786@debbugs.gnu.org,  mohamed.hibti@gmail.com
> Date: Mon, 29 Feb 2016 13:34:47 +1100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Except that I think it shouldn't be a toggle, but a tristate: nil is
> > also a useful value for bidi-paragraph-direction in this context.
> 
> But as an interactive function the nil value doesn't seem very useful.

The _result_ should be to have bidi-paragraph-direction bound to nil.

> eww defaults explicitly to left-to-right, so the user just needs
> something to switch to right-to-left...  (And back again, if needed.)

If you have a page whose paragraphs need different directions, neither
of the above 2 will DTRT, but nil will.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> eww defaults explicitly to left-to-right, so the user just needs
>> something to switch to right-to-left...  (And back again, if needed.)
>
> If you have a page whose paragraphs need different directions, neither
> of the above 2 will DTRT, but nil will.

Oh, that's true...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22786@debbugs.gnu.org,  mohamed.hibti@gmail.com
> Date: Mon, 29 Feb 2016 15:36:21 +1100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> eww defaults explicitly to left-to-right, so the user just needs
> >> something to switch to right-to-left...  (And back again, if needed.)
> >
> > If you have a page whose paragraphs need different directions, neither
> > of the above 2 will DTRT, but nil will.
> 
> Oh, that's true...

I suggest to call the 3rd state "auto" or "auto-detected".

Thanks.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I suggest to call the 3rd state "auto" or "auto-detected".

I've now done this in emacs-25.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Mohamed HIBTI]

[-- Attachment #1: Type: text/plain, Size: 394 bytes --]

Guys thank you so much for all this work !!
I will try as soon as possible.



2016-03-01 1:31 GMT+01:00 Lars Ingebrigtsen <larsi@gnus.org>:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> > I suggest to call the 3rd state "auto" or "auto-detected".
>
> I've now done this in emacs-25.
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>

[-- Attachment #2: Type: text/html, Size: 973 bytes --]

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The way around it is to support frames (or "side bars").  Then each
> frame could have its own base paragraph direction.  AFAICT, eww
> doesn't support that yet.

But would that help with the Wikipedia example?  There the paragraph
starts with Arabic text...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mohamed.hibti@gmail.com,  22786@debbugs.gnu.org
> Date: Mon, 29 Feb 2016 13:33:43 +1100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The way around it is to support frames (or "side bars").  Then each
> > frame could have its own base paragraph direction.  AFAICT, eww
> > doesn't support that yet.
> 
> But would that help with the Wikipedia example?  There the paragraph
> starts with Arabic text...

That paragraph is in a separate frame.  That frame could have a forced
left-to-right base direction.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That paragraph is in a separate frame.  That frame could have a forced
> left-to-right base direction.

I'm not sure what you mean by "frame" here.  DOM element?

Anyway, I looked at the Wikipedia source again, and it specifies "ltr"
as the direction.  And since shr respects those settings (now), perhaps
we should just remove the hardcoded left-to-right default in eww now,
and just let it be nil?  Then the aljazeera site would work
automatically.

There might be pages that render less well, though, but we'd be
following the Unicode recommendations (more)...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mohamed.hibti@gmail.com,  22786@debbugs.gnu.org
> Date: Mon, 29 Feb 2016 15:35:54 +1100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That paragraph is in a separate frame.  That frame could have a forced
> > left-to-right base direction.
> 
> I'm not sure what you mean by "frame" here.  DOM element?

Probably.  Forget it, I ws mistaken: there are no frames on that page.

> Anyway, I looked at the Wikipedia source again, and it specifies "ltr"
> as the direction.  And since shr respects those settings (now), perhaps
> we should just remove the hardcoded left-to-right default in eww now,
> and just let it be nil?  Then the aljazeera site would work
> automatically.
> 
> There might be pages that render less well, though, but we'd be
> following the Unicode recommendations (more)...

Removing the hardcoded value is probably a good idea (but maybe use it
as fallback if the HTML tag doesn't specify anything?).

However, note that support for this in shr is currently incomplete.
First, there's a 3rd value, "auto", which is unsupported -- it should
set bidi-paragraph-direction to nil.  Moreover, a document can use
several directives -- 'dir', 'bdi', and 'bdo' -- on the element level,
and that is entirely unsupported now.  What it means is that a page
that specifies special paragraph directions for some of its
paragraphs, and also mixes R2L and L2R text marked with 'bdi', will
not generally render correctly.

So I think for best results we should add support for the remaining
bidi directives.  Adding support for "dir=auto" in the HTML tag is
almost trivial.  To support the rest of the directives you need to add
bidirectional formatting control characters before and/or around the
text that is marked with these directives.  (If needed, I can provide
the details about the controls you need to insert in each case.)

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Removing the hardcoded value is probably a good idea (but maybe use it
> as fallback if the HTML tag doesn't specify anything?).

That's what it currently does.  I think I'm going to just have it
default to nil, and we'll see whether this makes things render badly in
practice.  My uninformed guess is that pages that mix scripts (like
Wikipedia) will explicitly say that the paragraph direction is
left-to-right, while pages with (mostly) only left-to-right (most pages)
and right-to-left (Al Jazeera) are the ones that leave out the spec, so
things should work out fine.

But we'll see.

> However, note that support for this in shr is currently incomplete.
> First, there's a 3rd value, "auto", which is unsupported -- it should
> set bidi-paragraph-direction to nil.

I've now added this to shr.

> So I think for best results we should add support for the remaining
> bidi directives.  Adding support for "dir=auto" in the HTML tag is
> almost trivial.  To support the rest of the directives you need to add
> bidirectional formatting control characters before and/or around the
> text that is marked with these directives.  (If needed, I can provide
> the details about the controls you need to insert in each case.)

I think I remember the control characters from past discussions.  But is
the dir attribute used much in practice?  I've tried to be very, very
restrictive in what features I add to the common paths in shr.  It's
already slow enough, and each new line of code in the common paths add
some slow down.  It's death by a thousand cuts.  :-) I don't oppose
adding support for this if it's really used a lot in the wild, but if
not, I'd rather not.

(The "dir" attribute can apply to (almost) any HTML element, so the code
to detect and react to it would go into `shr-descend', which is called
once for every single HTML node in the document.)

However, adding support for the <bdi> and <bdo> elements should be
trivial, and will cause no slowdown, so I'll add them now...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mohamed.hibti@gmail.com,  22786@debbugs.gnu.org
> Date: Tue, 01 Mar 2016 10:56:38 +1100
> 
> > So I think for best results we should add support for the remaining
> > bidi directives.  Adding support for "dir=auto" in the HTML tag is
> > almost trivial.  To support the rest of the directives you need to add
> > bidirectional formatting control characters before and/or around the
> > text that is marked with these directives.  (If needed, I can provide
> > the details about the controls you need to insert in each case.)
> 
> I think I remember the control characters from past discussions.  But is
> the dir attribute used much in practice?  I've tried to be very, very
> restrictive in what features I add to the common paths in shr.  It's
> already slow enough, and each new line of code in the common paths add
> some slow down.  It's death by a thousand cuts.  :-) I don't oppose
> adding support for this if it's really used a lot in the wild, but if
> not, I'd rather not.

I don't have any statistics.  I would expect it to be used in pages
that show paragraphs of different direction.

> (The "dir" attribute can apply to (almost) any HTML element, so the code
> to detect and react to it would go into `shr-descend', which is called
> once for every single HTML node in the document.)

Can you add its support to the <p> element?  I think this will go a
long way towards supporting the majority of use cases.

Thanks.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Can you add its support to the <p> element?  I think this will go a
> long way towards supporting the majority of use cases.

<p> isn't used a lot these days.  It's mostly <div> and <span> with CSS
properties, so I'm not sure that helps very much...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mohamed.hibti@gmail.com,  22786@debbugs.gnu.org
> Date: Wed, 02 Mar 2016 17:01:46 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Can you add its support to the <p> element?  I think this will go a
> > long way towards supporting the majority of use cases.
> 
> <p> isn't used a lot these days.  It's mostly <div> and <span> with CSS
> properties

OK, then make that <div> and <span>.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> So I think for best results we should add support for the remaining
> bidi directives.  Adding support for "dir=auto" in the HTML tag is
> almost trivial.  To support the rest of the directives you need to add
> bidirectional formatting control characters before and/or around the
> text that is marked with these directives.  (If needed, I can provide
> the details about the controls you need to insert in each case.)

Actually, I think I need help, anyway.  :-)

Adding support for BDO was easy enough -- I slap LRO/RLO at the start en
PDF at the end of the phrase.  This works fine:

<p>This goes <bdo dir=rtl>the wrong way</bdo>, doesn't it?

However, I'm not quite sure how the BDI is supposed to work.  With this
example:

<p><bdo dir=ltr>Here's something with <bdi>العربيّ</bdi> lri.</bdo>

I would have expected that slapping FSI/PDI pairs around the Arabic text
would isolate it from the LRO that the <bdo> introduced.  But it
doesn't.  Should it?  Using LRI/RLI instead works, but then I would have
to keep track of what direction the text is already in?  So I think
there's something I don't quite understand here.

Or perhaps I should have used LRM/RLM instead of LRO/RLO?

Anyway, this reminds me: Do we have a literal char syntax for these
control characters?  I know about ?\u2068, but that's not very...
clear...  Having something like ?\ucLRM (or... something) would make the
code a lot easier to read.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mohamed.hibti@gmail.com,  22786@debbugs.gnu.org
> Date: Tue, 01 Mar 2016 11:26:34 +1100
> 
> However, I'm not quite sure how the BDI is supposed to work.  With this
> example:
> 
> <p><bdo dir=ltr>Here's something with <bdi>العربيّ</bdi> lri.</bdo>
> 
> I would have expected that slapping FSI/PDI pairs around the Arabic text
> would isolate it from the LRO that the <bdo> introduced.  But it
> doesn't.  Should it?  Using LRI/RLI instead works, but then I would have
> to keep track of what direction the text is already in?

No, FSI..PDI is TRT in this case (unless you have <bdi dir=rtl> etc.).

There was a subtle bug in bidi.c which affected this case, now fixed
on the emacs-25 branch.  With that, you should see the expected result
in the above example.

> Anyway, this reminds me: Do we have a literal char syntax for these
> control characters?

Do we have a literal syntax for _any_ character?

Btw, these controls display as thin spaces on GUI frames, and as just
spaces on a TTY; for best results you could cover each control with an
invisible text property, which would make them entirely invisible.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> There was a subtle bug in bidi.c which affected this case, now fixed
> on the emacs-25 branch.  With that, you should see the expected result
> in the above example.

Great.

>> Anyway, this reminds me: Do we have a literal char syntax for these
>> control characters?
>
> Do we have a literal syntax for _any_ character?

We have ?\s (presumably because people found ?  too annoyingly
fragile).  Thinking about it a bit more, I think it would be kinda nice
to have a literal syntax for all Unicode characters.  Something like...
er...  ?\ucLEFT-TO-RIGHT-OVERRIDE and \?ucPILE-OF-POO?  We have that
table already, so perhaps that's feasible?

But it's, of course, more important for the control characters, because
they're just incomprehensible if you just ?💩 them...

> Btw, these controls display as thin spaces on GUI frames, and as just
> spaces on a TTY; for best results you could cover each control with an
> invisible text property, which would make them entirely invisible.

Hm...  perhaps it's good to (vaguely) show that they're there?  I'm not
sure.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22786: 25.1.50; eww arabic rendering

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: mohamed.hibti@gmail.com,  22786@debbugs.gnu.org
> Date: Wed, 02 Mar 2016 17:08:40 +0000
> 
> > Do we have a literal syntax for _any_ character?
> 
> We have ?\s (presumably because people found ?  too annoyingly
> fragile).  Thinking about it a bit more, I think it would be kinda nice
> to have a literal syntax for all Unicode characters.

We have categories, so you can use \cX where X is a single letter
which denotes a category.  Se "M-x describe-categories".  This is
similar to ?\s, but the available categories are rather ad-hoc.

> Something like...  er...  ?\ucLEFT-TO-RIGHT-OVERRIDE and
> \?ucPILE-OF-POO?  We have that table already, so perhaps that's
> feasible?

It's feasible, but is it really useful?  How is it different from just
typing the character in the first place?

> But it's, of course, more important for the control characters, because
> they're just incomprehensible if you just ?💩 them...

"C-x =" usually does the job for me.

> > Btw, these controls display as thin spaces on GUI frames, and as just
> > spaces on a TTY; for best results you could cover each control with an
> > invisible text property, which would make them entirely invisible.
> 
> Hm...  perhaps it's good to (vaguely) show that they're there?  I'm not
> sure.

Could be.  I think by default they shouldn't show, but we could have
an option to reveal them.

bug#22786: 25.1.50; eww arabic rendering

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Something like...  er...  ?\ucLEFT-TO-RIGHT-OVERRIDE and
>> \?ucPILE-OF-POO?  We have that table already, so perhaps that's
>> feasible?
>
> It's feasible, but is it really useful?  How is it different from just
> typing the character in the first place?
>
>> But it's, of course, more important for the control characters, because
>> they're just incomprehensible if you just ?💩 them...
>
> "C-x =" usually does the job for me.

I think perhaps we should take this part of the discussion to
emacs-devel.  I'll follow up there.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no