From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Itai Berli Newsgroups: gmane.emacs.bugs Subject: bug#27525: 25.1; Line wrapping of bidi paragraphs Date: Wed, 19 Jul 2017 15:59:14 +0300 Message-ID: References: <8337abobuz.fsf@gnu.org> <87eftpa30a.fsf@blei.turtle-trading.net> <83a84djweb.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="94eb2c1b47ce5fc6bd0554ab35a5" X-Trace: blaine.gmane.org 1500469284 12771 195.159.176.226 (19 Jul 2017 13:01:24 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Wed, 19 Jul 2017 13:01:24 +0000 (UTC) To: 27525@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Wed Jul 19 15:01:16 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXobK-0002rT-BV for geb-bug-gnu-emacs@m.gmane.org; Wed, 19 Jul 2017 15:01:14 +0200 Original-Received: from localhost ([::1]:33396 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dXobP-0001GU-SR for geb-bug-gnu-emacs@m.gmane.org; Wed, 19 Jul 2017 09:01:19 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:34007) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dXobD-0001Fb-27 for bug-gnu-emacs@gnu.org; Wed, 19 Jul 2017 09:01:13 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dXob9-0006u5-3i for bug-gnu-emacs@gnu.org; Wed, 19 Jul 2017 09:01:07 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:44595) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dXob8-0006tv-Ur for bug-gnu-emacs@gnu.org; Wed, 19 Jul 2017 09:01:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dXob8-0001f5-GZ for bug-gnu-emacs@gnu.org; Wed, 19 Jul 2017 09:01:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Itai Berli Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 19 Jul 2017 13:01:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 27525 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 27525-submit@debbugs.gnu.org id=B27525.15004692046260 (code B ref 27525); Wed, 19 Jul 2017 13:01:02 +0000 Original-Received: (at 27525) by debbugs.gnu.org; 19 Jul 2017 13:00:04 +0000 Original-Received: from localhost ([127.0.0.1]:47272 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXoaB-0001cp-Li for submit@debbugs.gnu.org; Wed, 19 Jul 2017 09:00:04 -0400 Original-Received: from mail-wr0-f175.google.com ([209.85.128.175]:33069) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXoa9-0001bp-9E for 27525@debbugs.gnu.org; Wed, 19 Jul 2017 09:00:02 -0400 Original-Received: by mail-wr0-f175.google.com with SMTP id v105so28091825wrb.0 for <27525@debbugs.gnu.org>; Wed, 19 Jul 2017 06:00:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=nnWM+oMYDs5eQRfx3C/xsLhoEwJn5qmJdj1XKApKUGs=; b=P/58dzt4L3RkomFkthzVcIogQwGWARiJrI7y/uRNBW8h3jPUb893mrEAZRKeAaB/BS +tWLNP/B1bU+xAXyrvzXfxGLzKpnoRjUMkgbAgtF0kjphFOXrD3bh3sU5sfqqIHP0aJu Ac89IQaE3fFOmjJE7c5fl2XRc6XZxCxz//yJEdxxF+4y/jzBKhO27E7CQhErcde4ViV+ 02wFfrcaWedbaN3VP7NU9MB4jL0EfHGf9lKQMtYX/yDQL/qii/ebGBGfxloB5dEm5uDJ J2C8LTNsKF6QqjUTyPSGjkRk4Hjj+kLsN5+oQ5ORVtR/C/uZn0Q3hggGLRVEzQ0epQUD uWog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=nnWM+oMYDs5eQRfx3C/xsLhoEwJn5qmJdj1XKApKUGs=; b=AQKMIqK+ONKU2KLDruf9MZo9S7BefHw6vaU0uc3cHjOJlV7UDrc4McjIPx0jaSQy8p WQDPXg7d4eF6IwAbGGUbzfVZDMqXACTBEFHQRZ21ZVudQRhxYuYBrYwwN5IeZbFKNJwb osvlC5bR8oHSX0U3KD+nWPM/aR1tmFTh/Mn5/MpRLy/BEqt/g2bXzamxpWVct5Y41KQ5 7kfU/64RJ+aSAYfCiO7YN7iYIZ4SJyj0bOLUm3SoEOg2PqUtP4Pm5w8cYt44FAC0Il7y xmgUf67qgsUbMYI0mFK1Fi2sP/5E+3DA7/VUAhU1qqYyDL3h+sR+qqP4oB9olxXRIbz6 tQaA== X-Gm-Message-State: AIVw111XVvcUF4sGVR3u9i5+DStYkUgDiM/NfRH62ORB9Q2Xa/Pv0zD7 zbxD53FY5qU2w8qAIsaF68OcswFyEGe6uKg= X-Received: by 10.223.170.150 with SMTP id h22mr4516857wrc.140.1500469195154; Wed, 19 Jul 2017 05:59:55 -0700 (PDT) Original-Received: by 10.28.197.9 with HTTP; Wed, 19 Jul 2017 05:59:14 -0700 (PDT) In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:134741 Archived-At: --94eb2c1b47ce5fc6bd0554ab35a5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable And in case you counter that Emacs takes advantage of the "higher-level protocol" clause, this clause doesn't apply to paragraph 3.4, which Emacs violates. But don't take my word for it. I contacted Mr. Aharon Lanin in the matter. Mr. Lanin is a senior software engineer at Google Tel Aviv as well as one of the three editors of the Unicode Bidirectional Algorithm from version 6.3.0 till the latest one, v. 10.0.0. I presented him with the following screenshot of a bidi paragraph in Emacs (it's the same screenshot as I posted originally in the present ticket): http://imgur.com/Bckn7zP I list below an excerp from our conversation, which Mr. Lanin has given me permission to quote. --- EXCERPT BEGIN --- ***Me***: Just to be clear, are the following statements correct for the Unicode Standard v. 8.0.0 and above? (I'm mentioning v. 8.0.0 because this is the version that Emacs claims conformance to.) 1. The way Emacs handles line wrapping of bidi paragraphs does not satisfy section 3.4 of the Unicode Bidirectional Algorithm. There are no provisions for higher-level protocol interpretation of this section. 2. If a candidate implementation of the Unicode Bidirectional Algorithm doesn't satisfy section 3.4, it does not conform to the Unicode Bidirectional Algorithm. 3. If a candidate implementation of the Unicode Standard does not conform to the Unicode Bidirectional Algorithm (of the same version), it does not conform to the Unicode Standard. ***Lanin*** I think so, but I am a programmer, not a lawyer :-) --- EXCERPT END --- The Emacs manual and all official Emacs publications should make it clear that Emacs does not conform to the Unicode Standard. Anything else is simply not true, and is a deliberate misleading. On Wed, Jul 19, 2017 at 11:50 AM, Itai Berli wrote: > Eli, in different bug report, namely 27526, I recently wrote the followin= g > remark: > > > the line-wrapping bug is still a major annoyance, at best, and until it > is fixed, Emacs cannot claim to be Unicode compliant. > > to which you replied: > > > I disagree, as I already said many times. > > You do agree, though, that Emacs does not conform to the Unicode > Bidirectional Algorithm as specified in the Unicode Standard Annex #9. > After all, the following paragraph appears in the bidi code itself ( > http://git.savannah.gnu.org/cgit/emacs.git/tree/src/bidi.c): > > Note that, because reordering is implemented below the level in > xdisp.c that breaks glyphs into screen lines, we are violating > paragraph 3.4 of UAX#9. which mandates that line breaking shall be > done before reordering each screen line separately. > > So the only thing you disagree with me is that non-conformance to the > Unicode Bidirectional Algorithm is tantamount to non-conformance to the > Unicode Standard. Well, this disagreement is easily settled by reading > article C12 'Bidirectional Text' of section 3.2 'Conformance Requirements= ' > of the Unicode Standard: > > A process that displays text containing supported right-to-left character= s > or embedding codes shall display all visible representations of character= s > (excluding format characters) in the same order as if the Bidirectional > Algorithm had been applied to the text, unless tailored by a higher-level > protocol as permitted by the specification. > > * The Bidirectional Algorithm is specified in Unicode Standard Annex #9, > =E2=80=9CUni- code Bidirectional Algorithm.=E2=80=9D > > > On Sun, Jul 9, 2017 at 9:30 PM, Eli Zaretskii wrote: > >> > From: Benjamin Riefenstahl >> > Cc: 27525@debbugs.gnu.org, itai.berli@gmail.com >> > Date: Sun, 09 Jul 2017 20:17:41 +0200 >> > >> > >> From: Richard Stallman >> > >> It might be good if Emacs could refill lines automatically the way >> > >> some other ediors do. >> > >> > Eli Zaretskii writes: >> > > We already have that: "M-x visual-line-mode RET". >> > >> > JFTR, even that does not help in this case. With visual-line-mode the >> > order of the lines is still wrong with the text that the OP gave. >> >> Of course. It isn't supposed to help. From the POV of the display >> engine, visual-line-mode is just a fancy kind of producing >> continuation lines, so all the problems you see with continued lines >> will still be there in visual-line-mode. >> > > --94eb2c1b47ce5fc6bd0554ab35a5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
And in case you counter that Emacs takes advantage of the = "higher-level protocol" clause, this clause doesn't apply to = paragraph 3.4, which Emacs violates.

But don't take = my word for it. I contacted Mr. Aharon Lanin in the matter. Mr. Lanin is a = senior software engineer at Google Tel Aviv as well as one of the three edi= tors of the Unicode Bidirectional Algorithm from version 6.3.0 till the lat= est one, v. 10.0.0. I presented him with the following screenshot of a bidi= paragraph in Emacs (it's the same screenshot as I posted originally in= the present ticket):=C2=A0http://imgur.com/Bckn7zP

I list below an excerp from our conversation, = which Mr. Lanin has given me permission to quote.

--- EXCERPT BEGIN ---

***Me***:=C2=A0= Just to be clear,=C2=A0are the follow= ing statements correct for the Unicode Standard v. 8.0.0 and above? (I'= m mentioning v. 8.0.0 because this is the version that Emacs claims conform= ance to.)

1. The way Emacs handles line wrapping of bidi paragr= aphs does not satisfy section 3.4 of the Unicode Bidirectional Algorithm. There are no provisions for higher-level pr= otocol interpretation of this section.

2. If a candidate implement= ation of the Unicode Bidirectional Algorithm doesn't satisfy section 3.= 4, it does not conform to the Unicode Bidirectional Algorithm.
=

3. = If a candidate implementation of the Unicode Standard does not conform to t= he Unicode Bidirectional Algorithm (of the same version), it does not confo= rm to the Unicode Standard.

***Lanin***=C2=A0I think so, but I am a programmer, not a lawyer :-)

--- EXCERPT END ---<= /span>

The Emacs manual and all official Emacs publications should make it cl= ear that Emacs does not conform to the Unicode Standard. Anything else is s= imply not true, and is a deliberate misleading.

On Wed, Jul 19, 2017 at 11= :50 AM, Itai Berli <itai.berli@gmail.com> wrote:
Eli, in different bug report, na= mely 27526, I recently wrote the following remark:

> the=C2=A0line-wrapping=C2=A0bug is still a major=C2=A0annoyance, at best, and until it is fixed, Emacs cannot claim to be Un= icode compliant.

to which you repl= ied:

>=C2=A0I = disagree, as I already said many times.

You= do agree, though, that Emacs does not conform to the Unicode Bidirectional= Algorithm as specified in the Unicode Standard Annex #9. After all, the fo= llowing paragraph appears in the bidi code itself (http://git= .savannah.gnu.org/cgit/emacs.git/tree/src/bidi.c):
= 
   Note that, because reordering is implemented below the level =
in
   xdisp.c that breaks glyphs into screen lines, we are violating
   paragraph 3.4 of UAX#9. which mandates that line breaking shall be
   done before reordering each screen line separately.
So the only thing you disagree with me is that non-conformance to the Un= icode Bidirectional Algorithm is tantamount to non-conformance to the Unico= de Standard. Well, this disagreement is easily settled by reading article C= 12 'Bidirectional Text' of section 3.2 'Conformance Requirement= s' of the Unicode Standard:

A process that displays text containing supported right-t= o-left characters or embedding codes shall display all visible representati= ons of characters (excluding format characters) in the same order as if the= Bidirectional Algorithm had been applied to the text, unless tailored by a= higher-level protocol as permitted by the specification.
=
* The Bidirectional Algorithm is specified in Unicode St= andard Annex #9, =E2=80=9CUni- code Bidirectional Algorithm.=E2=80=9D


On Sun, Jul 9, 2017 at 9:30 PM,= Eli Zaretskii <eliz@gnu.org> wrote:
> From: Benjamin Riefenstahl <b.riefenstahl@turtl= e-trading.net>
> Cc: 27525@d= ebbugs.gnu.org,=C2=A0 itai.berli@gmail.com
> Date: Sun, 09 Jul 2017 20:17:41 +0200
>
> >> From: Richard Stallman <rms@gnu.org>
> >> It might be good if Emacs could refill lines automatically th= e way
> >> some other ediors do.
>
> Eli Zaretskii writes:
> > We already have that: "M-x visual-line-mode RET".
>
> JFTR, even that does not help in this case.=C2=A0 With visual-line-mod= e the
> order of the lines is still wrong with the text that the OP gave.

Of course.=C2=A0 It isn't supposed to help.=C2=A0 From the POV o= f the display
engine, visual-line-mode is just a fancy kind of producing
continuation lines, so all the problems you see with continued lines
will still be there in visual-line-mode.


--94eb2c1b47ce5fc6bd0554ab35a5--