From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#27526: 25.1; Nonconformance to Unicode bidirectionality algorithm due to paragraph separator Date: Tue, 04 Jul 2017 18:03:58 +0300 Message-ID: <83inj8nt0h.fsf@gnu.org> References: Reply-To: Eli Zaretskii NNTP-Posting-Host: blaine.gmane.org X-Trace: blaine.gmane.org 1499180719 20041 195.159.176.226 (4 Jul 2017 15:05:19 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 4 Jul 2017 15:05:19 +0000 (UTC) Cc: 27526@debbugs.gnu.org To: Itai Berli Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Jul 04 17:05:15 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSPO0-0004gd-97 for geb-bug-gnu-emacs@m.gmane.org; Tue, 04 Jul 2017 17:05:08 +0200 Original-Received: from localhost ([::1]:41863 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dSPO5-0005XA-H8 for geb-bug-gnu-emacs@m.gmane.org; Tue, 04 Jul 2017 11:05:13 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:45120) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dSPNy-0005VG-Up for bug-gnu-emacs@gnu.org; Tue, 04 Jul 2017 11:05:08 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dSPNu-0002m1-3V for bug-gnu-emacs@gnu.org; Tue, 04 Jul 2017 11:05:06 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:49623) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dSPNt-0002lr-Vv for bug-gnu-emacs@gnu.org; Tue, 04 Jul 2017 11:05:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dSPNt-0005BO-LY for bug-gnu-emacs@gnu.org; Tue, 04 Jul 2017 11:05:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 04 Jul 2017 15:05:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 27526 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 27526-submit@debbugs.gnu.org id=B27526.149918065319831 (code B ref 27526); Tue, 04 Jul 2017 15:05:01 +0000 Original-Received: (at 27526) by debbugs.gnu.org; 4 Jul 2017 15:04:13 +0000 Original-Received: from localhost ([127.0.0.1]:52293 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSPN7-00059L-D3 for submit@debbugs.gnu.org; Tue, 04 Jul 2017 11:04:13 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:53989) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSPN5-000595-9w for 27526@debbugs.gnu.org; Tue, 04 Jul 2017 11:04:11 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dSPMw-0002Qn-T4 for 27526@debbugs.gnu.org; Tue, 04 Jul 2017 11:04:06 -0400 Original-Received: from fencepost.gnu.org ([2001:4830:134:3::e]:48447) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dSPMw-0002Qh-Pg; Tue, 04 Jul 2017 11:04:02 -0400 Original-Received: from 84.94.185.246.cable.012.net.il ([84.94.185.246]:1917 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dSPMw-0005Md-2m; Tue, 04 Jul 2017 11:04:02 -0400 In-reply-to: (message from Itai Berli on Tue, 4 Jul 2017 13:42:19 +0300) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:134171 Archived-At: > From: Itai Berli > Date: Tue, 4 Jul 2017 13:42:19 +0300 > > I'd like to add another reason why this behavior is problematic: it breaks interoperability with other plain text > editors, since the text will not be displayed the same way. Consider, for instance, the very same plain text file > in GEdit: http://imgur.com/Iw4yrdQ > in Emacs: http://imgur.com/7kfWseE As I already explained, the behavior of GEdit is unacceptable for Emacs, because most modes derived from Text mode tend to deal with buffers where lines are broken by newlines, so potentially switching paragraph direction just because a newline happens to be there would have devastating effect on the text as displayed. This is perhaps in contrast with other editors and word-processors which mostly deal with long lines without hard newlines. That's why the notion of paragraph in Emacs's UBA implementation was chosen to fit the traditional Emacs definition of paragraph in text-mode and its derivatives. > Finally, the question of whether Emacs behavior is consistent with the UBA specifications is debatable, since > when UBA section 3 states "Paragraphs may also be determined by higher-level protocols" the question is > what exactly the "also" means: is it that the higher-level protocols (HLP) can decide that a newline character is > not a paragraph boundary, as Emacs does, or is it that the HLP can only declare paragraph boundaries in > addition to paragraph separator characters? It is clear from the context and the example following the above sentence that "also" doesn't mean "in addition". However, the main issue is not the paragraph boundary, the main issue is how the base direction of the paragraph is determined. Because no matter where the paragraph boundary is, if the base direction is not recalculated there, then the fact that the boundary is there doesn't matter. >From Section 4.3 Higher-Level Protocols of the UAX#9: HL1. Override P3, and set the paragraph embedding level explicitly. This does not apply when deciding how to treat FSI in rule X5c. . A higher-level protocol may set any paragraph level. This can be done on the basis of the context, such as on a table cell, paragraph, document, or system level. (P2 may be skipped if P3 is overridden). [...] . A higher-level protocol may apply rules equivalent to P2 and P3 but default to level 1 (RTL) rather than 0 (LTR) to match overall RTL context. . A higher-level protocol may use an entirely different algorithm that heuristically auto-detects the paragraph embedding level based on the paragraph text and its context. For example, it could base it on whether there are more RTL characters in the text than LTR. As another example, when the paragraph contains no strong characters, its direction could be determined by the levels of the paragraphs before and after. And Section 3.3.1, which describes the P1, P2, and P3 paragraph-level rules, says: Whenever a higher-level protocol specifies the paragraph level, rules P2 and P3 may be overridden: see HL1. So an application is allowed to override _all_ of the paragraph-level rules, and do what suits it best. And based on some non-negligible experience with bidi-aware applications, I submit that an application that does _not_ employ some higher-level protocol for base paragraph direction will violate user expectations when working with plain text. E.g., try reading in MS Outlook an unformatted text message which has a lot of RTL text mixed with LTR. It's unreadable; I always copy/paste it into Emacs, and only then I'm able to read it.