From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Itai Berli Newsgroups: gmane.emacs.bugs Subject: bug#27544: 25.1; Visualization of Unicode bidirectional marks Date: Sat, 1 Jul 2017 12:58:28 +0300 Message-ID: NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Trace: blaine.gmane.org 1498903221 20328 195.159.176.226 (1 Jul 2017 10:00:21 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sat, 1 Jul 2017 10:00:21 +0000 (UTC) To: 27544@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Sat Jul 01 12:00:15 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dRFCG-0004mu-7o for geb-bug-gnu-emacs@m.gmane.org; Sat, 01 Jul 2017 12:00:12 +0200 Original-Received: from localhost ([::1]:53604 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dRFCL-0007Jt-Gt for geb-bug-gnu-emacs@m.gmane.org; Sat, 01 Jul 2017 06:00:17 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:44272) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dRFCB-0007FJ-Db for bug-gnu-emacs@gnu.org; Sat, 01 Jul 2017 06:00:12 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dRFC6-000152-VC for bug-gnu-emacs@gnu.org; Sat, 01 Jul 2017 06:00:07 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:44234) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dRFC6-00014g-RC for bug-gnu-emacs@gnu.org; Sat, 01 Jul 2017 06:00:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dRFC6-0002E4-Jt for bug-gnu-emacs@gnu.org; Sat, 01 Jul 2017 06:00:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Itai Berli Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 01 Jul 2017 10:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 27544 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.14989031638471 (code B ref -1); Sat, 01 Jul 2017 10:00:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 1 Jul 2017 09:59:23 +0000 Original-Received: from localhost ([127.0.0.1]:46911 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dRFBT-0002CZ-Cr for submit@debbugs.gnu.org; Sat, 01 Jul 2017 05:59:23 -0400 Original-Received: from eggs.gnu.org ([208.118.235.92]:52878) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dRFBR-0002CJ-Fe for submit@debbugs.gnu.org; Sat, 01 Jul 2017 05:59:21 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dRFBK-0007y3-P3 for submit@debbugs.gnu.org; Sat, 01 Jul 2017 05:59:16 -0400 Original-Received: from lists.gnu.org ([2001:4830:134:3::11]:50815) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dRFBK-0007xl-M7 for submit@debbugs.gnu.org; Sat, 01 Jul 2017 05:59:14 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:43835) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dRFBI-00074q-RF for bug-gnu-emacs@gnu.org; Sat, 01 Jul 2017 05:59:14 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dRFBH-0007qE-7f for bug-gnu-emacs@gnu.org; Sat, 01 Jul 2017 05:59:12 -0400 Original-Received: from mail-vk0-x233.google.com ([2607:f8b0:400c:c05::233]:33721) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dRFBH-0007m1-1i for bug-gnu-emacs@gnu.org; Sat, 01 Jul 2017 05:59:11 -0400 Original-Received: by mail-vk0-x233.google.com with SMTP id r126so77512027vkg.0 for ; Sat, 01 Jul 2017 02:59:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=hfuw7C5fOF/MCErAUlrKmWnnUGdh4FIihYooESLdk+Y=; b=WSpcAsdKvCMhxFGO71LbBAgyzkL0wd90QRd6MqsCXmqBUTvRagbvG2t5HIFPp+UCOW BTyt8NfniukGrGxf9Z9Aov0lIfaDHWdDVc0HbuNQsLhOduo7Tafi9lBlep+qFG0l11P/ wLnjOTGB5xyz08SiI0vquAmn3anDULj0IIyk/D8pBZcvmAuba0rbvgSjRslaIVd/YHa/ bsnVeqqqnjuAx0/fchQyuo0CUF844KMPZpfMN/TlRncZ373EmK/pQYBSzAtbzRB5P6g5 aW12qGgfRemXPMV1vLeXHYkeiWOtSOcTX0erwpt+jqArtTFFDbSmOaKb1dyTNzlp+rCQ fK+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=hfuw7C5fOF/MCErAUlrKmWnnUGdh4FIihYooESLdk+Y=; b=WZFr0QAEXyHltQhHVcuDea9RDvSzmvbJPjI0dMlcJG8HsQyhk8NKkVx8xhdmj4vPVB WKg+xuBpmjoygJ+w1qf+1q0iktpuzUxKvniSTmBcx2bl2wxC68GQiW1QEGB9S0wRe1Kt ypTzX+muRM69vspbRMQ8gl3FV9IaDogWE/YX+eIsLOOql7QA7laEVZf2Bjj8OWGIxgjy P1dYErzyvXA7fJZdp+mo2Z/ib3vudSMwee+mQDiBD4ciSHw2GpnJ1R86BZlrgFnhUkbY ENN2uSxXHE6g1BcTCXZoD4vIMMG9YiSiNRrxyl8ycNnvVa23EwK9i69IcnuVNgR8DFN/ yvAw== X-Gm-Message-State: AKS2vOx74s7R7shhKCtkkWxLfWgJpRD+8//QUrT47kB2TM/NemPDewCw TEjM0a1/dNaWwlqJ3xr1jCbVWPguwh3dbd4= X-Received: by 10.31.138.135 with SMTP id m129mr13150123vkd.84.1498903148603; Sat, 01 Jul 2017 02:59:08 -0700 (PDT) Original-Received: by 10.176.70.85 with HTTP; Sat, 1 Jul 2017 02:58:28 -0700 (PDT) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:134064 Archived-At: Emacs supports 12 Unicode bidirectional marks (ALM, RLM, LRM, LRE, RLE, LRO, RLO, PDF, FSI, LRI, RLI, and PDI), each of which displays as a very thin space. This raises two problems. 1. On the one hand, the fact that these inherently invisible marks manifest, by default, as thin spaces undermines attempts at precise alignment and positioning. Moreover, in the case of LRM, RLM and ALM, this behavior contradicts explicit directions given in the Unicode Bidirectional Algorithm 8.0.0 specifications (section 2.6 Implicit Directional Marks): > they do not appear in the display (To my understanding, this is meant to apply to all bidi marks, even if only stated explicitly for LRM, RLM and ALM.) 2. On the other hand, the fact that these spaces are so thin as to be barely noticeable, and the fact that they are indistinguishable from one another makes it difficult to debug and resolve strange and/or erroneous behavior that can happen in a bidi document, an example of which is given below. The solution to both problems is to make the bidi marks visible in `whitespace` mode only, and to give them glyphs that are (a) easy to notice, (b) distinguishable from other whitespace visualization glyphs, (c) distinct from one another. The following example exhibits strange behavior that can arise due to the use of bidi marks. This behavior is difficult to debug without visualizing the bidi marks. Consider the following paragraph. ILLUSTRATION #1: An English sentence that is formatted from right to left. http://imgur.com/is1OBtM The paragraph is entirely in English, then why is it formatted from right to left? Without visible bidi marks, it's hard to tell; however a savvy Unicode-aware person would realize that this must indicate the presence of a Right-To-Left Mark (U+200F). Therefore, if we position the cursor at the beginning of the paragraph (`C-a`), and delete the following character (`C-d`), the sentence should display normally. ILLUSTRATION #2: Deleting the first Right-To-Left mark at the beginning of the paragraph has no effect. http://imgur.com/eBVpdZA Against our expectations, nothing appears to have changed. There must be another Right-To-Left mark at the beginning of the paragraph. Let's delete it as well. (`C-d`) ILLUSTRATION #3: Deleting the second Right-To-Left Mark left-aligns the paragraph, but leave the comma misplaced. http://imgur.com/Klj3lZC The paragraph is now aligned to the left, as it should, and everything looks normal, except for the comma, which appears in the beginning of the paragraph. But this can be easily remedied: let's delete the comma and then retype it in its proper place. We position the cursor at the beginning of the paragraph (`C-a`) and delete the following character (`C-d`). ILLUSTRATION #4: After trying to delete the comma, the paragraph is finally displayed correctly. http://imgur.com/3w73MxM Instead of deleting the comma, this has shifted the comma to it's correct position. If we were able to visualize the whitespace, we would have realized from the beginning that the sequence of characters in this paragraph was, from left to right: RTL-RTL-RLO-RLO-H-e-l-l-o-PDO-,-PDO-SPACE-w-o-r-l-d-! Thus, our first three actions removed the first three characters, leaving us with: RLO-H-e-l-l-o-PDO-,-PDO-SPACE-w-o-r-l-d-! We now realize that even the final, correct form, is in fact littered with bidi errors and potential landmines! In GNU Emacs 25.1.1 (x86_64-apple-darwin13.4.0, NS appkit-1265.21 Version 10.9.5 (Build 13F1911)) of 2016-09-21 built on builder10-9.porkrind.org Windowing system distributor 'Apple', version 10.3.1504 Configured using: 'configure --with-ns '--enable-locallisppath=/Library/Application Support/Emacs/${version}/site-lisp:/Library/Application Support/Emacs/site-lisp' --with-modules' Configured features: NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES Important settings: value of $LANG: en_US.UTF-8 locale-coding-system: utf-8-unix Major mode: TeX/P Minor modes in effect: diff-auto-refine-mode: t TeX-PDF-mode: t ivy-mode: t shell-dirtrack-mode: t projectile-mode: t helm-descbinds-mode: t async-bytecomp-package-mode: t tooltip-mode: t global-eldoc-mode: t electric-indent-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t column-number-mode: t line-number-mode: t transient-mark-mode: t Recent messages: Applying style hooks... done Mark set C-> is undefined Mark set [2 times] Saving file /Users/itaiberli/Documents/GitHub/Thesis/test22.tex... Wrote /Users/itaiberli/Documents/GitHub/Thesis/test22.tex Undo! Auto-saving...done repeat-complex-command: There are no previous complex commands to repeat delete-backward-char: Text is read-only Load-path shadows: /Users/itaiberli/.emacs.d/elpa/seq-2.20/seq hides /Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/seq Features: (shadow sort mail-extr emacsbug message rfc822 mml mml-sec epg mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mail-utils vc-git diff-mode tex-bar toolbar-x font-latex plain-tex tex-buf latex tex-ispell tex-style tex crm tex-mode latexenc colir color counsel jka-compr esh-util etags xref project swiper reftex reftex-vars two-column ivy delsel ivy-overlay helm-projectile helm-files rx image-dired tramp tramp-compat tramp-loaddefs trampver shell pcomplete format-spec dired-x dired-aux ffap helm-tags helm-bookmark helm-adaptive helm-info bookmark pp helm-external helm-net browse-url xml url url-proxy url-privacy url-expand url-methods url-history url-cookie url-domsuf url-util url-parse auth-source gnus-util mm-util help-fns mail-prsvr password-cache url-vars mailcap helm-buffers helm-grep helm-regexp helm-utils helm-locate helm-help helm-types projectile grep compile comint ansi-color ring ibuf-ext ibuffer thingatpt helm-descbinds helm easy-mmode helm-source cl-seq eieio-compat eieio eieio-core helm-multi-match helm-lib dired helm-config helm-easymenu cl-macs async-bytecomp async advice edmacro kmacro finder-inf tex-site info package epg-config seq byte-opt gv bytecomp byte-compile cl-extra help-mode easymenu cconv cl-loaddefs pcase cl-lib time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel ns-win ucs-normalize term/common-win tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces cus-face macroexp files text-properties overlay sha1 md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote kqueue cocoa ns multi-tty make-network-process emacs) Memory information: ((conses 16 359730 16119) (symbols 48 34262 0) (miscs 40 100 221) (strings 32 65306 15883) (string-bytes 1 1997869) (vectors 16 60314) (vector-slots 8 1721804 214602) (floats 8 589 398) (intervals 56 269 0) (buffers 976 19))