bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Dov Grobgeld]

This bug report will be sent to the Bug-GNU-Emacs mailing list
and the GNU bug tracker at debbugs.gnu.org.  Please check that
the From: line contains a valid email address.  After a delay of up
to one day, you should receive an acknowledgement at that address.

Please write in English if possible, as the Emacs maintainers
usually do not have translators for other languages.

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from `emacs -Q':

The ordering of columns of an org-table is wrong in when
bidi-display-reordering is true and there RTL characters in the table.

Imagine you have a buffer with the following logical contents (using the
convention that capitals are RTL characters).

| abcdef | abc |
| ABCDEF | ABC |

I would like this to be displayed as:

| abcdef | abc |
| FEDCBA | CBA |

The problem is that I want to each column of the table to be isolated
(with regards to bidi influence) from other columns in the table. (Of
course we also want to choose the table direction, but that is a
different and solvable issue.) If there is no such separation, which
is the behaviour currently get in emacs HEAD, then the resulting
rendered buffer is:

| abcdef | abc |
| CBA | FEDCBA |

Is this even solvable in the current emacs bidi model?

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/local/public-dev/share/emacs/24.1.50/etc/DEBUG.


In GNU Emacs 24.1.50.1 (i686-pc-linux-gnu, GTK+ Version 2.24.10)
 of 2012-06-13 on grower
Windowing system distributor `Fedora Project', version 11.0.11200000
Configured using:
 `configure '--prefix=/usr/local/public-dev/' 'CFLAGS=-g -O0''

Important settings:
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Org

Minor modes in effect:
  buffer-face-mode: t
  show-paren-mode: t
  csv-field-index-mode: t
  partial-completion-mode: t
  xmsi-mode: t
  diff-auto-refine-mode: t
  delete-selection-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  visual-line-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo>
<help-echo> <help-echo> <menu-bar> <help-menu> <about-emacs>
<help-echo> <help-echo> <help-echo> C-x 1 C-x C-f C-a
C-k / t m p / f o o . o r g <return> | SPC a b c SPC
| SPC a b c d e f SPC | <return> | SPC א ב ג SPC |
SPC א ב ג ל <backspace> ד ה ו <tab> <down-mouse-1>
<mouse-1> d e f <C-right> <backspace> <backspace> <backspace>
<down> <left> <left> <right> <right> <right> <right>
<left> <left> <left> <left> <left> <left> <left> <left>
<left> <left> <left> <left> <left> <left> <right> <right>
ד ה ו <left> <left> <left> <right> <right> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> C-י C-d C-d C-a C-f C-f C-f C-b C-b
C-b C-b C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f C-f
C-f C-f C-f C-d C-d C-d C-_ C-d e <backspace> ג <tab>
<left> <up> <right> <right> <right> <right> <left>
<right> <right> <right> <S-right> <S-right> <S-left>
<S-left> <S-left> <left> <left> <left> <right> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> <backspace> <backspace> y <backspace> א
ב ב <backspace> ג SPC | SPC א ב ג ד ה ו <tab> <up>
<left> <left> <down> <left> <left> א ב ן <backspace>
ג ד ה ו SPC | SPC א ב ג C-k <tab> M-נ ם <backspace>
<backspace> <backspace> M-x r e p o <tab> r <tab>
<return>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
(New file)
OVERVIEW
Entities are displayed as UTF8 characers
Auto-saving...done
Undo!
if: To use shift-selection with Org-mode, customize
`org-support-shift-select' [5 times]
Auto-saving...done
Auto-saving...done
Making completion list...

Load-path shadows:
/home/dov/github/dov-env/emacs/longlines hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/longlines
/home/dov/github/dov-env/emacs/compile hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/progmodes/compile
/home/dov/github/dov-env/emacs/octave-mod hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/progmodes/octave-mod
/home/dov/github/dov-env/emacs/gdb-mi hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/progmodes/gdb-mi
/home/dov/github/dov-env/emacs/octave-inf hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/progmodes/octave-inf
/home/dov/github/dov-env/emacs//org-mode/ob-lilypond hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-lilypond
/home/dov/github/dov-env/emacs//org-mode/ob-C hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-C
/home/dov/github/dov-env/emacs//org-mode/org-rmail hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-rmail
/home/dov/github/dov-env/emacs//org-mode/org-beamer hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-beamer
/home/dov/github/dov-env/emacs//org-mode/ob-comint hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-comint
/home/dov/github/dov-env/emacs//org-mode/ob-keys hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-keys
/home/dov/github/dov-env/emacs//org-mode/org-macs hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-macs
/home/dov/github/dov-env/emacs//org-mode/ob-ref hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-ref
/home/dov/github/dov-env/emacs//org-mode/ob-js hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-js
/home/dov/github/dov-env/emacs//org-mode/org-mobile hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-mobile
/home/dov/github/dov-env/emacs//org-mode/org-plot hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-plot
/home/dov/github/dov-env/emacs//org-mode/org-capture hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-capture
/home/dov/github/dov-env/emacs//org-mode/ob-lisp hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-lisp
/home/dov/github/dov-env/emacs//org-mode/org-compat hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-compat
/home/dov/github/dov-env/emacs//org-mode/org-html hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-html
/home/dov/github/dov-env/emacs//org-mode/org-id hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-id
/home/dov/github/dov-env/emacs//org-mode/ob-mscgen hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-mscgen
/home/dov/github/dov-env/emacs//org-mode/org-entities hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-entities
/home/dov/github/dov-env/emacs//org-mode/org-colview hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-colview
/home/dov/github/dov-env/emacs//org-mode/org-faces hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-faces
/home/dov/github/dov-env/emacs//org-mode/ob-R hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-R
/home/dov/github/dov-env/emacs//org-mode/ob-awk hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-awk
/home/dov/github/dov-env/emacs//org-mode/org-xoxo hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-xoxo
/home/dov/github/dov-env/emacs//org-mode/org-timer hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-timer
/home/dov/github/dov-env/emacs//org-mode/org-icalendar hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-icalendar
/home/dov/github/dov-env/emacs//org-mode/ob-org hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-org
/home/dov/github/dov-env/emacs//org-mode/ob-screen hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-screen
/home/dov/github/dov-env/emacs//org-mode/org-habit hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-habit
/home/dov/github/dov-env/emacs//org-mode/ob-eval hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-eval
/home/dov/github/dov-env/emacs//org-mode/org-datetree hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-datetree
/home/dov/github/dov-env/emacs//org-mode/ob-table hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-table
/home/dov/github/dov-env/emacs//org-mode/ob-scheme hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-scheme
/home/dov/github/dov-env/emacs//org-mode/org-ctags hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-ctags
/home/dov/github/dov-env/emacs//org-mode/org-mouse hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-mouse
/home/dov/github/dov-env/emacs//org-mode/ob-sh hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-sh
/home/dov/github/dov-env/emacs//org-mode/org hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org
/home/dov/github/dov-env/emacs//org-mode/org-mhe hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-mhe
/home/dov/github/dov-env/emacs//org-mode/org-mac-message hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-mac-message
/home/dov/github/dov-env/emacs//org-mode/ob-exp hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-exp
/home/dov/github/dov-env/emacs//org-mode/org-freemind hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-freemind
/home/dov/github/dov-env/emacs//org-mode/org-remember hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-remember
/home/dov/github/dov-env/emacs//org-mode/org-attach hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-attach
/home/dov/github/dov-env/emacs//org-mode/org-wl hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-wl
/home/dov/github/dov-env/emacs//org-mode/ob-java hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-java
/home/dov/github/dov-env/emacs//org-mode/ob-octave hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-octave
/home/dov/github/dov-env/emacs//org-mode/ob-matlab hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-matlab
/home/dov/github/dov-env/emacs//org-mode/org-ascii hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-ascii
/home/dov/github/dov-env/emacs//org-mode/org-bbdb hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-bbdb
/home/dov/github/dov-env/emacs//org-mode/ob-haskell hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-haskell
/home/dov/github/dov-env/emacs//org-mode/ob-sqlite hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-sqlite
/home/dov/github/dov-env/emacs//org-mode/org-info hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-info
/home/dov/github/dov-env/emacs//org-mode/ob-perl hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-perl
/home/dov/github/dov-env/emacs//org-mode/org-archive hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-archive
/home/dov/github/dov-env/emacs//org-mode/org-exp-blocks hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-exp-blocks
/home/dov/github/dov-env/emacs//org-mode/org-mks hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-mks
/home/dov/github/dov-env/emacs//org-mode/ob-ruby hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-ruby
/home/dov/github/dov-env/emacs//org-mode/org-protocol hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-protocol
/home/dov/github/dov-env/emacs//org-mode/org-special-blocks hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-special-blocks
/home/dov/github/dov-env/emacs//org-mode/org-table hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-table
/home/dov/github/dov-env/emacs//org-mode/org-taskjuggler hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-taskjuggler
/home/dov/github/dov-env/emacs//org-mode/ob-clojure hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-clojure
/home/dov/github/dov-env/emacs//org-mode/org-mew hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-mew
/home/dov/github/dov-env/emacs//org-mode/ob-ditaa hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-ditaa
/home/dov/github/dov-env/emacs//org-mode/org-w3m hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-w3m
/home/dov/github/dov-env/emacs//org-mode/ob-plantuml hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-plantuml
/home/dov/github/dov-env/emacs//org-mode/org-gnus hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-gnus
/home/dov/github/dov-env/emacs//org-mode/ob-tangle hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-tangle
/home/dov/github/dov-env/emacs//org-mode/ob hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob
/home/dov/github/dov-env/emacs//org-mode/org-src hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-src
/home/dov/github/dov-env/emacs//org-mode/ob-calc hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-calc
/home/dov/github/dov-env/emacs//org-mode/org-crypt hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-crypt
/home/dov/github/dov-env/emacs//org-mode/org-list hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-list
/home/dov/github/dov-env/emacs//org-mode/ob-latex hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-latex
/home/dov/github/dov-env/emacs//org-mode/org-docbook hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-docbook
/home/dov/github/dov-env/emacs//org-mode/org-pcomplete hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-pcomplete
/home/dov/github/dov-env/emacs//org-mode/org-publish hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-publish
/home/dov/github/dov-env/emacs//org-mode/ob-emacs-lisp hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-emacs-lisp
/home/dov/github/dov-env/emacs//org-mode/org-exp hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-exp
/home/dov/github/dov-env/emacs//org-mode/org-jsinfo hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-jsinfo
/home/dov/github/dov-env/emacs//org-mode/org-indent hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-indent
/home/dov/github/dov-env/emacs//org-mode/org-feed hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-feed
/home/dov/github/dov-env/emacs//org-mode/org-clock hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-clock
/home/dov/github/dov-env/emacs//org-mode/ob-lob hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-lob
/home/dov/github/dov-env/emacs//org-mode/org-bibtex hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-bibtex
/home/dov/github/dov-env/emacs//org-mode/org-latex hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-latex
/home/dov/github/dov-env/emacs//org-mode/org-footnote hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-footnote
/home/dov/github/dov-env/emacs//org-mode/ob-maxima hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-maxima
/home/dov/github/dov-env/emacs//org-mode/ob-sass hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-sass
/home/dov/github/dov-env/emacs//org-mode/ob-gnuplot hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-gnuplot
/home/dov/github/dov-env/emacs//org-mode/ob-dot hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-dot
/home/dov/github/dov-env/emacs//org-mode/org-inlinetask hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-inlinetask
/home/dov/github/dov-env/emacs//org-mode/ob-asymptote hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-asymptote
/home/dov/github/dov-env/emacs//org-mode/ob-css hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-css
/home/dov/github/dov-env/emacs//org-mode/org-agenda hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-agenda
/home/dov/github/dov-env/emacs//org-mode/ob-python hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-python
/home/dov/github/dov-env/emacs//org-mode/org-irc hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-irc
/home/dov/github/dov-env/emacs//org-mode/org-vm hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-vm
/home/dov/github/dov-env/emacs//org-mode/ob-ocaml hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-ocaml
/home/dov/github/dov-env/emacs//org-mode/org-docview hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/org-docview
/home/dov/github/dov-env/emacs//org-mode/ob-ledger hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-ledger
/home/dov/github/dov-env/emacs//org-mode/ob-sql hides
/usr/local/public-dev/share/emacs/24.1.50/lisp/org/ob-sql

Features:
(shadow mail-extr emacsbug sendmail tabify org-table face-remap org-wl
org-w3m org-vm org-rmail org-mhe org-mew org-irc org-jsinfo org-infojs
org-html org-info org-gnus org-docview org-bibtex bibtex org-bbdb paren
zoom-frm qtdoc browse-url git-find-file org-mw ob-C ob-R ob-octave
ob-plantuml ob-asymptote ob-dot ob-ditaa ob-python ob-perl ob-sh
org-htmlslidy org-s5 persistent ipython executable org-export iimage
org-crypt org-install org-wikipedia org-git-hyperlink org-man screenshot
dmacro tramp tramp-compat tramp-loaddefs tex-mode color-moccur
matlab-load sourcepair csv-mode sort dired-details+ dired-details
mediawiki url-cache mm-url url-http tls url-auth url-gw python-mode url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap xml-parse doxymacs ack epa-file epa epg
epg-config epresent org-latex org-export-latex org-beamer footnote
org-exp ob-exp org-exp-blocks org-agenda org ob-emacs-lisp ob-tangle
ob-ref ob-lob ob-table org-footnote org-src ob-comint ob-keys ob ob-eval
org-pcomplete org-list org-faces org-compat org-entities org-macs
cal-menu calendar cal-loaddefs gnus-sum nnoo gnus-group gnus-undo nnmail
mail-source gnus-start gnus-spec gnus-int gnus-range gnus-win gnus
gnus-ems nnheader pde-load complete pde-abbv tempo-x tempo pde-patch
help-dwim woman man template-simple pde-vars xmsi-math-symbols-input
mo-git-blame scroll-all markdown-mode noutline outline magit-blame
magit-bisect magit-key-mode magit esh-var esh-io esh-cmd esh-ext
esh-proc esh-arg eldoc esh-groups eshell esh-module esh-mode esh-util
ido iswitchb diff-mode log-edit pcvs-util add-log vc-ediff octave-mod
csharp-mode cc-langs doc-mode derived sgml-mode mb-depth two-column gud
message rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev mail-utils gmm-utils
mailheader delsel server javascript-mode icicles icicles-mode warnings
dired+ dired-x dired-aux advice help-fns advice-preload icicles-cmd2
compile easy-mmode imenu edmacro icicles-cmd1 frame-cmds frame-fns avoid
cus-edit cus-start cus-load yow etags shell pcomplete cookie1 comint
ansi-color ring bookmark pp dabbrev recentf tree-widget icicles-mcmd
help-mode view image-dired format-spec dired filesets icicles-fn
icicles-var cl-macs icicles-opt kmacro ffap url-parse auth-source eieio
byte-opt bytecomp byte-compile cconv gnus-util mm-util mail-prsvr
password-cache url-vars wid-edit thingatpt icicles-face icicles-mac cl
macroexp cl-lib gtk-look info-look info vc ediff-merg ediff-diff
ediff-wind ediff-help ediff-util ediff-mult ediff-init ediff
vc-dispatcher cc-mode cc-fonts easymenu cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs time-date tooltip
ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
register page menu-bar rfn-eshadow timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer loaddefs button faces cus-face files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process dbusbind
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty emacs)

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Wed, 13 Jun 2012 22:26:35 +0300
> From: Dov Grobgeld <dov.grobgeld@gmail.com>
> 
> Imagine you have a buffer with the following logical contents (using the
> convention that capitals are RTL characters).
> 
> | abcdef | abc |
> | ABCDEF | ABC |
> 
> I would like this to be displayed as:
> 
> | abcdef | abc |
> | FEDCBA | CBA |
> 
> The problem is that I want to each column of the table to be isolated
> (with regards to bidi influence) from other columns in the table. (Of
> course we also want to choose the table direction, but that is a
> different and solvable issue.) If there is no such separation, which
> is the behaviour currently get in emacs HEAD, then the resulting
> rendered buffer is:
> 
> | abcdef | abc |
> | CBA | FEDCBA |
> 
> Is this even solvable in the current emacs bidi model?

Yes, it is.  The solution involves putting segment separators between
the table columns.  These could be TAB characters or a display
property whose value is (space . :width N) or (space . :align-to COL).

Org maintainers, please ask if you need help in fixing this.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Dov Grobgeld]

Do you think that you can provide some lisp code that builds the
buffer contents and the display properties for the three options that
ou refered to as an illustration?

I tried inserting tabs into the buffer before the vertical bars, but
after reordering it still didn't come out right.

Regards,
Dov

On Thu, Jun 14, 2012 at 6:10 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 13 Jun 2012 22:26:35 +0300
>> From: Dov Grobgeld <dov.grobgeld@gmail.com>
>>
>> Imagine you have a buffer with the following logical contents (using the
>> convention that capitals are RTL characters).
>>
>> | abcdef | abc |
>> | ABCDEF | ABC |
>>
>> I would like this to be displayed as:
>>
>> | abcdef | abc |
>> | FEDCBA | CBA |
>>
>> The problem is that I want to each column of the table to be isolated
>> (with regards to bidi influence) from other columns in the table. (Of
>> course we also want to choose the table direction, but that is a
>> different and solvable issue.) If there is no such separation, which
>> is the behaviour currently get in emacs HEAD, then the resulting
>> rendered buffer is:
>>
>> | abcdef | abc |
>> | CBA | FEDCBA |
>>
>> Is this even solvable in the current emacs bidi model?
>
> Yes, it is.  The solution involves putting segment separators between
> the table columns.  These could be TAB characters or a display
> property whose value is (space . :width N) or (space . :align-to COL).
>
> Org maintainers, please ask if you need help in fixing this.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Thu, 14 Jun 2012 21:10:46 +0300
> From: Dov Grobgeld <dov.grobgeld@gmail.com>
> Cc: 11700@debbugs.gnu.org
> 
> Do you think that you can provide some lisp code that builds the
> buffer contents and the display properties for the three options that
> ou refered to as an illustration?
> 
> I tried inserting tabs into the buffer before the vertical bars, but
> after reordering it still didn't come out right.

??? What exactly did you try?  You need to have a TAB before and after
each '|' that's between the cells (the outer ones do not need a TAB).
Like this (you should see this correctly in Emacs 24.1; move cursor
with C-f to see the logical order):

‎|      אבגד 	|	הוזחטי     |

Isn't this what you wanted?

As for the display properties, replace the TABs in the example above
with something like this:

  (insert (propertize " " '(display (space :align-to 10))))

The number 10 should be the column where you want the "|" separator to
be aligned.  For the replacement of the second TAB, increment the
value to get the desired alignment of the text of the second cell.

bug#11700: [O] bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Dov Grobgeld]

On Thu, Jun 14, 2012 at 10:37 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 14 Jun 2012 21:10:46 +0300
>> From: Dov Grobgeld <dov.grobgeld@gmail.com>
>> Cc: 11700@debbugs.gnu.org
>> [stuff deleted]
>> I tried inserting tabs into the buffer before the vertical bars, but
>> after reordering it still didn't come out right.
>
> ??? What exactly did you try?  You need to have a TAB before and after
> each '|' that's between the cells (the outer ones do not need a TAB).
> Like this (you should see this correctly in Emacs 24.1; move cursor
> with C-f to see the logical order):
>
> ‎|      אבגד    |       הוזחטי     |
>
> Isn't this what you wanted?

Yes. Great! This is indeed what I wanted. My mistake was that I tried
it with a tab character before OR after the vertical bar. This
solution should be really simple to implement in org-mode as it means
that instead of joining the table columns with "<space><vbar><space>"
as is currently done, you just need to use "<tab><vbar><tab>" as well
as setting the tab width to 1.

But I just wonder, is there any other character (preferably white
space character) with the same end-of-segment-boundary properties as
tab, in case tab is used for something else in org-mode? Or is it
possible to take an arbitrary character, e.g. U+E0020, and bless it
with end-of-segment boundary properties and then use that in the
org-mode buffer?

As a side note, I really like the idea of end of segment separator,
and I wasn't aware of it when I wrote fribidi a long time ago. I
wonder if the semantics of the emacs segment separator follows the
Unicode Bidi Algorithm?

bug#11700: [O] bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Fri, 15 Jun 2012 09:39:35 +0300
> From: Dov Grobgeld <dov.grobgeld@gmail.com>
> Cc: 11700@debbugs.gnu.org
> 
> Yes. Great! This is indeed what I wanted. My mistake was that I tried
> it with a tab character before OR after the vertical bar. This
> solution should be really simple to implement in org-mode as it means
> that instead of joining the table columns with "<space><vbar><space>"
> as is currently done, you just need to use "<tab><vbar><tab>" as well
> as setting the tab width to 1.

Yep.

> But I just wonder, is there any other character (preferably white
> space character) with the same end-of-segment-boundary properties as
> tab, in case tab is used for something else in org-mode?

That's the (space . :align-to COLUMN) display property I was talking
about.  With it, you can arrange for a single blank, say, to produce a
stretch of whitespace of arbitrary size, in character cell units,
aligned to a specified column.  If you use :width instead of
:align-to, you can have the size tuned in pixels.  Emacs treats text
covered by such display properties as segment separators, so they
produce the same effect as TAB does.  That's because conceptually,
such display properties are used to separate text into columnar
display, exactly like TAB is used in plain-text tables.

You can find examples of using these display properties in
minibuffer.el, where they are used to produce the display in the
*Completions* buffer.  Their documentation is in the ELisp manual.

> Or is it possible to take an arbitrary character, e.g. U+E0020, and
> bless it with end-of-segment boundary properties and then use that
> in the org-mode buffer?

Try using u+2029, it should do the trick, I think.

> As a side note, I really like the idea of end of segment separator,
> and I wasn't aware of it when I wrote fribidi a long time ago. I
> wonder if the semantics of the emacs segment separator follows the
> Unicode Bidi Algorithm?

Of course, it does; Emacs implements the UBA to the letter, taking
only the "high-level protocols" fire-escape to guide the reordering using
Emacs-specific context.  (But even the high-level protocols feature is
part of the UBA, see section 4.3 there.)

The Segment Separator feature is not an Emacs invention, it exists in
the UBA.  The key to understanding it is this part of the UBA:

  L1. On each line, reset the embedding level of the following
      characters to the paragraph embedding level:

  1. Segment separators,
  2. Paragraph separators,
  3. Any sequence of whitespace characters preceding a segment
     separator or paragraph separator, and 
  4. Any sequence of white space characters at the end of the line.

And the TAB character has "S", i.e. "segment separator", as its bidi
property.  Since its embedding level is reset to the base embedding
level of the paragraph, the result is that text on both sides of a TAB
is reordered separately and independently.

Moreover, the high-level protocols feature give us one more
possibility:

  HL4. Apply the Bidirectional Algorithm to segments.

    . The Bidirectional Algorithm can be applied independently to one
      or more segments of structured text. For example, when
      displaying a document consisting of textual data and visible
      markup in an editor, a higher-level process can handle syntactic
      elements in the markup separately from the textual data.

Emacs uses this to treat the `space' display properties as segment
separators.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Nicolas Goaziou]

Hello,

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 13 Jun 2012 22:26:35 +0300
>> From: Dov Grobgeld <dov.grobgeld@gmail.com>
>> 
>> Imagine you have a buffer with the following logical contents (using the
>> convention that capitals are RTL characters).
>> 
>> | abcdef | abc |
>> | ABCDEF | ABC |
>> 
>> I would like this to be displayed as:
>> 
>> | abcdef | abc |
>> | FEDCBA | CBA |
>> 
>> The problem is that I want to each column of the table to be isolated
>> (with regards to bidi influence) from other columns in the table. (Of
>> course we also want to choose the table direction, but that is a
>> different and solvable issue.) If there is no such separation, which
>> is the behaviour currently get in emacs HEAD, then the resulting
>> rendered buffer is:
>> 
>> | abcdef | abc |
>> | CBA | FEDCBA |
>> 
>> Is this even solvable in the current emacs bidi model?
>
> Yes, it is.  The solution involves putting segment separators between
> the table columns.  These could be TAB characters or a display
> property whose value is (space . :width N) or (space . :align-to COL).
>
> Org maintainers, please ask if you need help in fixing this.

*raises a hand*

I'd rather preserve structure of Org documents outside of Emacs. So,
`:align-to' is not an option. 

IIUC, I need to replace the closest space from vertical bars with 

  #(" " 0 1 (space :width 1))

This doesn't sound too difficult.

However, could someone provide tests cases so we get it right once and
for all?

Thank you.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Dov Grobgeld]

[-- Attachment #1: Type: text/plain, Size: 2059 bytes --]

The correct Unicode≥6.3 way to do this would be with the unicode isolation
characters. I.e. you would wrap each of the columns with <FSI>column
contents<BDI>. Does emacs honor these? Should be easy to test.

Regards,
Dov

On Mon, Dec 4, 2017 at 9:27 PM, Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Hello,
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> Date: Wed, 13 Jun 2012 22:26:35 +0300
> >> From: Dov Grobgeld <dov.grobgeld@gmail.com>
> >>
> >> Imagine you have a buffer with the following logical contents (using the
> >> convention that capitals are RTL characters).
> >>
> >> | abcdef | abc |
> >> | ABCDEF | ABC |
> >>
> >> I would like this to be displayed as:
> >>
> >> | abcdef | abc |
> >> | FEDCBA | CBA |
> >>
> >> The problem is that I want to each column of the table to be isolated
> >> (with regards to bidi influence) from other columns in the table. (Of
> >> course we also want to choose the table direction, but that is a
> >> different and solvable issue.) If there is no such separation, which
> >> is the behaviour currently get in emacs HEAD, then the resulting
> >> rendered buffer is:
> >>
> >> | abcdef | abc |
> >> | CBA | FEDCBA |
> >>
> >> Is this even solvable in the current emacs bidi model?
> >
> > Yes, it is.  The solution involves putting segment separators between
> > the table columns.  These could be TAB characters or a display
> > property whose value is (space . :width N) or (space . :align-to COL).
> >
> > Org maintainers, please ask if you need help in fixing this.
>
> *raises a hand*
>
> I'd rather preserve structure of Org documents outside of Emacs. So,
> `:align-to' is not an option.
>
> IIUC, I need to replace the closest space from vertical bars with
>
>   #(" " 0 1 (space :width 1))
>
> This doesn't sound too difficult.
>
> However, could someone provide tests cases so we get it right once and
> for all?
>
> Thank you.
>
> Regards,
>
> --
> Nicolas Goaziou                                                0x80A93738
>

[-- Attachment #2: Type: text/html, Size: 3200 bytes --]

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> From: Dov Grobgeld <dov.grobgeld@gmail.com>
> Date: Mon, 4 Dec 2017 21:35:40 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, 11700@debbugs.gnu.org
> 
> The correct Unicode≥6.3 way to do this would be with the unicode isolation characters. I.e. you would wrap
> each of the columns with <FSI>column contents<BDI>. Does emacs honor these?

Yes, Emacs implements Unicode 9.0, including the UBA with isolates.

However, I suspect that the results of wrapping with isolates will be
different from using the original proposal of using segment separators.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Mon, 04 Dec 2017 22:43:12 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: mail@nicolasgoaziou.fr, 11700@debbugs.gnu.org
> 
> Yes, Emacs implements Unicode 9.0, including the UBA with isolates.

Actually, the current development sources and the upcoming Emacs 26.1
already support Unicode 10.0.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Cc: Dov Grobgeld <dov.grobgeld@gmail.com>,  11700@debbugs.gnu.org
> Date: Mon, 04 Dec 2017 21:27:53 +0100
> 
> I'd rather preserve structure of Org documents outside of Emacs. So,
> `:align-to' is not an option. 
> 
> IIUC, I need to replace the closest space from vertical bars with 
> 
>   #(" " 0 1 (space :width 1))
> 
> This doesn't sound too difficult.
> 
> However, could someone provide tests cases so we get it right once and
> for all?

Such tests can only be run interactively, because bidi reordering is a
display-time feature in Emacs.  Is that OK with you?

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Nicolas Goaziou]

Eli Zaretskii <eliz@gnu.org> writes:

> Such tests can only be run interactively, because bidi reordering is a
> display-time feature in Emacs.  Is that OK with you?

That's better than no test at all in my book, so I'm fine with it, yes.

I can use isolation characters instead (if anyone cares to point me to
what those are), if you think that's better.

Regards,

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Cc: dov.grobgeld@gmail.com,  11700@debbugs.gnu.org
> Date: Mon, 04 Dec 2017 22:02:00 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Such tests can only be run interactively, because bidi reordering is a
> > display-time feature in Emacs.  Is that OK with you?
> 
> That's better than no test at all in my book, so I'm fine with it, yes.

OK.

Can one of you please provide a short Lisp snippet that generates a
2x2 Org table and inserts it in a buffer, which I could use as the
basis for the test?  That would get me off the ground quicker, since
I'm a very infrequent user of Org tables.

> I can use isolation characters instead (if anyone cares to point me to
> what those are), if you think that's better.

I will experiment with both and see which one is better.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Nicolas Goaziou]

Hello,

Eli Zaretskii <eliz@gnu.org> writes:

> Can one of you please provide a short Lisp snippet that generates a
> 2x2 Org table and inserts it in a buffer, which I could use as the
> basis for the test?  That would get me off the ground quicker, since
> I'm a very infrequent user of Org tables.

For tests, we use `org-test-with-temp-text' macro, e.g.,

  (org-test-with-temp-text "| a | b |\n| c | d |"
    ... do something in that buffer ...)

> I will experiment with both and see which one is better.

OK. Thank you.

Regards,

-- 
Nicolas Goaziou

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Date: Fri, 08 Dec 2017 18:08:57 +0100
> Cc: dov.grobgeld@gmail.com, 11700@debbugs.gnu.org
> 
> For tests, we use `org-test-with-temp-text' macro, e.g.,
> 
>   (org-test-with-temp-text "| a | b |\n| c | d |"
>     ... do something in that buffer ...)

You didn't say that this macro is only available in the Org's Git
repository...

Anyway, since it's a very simple macro, I just used its guts below.

I found both methods doing well, so I'm going to show both, and let
you decide which one is better.  The below provides a demonstration of
each method by displaying a buffer with a table whose columns include
both L2R and R2L text, such that the table columns are still laid out
left to right, unlike when one just types the characters in the cells.

Method 1: wrap each string with (invisible) bidi formatting control
characters which isolate each string from the surrounding text.

(defun bidi-isolate-string (str)
  (concat (propertize (string ?\x2068) 'invisible t)
	  str
	  (propertize (string ?\x2069) 'invisible t)))

(with-current-buffer (get-buffer-create "bidi-org-table1")
  (org-mode)
  (insert (concat "| "
		(bidi-isolate-string "abcd")
		" | "
		(bidi-isolate-string "efgh")
		" |\n| "
		(bidi-isolate-string "אבגד")
		" | "
		(bidi-isolate-string "הוזח")
		" |"))
  (pop-to-buffer "bidi-org-table"))

This has a minor issue: it fails to conceal the bidi control
characters on display, although I used the 'invisible' property for
that purpose.  I'm guessing that Org takes control of the 'invisible'
properties, in which case perhaps this method should use some other
property, if possible.  If it is not practical to conceal the bidi
controls on display, the following method is preferable.

Method 2: give the spaces around the cell text the display property
which makes the spaces serve as segment separators for the purposes of
the bidirectional reordering.

(defun bidi-separator-space ()
  (propertize " " 'display '(space :width 1)))

(with-current-buffer (get-buffer-create "bidi-org-table2")
  (org-mode)
  (insert (concat "|"
		  (bidi-separator-space)
		  "abcd"
		  (bidi-separator-space)
		  "|"
		  (bidi-separator-space)
		  "efgh"
		  (bidi-separator-space)
		  "|\n|"
		  (bidi-separator-space)
		  "אבגד"
		  (bidi-separator-space)
		  "|"
		  (bidi-separator-space)
		  "הוזח"
		  (bidi-separator-space)
		  "|"))
  (pop-to-buffer "bidi-org-table2"))

Let me know if I can help you further, or if you have additional
questions.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Sat, 23 Dec 2017 15:38:11 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dov.grobgeld@gmail.com, 11700@debbugs.gnu.org
> 
> I found both methods doing well, so I'm going to show both, and let
> you decide which one is better.

On second thought, I think Method 2 is better, because it does exactly
what segment separators were invented for: to separate cells in
tables.  By contrast, the bidi formatting control characters are for
controlling the display order in general text, not necessarily in
tables.

bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Nicolas Goaziou]

Hello,

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
>> Date: Fri, 08 Dec 2017 18:08:57 +0100
>> Cc: dov.grobgeld@gmail.com, 11700@debbugs.gnu.org
>> 
>> For tests, we use `org-test-with-temp-text' macro, e.g.,
>> 
>>   (org-test-with-temp-text "| a | b |\n| c | d |"
>>     ... do something in that buffer ...)
>
> You didn't say that this macro is only available in the Org's Git
> repository...

I believed our test suite had been merged with Emacs. Sorry about that.

> Method 2: give the spaces around the cell text the display property
> which makes the spaces serve as segment separators for the purposes of
> the bidirectional reordering.

Sounds good. I implemented it in Org. Hopefully, Org is now friendlier
to BiDi.

Please let me know if something is missing.

Thank you.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738