mis-alignment in org-tables with Tibetan characters

mis-alignment in org-tables with Tibetan characters

[Steffan Iverson]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

Hello all,


I've made an org table that includes both English and Tibetan characters,
and the columns don't line up. I suspect this is because of the way Tibetan
characters are displayed - they "stack" below and above each other, making
the characters vertically taller than the English characters. You can see
this
same problem in the text pasted below, it behaves the same way in an org
table. Any way to solve this problem? Perhaps somehow changing the height
of all the text in the table? Thanks!

| རྐུ་བ་    | rku-ba     | བརྐུས་པ་  | brkus-pa    | བརྐུ་བ་   | brku-ba
| རྐུས་   | rkus    |
| གསོ་བ་   | gso-ba     | གསོས་པ་  | gsos-pa     | གསོ་བ་   | gso-ba     |
གསོས་  | gsos    |
| འཆོར་བ་  | 'chor-ba   | ཤོར་བ་   | shor-ba     | འཆོར་བ་  | 'chor-ba   |

[-- Attachment #2: Type: text/html, Size: 1081 bytes --]

Re: mis-alignment in org-tables with Tibetan characters

[Steffan Iverson]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

Screenshot: http://oi59.tinypic.com/lz893.jpg


On Sun, Feb 2, 2014 at 7:29 PM, Steffan Iverson
<steffan.iverson@gmail.com>wrote:

> Hello all,
>
>
> I've made an org table that includes both English and Tibetan characters,
> and the columns don't line up. I suspect this is because of the way
> Tibetan
> characters are displayed - they "stack" below and above each other, making
> the characters vertically taller than the English characters. You can see
> this
> same problem in the text pasted below, it behaves the same way in an org
> table. Any way to solve this problem? Perhaps somehow changing the height
> of all the text in the table? Thanks!
>
> | རྐུ་བ་    | rku-ba     | བརྐུས་པ་  | brkus-pa    | བརྐུ་བ་   |
> brku-ba    | རྐུས་   | rkus    |
> | གསོ་བ་   | gso-ba     | གསོས་པ་  | gsos-pa     | གསོ་བ་   | gso-ba     |
> གསོས་  | gsos    |
> | འཆོར་བ་  | 'chor-ba   | ཤོར་བ་   | shor-ba     | འཆོར་བ་  | 'chor-ba   |
>

[-- Attachment #2: Type: text/html, Size: 1599 bytes --]

Re: mis-alignment in org-tables with Tibetan characters

[Michael Brand]

Hi Steffan

On Mon, Feb 3, 2014 at 6:33 AM, Steffan Iverson
<steffan.iverson@gmail.com> wrote:
> Screenshot: http://oi59.tinypic.com/lz893.jpg

Judging from e. g. this part
| རྐུས་  | rkus |
| གསོས་ | gsos |
in your screenshot my speculation is that you get glyphs from not only your
default monospace font but also from other (maybe variable width) font(s).
Either because some glyphs are not available in your default monospace font
or are not used due to an incomplete configuration with set-fontset-font.

For a diagnosis in this direction I would check the alignment of "|" after
some characters just repeated like here:

xxxxxxxxxx|
་་་་་་་་་་|
གགགགགགགགགག|
རརརརརརརརརར|
སསསསསསསསསས|
ུུུུུུུུུུ|
ོོོོོོོོོོ|
ྐྐྐྐྐྐྐྐྐྐ|
xxxxxxxxxx|
∞∞∞∞∞∞∞∞∞∞|

Michael

Re: mis-alignment in org-tables with Tibetan characters

[Steffan Iverson]

[-- Attachment #1: Type: text/plain, Size: 1375 bytes --]

Michael, that test case you suggest does indeed cause the same spacing
problem. So this isn't a problem from the vowel stacking in Tibetan,
because of the type of font it is? Sorry, generally new to Emacs so I'm not
sure how to move forward with that insight.


On Tue, Feb 4, 2014 at 10:13 AM, Michael Brand
<michael.ch.brand@gmail.com>wrote:

> Hi Steffan
>
> On Mon, Feb 3, 2014 at 6:33 AM, Steffan Iverson
> <steffan.iverson@gmail.com> wrote:
> > Screenshot: http://oi59.tinypic.com/lz893.jpg
>
> Judging from e. g. this part
> | རྐུས་  | rkus |
> | གསོས་ | gsos |
> in your screenshot my speculation is that you get glyphs from not only your
> default monospace font but also from other (maybe variable width) font(s).
> Either because some glyphs are not available in your default monospace font
> or are not used due to an incomplete configuration with set-fontset-font.
>
> For a diagnosis in this direction I would check the alignment of "|" after
> some characters just repeated like here:
>
> xxxxxxxxxx|
> ་་་་་་་་་་|
> གགགགགགགགགག|
> རརརརརརརརརར|
> སསསསསསསསསས|
> ུུུུུུུུུུ|
> ོོོོོོོོོོ|
> ྐྐྐྐྐྐྐྐྐྐ|
> xxxxxxxxxx|
> ∞∞∞∞∞∞∞∞∞∞|
>
> Michael
>

[-- Attachment #2: Type: text/html, Size: 1940 bytes --]

Re: mis-alignment in org-tables with Tibetan characters

[Michael Brand]

Hi Steffan

On Wed, Feb 5, 2014 at 1:00 AM, Steffan Iverson
<steffan.iverson@gmail.com> wrote:
> Michael, that test case you suggest does indeed cause the same spacing
> problem. So this isn't a problem from the vowel stacking in Tibetan,

Such combining characters and and also double width characters may
still remain as separate issues for Org table alignment and maybe the
patches from Eric Abrahamsen are required to resolve one or both.

> because of the type of font it is? Sorry, generally new to Emacs so
> I'm not sure how to move forward with that insight.

Several steps to go with choosing font(s) and probably tweaking
set-fontset-font which I don't have much insight neither.

Regarding the choice of font: Is there any font that is monospaced
even in the Tibetan range at all? What does a "cat" (not "less" or any
other ncurses) of your file look like in different terminal emulators
like xterm, gnome-terminal etc. on different OSes? What does "locale"
from these terminal emulators tell?

Regarding a possible first step with Emacs set-fontset-font: How did
the glyph for INFINITY from the test look like, was this aligned for
you?:

xxxxxxxxxx|
∞∞∞∞∞∞∞∞∞∞|

Michael

Re: mis-alignment in org-tables with Tibetan characters

[Michael Brand]

Hi Steffan

On Wed, Feb 5, 2014 at 6:35 AM, Michael Brand
<michael.ch.brand@gmail.com> wrote:
> Is there any font that is monospaced even in the Tibetan range at all?

On one of my "trips" of learning more about unicode I just discovered
the monospaced bitmap font GNU Unifont [1] [2] which implements most
of (!) the range U+0000 to U+FFFF and should be ok for a whole lot of
tests. See PNG image of e. g. version 6.3.20131006 [3]. The unicode
script Tibetan you need starts at 0F00.

Michael

[1] http://en.wikipedia.org/wiki/GNU_Unifont
[2] http://unifoundry.com/unifont.html
[3] http://upload.wikimedia.org/wikipedia/commons/e/e3/Unifont-6.3.20131006.png

Re: mis-alignment in org-tables with Tibetan characters

[Bastien]

Hi Steffan,

thanks for reporting this.

Steffan Iverson <steffan.iverson@gmail.com> writes:

> Any way to solve this problem?

This is a recurring problem, and Eric is the one who tried
to solve it.

Eric, I don't remember why we didn't follow-up on your patch
here:

http://lists.gnu.org/archive/html/emacs-orgmode/2013-02/msg00664.html

Would that help in this situation?

Thanks,

-- 
 Bastien

Re: mis-alignment in org-tables with Tibetan characters

[Eric Abrahamsen]

Bastien <bzg@gnu.org> writes:

> Hi Steffan,
>
> thanks for reporting this.
>
> Steffan Iverson <steffan.iverson@gmail.com> writes:
>
>> Any way to solve this problem?
>
> This is a recurring problem, and Eric is the one who tried
> to solve it.
>
> Eric, I don't remember why we didn't follow-up on your patch
> here:
>
> http://lists.gnu.org/archive/html/emacs-orgmode/2013-02/msg00664.html
>
> Would that help in this situation?

I've been using that patch or something like it for nearly a year now,
with no adverse effects. I'm on the road right now, give me a day and
I'll take a closer look at what I've got...

E

Re: mis-alignment in org-tables with Tibetan characters

[Bastien]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I've been using that patch or something like it for nearly a year now,
> with no adverse effects. I'm on the road right now, give me a day and
> I'll take a closer look at what I've got...

Great -- thanks in advance!  I'll then wait before releasing a new
minor version and merging it into Emacs for Emacs 24.4.

To other core maintainers: if you see important issues that needs to
be fixed in maint, let me know.

-- 
 Bastien

Re: mis-alignment in org-tables with Tibetan characters

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

Bastien <bzg@gnu.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I've been using that patch or something like it for nearly a year now,
>> with no adverse effects. I'm on the road right now, give me a day and
>> I'll take a closer look at what I've got...
>
> Great -- thanks in advance!  I'll then wait before releasing a new
> minor version and merging it into Emacs for Emacs 24.4.
>
> To other core maintainers: if you see important issues that needs to
> be fixed in maint, let me know.

Sorry this took a while to get to...

I think it was a little simpler than I thought -- at least I hope that's
true, and I'm not missing something really obvious. There are two
patches attached, a simple one that handles re-justification of table
fields during field movement, and another that allows for narrowing of
columns with double-width strings. The second patch is uglier, and
doesn't work 100% well (you get misalignment if you try to narrow a
double-wide to an odd number of single-width characters), but it's
better than nothing.

Please test!

Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-table-field-justification-respect-string-width.patch --]
[-- Type: text/x-diff, Size: 1285 bytes --]

From f03f4755b2d3e62d9f88845d93390a364805c131 Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Date: Wed, 13 Feb 2013 11:56:18 +0800
Subject: [PATCH 1/9] Make table field justification respect string width

---
 lisp/org-table.el | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 520ac8a..33ae63e 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -973,14 +973,16 @@ Optional argument NEW may specify text to replace the current field content."
 	    (progn
 	      (setq s (match-string 1)
 		    o (match-string 0)
-		    l (max 1 (- (match-end 0) (match-beginning 0) 3))
+		    l (max 1
+			   (- (org-string-width
+			       (buffer-substring
+				(match-end 0) (match-beginning 0))) 3))
 		    e (not (= (match-beginning 2) (match-end 2))))
 	      (setq f (format (if num " %%%ds %s" " %%-%ds %s")
 			      l (if e "|" (setq org-table-may-need-update t) ""))
 		    n (format f s))
 	      (if new
-		  (if (<= (length new) l)      ;; FIXME: length -> str-width?
+		  (if (<= (org-string-width new) l)
 		      (setq n (format f new))
 		    (setq n (concat new "|") org-table-may-need-update t)))
 	      (if (equal (string-to-char n) ?-) (setq n (concat " " n)))
-- 
1.8.5.4


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Make-table-column-narrowing-play-nice-with-variable-.patch --]
[-- Type: text/x-diff, Size: 1876 bytes --]

From 85aee797a97194d9c29d7baac6415555f3849f7f Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Date: Wed, 13 Feb 2013 16:46:33 +0800
Subject: [PATCH 2/9] Make table column narrowing play nice with variable-width
 strings

---
 lisp/org-table.el | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 33ae63e..32dd025 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -758,7 +758,7 @@ When nil, simply write \"#ERROR\" in corrupted fields.")
 	 (hfmt1 (concat
 		 (make-string sp2 ?-) "%s" (make-string sp1 ?-) "+"))
 	 emptystrings links dates emph raise narrow
-	 falign falign1 fmax f1 len c e space)
+	 falign falign1 fmax f1 f2 len c e space)
     (untabify beg end)
     (remove-text-properties beg end '(org-cwidth t org-dwidth t display t))
     ;; Check if we have links or dates
@@ -843,10 +843,19 @@ When nil, simply write \"#ERROR\" in corrupted fields.")
 		  (unless (> f1 1)
 		    (user-error "Cannot narrow field starting with wide link \"%s\""
 			   (match-string 0 xx)))
-		  (add-text-properties f1 (length xx) (list 'org-cwidth t) xx)
-		  (add-text-properties (- f1 2) f1
-				       (list 'display org-narrow-column-arrow)
-				       xx)))))
+		  (setq f2 (length xx))
+		  (if (= (org-string-width xx)
+			 f2)
+		      (setq f2 f1)
+		    (setq f2 1)
+		    (while (< (org-string-width (substring xx 0 f2))
+			      f1)
+		      (setq f2 (1+ f2))))
+		  (add-text-properties f2 (length xx) (list 'org-cwidth t) xx)
+		  (add-text-properties (if (>= (string-width (substring xx (1- f2) f2)) 2)
+					   (1- f2) (- f2 2)) f2
+					   (list 'display org-narrow-column-arrow)
+					   xx)))))
       ;; Get the maximum width for each column
       (push (apply 'max (or fmax 1) 1 (mapcar 'org-string-width column))
 	    lengths)
-- 
1.8.5.4

Re: mis-alignment in org-tables with Tibetan characters

[Steffan Iverson]

[-- Attachment #1: Type: text/plain, Size: 1434 bytes --]

Thanks Eric - I've used this patch but I doesn't seem to solve the problem.
I'm working on an earlier suggestion by Michael about the unicode type that
my Tibetan font is. I very much appreciate all this help!

Steffan


On Sun, Feb 9, 2014 at 8:09 PM, Eric Abrahamsen <eric@ericabrahamsen.net>wrote:

> Bastien <bzg@gnu.org> writes:
>
> > Eric Abrahamsen <eric@ericabrahamsen.net> writes:
> >
> >> I've been using that patch or something like it for nearly a year now,
> >> with no adverse effects. I'm on the road right now, give me a day and
> >> I'll take a closer look at what I've got...
> >
> > Great -- thanks in advance!  I'll then wait before releasing a new
> > minor version and merging it into Emacs for Emacs 24.4.
> >
> > To other core maintainers: if you see important issues that needs to
> > be fixed in maint, let me know.
>
> Sorry this took a while to get to...
>
> I think it was a little simpler than I thought -- at least I hope that's
> true, and I'm not missing something really obvious. There are two
> patches attached, a simple one that handles re-justification of table
> fields during field movement, and another that allows for narrowing of
> columns with double-width strings. The second patch is uglier, and
> doesn't work 100% well (you get misalignment if you try to narrow a
> double-wide to an odd number of single-width characters), but it's
> better than nothing.
>
> Please test!
>
> Eric
>
>

[-- Attachment #2: Type: text/html, Size: 2141 bytes --]

Re: mis-alignment in org-tables with Tibetan characters

[Eric Abrahamsen]

Ah yes, this won't work if your font draws characters at variable
widths. I use Chinese a lot, and some Chinese fonts will still create
misalignment, simply because a Chinese character as drawn as 192% the
width of an ascii character. Dunno how to get around that.

On 02/11/14 10:50 AM, Steffan Iverson wrote:
> Thanks Eric - I've used this patch but I doesn't seem to solve the
> problem. I'm working on an earlier suggestion by Michael about the
> unicode type that my Tibetan font is. I very much appreciate all this
> help!
>
> Steffan
>
>
> On Sun, Feb 9, 2014 at 8:09 PM, Eric Abrahamsen <
> eric@ericabrahamsen.net> wrote:
>
>     Bastien <bzg@gnu.org> writes:
>    
>     > Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>     >
>     >> I've been using that patch or something like it for nearly a
>     year now,
>     >> with no adverse effects. I'm on the road right now, give me a
>     day and
>     >> I'll take a closer look at what I've got...
>     >
>     > Great -- thanks in advance!  I'll then wait before releasing a
>     new
>     > minor version and merging it into Emacs for Emacs 24.4.
>     >
>     > To other core maintainers: if you see important issues that
>     needs to
>     > be fixed in maint, let me know.
>    
>     Sorry this took a while to get to...
>    
>     I think it was a little simpler than I thought -- at least I hope
>     that's
>     true, and I'm not missing something really obvious. There are two
>     patches attached, a simple one that handles re-justification of
>     table
>     fields during field movement, and another that allows for
>     narrowing of
>     columns with double-width strings. The second patch is uglier,
>     and
>     doesn't work 100% well (you get misalignment if you try to narrow
>     a
>     double-wide to an odd number of single-width characters), but
>     it's
>     better than nothing.
>    
>     Please test!
>    
>     Eric
>    

Re: mis-alignment in org-tables with Tibetan characters

[Bastien]

Hi Eric,

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Sorry this took a while to get to...
>
> I think it was a little simpler than I thought -- at least I hope that's
> true, and I'm not missing something really obvious. There are two
> patches attached, a simple one that handles re-justification of table
> fields during field movement, and another that allows for narrowing of
> columns with double-width strings. The second patch is uglier, and
> doesn't work 100% well (you get misalignment if you try to narrow a
> double-wide to an odd number of single-width characters), but it's
> better than nothing.

I want to apply the patches on master -- can you give them a commit
message with an Emacs changelog?  Also, the first one (0001-...) does
not apply on master anymore.

Thanks in advance!

-- 
 Bastien

Re: mis-alignment in org-tables with Tibetan characters

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 1148 bytes --]

Bastien <bzg@gnu.org> writes:

> Hi Eric,
>
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Sorry this took a while to get to...
>>
>> I think it was a little simpler than I thought -- at least I hope that's
>> true, and I'm not missing something really obvious. There are two
>> patches attached, a simple one that handles re-justification of table
>> fields during field movement, and another that allows for narrowing of
>> columns with double-width strings. The second patch is uglier, and
>> doesn't work 100% well (you get misalignment if you try to narrow a
>> double-wide to an odd number of single-width characters), but it's
>> better than nothing.
>
> I want to apply the patches on master -- can you give them a commit
> message with an Emacs changelog?  Also, the first one (0001-...) does
> not apply on master anymore.
>
> Thanks in advance!

The patch series that would not die!

I'm not a git wizard, but I think I was able to apply both these patches
to master with no problem: they're now sitting on top of 56b47fb and
nothing is going wrong (I think). Can you try again?

Here they are with better commit messages.

E


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Use-string-width-in-table-field-justification.patch --]
[-- Type: text/x-diff, Size: 1468 bytes --]

From 3e6fd0e2790156a9bd68401bdabbf1b170408ba6 Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Date: Wed, 13 Feb 2013 11:56:18 +0800
Subject: [PATCH 1/2] Use string width in table field justification

lisp/org-table.el (org-table-justify-field-maybe): Use
org-string-width instead of length to calculate the visual width of
table cells -- helps with double-width charsets, etc.
---
 lisp/org-table.el | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 6a7935e..5e0177f 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -981,13 +981,16 @@ Optional argument NEW may specify text to replace the current field content."
 	    (progn
 	      (setq s (match-string 1)
 		    o (match-string 0)
-		    l (max 1 (- (match-end 0) (match-beginning 0) 3))
+		    l (max 1
+			   (- (org-string-width
+			       (buffer-substring-no-properties
+				(match-end 0) (match-beginning 0))) 3))
 		    e (not (= (match-beginning 2) (match-end 2))))
 	      (setq f (format (if num " %%%ds %s" " %%-%ds %s")
 			      l (if e "|" (setq org-table-may-need-update t) ""))
 		    n (format f s))
 	      (if new
-		  (if (<= (length new) l)      ;; FIXME: length -> str-width?
+		  (if (<= (org-string-width new) l)
 		      (setq n (format f new))
 		    (setq n (concat new "|") org-table-may-need-update t)))
 	      (if (equal (string-to-char n) ?-) (setq n (concat " " n)))
-- 
1.9.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Improve-table-column-narrowing-with-variable-width-s.patch --]
[-- Type: text/x-diff, Size: 2143 bytes --]

From 7850f739a6ef21bcc92763150861a67798282f12 Mon Sep 17 00:00:00 2001
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Date: Wed, 13 Feb 2013 16:46:33 +0800
Subject: [PATCH 2/2] Improve table column narrowing with variable-width
 strings

lisp/org-table.el (org-table-align): Use org-string-width to make a
better guess as to how wide a narrowed table column should actually
be. This won't work perfectly with variable-width scripts, as it still
only narrows in full-column increments, but it's better than before.
---
 lisp/org-table.el | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 5e0177f..d1f9686 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -765,7 +765,7 @@ When nil, simply write \"#ERROR\" in corrupted fields.")
 	 (hfmt1 (concat
 		 (make-string sp2 ?-) "%s" (make-string sp1 ?-) "+"))
 	 emptystrings links dates emph raise narrow
-	 falign falign1 fmax f1 len c e space)
+	 falign falign1 fmax f1 f2 len c e space)
     (untabify beg end)
     (remove-text-properties beg end '(org-cwidth t org-dwidth t display t))
     ;; Check if we have links or dates
@@ -851,10 +851,19 @@ When nil, simply write \"#ERROR\" in corrupted fields.")
 		  (unless (> f1 1)
 		    (user-error "Cannot narrow field starting with wide link \"%s\""
 			   (match-string 0 xx)))
-		  (add-text-properties f1 (length xx) (list 'org-cwidth t) xx)
-		  (add-text-properties (- f1 2) f1
-				       (list 'display org-narrow-column-arrow)
-				       xx)))))
+		  (setq f2 (length xx))
+		  (if (= (org-string-width xx)
+			 f2)
+		      (setq f2 f1)
+		    (setq f2 1)
+		    (while (< (org-string-width (substring xx 0 f2))
+			      f1)
+		      (setq f2 (1+ f2))))
+		  (add-text-properties f2 (length xx) (list 'org-cwidth t) xx)
+		  (add-text-properties (if (>= (string-width (substring xx (1- f2) f2)) 2)
+					   (1- f2) (- f2 2)) f2
+					   (list 'display org-narrow-column-arrow)
+					   xx)))))
       ;; Get the maximum width for each column
       (push (apply 'max (or fmax 1) 1 (mapcar 'org-string-width column))
 	    lengths)
-- 
1.9.3

Re: mis-alignment in org-tables with Tibetan characters

[Bastien]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> The patch series that would not die!

:)

Applied, thanks!

-- 
 Bastien