Spacing within org-mode tables when diacritics are used

Spacing within org-mode tables when diacritics are used

[Leo Vivier]

Hi everyone,

I'm quite excited because it is the first time that I get to ask a 
question on a mailing list!
I've been wandering between this list, the wikis, and SE to get my 
answers so far, but the problem I've encountered now wasn't addressed in 
any of these.
Please tell me if the format of the email is not up to standard, and 
sorry for the inconvenience if this is the case.

My problems concerns the spacing within org-mode tables when diacritics 
(e.g. accented letters) are used.

Here's a MWE with emacs -q.

| 1 | test  | test |
| 2 | tést | test |
| 3 | test  | test |
| 4 | test  | test |

The 2nd line (with the `é') as an offset of 1 space compared to the others.
Pressing <tab> within any of the cells to realign the table doesn't fix it.

I think this might have something to do with multibyte characters.
Running (toggle-enable-multibyte-characters) shows that the é does 
indeed take two bytes.
Setting the language environment to French via (set-language-environment 
'french) solves this issue (i.e. pressing <tab> realigns the table 
properly), but this only last whilst the file remains opened.

Would you have an idea on how to fix this?

Thanks for the help.


-- 
Leo Vivier
English Studies & General Linguistics
Language Assistant, French Department
Reed College

Re: Spacing within org-mode tables when diacritics are used

[Eli Zaretskii]

> From: Leo Vivier <leo.vivier@gmail.com>
> Date: Thu, 31 Aug 2017 12:19:58 -0700
> 
> Here's a MWE with emacs -q.
> 
> | 1 | test  | test |
> | 2 | tést | test |
> | 3 | test  | test |
> | 4 | test  | test |
> 
> The 2nd line (with the `é') as an offset of 1 space compared to the others.
> Pressing <tab> within any of the cells to realign the table doesn't fix it.

Please show all the steps for reproducing this, not just the resulting
display.  From the rest of your text, I'm guessing that one needs to
turn on Org mode, create a table, and populate it with the above
contents.  But if I do this by hand, I don't see the problem, so maybe
I missed some step.

It is also important to describe how you typed é, assuming that you
typed it inside Emacs.  OTOH, if the table was created by importing a
file, please tell which bytes stand for é there.

Finally, please tell which versions of Emacs and Org you have
installed.

> I think this might have something to do with multibyte characters.
> Running (toggle-enable-multibyte-characters) shows that the é does 
> indeed take two bytes.

That couldn't be the reason.

> Setting the language environment to French via (set-language-environment 
> 'french) solves this issue (i.e. pressing <tab> realigns the table 
> properly), but this only last whilst the file remains opened.

What is your default language-environment?

Re: Spacing within org-mode tables when diacritics are used

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Leo Vivier <leo.vivier@gmail.com>
>> Date: Thu, 31 Aug 2017 12:19:58 -0700
>> 
>> Here's a MWE with emacs -q.
>> 
>> | 1 | test  | test |
>> | 2 | tést | test |
>> | 3 | test  | test |
>> | 4 | test  | test |
>> 
>> The 2nd line (with the `é') as an offset of 1 space compared to the others.
>> Pressing <tab> within any of the cells to realign the table doesn't fix it.
>
> Please show all the steps for reproducing this, not just the resulting
> display.  From the rest of your text, I'm guessing that one needs to
> turn on Org mode, create a table, and populate it with the above
> contents.  But if I do this by hand, I don't see the problem, so maybe
> I missed some step.
>
> It is also important to describe how you typed é, assuming that you
> typed it inside Emacs.  OTOH, if the table was created by importing a
> file, please tell which bytes stand for é there.
>
> Finally, please tell which versions of Emacs and Org you have
> installed.
>
>> I think this might have something to do with multibyte characters.
>> Running (toggle-enable-multibyte-characters) shows that the é does 
>> indeed take two bytes.
>
> That couldn't be the reason.

Why not? Org table formatting used to be done with `length' on the
string, I changed it to `string-width' precisely to fix wonkiness with
multibyte characters. But that was ages ago, he'd have to have a really
old Org.

Eric

Re: Spacing within org-mode tables when diacritics are used

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Fri, 01 Sep 2017 14:02:20 -0700
> 
> >> I think this might have something to do with multibyte characters.
> >> Running (toggle-enable-multibyte-characters) shows that the é does 
> >> indeed take two bytes.
> >
> > That couldn't be the reason.
> 
> Why not? Org table formatting used to be done with `length' on the
> string, I changed it to `string-width' precisely to fix wonkiness with
> multibyte characters.

Because 'length' returns the number of characters in a string, not the
number of bytes.  See its doc string.

Re: Spacing within org-mode tables when diacritics are used

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Fri, 01 Sep 2017 14:02:20 -0700
>> 
>> >> I think this might have something to do with multibyte characters.
>> >> Running (toggle-enable-multibyte-characters) shows that the é does 
>> >> indeed take two bytes.
>> >
>> > That couldn't be the reason.
>> 
>> Why not? Org table formatting used to be done with `length' on the
>> string, I changed it to `string-width' precisely to fix wonkiness with
>> multibyte characters.
>
> Because 'length' returns the number of characters in a string, not the
> number of bytes.  See its doc string.

I know... Obviously this isn't worth arguing about, but I still suspect
this is the change OP doesn't have:

http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=4af31090b4420978067bc24512467bcc9a78a7f1

Re: Spacing within org-mode tables when diacritics are used

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Sat, 02 Sep 2017 07:58:17 -0700
> 
> > Because 'length' returns the number of characters in a string, not the
> > number of bytes.  See its doc string.
> 
> I know... Obviously this isn't worth arguing about, but I still suspect
> this is the change OP doesn't have:
> 
> http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=4af31090b4420978067bc24512467bcc9a78a7f1

Maybe you are right, although string-width shouldn't affect display of
normal-width characters such as é.  It only affects double-width
characters, or characters that Emacs composes into a single grapheme
cluster.  Multibyte-ness per se doesn't come into play here.  That's
why I asked how that é was typed, and what was its encoding on the
file (if the table was imported from a file).  If there were actually 
2 characters, e and ́, then yes, using length would produce wrong
results that string-width would fix.

Re: Spacing within org-mode tables when diacritics are used

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Sat, 02 Sep 2017 07:58:17 -0700
>> 
>> > Because 'length' returns the number of characters in a string, not the
>> > number of bytes.  See its doc string.
>> 
>> I know... Obviously this isn't worth arguing about, but I still suspect
>> this is the change OP doesn't have:
>> 
>> http://orgmode.org/cgit.cgi/org-mode.git/commit/?id=4af31090b4420978067bc24512467bcc9a78a7f1
>
> Maybe you are right, although string-width shouldn't affect display of
> normal-width characters such as é.  It only affects double-width
> characters, or characters that Emacs composes into a single grapheme
> cluster.  Multibyte-ness per se doesn't come into play here.  That's
> why I asked how that é was typed, and what was its encoding on the
> file (if the table was imported from a file).  If there were actually 
> 2 characters, e and ́, then yes, using length would produce wrong
> results that string-width would fix.

Ah, interesting -- thanks for the clarification.

Re: Spacing within org-mode tables when diacritics are used

[Mario Castelán Castro]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

On 02/09/17 10:13, Eli Zaretskii wrote:
> Maybe you are right, although string-width shouldn't affect display of
> normal-width characters such as é.  It only affects double-width
> characters, or characters that Emacs composes into a single grapheme
> cluster.  Multibyte-ness per se doesn't come into play here.  That's
> why I asked how that é was typed, and what was its encoding on the
> file (if the table was imported from a file).  If there were actually 
> 2 characters, e and ́, then yes, using length would produce wrong
> results that string-width would fix.

So what you are saying is that GNU Emacs is broken in that it assumes
that 1 Unicode code point corresponds to 1 glyph.

-- 
Do not eat animals; respect them as you respect people.
https://duckduckgo.com/?q=how+to+(become+OR+eat)+vegan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Spacing within org-mode tables when diacritics are used

[Eli Zaretskii]

> From: Mario Castelán Castro <marioxcc.MT@yandex.com>
> Date: Sat, 2 Sep 2017 13:52:37 -0500
> 
> On 02/09/17 10:13, Eli Zaretskii wrote:
> > Maybe you are right, although string-width shouldn't affect display of
> > normal-width characters such as é.  It only affects double-width
> > characters, or characters that Emacs composes into a single grapheme
> > cluster.  Multibyte-ness per se doesn't come into play here.  That's
> > why I asked how that é was typed, and what was its encoding on the
> > file (if the table was imported from a file).  If there were actually 
> > 2 characters, e and ́, then yes, using length would produce wrong
> > results that string-width would fix.
> 
> So what you are saying is that GNU Emacs is broken in that it assumes
> that 1 Unicode code point corresponds to 1 glyph.

Which part of what I wrote led you to that conclusion?  That's not
what I meant, not at all.  What I meant was that a Lisp program is
broken if it uses 'length' to compute how many glyphs (the accurate
term is "grapheme clusters", not "glyphs") correspond to the
characters of a string.  It should use 'string-width' instead.

Re: Spacing within org-mode tables when diacritics are used

[Mario Castelán Castro]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

On 02/09/17 14:21, Eli Zaretskii wrote:
>> From: Mario Castelán Castro <marioxcc.MT@yandex.com>
>> So what you are saying is that GNU Emacs is broken in that it assumes
>> that 1 Unicode code point corresponds to 1 glyph.
> 
> Which part of what I wrote led you to that conclusion?  That's not
> what I meant, not at all.  What I meant was that a Lisp program is
> broken if it uses 'length' to compute how many glyphs (the accurate
> term is "grapheme clusters", not "glyphs") correspond to the
> characters of a string.  It should use 'string-width' instead.

My apologies, I should have said “org-mode”, not “GNU Emacs” in the
message of mine that you quoted.

-- 
Do not eat animals; respect them as you respect people.
https://duckduckgo.com/?q=how+to+(become+OR+eat)+vegan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Spacing within org-mode tables when diacritics are used

[Eric Abrahamsen]

Mario Castelán Castro <marioxcc.MT@yandex.com> writes:

> On 02/09/17 14:21, Eli Zaretskii wrote:
>>> From: Mario Castelán Castro <marioxcc.MT@yandex.com>
>>> So what you are saying is that GNU Emacs is broken in that it assumes
>>> that 1 Unicode code point corresponds to 1 glyph.
>> 
>> Which part of what I wrote led you to that conclusion?  That's not
>> what I meant, not at all.  What I meant was that a Lisp program is
>> broken if it uses 'length' to compute how many glyphs (the accurate
>> term is "grapheme clusters", not "glyphs") correspond to the
>> characters of a string.  It should use 'string-width' instead.
>
> My apologies, I should have said “org-mode”, not “GNU Emacs” in the
> message of mine that you quoted.

Org mode pre-early-2013, yes.

Re: Spacing within org-mode tables when diacritics are used

[Mario Castelán Castro]

[-- Attachment #1: Type: text/plain, Size: 393 bytes --]

On 02/09/17 16:16, Eric Abrahamsen wrote:
> Mario Castelán Castro <marioxcc.MT@yandex.com> writes:
>> My apologies, I should have said “org-mode”, not “GNU Emacs” in the
>> message of mine that you quoted.
> 
> Org mode pre-early-2013, yes.

Understood.

-- 
Do not eat animals; respect them as you respect people.
https://duckduckgo.com/?q=how+to+(become+OR+eat)+vegan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]