character composition seems much worse than before

character composition seems much worse than before

[Miles Bader]

For a while, maybe 4-5 months ago (?), automatic composition
in Emacs seemed to be working really well -- things just
_worked_, and worked correctly in almost every case.

Now it seems to be a fair bit worse... E.g.:

I do (using emacs -Q, latest trunk):

 (1) Insert a letter "e" by typing it

 (2) Insert a unicode combinging accute accent by doing "C-x 8 RET"
     and then "COMBINING ACCUTE ACCENT" RET

*Previously* (meaning N months ago), that would then properly
display an accented "e" character, which looked exactly like
the precombined unicode "é".  Perfect!  One could even add
multiple combining accents and they all displayed correctly
stacked on top of each other.  It was impressive.

But now:

 + It doesn't work by default -- it seems that I need to do
   "M-x auto-composition-mode" first.  Why is this necessary?
   Aren't combining characters almost always intended to, well,
   combine?  Why would we want them to _not_ combine by
   default?

 + The accent positioning is fairly clearly wrong.  I know this
   is font-specific, but this is using "Bitstream Vera Sans
   Mono" which is a pretty standard free font, and with which
   the positioning seemed to be perfect before.

 + Some combining accents seem to simply not work at all.  If I
   insert an "e", and then a "COMBINING CIRCUMFLEX ACCENT"
   character, the latter simply displays as a separate
   character, not stacked on top of the "e".  This used to work
   properly (and is a legimate combination -- there's a
   precombined "LATIN SMALL LETTER E WITH CIRCUMFLEX"
   character.

 + Multiple combining accents seem to display very badly, not
   stacked properly, but instead seemingly all at the same
   location (resulting in an unreadable blob).  This also used
   to work "properly."

Maybe there's something wrong with my system or my font, but I
think it's a reasonably standard Debian distro, with a standard
font...

Do these things work for other people?

[Well, I guess the question of whether "auto-composition-mode"
is valid regardless.]

Thanks,

-Miles

-- 
Patience, n. A minor form of despair, disguised as a virtue.

Re: character composition seems much worse than before

[Miles Bader]

Oh, one more issue:

 + Previous, cursor-movement would treat char+combining-accent as a
   single unit (whereas text modification commands would not); this was
   pretty convenient, and basically seemed the right thing.

   Now cursor movement movement seems to stop "between" the character
   and its combining accents, with a little thin-line cursor.  If there
   are multiple combining accents, it stops at _each_ position, moving
   the thin cursor one pixel for each -- and even worse, there seems to
   be a single-pixel of "whitespace" for each accent!  I think it's very
   wrong for multiple accents to result in extra whitespace....!

[Ok, maybe that's two related issues...]

Thanks,

-Miles
   
-- 
Come now, if we were really planning to harm you, would we be waiting here,
 beside the path, in the very darkest part of the forest?

Re: character composition seems much worse than before

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Date: Mon, 08 Nov 2010 12:53:25 +0900
> 
>  + Previous, cursor-movement would treat char+combining-accent as a
>    single unit (whereas text modification commands would not); this was
>    pretty convenient, and basically seemed the right thing.
> 
>    Now cursor movement movement seems to stop "between" the character
>    and its combining accents, with a little thin-line cursor.

This seems to indicate that compositions are turned off, and what you
see is the font backend's features, not Emacs features.

The fact that you needed to invoke auto-composition-mode seems
consistent with this, because auto-composition-mode is turned on by
default (AFAIK), and this invocation turns it off, I think.

Re: character composition seems much worse than before

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
> This seems to indicate that compositions are turned off, and what you
> see is the font backend's features, not Emacs features.

Hmm, how can I distinguish?

> The fact that you needed to invoke auto-composition-mode seems
> consistent with this, because auto-composition-mode is turned on by
> default (AFAIK), and this invocation turns it off, I think.

You seem to be right -- auto-composition-mode _is_ turned on by default.

So the question becomes:  "Why don't combining characters work at all
when auto-composition-mode is turned on?"  [an odd situation to be sure!]

-Miles

-- 
.Numeric stability is probably not all that important when you're guessing.

Re: character composition seems much worse than before

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Cc: emacs-devel@gnu.org
> System-Type: x86_64-unknown-linux-gnu
> Blat: Foop
> Date: Mon, 08 Nov 2010 13:11:58 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > This seems to indicate that compositions are turned off, and what you
> > see is the font backend's features, not Emacs features.
> 
> Hmm, how can I distinguish?

The fact that cursor motion doesn't treat the composed character as a
single unit is the telltale sign.

Re: character composition seems much worse than before

[Kenichi Handa]

In article <buoeiawcv3l.fsf@dhlpc061.dev.necel.com>, Miles Bader <miles@gnu.org> writes:

> You seem to be right -- auto-composition-mode _is_ turned on by default.

> So the question becomes:  "Why don't combining characters work at all
> when auto-composition-mode is turned on?"  [an odd situation to be sure!]

It works fine for me with the latest code.  I have no idea
why it doesn't for you.  I've not modified the composition
related codes for a while.  At the moment, I don't have
"bitstream vera sans mono" font.  Could you please try with
"dejavu sans mono" or "freemono" fonts?

---
Kenichi Handa
handa@m17n.org

Re: character composition seems much worse than before

[Miles Bader]

Kenichi Handa <handa@m17n.org> writes:
> It works fine for me with the latest code.  I have no idea
> why it doesn't for you.  I've not modified the composition
> related codes for a while.  At the moment, I don't have
> "bitstream vera sans mono" font.  Could you please try with
> "dejavu sans mono" or "freemono" fonts?

Ok (though AFAIK, "dejavu sans mono" is just "bitstream vera sans mono"
with extra characters added).

-Miles

-- 
Selfish, adj. Devoid of consideration for the selfishness of others.

Re: character composition seems much worse than before

[Miles Bader]

Miles Bader <miles@gnu.org> writes:
>> It works fine for me with the latest code.  I have no idea
>> why it doesn't for you.  I've not modified the composition
>> related codes for a while.  At the moment, I don't have
>> "bitstream vera sans mono" font.  Could you please try with
>> "dejavu sans mono" or "freemono" fonts?
>
> Ok (though AFAIK, "dejavu sans mono" is just "bitstream vera sans mono"
> with extra characters added).

Argh.

So with the following command:

   emacs -Q -font 'dejavu sans mono 12'

everything works properly!

but this:

   emacs -Q -font 'bitstream vera sans mono 12'

fails as I described before (most importantly, fails to combine
characters at all when auto-composition-mode is enabled).

This is pretty irritating, as Bitstream Vera Sans Mono seems to be what
Emacs uses by default on Debian...!

-Miles

-- 
Optimist, n. A proponent of the doctrine that black is white.

Re: character composition seems much worse than before

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Mon, 08 Nov 2010 16:51:51 +0900
> 
>    emacs -Q -font 'bitstream vera sans mono 12'
> 
> fails as I described before (most importantly, fails to combine
> characters at all when auto-composition-mode is enabled).

Do you see any problems in how Emacs displays the HELLO file?  E.g.,
does the cursor move across the composed characters as a single unit
there?

In other words, is the problem with auto-composing only some
characters, or is it with compositions in general, with the "bitstream
vera sans mono 12" font?

Re: character composition seems much worse than before

[Andreas Schwab]

Miles Bader <miles@gnu.org> writes:

> So with the following command:
>
>    emacs -Q -font 'dejavu sans mono 12'
>
> everything works properly!
>
> but this:
>
>    emacs -Q -font 'bitstream vera sans mono 12'
>
> fails as I described before (most importantly, fails to combine
> characters at all when auto-composition-mode is enabled).

Note that bistream vera contains very few characters, and none of the
combining characters.  Perhaps the issue is that the two characters are
taken from different fonts?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: character composition seems much worse than before

[Miles Bader]

Andreas Schwab <schwab@linux-m68k.org> writes:
>> fails as I described before (most importantly, fails to combine
>> characters at all when auto-composition-mode is enabled).
>
> Note that bistream vera contains very few characters, and none of the
> combining characters.  Perhaps the issue is that the two characters are
> taken from different fonts?

You are right!

That explains things.

It seems that since Dejavu is pretty much a superset of Bitstream Vera,
the former should always be preferred if both are installed (I have both
installed here); I wonder where that should be done...

Thanks,

-Miles

-- 
Happiness, n. An agreeable sensation arising from contemplating the misery of
another.

Re: character composition seems much worse than before

[Andreas Schwab]

Miles Bader <miles@gnu.org> writes:

> It seems that since Dejavu is pretty much a superset of Bitstream Vera,
> the former should always be preferred if both are installed (I have both
> installed here); I wonder where that should be done...

I guess you need to adjust the monospace/sans-serif/serif aliases.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: character composition seems much worse than before

[Kenichi Handa]

In article <buooca0b6co.fsf@dhlpc061.dev.necel.com>, Miles Bader <miles@gnu.org> writes:

> So with the following command:

>    emacs -Q -font 'dejavu sans mono 12'

> everything works properly!

> but this:

>    emacs -Q -font 'bitstream vera sans mono 12'

> fails as I described before (most importantly, fails to combine
> characters at all when auto-composition-mode is enabled).

Then, I suspect that the font doesn't have glyphs for
combining characters (U+0300...U+037F), and thus Emacs uses
the other font for those characters, which results in
no-composition (Emacs can compose characters of the same
font only).  If you have a tool to see glyphs of ttf fonts,
please check it.  If you don't have such a tool, please try
the program "otfview" included in libotf-bin package.

> This is pretty irritating, as Bitstream Vera Sans Mono seems to be what
> Emacs uses by default on Debian...!

How about the other programs?  For instance, can gedit
display the same text correctly with that font?

---
Kenichi Handa
handa@m17n.org

Re: character composition seems much worse than before

[Andreas Schwab]

Kenichi Handa <handa@m17n.org> writes:

> If you have a tool to see glyphs of ttf fonts,

Like xfd.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: character composition seems much worse than before

[Miles Bader]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

Kenichi Handa <handa@m17n.org> writes:
>> This is pretty irritating, as Bitstream Vera Sans Mono seems to be
>> what Emacs uses by default on Debian...!
>
> How about the other programs?  For instance, can gedit display the
> same text correctly with that font?

Yes, it seems to.

I tried 3 different fonts, and two different character combinations
("e" + COMBINING ACUTE ACCENT [0x301], and "e" + COMBINING CIRCUMFLEX
ACCENT [0x302] + COMBINING ACUTE ACCENT), in Emacs and gedit:

                                 -Emacs-     -gedit-
  "Dejavu Sans Mono"            PERFECT     PERFECT
  "Bitstream Vera Sans Mono"    NONE        OK
  "Lucida Typewriter"           NONE        PASSABLE

According to "otfview", only Dejavu Sans Mono has those combining
accents (0x301 and 0x302), so gedit must have been doing some
substitution -- it looked pretty good though, especially with
Bitstream Vera Sans Mono.  [I tried Lucida Sans too because while the
letter shapes in Dejavu and Vera are basically identical, Lucida Sans
looks different, so it's easier to tell if substitutions are being
made.]

   "PERFECT" means:  correct letter-shape used (no wacky font
   substitution), perfect spacing/alignment/accent-positioning.

   "OK" means:  correct letter-shape used, correct spacing,
   accent-positioning slightly (~1 pixel) off but basically OK.

   "PASSABLE" means:  correct letter-shape used, correct spacing,
   accent-positioning slightly (~1 pixel) off but basically OK for a
   single accent, *but* smashed-together/overstruck for two
   simultaneous accents.

   "NONE" means: no combining done, letter and accents displayed as
   separate characters.

So gedit seems to be doing some substitution or something, but it does
a fairly good job of it except in the case of two accents being used
simultaneously.

-Miles

p.s. I've attached the text file I used for testing...



[-- Attachment #2: test file for combining accents --]
[-- Type: application/octet-stream, Size: 92 bytes --]

this is a test é 1lI|0Oabcdef
ếếếếếế|
éééééé|
eeeeee|

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]



-- 
"Most attacks seem to take place at night, during a rainstorm, uphill,
 where four map sheets join."   -- Anon. British Officer in WW I

Re: character composition seems much worse than before

[Miles Bader]

Miles Bader <miles@gnu.org> writes:
> So gedit seems to be doing some substitution or something, but it does
> a fairly good job of it except in the case of two accents being used
> simultaneously.

Correction -- gedit _does_ do a good job of two simultaneous accents
with Bitstream Vera Sans Mono (maybe it knows to grab the accent info
from Dejavu Sans Mono?), but two accents look a bit screwed up using
Lucida Typewriter.

-Miles

-- 
.Numeric stability is probably not all that important when you're guessing.

Re: character composition seems much worse than before

[Julien Danjou]

On Tue, Nov 09 2010, Miles Bader wrote:

> Correction -- gedit _does_ do a good job of two simultaneous accents
> with Bitstream Vera Sans Mono (maybe it knows to grab the accent info
> from Dejavu Sans Mono?), but two accents look a bit screwed up using
> Lucida Typewriter.

I think it's not gedit, but pango which does that behinds.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: character composition seems much worse than before

[Kenichi Handa]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]

In article <buofwvbgiby.fsf@dhlpc061.dev.necel.com>, Miles Bader <miles@gnu.org> writes:

> I tried 3 different fonts, and two different character combinations
> ("e" + COMBINING ACUTE ACCENT [0x301], and "e" + COMBINING CIRCUMFLEX
> ACCENT [0x302] + COMBINING ACUTE ACCENT), in Emacs and gedit:

>                                  -Emacs-     -gedit-
>   "Dejavu Sans Mono"            PERFECT     PERFECT
>   "Bitstream Vera Sans Mono"    NONE        OK
>   "Lucida Typewriter"           NONE        PASSABLE

I downloaded "Bitstream Vera Sans Mono" from
http://ftp.gnome.org/pub/GNOME/sources/ttf-bitstream-vera/1.10/
and tried it with gedit, but when I choose that font through
the menu: "Edit->Preference->Font&Colors->Editor Font",
gedit doesn't compose the characters (see the attached
image).  How did you specify the font for gedit?

---
Kenichi Handa
handa@m17n.org


[-- Attachment #2: temp.png --]
[-- Type: image/png, Size: 15265 bytes --]