Placement of HEBREW MAQAF (diacritical)

Placement of HEBREW MAQAF (diacritical)

[Amit Ramon]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

Hello,

The version of Emacs that I pulled and built two days ago doesn't
place the HEBREW PUNCTUATION MAQAF (05BE) in the right place. The
maqaf is like a dash and it should be placed between letters, but it
seems that now Emacs places it above the letter that precedes it. It
was working fine with the version that I built ~2 weeks ago. I'm
running Emacs on Debian Linux and X. I checked this with both Lucida
Sans and DejaVu Sans fonts. What also seems strange to me is that it
is not counted in the columns - what I mean is that the cursor column
position is not taking the maqaf into account - like it is not there.

I'm attaching two images of the same text - one inside Emacs and the
other in mlterm (that's a RTL-aware terminal emulator), both using the
same fonts. mlterm displays the maqaf correctly, but not Emacs. Also
attached is the text file that contains the text (mot.txt).

I'm also attaching the output of what-cursor-position, which again
looks strange to me since it talks about 'composing', which as far as
I understand should not occur in this case.

This problem occurs whether or not the bidi reordering is
active. Revision is 100853.

Thanks,

Amit

[-- Attachment #2: mot-emacs-24.png --]
[-- Type: image/png, Size: 854 bytes --]

[-- Attachment #3: mot-mlterm.png --]
[-- Type: image/png, Size: 876 bytes --]

[-- Attachment #4: mot.txt --]
[-- Type: text/plain, Size: 16 bytes --]

מוט־קטן

[-- Attachment #5: tet-maqaf --]
[-- Type: text/plain, Size: 810 bytes --]

        character: ט (1496, #o2730, #x5d8)
preferred charset: unicode (Unicode (ISO10646))
       code point: 0x05D8
           syntax: w 	which means: word
         category: .:Base
         to input: type "y" with hebrew-lyx
      buffer code: #xD7 #x98
        file code: #xD7 #x98 (encoded by coding system utf-8-unix)
          display: composed to form "ט־" (see below)

Composed with the following character(s) "־" using this font:
  xft:-unknown-DejaVu Sans-normal-normal-normal-*-13-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 1 1496 1324 8 1 7 7 0 nil]
  [0 1 1470 1308 5 0 5 8 -6 [-7 0 0]]

Character code properties: customize what to show
  name: HEBREW LETTER TET
  general-category: Lo (Letter, Other)

There are text properties here:
  fontified            t

[back]

Re: Placement of HEBREW MAQAF (diacritical)

[Yair F.]

Amit Ramon <amit.ramon <at> gmail.com> writes:

> Hello,
> 
> The version of Emacs that I pulled and built two days ago doesn't
> place the HEBREW PUNCTUATION MAQAF (05BE) in the right place.

Hello,

I know about this problem as well. There are some more characters which
are not defined properly for composition. I wasn't able to fully
understand how to define the auto-composition characters.  Take a look at
lisp/language/hebrew.el and how composition-function-table is defined there.

Yair

Re: Placement of HEBREW MAQAF (diacritical)

[Eli Zaretskii]

> From: Yair F. <yair.f.lists@gmail.com>
> Date: Wed, 28 Jul 2010 15:39:33 +0000 (UTC)
> 
> I wasn't able to fully understand how to define the auto-composition
> characters.

This area in Emacs is notoriously under-documented.

How about if you ask questions about things you don't understand?  We
could then ask Handa-san to answer them, and use the answers as a
starting point for documenting this in the ELisp manual.

TIA

Re: Placement of HEBREW MAQAF (diacritical)

[Yair F]

On Wed, Jul 28, 2010 at 8:51 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Yair F. <yair.f.lists@gmail.com>
> How about if you ask questions about things you don't understand?  We
> could then ask Handa-san to answer them, and use the answers as a
> starting point for documenting this in the ELisp manual.

Here are my questions:
1. Given the regular expressions below, how should
composition-function-table be adjust so only those patterns are
composed?
2. How to optimize the patterns in order not to slow down display of
letters without points and give hebrew-composable-pattern-basic priority
over hebrew-composable-pattern-full performance-wise?
3. Is the composition of script-specific characters from Hebrew bloc
(0590) with non-script-specific marks from the combining marks bloc
(0300) currently possible?
4. How to set the language environment so composition displays the correct
presentation form for the given  language?

I believe if we get the answers for these questions we are nearly done
with Hebrew composition. The only missing thing would be influence of
ZWJ, ZWNJ and CGJ on certain combinations.

Yair

(defconst hebrew-composable-no-rafe-pattern
  (concat
   "[\u05D6-\u05D9\u05DC-\u05E2\u05E5-\u05E8]" ;; base
   "\u05BC?"                        ;; 0-1 marks of dagesh
   "[\u05B0-\u05B9\u05BB\u05C7]?"   ;; 0-1 marks of niqqud
   )
  "Regexp matching letters that cannot accept rafe and basic
composition order.")
(defconst hebrew-composable-rafe-pattern
  (concat
   "[\u05D0-\u05D4\u05DA\u05DB\u05E4\u05E5\u05EA]" ;; base
   "[\u05BC\u05BF]?"                ;; 0-1 marks of dagesh/rafe
   "[\u05B0-\u05B9\u05BB\u05C7]?"   ;; 0-1 marks of niqqud
   )
  "Regexp matching letters that cannot accept rafe and basic
composition order.")
(defconst hebrew-composable-vav-pattern
  (concat
   "\u05D5"                         ;; base (vav)
   "\u05BC?"                        ;; 0-1 marks of dagesh (actually shuruq)
   "[\u05B0-\u05BB\u05C7]?"         ;; 0-1 marks of extended niqqud
   )
  "Regexp matching composition order of letter vav.")
(defconst hebrew-composable-shin-pattern
  (concat
   "\u05E9"                         ;; base (shin)
   "\u05BC?"                        ;; 0-1 marks of dagesh
   "[\u05C1\u05C2]?"                ;; 0-1 marks of shin dot
   "[\u05B0-\u05B9\u05BB\u05C7]?"   ;; 0-1 marks of niqqud
   )
  "Regexp matching composition order of letter Shin.")
(defconst  hebrew-composable-cantillation
  (concat
     "[\u0591\u0596\u059B\u05A3-\u05A7\u05AA\u05BD\u05C5]*" ;;  marks
of low-center cantillation marks
     "[\u0323-\u0325\u0332\u0333]?" ;; 0-1 marks of other low marks
     "[\u059A\u05AD]?" ;; 0-1 marks of low-right cantillation marks
     "[\u059D\u05A0]?" ;; 0-1 marks of high-right cantillation marks
     "[\u0593-\u0595\u0597\u0598\u059C\u059E\u05A1\u05A8\u05A9\u05AB\u05AC\u05AF\u05C4]*"
;; marks of high-center cantillation marks
     "[\u0305\u0307\u0308\u030A]?" ;; 0-1 marks of other high marks
     "[\u0592\u0599\u05A9\u5AE]?" ;; 0-1 marks of high-left cantillation marks
   )
  "Regexp matching composition order of cantillation and other marks.")
(defconst hebrew-maqaf-compisition
  "\u05BE\u05AF?"
  "Regexp matching composition order of Maqaf.")
(defconst hebrew-nun-hafukha-compisition
  "\u05C6\u0307?"
  "Regexp matching composition order of Nun Hafukha.")
(defconst hebrew-composable-pattern-basic
  (concat
   "\\("
   hebrew-composable-no-rafe-pattern
   "\\|"
   hebrew-composable-rafe-pattern
   "\\|"
   hebrew-composable-vav-pattern
   "\\|"
   hebrew-composable-shin-pattern
   "\\)")
  "Regexp matching a composable sequence of Hebrew characters basic level.")
(defconst hebrew-composable-pattern-full
  (concat
   "\\("
   hebrew-composable-no-rafe-pattern
   hebrew-composable-cantillation
   "\\|"
   hebrew-composable-rafe-pattern
   hebrew-composable-cantillation
   "\\|"
   hebrew-composable-vav-pattern
   hebrew-composable-cantillation
   "\\|"
   hebrew-composable-shin-pattern
   hebrew-composable-cantillation
   "\\|"
   hebrew-maqaf-compisition
   "\\|"
   hebrew-nun-hafukha-compisition
   "\\)")
  "Regexp matching a composable sequence of Hebrew characters basic level.")
(defconst hebrew-yiddish-composable-pattern
    (concat
     "\u05D0"	       ;; alef
     "[\u05B7\u05B8]?" ;; qamats or Patah
     "\\|"
     "[\u05D1\u05DA\u05DB\u05E4\u05EA]" ;; base
     "[\u05BC\u05BF]?" 			;; dagesh-rafe
     "\\|"
     "\u05D5"                         ;; base (vav)
     "\u05BC?"                        ;; 0-1 marks of dagesh (actually shuruq)
     "\\|"
     "\u05D9"                         ;; base (yod)
     "\u05B4"                        ;; This should be composed as  \uFB1D
     "\\|"
     "\u05E9"                         ;; base (shin)
     "[\u05C1\u05C2]?"                ;; 0-1 marks of shin dot
     "\\|"
     "\u05F2"                         ;; base (zwei yoden)
     "\u05B7"                        ;; This should be composed as  \uFB1F
     )
  "Regexp matching a composable sequence of Hebrew characters for
Yiddish language.")
(defconst hebrew-ladino-composable-pattern
  "[\u05D1\u05D2\u05D6\u05E4]\u05BF"  ;; all sould use \uFB1E insread of \05BF
  "Regexp matching a composable sequence of Hebrew characters for
Ladino (Judezmo) language.")