Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Kenichi Handa]

In article <AANLkTinrGPzGquxPmfigvZLzGbids39yU942sfwKGMYk@mail.gmail.com>, Yair F <yair.f.lists@gmail.com> writes:

> On Wed, Aug 18, 2010 at 5:06 AM, Kenichi Handa <handa@m17n.org> wrote:
> > I think it's about the time to decide how to display these
> > formatting characters: LRE, RLE, LRO, RLO, PDF, LRM, RLM.

> This is not a bidi-specific issue.

Yes.  I included emacs-devel@gnu.org in CC:.

> It is possible that these characters
> and other Unicode control characters needs to be trated as non-visiblle
> characters. The assitional characters are CGJ, IAA, IAS, ZWNBSP (BOM),
> IAT LSEP, PSEP, WJ, Invisible Operators, ans all Zero width characters:
> ZWSP, ZWNJ, ZWJ.

> All of these characters modify the environment aboud them but do not
> display glyphs. In some way they are like the TAB character.

> I am thinking of extending whitespace.el to make them visible if the
> user wishes, but IMO, they should bot be visible by default. See the
> impact on the HELLO fle.

Extending whitespace.el will be good.  But, anyway we must
hide those characters by default, so we need some char-table
to specify that.  Currently standard-display-table is not
made by default.  I think the first step is to made it by
default, and specify [] for all of those characters.

> > (1) invisible-mode (perhaps the default)
> >
> > Hide them, for instance, by
> >  (aset standard-display-table #x202e [])
> >
> > Then, you have to type C-f or C-b twice to pass over those
> > characters.  That means users can still put cursor anywhere
> > if he moves cursor carefully.
> Yes, but currently the cursore is "gone" if it is on invisible character.

"Gone"?  The cursor is displayed on the next visible
character.  It may be good to show a bar cursor of 1-pixel
width when it's on a zero-width character.

> > (2) light-visible-mode
> >
> > Show them by a space of 1-pixel width.
> >
> > (3) heavy-visible-mode
> >
> > Show them, for instance, by
> >  (aset standard-display-table #x202e [?[ ?R ?L ?O ?]])
> > perhaps with some color attribute (forground? underline?).
> >

> These modes seems less usable.

I may use (2) for my default setting.  And (3) may be usable
on terminal where we can't show a fancy glyph.

> > (4) fancy-visible-mode
> >
> > Show them by a glyph something like what you can see by this code:
> >
> Yes, SI 1452 indeed specify characters look U21B2 and U21B3 for
> keyboard marking for LRM and RLM.

What is "SI 1452"?  Anyway, for that we must implement a
fundamental mechanism in display engine.  I think it's not a
hard work.

I'd like to extend the elements of a display table.
Currently only a glyph vector or nil is allowed.  It seems
good to extend it so that it can completely control the
displaying of a character (like by face and display text
properties).

At least, for (2), I want to specify a space width (relative
or absolute), and for (4) I want to specify a special form
(list?) containing a mnemonic label.

---
Kenichi Handa
handa@m17n.org

Re: Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Kenichi Handa]

In article <tl7vd77z591.fsf@m17n.org>, Kenichi Handa <handa@m17n.org> writes:

> I'd like to extend the elements of a display table.
> Currently only a glyph vector or nil is allowed.  It seems
> good to extend it so that it can completely control the
> displaying of a character (like by face and display text
> properties).

> At least, for (2), I want to specify a space width (relative
> or absolute), and for (4) I want to specify a special form
> (list?) containing a mnemonic label.

I've just committed a basic infrastructure for displaying
non-graphic and no-font characters.  For that, instead of
extending the current display table, I implemented a new
char table glyphless-char-display.

------------------------------------------------------------
Char-table to control displaying of glyphless characters.
Each element, if non-nil, is an ASCII acronym string (displayed in a box)
or one of these symbols:
  hexa-code: display with hexadecimal character code in a box
  empty-box: display with an empty box
  thin-space: display with 1-pixel width space
  zero-width: don't display

It has one extra slot to control the display of a character for which
no font is found.  The value of the slot is `hexa-code' or `empty-box'.
The default is `empty-box'.
------------------------------------------------------------

It is easy to merge such functionality to a display table
later if we decide to do so.

But, there left these works.

(1) As it is not convenient for a user to specify each
    element of glyphless-char-display, I made variables
    char-acronym-table and glyphless-char-control.  The
    former is a simple char-table that contains acronym
    strings.  The latter is as this:

------------------------------------------------------------
glyphless-char-control is a variable defined in `characters.el'.
Its value is 
((format-control . thin-space)
 (no-font . hexa-code))

Documentation:
List of directives to control displaying of glyphless characters.

Each element has the form (TARGET . METHOD), where TARGET is a
symbol specifying the target character group to control, and
METHOD is a symbol specifying the method of displaying them.

TARGET must be one of these symbols:
  `c0-control': U+0000..U+001F.
  `c1-control': U+0080..U+009F.
  `format-control': Characters of Unicode General Category `Cf'.
     Ex: U+200C (ZWNJ), U+200E (LRM)), but don't include characters
     that have graphic image such as U+00AD (SHY).
  `no-font': characters for which no suitable font is found.

METHOD must be one of these symbols:
  `zero-width': don't display.
  `thin-space': display a thin space (1-pixel width).
  `empty-box': display an empty box.
  `acronym': display an acronum string in a box.
  `hexa-code': display a hexadecimal character code in a box.

Just setting this variable does not take effect.  Call the
function `update-glyphless-char-display' (which see) after
setting this variable.
------------------------------------------------------------

When acronum is specified, the function
update-glyphless-char-display sets elements in
glyphless-char-display to acronyms stored in
char-acronym-table.

Actually, glyphless-char-control should be a customizable
variable with proper set: function.  But, I don't know how
to write the correct 'defcustom' form.  Could someone DTRT?

(2) For Windows port:

I wrote x_draw_glyphless_glyph_string_foreground (in
w32term.c) by referring to x_draw_glyph_string_foreground in
the same file.  But, it doesn't draw acronym nor hexa-code.
I couldn't figure out what is wrong.  Please someone who
knows Windows code fix it.

(3) For mac port:

As I have no idea how to write a code for mac, the 'case
GLYPHLESS_GLYPH:' part in ns_draw_glyph_string (in nsterm.m)
is not finished.  Please fill in the correct code by
referring to x_draw_glyph_string_foreground in xterm.c.

---
Kenichi Handa
handa@m17n.org

Re: Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Mon, 01 Nov 2010 17:15:13 +0900
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> I've just committed a basic infrastructure for displaying
> non-graphic and no-font characters.

Thanks!

> Char-table to control displaying of glyphless characters.
> Each element, if non-nil, is an ASCII acronym string (displayed in a box)
> or one of these symbols:
>   hexa-code: display with hexadecimal character code in a box
    ^^^^^^^^^
Suggest to name this "hex-code" instead.

>   empty-box: display with an empty box
>   thin-space: display with 1-pixel width space
>   zero-width: don't display
> 
> It has one extra slot to control the display of a character for which
> no font is found.  The value of the slot is `hexa-code' or `empty-box'.
> The default is `empty-box'.

What will happen on a TTY?

> glyphless-char-control is a variable defined in `characters.el'.
  ^^^^^^^^^^^^^^^^^^^^^^
Suggest to name this "glyphless-char-display-control".

Re: Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Kenichi Handa]

In article <E1PCr8f-0003bZ-UC@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > Char-table to control displaying of glyphless characters.
> > Each element, if non-nil, is an ASCII acronym string (displayed in a box)
> > or one of these symbols:
> >   hexa-code: display with hexadecimal character code in a box
>     ^^^^^^^^^
> Suggest to name this "hex-code" instead.

As Google search found much more pages of "hex code" than
"hexa code", ok, I'll change the name..

> >   empty-box: display with an empty box
> >   thin-space: display with 1-pixel width space
> >   zero-width: don't display
> > 
> > It has one extra slot to control the display of a character for which
> > no font is found.  The value of the slot is `hexa-code' or `empty-box'.
> > The default is `empty-box'.

> What will happen on a TTY?

Ah, I forgot to mention about that.  At first, empty-box,
hexa-code, and acronym are displayed by using a new face
glyphless-char which is defined as this.

(defface glyphless-char
  '((((type tty)) :inherit underline)
    (t :height 0.6))
   ...)

And, for tty, as it's impossible to do the same thing as
graphic terminal, the current code does this:

thin-space: same as empty-box
hexa-code: display "U+XX", "U+XXXX", "U+XXXXXX" ,
	"E+XXXXXX" depends on the character code (the last
        one is for a character of code >= #x110000).
acronym: surround an acronym by "[" and "]" as this:
	"[ZWNJ]", "[LRE]"

At the moment, that is hardcoded in the function
produce_glyphless_glyph of term.c.

And, for tty, `no-font' means a character not encodable by
the terminal coding system.

> > glyphless-char-control is a variable defined in `characters.el'.
>   ^^^^^^^^^^^^^^^^^^^^^^
> Suggest to name this "glyphless-char-display-control".

I thought that name was too long, but if people prefer it, I
don't mind renaming to it.

---
Kenichi Handa
handa@m17n.org

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> Date: Mon, 01 Nov 2010 20:16:57 +0900
> 
> And, for tty, as it's impossible to do the same thing as
> graphic terminal, the current code does this:
> 
> thin-space: same as empty-box
> hexa-code: display "U+XX", "U+XXXX", "U+XXXXXX" ,
> 	"E+XXXXXX" depends on the character code (the last
>         one is for a character of code >= #x110000).
> acronym: surround an acronym by "[" and "]" as this:
> 	"[ZWNJ]", "[LRE]"
> 
> At the moment, that is hardcoded in the function
> produce_glyphless_glyph of term.c.
> 
> And, for tty, `no-font' means a character not encodable by
> the terminal coding system.

There are a few issues that perhaps need to be fixed:

  . If the default value of terminal-coding-system is nil, glyphless
    character display does not take effect: all the non-ASCII
    characters are displayed as question marks.  I think this is
    because safe_terminal_coding claims it can safely encode any
    character.  This look inconsistent and confusing, so I think we
    should fix that.

  . Composite characters are displayed as question marks regardless of
    the setting of glyphless-char-display-control.  I think this is
    because term.c:produce_composite_glyph does not consider the new
    glyphless-char display feature.  I think users will expect that
    composite characters behave like un-encodable characters on a TTY.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Kenichi Handa]

In article <83aalde3i0.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> There are a few issues that perhaps need to be fixed:

>   . If the default value of terminal-coding-system is nil, glyphless
>     character display does not take effect: all the non-ASCII
>     characters are displayed as question marks.  I think this is
>     because safe_terminal_coding claims it can safely encode any
>     character.  This look inconsistent and confusing, so I think we
>     should fix that.

There was a bug in setting term->charset_list.  I've just
installed a fix.

>   . Composite characters are displayed as question marks regardless of
>     the setting of glyphless-char-display-control.  I think this is
>     because term.c:produce_composite_glyph does not consider the new
>     glyphless-char display feature.  I think users will expect that
>     composite characters behave like un-encodable characters on a TTY.

I think composite characters should be sent to a tty as is
(i.e. just sending encoded characters), then terminal may
correctly compose them.

By the way, with the latest trunk code, on tty terminal,
Emacs positions cursor incorrectly (at column 1) on empty
lines except for end-of-buffer.  I don't know which code is
wrong but, at least, it didn't happen when I committed the
big changes for glyphless-char-display.

---
Kenichi Handa
handa@m17n.org

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Kenichi Handa]

In article <tl78w0sqyo8.fsf@m17n.org>, Kenichi Handa <handa@m17n.org> writes:
> >   . Composite characters are displayed as question marks regardless of
> >     the setting of glyphless-char-display-control.  I think this is
> >     because term.c:produce_composite_glyph does not consider the new
> >     glyphless-char display feature.  I think users will expect that
> >     composite characters behave like un-encodable characters on a TTY.

> I think composite characters should be sent to a tty as is
> (i.e. just sending encoded characters), then terminal may
> correctly compose them.

Ah, no, I misunderstood what you mean.  If the tty doesn't
support those characters, we should consult
glyphless-char-display.  I'll fix it soon.

---
Kenichi Handa
handa@m17n.org

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 17 Nov 2010 14:58:15 +0900
> 
> By the way, with the latest trunk code, on tty terminal,
> Emacs positions cursor incorrectly (at column 1) on empty
> lines except for end-of-buffer.  I don't know which code is
> wrong but, at least, it didn't happen when I committed the
> big changes for glyphless-char-display.

This is probably bug#7417.  It was introduced when I fixed the problem
with cursor positioning when displaying glyphless characters using the
zero-width method.

I will try to fix this as soon as I can.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Wed, 17 Nov 2010 07:21:50 -0500
> Cc: emacs-devel@gnu.org
> 
> > From: Kenichi Handa <handa@m17n.org>
> > Cc: emacs-devel@gnu.org
> > Date: Wed, 17 Nov 2010 14:58:15 +0900
> > 
> > By the way, with the latest trunk code, on tty terminal,
> > Emacs positions cursor incorrectly (at column 1) on empty
> > lines except for end-of-buffer.  I don't know which code is
> > wrong but, at least, it didn't happen when I committed the
> > big changes for glyphless-char-display.
> 
> This is probably bug#7417.  It was introduced when I fixed the problem
> with cursor positioning when displaying glyphless characters using the
> zero-width method.
> 
> I will try to fix this as soon as I can.

Done.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Mon, 01 Nov 2010 17:15:13 +0900
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> Actually, glyphless-char-control should be a customizable
> variable with proper set: function.  But, I don't know how
> to write the correct 'defcustom' form.  Could someone DTRT?

I didn't do this; volunteers are welcome.

> (2) For Windows port:
> 
> I wrote x_draw_glyphless_glyph_string_foreground (in
> w32term.c) by referring to x_draw_glyph_string_foreground in
> the same file.  But, it doesn't draw acronym nor hexa-code.
> I couldn't figure out what is wrong.  Please someone who
> knows Windows code fix it.

I fixed this.  The problem was that w32_draw_rectangle wiped out what
was drawn inside it; I switched the order, so the box is drawn before
the glyphs inside it.  Jason, could you please take a look and see if
I did everything right?

Assuming that on X the order doesn't matter (does it?), perhaps we
should switch the order there as well, for uniformity.

> (3) For mac port:
> 
> As I have no idea how to write a code for mac, the 'case
> GLYPHLESS_GLYPH:' part in ns_draw_glyph_string (in nsterm.m)
> is not finished.  Please fill in the correct code by
> referring to x_draw_glyph_string_foreground in xterm.c.

Not done; I have no idea how to do this.

In addition, I did this:

 . renamed glyphless-char-control to glyphless-char-display-control

 . renamed hexa-code to hex-code.

 . changed TTY display top enclose U+nnnn and "empty box" displays in
   "[]", to simulate a box and make the display easier to read.

 . documented this feature in NEWS and in the manual.

There are a few issues with TTY display, see my other mail.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> Date: Sat, 13 Nov 2010 15:44:55 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: jasonr@gnu.org, emacs-devel@gnu.org
> 
> In addition, I did this:
> 
>  . renamed glyphless-char-control to glyphless-char-display-control
> 
>  . renamed hexa-code to hex-code.
> 
>  . changed TTY display top enclose U+nnnn and "empty box" displays in
>    "[]", to simulate a box and make the display easier to read.
> 
>  . documented this feature in NEWS and in the manual.

Oh, and one other thing: there was a bug with positioning the cursor
on characters for which the zero-width method was specified in
glyphless-char-display-control.  (The cursor would jump to the end of
the line.)  I fixed that as well.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Kenichi Handa]

In article <83bp5te3s8.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > (2) For Windows port:
> > 
> > I wrote x_draw_glyphless_glyph_string_foreground (in
> > w32term.c) by referring to x_draw_glyph_string_foreground in
> > the same file.  But, it doesn't draw acronym nor hexa-code.
> > I couldn't figure out what is wrong.  Please someone who
> > knows Windows code fix it.

> I fixed this.  The problem was that w32_draw_rectangle wiped out what
> was drawn inside it; I switched the order, so the box is drawn before
> the glyphs inside it.

Are, I see.  Thank you for fixing it.

> Assuming that on X the order doesn't matter (does it?), perhaps we
> should switch the order there as well, for uniformity.

Perhaps.

> In addition, I did this:

>  . renamed glyphless-char-control to glyphless-char-display-control

>  . renamed hexa-code to hex-code.

Thank you.

>  . changed TTY display top enclose U+nnnn and "empty box" displays in
>    "[]", to simulate a box and make the display easier to read.

For U+NNNN, I decided not to use "[]" because it takes too
many columns.  I thought underline or some background color
(customizable through a face) was enough.  Don't you think 8
columns (instead of 6 columns) is annoying?

>  . documented this feature in NEWS and in the manual.

I dared not write that because I have not yet made my mind
which is better; current glyphless-char-display or extending
the normal display-table.

---
Kenichi Handa
handa@m17n.org

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: emacs-devel@gnu.org, jasonr@gnu.org
> Date: Wed, 17 Nov 2010 12:57:27 +0900
> 
> >  . changed TTY display top enclose U+nnnn and "empty box" displays in
> >    "[]", to simulate a box and make the display easier to read.
> 
> For U+NNNN, I decided not to use "[]" because it takes too
> many columns.  I thought underline or some background color
> (customizable through a face) was enough.  Don't you think 8
> columns (instead of 6 columns) is annoying?

It is annoying allright, but [U+1234][U+5678] is more readable than
U+1234U+5678.  However, if others disagree, I can change that back.

> >  . documented this feature in NEWS and in the manual.
> 
> I dared not write that because I have not yet made my mind
> which is better; current glyphless-char-display or extending
> the normal display-table.

By the way, one other issue is that display tables take precedence
over glyphless-char-display, in the sense that characters for which
there are non-trivial entries in the current display table are
displayed using the display table, disregarding any
glyphless-char-display-control settings.  If this is what we want, we
should probably document that.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Kenichi Handa <handa@m17n.org>
>> Cc: emacs-devel@gnu.org, jasonr@gnu.org
>> Date: Wed, 17 Nov 2010 12:57:27 +0900
>> 
>> >  . changed TTY display top enclose U+nnnn and "empty box" displays in
>> >    "[]", to simulate a box and make the display easier to read.
>> 
>> For U+NNNN, I decided not to use "[]" because it takes too
>> many columns.  I thought underline or some background color
>> (customizable through a face) was enough.  Don't you think 8
>> columns (instead of 6 columns) is annoying?
>
> It is annoying allright, but [U+1234][U+5678] is more readable than
> U+1234U+5678.  However, if others disagree, I can change that back.

\u1234\u5678 is even better.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Stefan Monnier]

>> It is annoying allright, but [U+1234][U+5678] is more readable than
>> U+1234U+5678.  However, if others disagree, I can change that back.

Even more so for chars followed by digits where the display could be
fairly confusing: U+123456 rather than [U+1234]56.

> \u1234\u5678 is even better.

I largely agree, since it's more in keeping with the Emacs tradition,
but again if digits follow, this is ambiguous: \u123456 (tho this could
also be displayed as \u1234\ 56).


        Stefan

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
>         Kenichi Handa <handa@m17n.org>
> Date: Wed, 17 Nov 2010 08:24:57 -0500
> 
> I largely agree, since it's more in keeping with the Emacs tradition,
> but again if digits follow, this is ambiguous: \u123456 (tho this could
> also be displayed as \u1234\ 56).

I think it's extremely unlikely to have \u1234 followed by 56, and we
already have that problem with octal escapes anyway.  Also, don't
forget that \u1234 is displayed in a special face, which makes it
stand out when followed by normal characters.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Kenichi Handa <handa@m17n.org>,  emacs-devel@gnu.org
> Date: Wed, 17 Nov 2010 13:55:07 +0100
> 
> > It is annoying allright, but [U+1234][U+5678] is more readable than
> > U+1234U+5678.  However, if others disagree, I can change that back.
> 
> \u1234\u5678 is even better.

I'm fine with that, too.  If no one objects, I will move to \uNNNN in
a couple of days.

Btw, how to display code points smaller than 256? \uNN or \u00NN?

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Stefan Monnier]

> Btw, how to display code points smaller than 256? \uNN or \u00NN?

I think \u00NN is the clear winner.


        Stefan

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Andreas Schwab <schwab@linux-m68k.org>, emacs-devel@gnu.org,
>         handa@m17n.org
> Date: Wed, 17 Nov 2010 18:59:15 -0500
> 
> > Btw, how to display code points smaller than 256? \uNN or \u00NN?
> 
> I think \u00NN is the clear winner.

It's according to RFC 5137, so yes.

But what about code points above MAX_UNICODE_CHAR?  Currently we
display them as [E+NNNNNN].  Use \eNNNNNN?

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Stefan Monnier]

>> > Btw, how to display code points smaller than 256? \uNN or \u00NN?
>> I think \u00NN is the clear winner.

> It's according to RFC 5137, so yes.

> But what about code points above MAX_UNICODE_CHAR?  Currently we
> display them as [E+NNNNNN].  Use \eNNNNNN?

I think \xNNNNNN would be preferable, since that's what we
use elsewhere.


        Stefan

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: schwab@linux-m68k.org, emacs-devel@gnu.org, handa@m17n.org
> Date: Thu, 18 Nov 2010 17:15:44 -0500
> 
> >> > Btw, how to display code points smaller than 256? \uNN or \u00NN?
> >> I think \u00NN is the clear winner.
> 
> > It's according to RFC 5137, so yes.
> 
> > But what about code points above MAX_UNICODE_CHAR?  Currently we
> > display them as [E+NNNNNN].  Use \eNNNNNN?
> 
> I think \xNNNNNN would be preferable, since that's what we
> use elsewhere.

Elsewhere where?

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: schwab@linux-m68k.org, emacs-devel@gnu.org, handa@m17n.org
> Date: Thu, 18 Nov 2010 17:15:44 -0500
> 
> >> > Btw, how to display code points smaller than 256? \uNN or \u00NN?
> >> I think \u00NN is the clear winner.
> 
> > It's according to RFC 5137, so yes.
> 
> > But what about code points above MAX_UNICODE_CHAR?  Currently we
> > display them as [E+NNNNNN].  Use \eNNNNNN?
> 
> I think \xNNNNNN would be preferable, since that's what we
> use elsewhere.

Done.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> But what about code points above MAX_UNICODE_CHAR?  Currently we
> display them as [E+NNNNNN].  Use \eNNNNNN?

\U00NNNNNN

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,  emacs-devel@gnu.org,  handa@m17n.org
> Date: Fri, 19 Nov 2010 10:53:57 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But what about code points above MAX_UNICODE_CHAR?  Currently we
> > display them as [E+NNNNNN].  Use \eNNNNNN?
> 
> \U00NNNNNN

That'd be a lie, wouldn't it?  These are not Unicode code points.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,  emacs-devel@gnu.org,  handa@m17n.org
>> Date: Fri, 19 Nov 2010 10:53:57 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > But what about code points above MAX_UNICODE_CHAR?  Currently we
>> > display them as [E+NNNNNN].  Use \eNNNNNN?
>> 
>> \U00NNNNNN
>
> That'd be a lie, wouldn't it?  These are not Unicode code points.

Does it matter?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Wed, 17 Nov 2010 07:26:15 -0500
> Cc: emacs-devel@gnu.org
> 
> By the way, one other issue is that display tables take precedence
> over glyphless-char-display, in the sense that characters for which
> there are non-trivial entries in the current display table are
> displayed using the display table, disregarding any
> glyphless-char-display-control settings.  If this is what we want, we
> should probably document that.

And another issue: the character group c0-control includes the
newline.  So if the display of this group is set to anything at all,
the newlines are not displayed as such.  I doubt that users would
expect or want that when they customize the display of c0-control.  So
maybe we should exempt newline (and perhaps also TAB) from this group;
users who really want that can always set glyphless-char-display
directly.

WDYT?

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

Ping!

> Date: Wed, 17 Nov 2010 21:39:49 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 
> 
> And another issue: the character group c0-control includes the
> newline.  So if the display of this group is set to anything at all,
> the newlines are not displayed as such.  I doubt that users would
> expect or want that when they customize the display of c0-control.  So
> maybe we should exempt newline (and perhaps also TAB) from this group;
> users who really want that can always set glyphless-char-display
> directly.
> 
> WDYT?

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Kenichi Handa]

Sorry for the late response on this matter.

In article <834obfd9iy.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: Eli Zaretskii <eliz@gnu.org>
> > Date: Wed, 17 Nov 2010 07:26:15 -0500
> > Cc: emacs-devel@gnu.org
> > 
> > By the way, one other issue is that display tables take precedence
> > over glyphless-char-display, in the sense that characters for which
> > there are non-trivial entries in the current display table are
> > displayed using the display table, disregarding any
> > glyphless-char-display-control settings.  If this is what we want, we
> > should probably document that.

IF that is what we want, we surely should document that.
But, as I wrote before, I'm still hesitating over which is
better; keeping glyphless-char-display separate from display
table, or integrating that functionality to display table.

> And another issue: the character group c0-control includes the
> newline.  So if the display of this group is set to anything at all,
> the newlines are not displayed as such.  I doubt that users would
> expect or want that when they customize the display of c0-control.  So
> maybe we should exempt newline (and perhaps also TAB) from this group;
> users who really want that can always set glyphless-char-display
> directly.

> WDYT?

I agree on exempting TAB and NL from c0-control group.

---
Kenichi Handa
handa@m17n.org

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 26 Nov 2010 21:29:43 +0900
> 
> But, as I wrote before, I'm still hesitating over which is
> better; keeping glyphless-char-display separate from display
> table, or integrating that functionality to display table.

What are the pros and cons, which make you hesitate?

> > And another issue: the character group c0-control includes the
> > newline.  So if the display of this group is set to anything at all,
> > the newlines are not displayed as such.  I doubt that users would
> > expect or want that when they customize the display of c0-control.  So
> > maybe we should exempt newline (and perhaps also TAB) from this group;
> > users who really want that can always set glyphless-char-display
> > directly.
> 
> > WDYT?
> 
> I agree on exempting TAB and NL from c0-control group.

Done.

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Kenichi Handa]

In article <83wrnz5fac.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > But, as I wrote before, I'm still hesitating over which is
> > better; keeping glyphless-char-display separate from display
> > table, or integrating that functionality to display table.

> What are the pros and cons, which make you hesitate?

Both glyphless-char-display and display table control the
displaying of each character.  I think such a control should
be done by a single mechanism.  At least it will benefit
users in a long run.

But, the biggest concern is about the backward
compatibility.  Currently, a display table element is nil or
vector of characters.  If there's a code that assumes that a
non-nil element is a vector of characters, it will be
broken.  Next, a display table is not inherited.  So, if
buffer-display-table or a window-specific display is set,
standard-display-table is not looked up.  At last, there are
several standard-display-XXX functions
(e.g. standard-display-8bit).  At the moment, I don't know
how to make the functionality of
glyphless-char-display-control go with them.

> > I agree on exempting TAB and NL from c0-control group.

> Done.

Thank you.

---
Kenichi Handa
handa@m17n.org

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Stefan Monnier]

> Both glyphless-char-display and display table control the
> displaying of each character.  I think such a control should
> be done by a single mechanism.  At least it will benefit
> users in a long run.

Agreed.

> Currently, a display table element is nil or vector of characters.
> If there's a code that assumes that a non-nil element is a vector of
> characters, it will be broken.

Let's not worry about that case for now.  Such code is probably rare and
easy to fix.

> Next, a display table is not inherited.  So, if
> buffer-display-table or a window-specific display is set,
> standard-display-table is not looked up.

IIRC that's a bug in itself that should be fixed.

> At last, there are several standard-display-XXX functions
> (e.g. standard-display-8bit).  At the moment, I don't know how to make
> the functionality of glyphless-char-display-control go with them.

Most of those functions are largely historical baggage whose precise
meaning has evolved over time, so I think it's OK to change it yet a bit
further (and to deprecate them more if needed).


        Stefan

Re: [emacs-bidi] Treatment of LRE,RLE,LRO,RLO,PDF,LRM,RLM

[Eli Zaretskii]

I now made glyphless-char-display-control a proper defcustom.