change in X character input processing

change in X character input processing

[Dave Love]

I've changed the handling of character input under X to consider the
definitions of keysyms rather than trying to decode everything for
some locale which may be inappropriate.  Thus, for instance, my euro
key will work in a Latin-1 environment like the pound key (and doesn't
depend on Latin-9 support in the X libraries).

I think this DTRT generally, but if some sort of input doesn't work
right, please let me know.  There seems to be a shortage of useful
documentation in this area.

Note that a given keysym produces a character from a fixed charset,
e.g. Odiaeresis always produces (make-char 'latin-iso8859-1 86).  The
Kana keysyms produce mule-unicode since I'm not sure how to map them
onto CJK charsets from the defined unicodes; if anyone is sure how to
translate them, please update the table in term/x-win.el which
populates `x-keysym-table'.

Note also that input via a Compose key and other sorts of X input
methods will be different.  Thus with typical XKB rules, `Compose e ='
won't produce € in a Latin-1 locale but `AltGr 4' will.  (That's using
X's Multi_key and Mode_switch.)

Re: change in X character input processing

[Stefan Monnier]

> I've changed the handling of character input under X to consider the
> definitions of keysyms rather than trying to decode everything for
> some locale which may be inappropriate.  Thus, for instance, my euro
> key will work in a Latin-1 environment like the pound key (and doesn't
> depend on Latin-9 support in the X libraries).

IIUC this doesn't prevent the usual Xlib decoding of keysym, but
only uses an auxiliary table x-keysym-table to map keysyms
to characters in case where Xlib failed to decode the keysym
(typically because it can't be represented in the user's
locale).

I.e. it is similar in behavior to my earlier patch which tried
Xutf8LookupString if XmbLookupString failed (except that your patch
doesn't rely on the deprecated Xutf8LookupString function or any
other extra support in Xlib for that matter).

Is that right ?


	Stefan

Re: change in X character input processing

[Stefan Monnier]

> I've changed the handling of character input under X to consider the
> definitions of keysyms rather than trying to decode everything for
> some locale which may be inappropriate.  Thus, for instance, my euro
> key will work in a Latin-1 environment like the pound key (and doesn't
> depend on Latin-9 support in the X libraries).

BTW, in the code I see you do:

 			  /* First deal with keysyms which have
 			     defined translations to characters.  */
			  if (keysym >= 32 && keysym < 128)
			    /* Avoid explicitly decoding each ASCII
			       character.  */
			    {
			      bufp->kind = ASCII_KEYSTROKE_EVENT;
			      bufp->code = c;
			    }

Have you encountered cases where this can happen ?

I thought this branch of the code can never be taken if the keysym
is an ASCII char (except for special keysyms explicitly listed,
it's taken only if (keysym != NoSymbol && nbytes == 0) which means that
XLookupString failed to decode the keysym into a char which seems
impossible if the keysym is an ASCII char).


	Stefan

Re: change in X character input processing

[Dave Love]

"Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu> writes:

> IIUC this doesn't prevent the usual Xlib decoding of keysym, but
> only uses an auxiliary table x-keysym-table to map keysyms
> to characters in case where Xlib failed to decode the keysym
> (typically because it can't be represented in the user's
> locale).

The idea is to translate known character keysyms directly to the
appropriate characters.  This should be entirely straightforward,
unlike the situation where you have to try to match X's and Emacs's
idea of the encoding, potentially with input from a different locale
from the one in which Emacs is running.

Note that I don't understand XIM -- is there a good introduction? --
and I don't know whether Emacs really DTRT with XIM.  I don't think
what I've done affects it, anyhow, and if it causes problems it can be
fixed or reverted.

> I.e. it is similar in behavior to my earlier patch which tried
> Xutf8LookupString if XmbLookupString failed (except that your patch
> doesn't rely on the deprecated Xutf8LookupString function or any
> other extra support in Xlib for that matter).

I don't know about that, but presumably not.  Avoiding Xlib
restrictions is part of the idea, e.g. you can use a euro key without
Latin-9 support from Xlib.

Re: change in X character input processing

[Dave Love]

"Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu> writes:

> Have you encountered cases where this can happen ?

Well spotted.  I had completely messed up a merge and didn't realize
on a quick test, but I think it's right now.  I _had_ exercised it
properly in a different context.  It's a nightmare keeping several
things in sync mostly down a phone line and trying to get it right in
a code base I don't run.

By the way, why were the event kinds all renamed like that?  It would
have made more sense to me to remove `_event' suffixes rather than add
them.

Re: change in X character input processing

[Stefan Monnier]

> "Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu> writes:
> 
> > IIUC this doesn't prevent the usual Xlib decoding of keysym, but
> > only uses an auxiliary table x-keysym-table to map keysyms
> > to characters in case where Xlib failed to decode the keysym
> > (typically because it can't be represented in the user's
> > locale).
> 
> The idea is to translate known character keysyms directly to the
> appropriate characters.  This should be entirely straightforward,
> unlike the situation where you have to try to match X's and Emacs's
> idea of the encoding, potentially with input from a different locale
> from the one in which Emacs is running.

Yes, I understand that part.  The question is: when is your code used;
in which case(s) does it change the previous behavior.

> > I.e. it is similar in behavior to my earlier patch which tried
> > Xutf8LookupString if XmbLookupString failed (except that your patch
> > doesn't rely on the deprecated Xutf8LookupString function or any
> > other extra support in Xlib for that matter).
> I don't know about that, but presumably not.

Actually, having looked at the code some more, I'd say "yes, it does the
same thing, except it does it itself rather than with Xlib's help"

> Avoiding Xlib
> restrictions is part of the idea, e.g. you can use a euro key without
> Latin-9 support from Xlib.

Yes, but the behavior is the same: if XmbLookupString manages to decode
the keysym into a char, then the old code is used.  If it doesn't,
my code tried Xutf8StringLookup whereas your code uses x-keysym-table.
When the two table are similar, you get pretty much the same behavior.
Not relying on Xlib is of course an advantage right now since unicode
support in Xlib can't be relied upon.


	Stefan

Re: change in X character input processing

[Stefan Monnier]

The new code you installed works very differently from the one you had before.
Now your x-keysym-table overrides the usual XmbLookupString+decode process
This means that when you do

	LANG=fr_FR@euro emacs -q --no-site-file

with your code an eacute will insert a latin-1 eacute whereas with
the previous code it inserted a latin-9 eacute.

I'm not sure we want to do that just now.
I suggest we revert to the code you had before, so that the
XmbLookupString+decode is used as before and x-keysym-table is
only used if that failed (which is only the case if the key you hit
translates into a char that doesn't exist in the coding-system
corresponding to your locale, such as when you hit EuroSign in
a latin-1 locale).

The reason why I'd like to revert is that your code basically does
a form of unify-on-keyboard-input and I'm not sure that we want to do
that already since we don't have unify-on-decoding turned on by default.

I.e. I suggest the patch below (against the version before your latest fix).
See after my sig for the patch against the current version.

*** src/xterm.c	30 Oct 2002 19:12:37 -0000	1.759
--- src/xterm.c	1 Nov 2002 19:47:22 -0000
***************
*** 10821,10850 ****
  			  if (temp_index == sizeof temp_buffer / sizeof (short))
  			    temp_index = 0;
  			  temp_buffer[temp_index++] = keysym;
!  			  /* First deal with keysyms which have
!  			     defined translations to characters.  */
! 			  if (keysym >= 32 && keysym < 128)
! 			    /* Avoid explicitly decoding each ASCII
! 			       character.  */
  			    {
! 			      bufp->kind = ASCII_KEYSTROKE_EVENT;
! 			      bufp->code = c;
! 			    }
! 			  else if (! EQ ((c = Fgethash (make_number (keysym),
! 							Vx_keysym_table,
! 							Qnil)),
! 					 Qnil))
! 			    {
! 			      bufp->kind = (SINGLE_BYTE_CHAR_P (c)
  					    ? ASCII_KEYSTROKE_EVENT
  					    : MULTIBYTE_CHAR_KEYSTROKE_EVENT);
- 			      bufp->code = c;
  			    }
  			  else
  			    {
! 			      /* Not a character keysym.
! 				 make_lispy_event will convert it to a
! 				 symbolic key.  */
  			      bufp->kind = NON_ASCII_KEYSTROKE_EVENT;
  			      bufp->code = keysym;
  			    }
--- 10822,10840 ----
  			  if (temp_index == sizeof temp_buffer / sizeof (short))
  			    temp_index = 0;
  			  temp_buffer[temp_index++] = keysym;
! 			  if (NATNUMP (c = Fgethash (make_number (keysym),
! 						     Vx_keysym_table,
! 						     Qnil)))
  			    {
! 			      bufp->code = XFASTINT (c);
! 			      bufp->kind = (SINGLE_BYTE_CHAR_P (bufp->code)
  					    ? ASCII_KEYSTROKE_EVENT
  					    : MULTIBYTE_CHAR_KEYSTROKE_EVENT);
  			    }
  			  else
  			    {
! 			      /* Not a character keysym.  make_lispy_event
! 				 will convert it to a symbolic key.  */
  			      bufp->kind = NON_ASCII_KEYSTROKE_EVENT;
  			      bufp->code = keysym;
  			    }

As you can see all it does is remove the ASCII handling (because it
basically can't happen here and because it uses variable `c' before
it's initialized) and fixes some int/Lisp_Object mixup (and uses NATNUMP
rather then EQ (..., Qnil)).


	Stefan


Index: src/xterm.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/xterm.c,v
retrieving revision 1.760
diff -c -r1.760 xterm.c
*** src/xterm.c	31 Oct 2002 17:59:30 -0000	1.760
--- src/xterm.c	1 Nov 2002 19:45:01 -0000
***************
*** 10738,10871 ****
  
  		  if (numchars > 1)
  		    {
! 		      Lisp_Object c;
! 
! 		      /* First deal with keysyms which have defined
! 			 translations to characters.  */
! 		      if (keysym >= 32 && keysym < 128)
! 			/* Avoid explicitly decoding each ASCII character.  */
! 			{
! 			  bufp->kind = ASCII_KEYSTROKE_EVENT;
! 			  bufp->code = keysym;
! 			  XSETFRAME (bufp->frame_or_window, f);
! 			  bufp->arg = Qnil;
! 			  bufp->modifiers
! 			    = x_x_to_emacs_modifiers (FRAME_X_DISPLAY_INFO (f),
! 						      modifiers);
! 			  bufp->timestamp = event.xkey.time;
! 			  bufp++;
! 			  count++;
! 			  numchars--;
! 			}
! 		      /* Now non-ASCII.  */
! 		      else if (! EQ ((c = Fgethash (make_number (keysym),
! 						    Vx_keysym_table, Qnil)),
! 				     Qnil))
! 			{
! 			  bufp->kind = (SINGLE_BYTE_CHAR_P (c)
! 					? ASCII_KEYSTROKE_EVENT
! 					: MULTIBYTE_CHAR_KEYSTROKE_EVENT);
! 			  bufp->code = c;
! 			  XSETFRAME (bufp->frame_or_window, f);
! 			  bufp->arg = Qnil;
! 			  bufp->modifiers
! 			    = x_x_to_emacs_modifiers (FRAME_X_DISPLAY_INFO (f),
! 						      modifiers);
! 			  bufp->timestamp = event.xkey.time;
! 			  bufp++;
! 			  count++;
! 			  numchars--;
! 			}
! 		      /* Random non-modifier sorts of keysyms.  */
! 		      else if (((keysym >= XK_BackSpace && keysym <= XK_Escape)
! 				|| keysym == XK_Delete
  #ifdef XK_ISO_Left_Tab
! 				|| (keysym >= XK_ISO_Left_Tab
! 				    && keysym <= XK_ISO_Enter)
  #endif
! 				|| IsCursorKey (keysym) /* 0xff50 <= x < 0xff60 */
! 				|| IsMiscFunctionKey (keysym) /* 0xff60 <= x < VARIES */
  #ifdef HPUX
! 				/* This recognizes the "extended function
! 				   keys".  It seems there's no cleaner way.
! 				   Test IsModifierKey to avoid handling
! 				   mode_switch incorrectly.  */
! 				|| ((unsigned) (keysym) >= XK_Select
! 				    && (unsigned)(keysym) < XK_KP_Space)
  #endif
  #ifdef XK_dead_circumflex
! 				|| orig_keysym == XK_dead_circumflex
  #endif
  #ifdef XK_dead_grave
! 				|| orig_keysym == XK_dead_grave
  #endif
  #ifdef XK_dead_tilde
! 				|| orig_keysym == XK_dead_tilde
  #endif
  #ifdef XK_dead_diaeresis
! 				|| orig_keysym == XK_dead_diaeresis
  #endif
  #ifdef XK_dead_macron
! 				|| orig_keysym == XK_dead_macron
  #endif
  #ifdef XK_dead_degree
! 				|| orig_keysym == XK_dead_degree
  #endif
  #ifdef XK_dead_acute
! 				|| orig_keysym == XK_dead_acute
  #endif
  #ifdef XK_dead_cedilla
! 				|| orig_keysym == XK_dead_cedilla
  #endif
  #ifdef XK_dead_breve
! 				|| orig_keysym == XK_dead_breve
  #endif
  #ifdef XK_dead_ogonek
! 				|| orig_keysym == XK_dead_ogonek
  #endif
  #ifdef XK_dead_caron
! 				|| orig_keysym == XK_dead_caron
  #endif
  #ifdef XK_dead_doubleacute
! 				|| orig_keysym == XK_dead_doubleacute
  #endif
  #ifdef XK_dead_abovedot
! 				|| orig_keysym == XK_dead_abovedot
  #endif
! 				|| IsKeypadKey (keysym) /* 0xff80 <= x < 0xffbe */
! 				|| IsFunctionKey (keysym) /* 0xffbe <= x < 0xffe1 */
! 				/* Any "vendor-specific" key is ok.  */
! 				|| (orig_keysym & (1 << 28))
! 				|| (keysym != NoSymbol && nbytes == 0))
! 			       && ! (IsModifierKey (orig_keysym)
  #ifndef HAVE_X11R5
  #ifdef XK_Mode_switch
! 				     || ((unsigned)(orig_keysym) == XK_Mode_switch)
  #endif
  #ifdef XK_Num_Lock
! 				     || ((unsigned)(orig_keysym) == XK_Num_Lock)
  #endif
  #endif /* not HAVE_X11R5 */
! 				     /* The symbols from XK_ISO_Lock
! 					to XK_ISO_Last_Group_Lock
! 					don't have real modifiers but
! 					should be treated similarly to
! 					Mode_switch by Emacs. */
  #if defined XK_ISO_Lock && defined XK_ISO_Last_Group_Lock
! 				     || ((unsigned)(orig_keysym)
! 					 >=  XK_ISO_Lock
! 					 && (unsigned)(orig_keysym)
! 					 <= XK_ISO_Last_Group_Lock)
  #endif
! 				     ))
  			{
  			  if (temp_index == sizeof temp_buffer / sizeof (short))
  			    temp_index = 0;
  			  temp_buffer[temp_index++] = keysym;
! 			  /* make_lispy_event will convert this to a symbolic
! 			     key.  */
! 			  bufp->kind = NON_ASCII_KEYSTROKE_EVENT;
! 			  bufp->code = keysym;
  			  XSETFRAME (bufp->frame_or_window, f);
  			  bufp->arg = Qnil;
  			  bufp->modifiers
--- 10738,10843 ----
  
  		  if (numchars > 1)
  		    {
! 		      if (((keysym >= XK_BackSpace && keysym <= XK_Escape)
! 			   || keysym == XK_Delete
  #ifdef XK_ISO_Left_Tab
! 			   || (keysym >= XK_ISO_Left_Tab && keysym <= XK_ISO_Enter)
  #endif
! 			   || (keysym >= XK_Kanji && keysym <= XK_Eisu_toggle)
! 			   || IsCursorKey (keysym) /* 0xff50 <= x < 0xff60 */
! 			   || IsMiscFunctionKey (keysym) /* 0xff60 <= x < VARIES */
  #ifdef HPUX
! 			   /* This recognizes the "extended function keys".
! 			      It seems there's no cleaner way.
! 			      Test IsModifierKey to avoid handling mode_switch
! 			      incorrectly.  */
! 			   || ((unsigned) (keysym) >= XK_Select
! 			       && (unsigned)(keysym) < XK_KP_Space)
  #endif
  #ifdef XK_dead_circumflex
! 			   || orig_keysym == XK_dead_circumflex
  #endif
  #ifdef XK_dead_grave
! 			   || orig_keysym == XK_dead_grave
  #endif
  #ifdef XK_dead_tilde
! 			   || orig_keysym == XK_dead_tilde
  #endif
  #ifdef XK_dead_diaeresis
! 			   || orig_keysym == XK_dead_diaeresis
  #endif
  #ifdef XK_dead_macron
! 			   || orig_keysym == XK_dead_macron
  #endif
  #ifdef XK_dead_degree
! 			   || orig_keysym == XK_dead_degree
  #endif
  #ifdef XK_dead_acute
! 			   || orig_keysym == XK_dead_acute
  #endif
  #ifdef XK_dead_cedilla
! 			   || orig_keysym == XK_dead_cedilla
  #endif
  #ifdef XK_dead_breve
! 			   || orig_keysym == XK_dead_breve
  #endif
  #ifdef XK_dead_ogonek
! 			   || orig_keysym == XK_dead_ogonek
  #endif
  #ifdef XK_dead_caron
! 			   || orig_keysym == XK_dead_caron
  #endif
  #ifdef XK_dead_doubleacute
! 			   || orig_keysym == XK_dead_doubleacute
  #endif
  #ifdef XK_dead_abovedot
! 			   || orig_keysym == XK_dead_abovedot
  #endif
! 			   || IsKeypadKey (keysym) /* 0xff80 <= x < 0xffbe */
! 			   || IsFunctionKey (keysym) /* 0xffbe <= x < 0xffe1 */
! 			   /* Any "vendor-specific" key is ok.  */
! 			   || (orig_keysym & (1 << 28))
! 			   || (keysym != NoSymbol && nbytes == 0))
! 			  && ! (IsModifierKey (orig_keysym)
  #ifndef HAVE_X11R5
  #ifdef XK_Mode_switch
! 				|| ((unsigned)(orig_keysym) == XK_Mode_switch)
  #endif
  #ifdef XK_Num_Lock
! 				|| ((unsigned)(orig_keysym) == XK_Num_Lock)
  #endif
  #endif /* not HAVE_X11R5 */
! 				/* The symbols from XK_ISO_Lock to
! 				   XK_ISO_Last_Group_Lock doesn't have real
! 				   modifiers but should be treated similarly
! 				   to Mode_switch by Emacs. */
  #if defined XK_ISO_Lock && defined XK_ISO_Last_Group_Lock
! 				|| ((unsigned)(orig_keysym) >=  XK_ISO_Lock
! 				    && (unsigned)(orig_keysym) <= XK_ISO_Last_Group_Lock)
  #endif
! 				))
  			{
+ 			  Lisp_Object c;
+ 
  			  if (temp_index == sizeof temp_buffer / sizeof (short))
  			    temp_index = 0;
  			  temp_buffer[temp_index++] = keysym;
! 			  if (NATNUMP (c = Fgethash (make_number (keysym),
! 						     Vx_keysym_table,
! 						     Qnil)))
! 			    {
! 			      bufp->code = XFASTINT (c);
! 			      bufp->kind = (SINGLE_BYTE_CHAR_P (bufp->code)
! 					    ? ASCII_KEYSTROKE_EVENT
! 					    : MULTIBYTE_CHAR_KEYSTROKE_EVENT);
! 			    }
! 			  else
! 			    {
! 			      /* Not a character keysym.  make_lispy_event
! 				 will convert it to a symbolic key.  */
! 			      bufp->kind = NON_ASCII_KEYSTROKE_EVENT;
! 			      bufp->code = keysym;
! 			    }
  			  XSETFRAME (bufp->frame_or_window, f);
  			  bufp->arg = Qnil;
  			  bufp->modifiers
***************
*** 10877,10883 ****
  			  numchars--;
  			}
  		      else if (numchars > nbytes)
! 			{	/* Raw bytes, not keysym.  */
  			  register int i;
  			  register int c;
  			  int nchars, len;
--- 10849,10855 ----
  			  numchars--;
  			}
  		      else if (numchars > nbytes)
! 			{
  			  register int i;
  			  register int c;
  			  int nchars, len;

Re: change in X character input processing

[Dave Love]

"Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu> writes:

> The new code you installed works very differently from the one you had before.
> Now your x-keysym-table overrides the usual XmbLookupString+decode process
> This means that when you do
> 
> 	LANG=fr_FR@euro emacs -q --no-site-file
> 
> with your code an eacute will insert a latin-1 eacute whereas with
> the previous code it inserted a latin-9 eacute.

Sorry, I forgot the input translation table never got used properly.
It should affect self-inserting characters, but I couldn't do that
originally (in Lisp).  The situation above is the same as for Quail
methods, and I think it's strictly correct insofar as X uses iso2022
and defines the keysym to be Latin-1, but obviously it's not what you
want.

I'll sort out the translation table, then you'll get a high
probability of translating the input character appropriately for the
buffer context, regardless of the locale.  I probably can't test that
under X before Monday.

If the current code is causing anyone problems, setting
`x-keysym-table' to an empty hash table should give the same behaviour
as before.

[The input translation may be useful for more charsets than currently
supported.  See ucs-tables.el if anyone wants to add more tables that
are useful and not too big.  If they are big (2-dimensional charsets),
they should probably just be added to the translation tables in
appropriate language environments.]

Re: change in X character input processing

[Stefan Monnier]

> > The new code you installed works very differently from the one you had before.
> > Now your x-keysym-table overrides the usual XmbLookupString+decode process
> > This means that when you do
> > 
> > 	LANG=fr_FR@euro emacs -q --no-site-file
> > 
> > with your code an eacute will insert a latin-1 eacute whereas with
> > the previous code it inserted a latin-9 eacute.
> 
> Sorry, I forgot the input translation table never got used properly.
> It should affect self-inserting characters, but I couldn't do that
> originally (in Lisp).  The situation above is the same as for Quail
> methods, and I think it's strictly correct insofar as X uses iso2022
> and defines the keysym to be Latin-1, but obviously it's not what you
> want.
> 
> I'll sort out the translation table, then you'll get a high
> probability of translating the input character appropriately for the
> buffer context, regardless of the locale.  I probably can't test that
> under X before Monday.

If I understand correctly, once the input translation table is setup,
then the difference I pointed out above between the old code and the new
code would mostly vanish: both would return a char in the charset most
closely related to the buffer's coding system.  Right?

> If the current code is causing anyone problems, setting
> `x-keysym-table' to an empty hash table should give the same behaviour
> as before.

That's a good point.  I think my main question is: which mapping table
should be preferred.  The options we have already seen are:
- originally, only one table was used: the Xlib table for the current locale.
- my utf8 patch temporarily changed it to: first try the Xlib table for
  the current locale and if that fails, try the Xlib table for a utf-8 locale.
- your first patch said: first try the Xlib table for the current locale
  and if that fails, try the x-keysym-table.
- the current code does: try x-keysym-table and if that fails, use
  the Xlib table for the current locale.

The use of x-keysym-table has the advantage of flexibility, since we don't
depend on the Xlib at all.  But I tend to prefer the Xlib table because
I think of it as canonical since every application uses it.  Among other
things, that means that it is kept uptodate and populated without
any work on our part.  For that reason I'd prefer using the code that you
first installed, which prefers the Xlib table to the x-keysym-table.
Of course I also prefer it because it offers better backward compatibility.


	Stefan

Re: change in X character input processing

[Dave Love]

"Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu> writes:

> If I understand correctly, once the input translation table is setup,
> then the difference I pointed out above between the old code and the new
> code would mostly vanish: both would return a char in the charset most
> closely related to the buffer's coding system.  Right?

No.  Previously nothing outside Quail tried to conform to the file
coding system of the current buffer.

> The use of x-keysym-table has the advantage of flexibility, since we
> don't depend on the Xlib at all.  But I tend to prefer the Xlib
> table because I think of it as canonical since every application
> uses it.

As I understand it, there currently isn't a proper definition of the
translation of many keysyms, and thus no guarantee of consistency
across platforms.  For instance, EuroSign, which my Sun keyboard
generates under XFree, isn't defined on Solaris 8.

> Among other things, that means that it is kept uptodate
> and populated without any work on our part.

But there isn't a single Xlib, especially one that can be relied to be
at all `up-to-date' on random systems, and the input may be coming
from a different system.  I don't think it's any more reasonable to
depend on Xlib than on iconv routines on the system, say.

> For that reason I'd prefer using the code that you
> first installed, which prefers the Xlib table to the x-keysym-table.
> Of course I also prefer it because it offers better backward compatibility.

I don't want it backwards-compatible, because that doesn't DTRT with
high probability.  Also it doesn't seem sensible to decode every
character, though I don't know whether that's a significant
inefficiency.

Afraid I haven't got the keyboard translation sorted out yet, partly
because I realized it should use `keyboard-translate-table'.

Re: change in X character input processing

[Stefan Monnier]

> > If I understand correctly, once the input translation table is setup,
> > then the difference I pointed out above between the old code and the new
> > code would mostly vanish: both would return a char in the charset most
> > closely related to the buffer's coding system.  Right?
> 
> No.  Previously nothing outside Quail tried to conform to the file
> coding system of the current buffer.

I meant to compare

	old xterm.c code with new input translation table setup
vs
	new xterm.c code with new input translation table setup

> > The use of x-keysym-table has the advantage of flexibility, since we
> > don't depend on the Xlib at all.  But I tend to prefer the Xlib
> > table because I think of it as canonical since every application
> > uses it.
> 
> As I understand it, there currently isn't a proper definition of the
> translation of many keysyms, and thus no guarantee of consistency
> across platforms.  For instance, EuroSign, which my Sun keyboard
> generates under XFree, isn't defined on Solaris 8.
> 
> > Among other things, that means that it is kept uptodate
> > and populated without any work on our part.
> 
> But there isn't a single Xlib, especially one that can be relied to be
> at all `up-to-date' on random systems, and the input may be coming
> from a different system.  I don't think it's any more reasonable to
> depend on Xlib than on iconv routines on the system, say.

There is one significant way in which it is more reasonable: 99% of
the programs rely on it.  So if your Xlib doesn't understand EuroSign,
then none of your programs will understand it, so I don't see why
Emacs should go through extra trouble: the problem should obviously
be fixed in Xlib anyway.

> > For that reason I'd prefer using the code that you
> > first installed, which prefers the Xlib table to the x-keysym-table.
> > Of course I also prefer it because it offers better backward compatibility.
> I don't want it backwards-compatible, because that doesn't DTRT with
> high probability.

Could you give some example where the old code does turn the event into
a char but doesn't do it right ?
That's obviously the only relevant case since if it fails to turn it into
a char we all agree that we should then use x-keysym-table.

> Also it doesn't seem sensible to decode every
> character, though I don't know whether that's a significant
> inefficiency.

Indeed, the efficiency aspect is irrelevant.

> Afraid I haven't got the keyboard translation sorted out yet, partly
> because I realized it should use `keyboard-translate-table'.

I think as long as this is not fixed, your new code's behavior
is wrong.  Preferring the decode rather than the x-keysym-table
would not suffer from this problem.  That's one of the ways in which
backward compatibility is beneficial.


	Stefan

Re: change in X character input processing

[Kim F. Storm]

"Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu> writes:

> There is one significant way in which it is more reasonable: 99% of
> the programs rely on it.  So if your Xlib doesn't understand EuroSign,
> then none of your programs will understand it, so I don't see why
> Emacs should go through extra trouble: the problem should obviously
> be fixed in Xlib anyway.

If you've got emacs, why care about those other 99% of the programs
which you probably only use 1% of the time :-) :-) :-)  

Anyway, on my redhat 6.2 system the Xkeymap (or whatever it is called)
doesn't even have a keycode for the AltGr-E on the DK-keymap, so emacs
never get any keysym to process - Xlib or no Xlib...

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: change in X character input processing

[Dave Love]

"Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu> writes:

> > > If I understand correctly, once the input translation table is setup,
> > > then the difference I pointed out above between the old code and the new
> > > code would mostly vanish: both would return a char in the charset most
> > > closely related to the buffer's coding system.  Right?
> > 
> > No.  Previously nothing outside Quail tried to conform to the file
> > coding system of the current buffer.
> 
> I meant to compare
> 
> 	old xterm.c code with new input translation table setup
> vs
> 	new xterm.c code with new input translation table setup

In that case, yes.

> > But there isn't a single Xlib, especially one that can be relied to be
> > at all `up-to-date' on random systems, and the input may be coming
> > from a different system.  I don't think it's any more reasonable to
> > depend on Xlib than on iconv routines on the system, say.
> 
> There is one significant way in which it is more reasonable: 99% of
> the programs rely on it.

99% of programs probably aren't multilingual.  I don't know of another
one that is, like Emacs, or guidance on writing one.  I don't think
that invalidates the point anyhow.

> So if your Xlib doesn't understand EuroSign,
> then none of your programs will understand it,

Whether or not they understand it, they probably aren't going to do
anything useful with it in a locale whose charset doesn't represent
it.  That isn't specific to euro.

> so I don't see why Emacs should go through extra trouble:

I've gone to the trouble anyway.

> the problem should obviously be fixed in Xlib anyway.

I think that's unrealistic.  There are assorted things that doubtless
should be done differently outside Emacs for which Emacs compensates.

> > I don't want it backwards-compatible, because that doesn't DTRT with
> > high probability.
> 
> Could you give some example where the old code does turn the event into
> a char but doesn't do it right ?

I'm not sure you're talking about the same thing, but what I meant
was: typing £ from a Latin-1 keyboard into a Latin-9 buffer didn't
DTRT previously.

> That's obviously the only relevant case since if it fails to turn it
> into a char we all agree that we should then use x-keysym-table.

I know how to DTRT at least as well as old Xlibs, the result will be
consistent, and it can be customized.  There may or may not be Xlibs
which translate certain keysyms wrongly in given locales.  I don't
have examples, but I'd be surprised if they didn't exist.  You're not
at their mercy.  If you think that character conversions are trivial
and well-defined, consider alternativnyj as an 8-bit example, and
jisx0208 as a multibyte one.  Note also that there aren't even
unicodes for some of the defined keysyms.

> Indeed, the efficiency aspect is irrelevant.

What are the figures?

> > Afraid I haven't got the keyboard translation sorted out yet, partly
> > because I realized it should use `keyboard-translate-table'.
> 
> I think as long as this is not fixed, your new code's behavior
> is wrong.

I said that should be used.  It is now (reverting to the original
translation table usage following handa's comment).

Are there now cases where it fails to DTRT according to what a user is
likely to expect and where it reasonably could?

Re: change in X character input processing

[Dave Love]

storm@cua.dk (Kim F. Storm) writes:

> If you've got emacs, why care about those other 99% of the programs
> which you probably only use 1% of the time :-) :-) :-)  

Why the emoticons?  I don't use them at all for multilingual input,
other than occasional tests of things in Yudit, I guess.

(For what it's worth, the major non-English script I'm actually
interested in using is maths, which has a lot in common with CJK.  I
partly pick examples like euro for familiarity, though I also want
them to work well.)

Re: change in X character input processing

[Stefan Monnier]

> > > But there isn't a single Xlib, especially one that can be relied to be
> > > at all `up-to-date' on random systems, and the input may be coming
> > > from a different system.  I don't think it's any more reasonable to
> > > depend on Xlib than on iconv routines on the system, say.
> > 
> > There is one significant way in which it is more reasonable: 99% of
> > the programs rely on it.
> 
> 99% of programs probably aren't multilingual.  I don't know of another
> one that is, like Emacs, or guidance on writing one.  I don't think
> that invalidates the point anyhow.

No, indeed it doesn't invalidate the point because the point was not
about multilingualism but about what happens when you generate a particular
keysym_press event perfectly understood by Xlib in the current locale.

> > So if your Xlib doesn't understand EuroSign,
> > then none of your programs will understand it,
> 
> Whether or not they understand it, they probably aren't going to do
> anything useful with it in a locale whose charset doesn't represent
> it.  That isn't specific to euro.

I'm not talking about the case where the locale cannot represent the
character.  As I've said repeatedly, in that case clearly the Emacs-21.3
code is not sufficient and something like x-keysym-table is necessary.
I don't object at all to that part of your code.

What I object to is that your code bypasses Xlib even in the cases
where Xlib used to do things just fine.  It is unnecessary and subtly
changes the behavior of the code, leading to potential unexpected
behaviors.  My point is that we don't need to make such behavioral
changes.

> > > I don't want it backwards-compatible, because that doesn't DTRT with
> > > high probability.
> > 
> > Could you give some example where the old code does turn the event into
> > a char but doesn't do it right ?
> 
> I'm not sure you're talking about the same thing, but what I meant
> was: typing £ from a Latin-1 keyboard into a Latin-9 buffer didn't
> DTRT previously.

I think you know by now that "doesn't DTRT" is not a useful description
of a problem.  AFAIK, it worked fine: it inserted a latin-1 £ and
your x-keysym-table code doesn't make any change here.

The translation-table-for-input change does make a difference, but it
is unrelated to the topic I'm discussing. (except for the fact that it
hides most of the incompatible changes introduced by your x-keysym-table
change that I'm objecting to).

> and well-defined, consider alternativnyj as an 8-bit example, and
> jisx0208 as a multibyte one.  Note also that there aren't even
> unicodes for some of the defined keysyms.

I don't see in what way this is relevant.

> > Indeed, the efficiency aspect is irrelevant.
> What are the figures?

It's done once per keypress.  If decoding a couple of bytes takes
a significant part of the keypress-handling time, then we're in
real trouble.

> > > Afraid I haven't got the keyboard translation sorted out yet, partly
> > > because I realized it should use `keyboard-translate-table'.
> > 
> > I think as long as this is not fixed, your new code's behavior
> > is wrong.
> 
> I said that should be used.  It is now (reverting to the original
> translation table usage following handa's comment).
> 
> Are there now cases where it fails to DTRT according to what a user is
> likely to expect and where it reasonably could?

I don't think so, but if the code only used x-keysym-table when Xlib
failed it would work just as well, so I really don't see why we should
introduce this risk of incompatible change.
What is the big deal with reverting to "try Xlib first and if that
fails, try x-keysym-table" ?  It would turn your x-keysym-table feature
into an "obviously safe" thing, whereas right now, it's only "probably safe
provided translation-table-for-input is setup corrctly".


	Stefan


PS: the current code still behaves differently in that the events corresponding
    to a keypress have not gone through translation-table-for-input yet, so
    in a latin-9 locale, those events used to be latin-9 chars whereas
    they are now mostly-latin-1-chars.  This shouldn't make too much of
    a difference in practice (except when people create bindings for
    non-ASCII keys maybe, or a few pieces of code that do something
    similar to self-insert-command).