Please avoid using Xutf8* API.

Please avoid using Xutf8* API.

[MIYASHITA Hisashi]

Hello.

I'd like to suspend that issue ("PWD") right now,
because I found more serious problem in the current Emacs.

The latest xterm.c uses Xutf8* API(Xutf8LookupString) to obtain
lookup strings if XmbLookupString, but I'd like to strongly recommend
not to use such APIs as a maintainer of Xlib-I18N.

Xutf8* APIs was suddenly introduced into XFree86 abruptly without
concensus.  After the developper noticed the change, there had been
tremendous discussions including flames to revoke these APIs.

Unfortunately, without any sound discussions, these APIs were
incorpolated and shipped with XFree86-4.0, thus these are very very
controversial APIs even now.  X.Org will never accept these APIs.
Xlib-I18N distributed by Li18nux.org does not contain those neither.
Furthermore, any works on these APIs are completely stopped because
many developpers make objections to these APIs.  

I believe that XmbLookupString() is sufficient in the
current Emacs.  In order not to cause serious compatibility
problems, please please refrain to use these APIs in Emacs.

I couldn't find the discussions that corresponds to this change
in emacs-devel ML, but from ChangeLog, this change seems to be
introduced on 4/16.  Could you please teach us why do you need
this change, Mr.Monnier?

For the requirements that we should be able to use UTF-8 string
independently from the locale, we will propose more reliable APIs by
extending XLC APIs.  If you'd like to require it, please wait for that.

  With regards,

from himi

Re: Please avoid using Xutf8* API.

[Stefan Monnier]

> Xutf8* APIs was suddenly introduced into XFree86 abruptly without
> concensus.  After the developper noticed the change, there had been
> tremendous discussions including flames to revoke these APIs.

OK!  I'll remove the change, then (it was somewhat experimental anyway).
Note that it's conditionalized on X_HAVE_UTF8_STRING.
Thanks for warning us about the problem.

> I couldn't find the discussions that corresponds to this change
> in emacs-devel ML, but from ChangeLog, this change seems to be
> introduced on 4/16.  Could you please teach us why do you need
> this change, Mr.Monnier?

The reason is basically as follows:
let's say your locale is fr_CH.iso-8859-1 and you use xmodmap
to bind some keycode to EuroSign.  Clearly XmbLookupString can't
return a string containing the euro char since it has
to return an iso-8859-1 string (where the euro char doesn't exist).
So we if XmbLookupString didn't return a string, we try again
with Xutf8LookupString which gives us the euro char (in utf-8).

Maybe an alternative would be to replace Xutf8LookupString with

	XmbLookupString(...)
	if (status_return == XLookupKeySym) {
	  ;; Try again with unicode.
	  <switch to utf-8 version of the same locale>
	  XmbLookupString(...)
	  <switch back to the original locale>
	}

but my X (and locale) programming knowledge is about as good as
inexistent so I have no idea how to do the switches.

Also we'd need to be sure that doing the locale-switches and the
additional call to XmbLookupString doesn't mess up any internal state.
This seemed to work with Xutf8LookupString, although we couldn't
find any doc that guarantees it.


	Stefan

Re: Please avoid using Xutf8* API.

[MIYASHITA Hisashi]

Sorry for the delay, I've contacted with one of the developpers
that had worked on I18N of X11R5.

"Stefan Monnier" <monnier+gnu/emacs@RUM.cs.yale.edu> writes:

>> Xutf8* APIs was suddenly introduced into XFree86 abruptly without
>> concensus.  After the developper noticed the change, there had been
>> tremendous discussions including flames to revoke these APIs.
>
> OK!  I'll remove the change, then (it was somewhat experimental anyway).
> Note that it's conditionalized on X_HAVE_UTF8_STRING.
> Thanks for warning us about the problem.

Thank you very much.  Emacs is a productive software that is used
by many people.  So your mail relaxed me very well.

> The reason is basically as follows:
> let's say your locale is fr_CH.iso-8859-1 and you use xmodmap
> to bind some keycode to EuroSign.  Clearly XmbLookupString can't
> return a string containing the euro char since it has
> to return an iso-8859-1 string (where the euro char doesn't exist).
> So we if XmbLookupString didn't return a string, we try again
> with Xutf8LookupString which gives us the euro char (in utf-8).

I understand.  Of course, we should be able to deal with the case.

> Maybe an alternative would be to replace Xutf8LookupString with
>
> 	XmbLookupString(...)
> 	if (status_return == XLookupKeySym) {
> 	  ;; Try again with unicode.
> 	  <switch to utf-8 version of the same locale>
> 	  XmbLookupString(...)
> 	  <switch back to the original locale>
> 	}
>
> but my X (and locale) programming knowledge is about as good as
> inexistent so I have no idea how to do the switches.

Hmmm, this code does not work because XLC is determined by the current locale
when XOpenOM/IM() is called.  So you have to call setlocale() when calling
XOpenIM() in xim_open_dpy()@xterm.c

  char *orig, *modified;

  orig = SetLocale(LC_CTYPE, NULL);
  modified = get_utf_8_codeset_locale_name(orig);
  setlocale(LC_CTYPE, modified);
  xim = XOpenIM (dpyinfo->display, dpyinfo->xrdb, resource_name, EMACS_CLASS);
  setlocale(LC_CTYPE, orig);
  free(modified);

After doing it, you will be able to use UTF-8 string in XIM API by Xmb* functions.

But this solution does not seem simple.  The character repertorie of UTF-8 codeset
completely covers that of ISO-8859-1 and almost all of other codesets.
So when Emacs supports UTF-8 codeset without special modules like Mule-UCS ;-) in the
near future, we have only to swith the codeset of the current locale --- ISO-8859-1 or
something like --- to UTF-8 at the startup.  Of course, even in this case, Emacs dose
not need to change the default coding system to utf-8 at all.

  With regards,

from himi

Re: Please avoid using Xutf8* API.

[Stefan Monnier]

> > 	XmbLookupString(...)
> > 	if (status_return == XLookupKeySym) {
> > 	  ;; Try again with unicode.
> > 	  <switch to utf-8 version of the same locale>
> > 	  XmbLookupString(...)
> > 	  <switch back to the original locale>
> > 	}
> >
> > but my X (and locale) programming knowledge is about as good as
> > inexistent so I have no idea how to do the switches.
> 
> Hmmm, this code does not work because XLC is determined by the current locale
> when XOpenOM/IM() is called.  So you have to call setlocale() when calling
> XOpenIM() in xim_open_dpy()@xterm.c
> 
>   char *orig, *modified;
> 
>   orig = SetLocale(LC_CTYPE, NULL);
>   modified = get_utf_8_codeset_locale_name(orig);
>   setlocale(LC_CTYPE, modified);
>   xim = XOpenIM (dpyinfo->display, dpyinfo->xrdb, resource_name, EMACS_CLASS);
>   setlocale(LC_CTYPE, orig);
>   free(modified);
> 
> After doing it, you will be able to use UTF-8 string in XIM API
> by Xmb* functions.

Right, but since we currently do not support UTF-8 fully, we cannot
rely on it.  We have to use the "orig" locale and only revert to
UTF-8 when the orig locale failed.  Or maybe XmbLookupString
could do the defaulting for us and return a special indicator that
utf-8 was used instead of the locale-specified charset.

In any case, from what you say, there is currently no way for
Emacs to do the right thing, short of using the deprecated
Xutf8LookupString interface, so we'll just have to wait.


	Stefan

Re: Please avoid using Xutf8* API.

[Hideki Hiura]

> From: "Stefan Monnier" <monnier+gnu/emacs@rum.cs.yale.edu>
> Right, but since we currently do not support UTF-8 fully, we cannot
> rely on it.  We have to use the "orig" locale and only revert to
> UTF-8 when the orig locale failed.

How about something mixture of both approach then?

      orig_locale = SetLocale(LC_CTYPE, NULL);
      utf8_locale = get_utf_8_codeset_locale_name(orig);
      orig_xim = XOpenIM (dpyinfo->display, dpyinfo->xrdb,
	             resource_name, EMACS_CLASS);
      orig_xic = XCreateIC(orig_xim, .....);
      setlocale(LC_CTYPE, utf8_locale);
      utf8_xim = XOpenIM (dpyinfo->display, dpyinfo->xrdb,
	             resource_name, EMACS_CLASS);
      utf8_xic = XCreateIC(utf8_xim, .....);
      setlocale(LC_CTYPE, orig_locale);

      
      XmbLookupString(orig_xic...)
      if (status_return == XLookupKeySym) {
	  ;; Try again with xic created on UTF-8 locale
          setlocale(LC_CTYPE, utf8_locale); just in case for XLC function
 	  XmbLookupString(utf8_xic...)
          setlocale(LC_CTYPE, orig_locale);

I have not verified this code would work with current implementation.

Or if the short term goal is to convert KeySym of Euro sign to the
Euro sign string, or something like such level(means that the case XIM
server does not get involved with), there may be an alternative.

--
hiura@{li18nux.org,sun.com,kondara.org,unicode.org} http://www.li18nux.org
Chair, Li18nux/Linux Internationalization Initiative, Free Standards Group
Architect/Sr. Staff Engineer, Sun Microsystems, Inc, USA  FAX 650-786-9553