bug#7380: 23.2; Dead keys misinterpreted in gtk emacs

bug#7380: 23.2; Dead keys misinterpreted in gtk emacs

[Ian D. Leroux]

Under X11 with the us_intl keyboard, dead keys are not correctly
combined with the following characters.  For instance <dead-acute> e,
which ought to give é, instead produces the message "<dead-acute> is
undefined" followed by an undecorated e.  The problem is specific to
emacs: all other applications in the same X11 session (Firefox, xterm,
urxvt, miscellaneous gtk apps like exfalso) accept accented input
typed with dead keys without special customization.  This is a
plain-vanilla install of emacs under pkgsrc.  The auto-collected data
reported below were generated by M-x report-emacs-bug from an emacs -Q
instance displaying this behaviour.

Workarounds: I can run emacs in a terminal, allowing xterm or urxvt to
handle keyboard input.  This gives me consistent keyboard behaviour at
the price of the graphical comforts of gtk emacs.  I can also activate
iso-transl and have emacs handle the composition of characters from
dead keys by its own internal mechanism, independent of X11, but then
I get a subtly different keyboard layout in emacs relative to other
software on the system.

Thank you for any insights and assistance you may be able to supply,

Ian Leroux

In GNU Emacs 23.2.1 (x86_64--netbsd, GTK+ Version 2.20.1)
 of 2010-11-09 on spip.homeunix.net
Windowing system distributor `The Xorg Foundation', version 11.0.10603000
configured using `configure  '--srcdir=/pkg_comp/obj/pkgsrc/editors/emacs/default/emacs-23.2' '--localstatedir=/var' '--x-includes=/usr/X11R7/include' '--x-libraries=/usr/X11R7/lib' '--with-x' '--with-xpm' '--with-jpeg' '--with-tiff' '--with-gif' '--with-png' '--with-x-toolkit=gtk' '--without-libiconv-prefix' '--without-libintl-prefix' '--prefix=/usr/pkg' '--build=x86_64--netbsd' '--host=x86_64--netbsd' '--infodir=/usr/pkg/info' '--mandir=/usr/pkg/man' 'build_alias=x86_64--netbsd' 'host_alias=x86_64--netbsd' 'CC=cc' 'CFLAGS=-O2 -mfpmath=sse -msse3 -march=athlon64 -pipe -I/usr/pkg/include -I/usr/include -I/usr/X11R7/include -I/usr/X11R7/include/freetype2' 'LDFLAGS=-L/usr/pkg/lib -L/usr/lib -Wl,-R/usr/lib -Wl,-R/usr/pkg/lib -L/usr/X11R7/lib -Wl,-R/usr/X11R7/lib' 'LIBS=' 'CPPFLAGS=-I/usr/pkg/include -I/usr/include -I/usr/X11R7/include -I/usr/X11R7/include/freetype2''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: C
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: C
  value of $LC_TIME: nil
  value of $LANG: nil
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<dead-acute> e M-x r e p o r t - e m a c s - b u g 
<return>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
/usr/pkg/share/emacs/site-lisp/flim/sha1 hides /usr/pkg/share/emacs/23.2/lisp/sha1
/usr/pkg/share/emacs/site-lisp/semi/pgg hides /usr/pkg/share/emacs/23.2/lisp/pgg
/usr/pkg/share/emacs/site-lisp/semi/pgg-pgp5 hides /usr/pkg/share/emacs/23.2/lisp/pgg-pgp5
/usr/pkg/share/emacs/site-lisp/semi/pgg-pgp hides /usr/pkg/share/emacs/23.2/lisp/pgg-pgp
/usr/pkg/share/emacs/site-lisp/semi/pgg-parse hides /usr/pkg/share/emacs/23.2/lisp/pgg-parse
/usr/pkg/share/emacs/site-lisp/semi/pgg-gpg hides /usr/pkg/share/emacs/23.2/lisp/pgg-gpg
/usr/pkg/share/emacs/site-lisp/semi/pgg-def hides /usr/pkg/share/emacs/23.2/lisp/pgg-def
/usr/pkg/share/emacs/site-lisp/flim/hex-util hides /usr/pkg/share/emacs/23.2/lisp/hex-util
/usr/pkg/share/emacs/site-lisp/flim/sasl hides /usr/pkg/share/emacs/23.2/lisp/net/sasl
/usr/pkg/share/emacs/site-lisp/flim/sasl-digest hides /usr/pkg/share/emacs/23.2/lisp/net/sasl-digest
/usr/pkg/share/emacs/site-lisp/flim/sasl-cram hides /usr/pkg/share/emacs/23.2/lisp/net/sasl-cram
/usr/pkg/share/emacs/site-lisp/flim/hmac-md5 hides /usr/pkg/share/emacs/23.2/lisp/net/hmac-md5
/usr/pkg/share/emacs/site-lisp/flim/hmac-def hides /usr/pkg/share/emacs/23.2/lisp/net/hmac-def
/usr/pkg/share/emacs/site-lisp/wl/rfc2368 hides /usr/pkg/share/emacs/23.2/lisp/mail/rfc2368
/usr/pkg/share/emacs/site-lisp/wl/utf7 hides /usr/pkg/share/emacs/23.2/lisp/gnus/utf7
/usr/pkg/share/emacs/site-lisp/semi/smime hides /usr/pkg/share/emacs/23.2/lisp/gnus/smime

Features:
(shadow sort mail-extr message sendmail regexp-opt ecomplete rfc822 mml
easymenu mml-sec password-cache mm-decode mm-bodies mm-encode mailcap
mail-parse rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader
gnus-util netrc time-date mm-util mail-prsvr gmm-utils wid-edit
mailheader canlock sha1 sha1-el hex-util hashcash mail-utils emacsbug
tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd
font-setting tool-bar dnd fontset image fringe lisp-mode register page
menu-bar rfn-eshadow timer select scroll-bar mldrag mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button
minibuffer faces cus-face files text-properties overlay md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind font-render-setting gtk
x-toolkit x multi-tty emacs)

bug#7380: 23.2; Dead keys misinterpreted in gtk emacs

[Stefan Monnier]

> Under X11 with the us_intl keyboard, dead keys are not correctly
> combined with the following characters.  For instance <dead-acute> e,
> which ought to give é, instead produces the message "<dead-acute> is
> undefined" followed by an undecorated e.  The problem is specific to
> emacs: all other applications in the same X11 session (Firefox, xterm,
> urxvt, miscellaneous gtk apps like exfalso) accept accented input
> typed with dead keys without special customization.  This is a
> plain-vanilla install of emacs under pkgsrc.  The auto-collected data
> reported below were generated by M-x report-emacs-bug from an emacs -Q
> instance displaying this behaviour.

It vaguely reminds me of some other bug-report.  But that's about as far
as it goes.  I don't use dead keys, but I do use the <Multi_key> daily to
enter most of my non-ASCII letters, which should rely on the same code.

> the price of the graphical comforts of gtk emacs.  I can also activate
> iso-transl and have emacs handle the composition of characters from
> dead keys by its own internal mechanism, independent of X11, but then
> I get a subtly different keyboard layout in emacs relative to other
> software on the system.

iso-transl is at best a workaround.

I just played with xmodmap to add a dead-acute key to my keyboard, and
"it works here" with all versions of Emacs I threw at it.

Now, as to why this X11 key composition does not work for you.
Could you maybe try to rebuild it and show us the output of "configure"?
Not sure it'll help, tho.  We'll need either someone to be able to
reproduce it, or you'll need to dig in the code, play with GDB to try
and see what's going on there.  If you're up to it, you can try and
place breakpoints near the call to XmbLookupString in xterm.c and single
step there.  Normally, the dead-acute event should not escape from this
part of the code: instead it should turn into "nothing" (just change
some state somewhere either in compose_status or in "FRAME_XIC (f)"
depending on whether that frame uses XIM/XIC), and subsequent "e" should
in that same part of the code be turned into an "é" (so the Elisp code
never even gets to know that this é was input as two separate key
presses).


        Stefan

bug#7380: 23.2; Dead keys misinterpreted in gtk emacs

[Ian D. Leroux]

On Tue, 16 Nov 2010 11:57 -0500, "Stefan Monnier"
<monnier@IRO.UMontreal.CA> wrote:
> Now, as to why this X11 key composition does not work for you. Could
> you maybe try to rebuild it and show us the output of "configure"? Not
> sure it'll help, tho.  We'll need either someone to be able to
> reproduce it, or you'll need to dig in the code, play with GDB to try
> and see what's going on there.  If you're up to it, you can try and
> place breakpoints near the call to XmbLookupString in xterm.c and
> single step there.  Normally, the dead-acute event should not escape
> from this part of the code: instead it should turn into "nothing"
> (just change some state somewhere either in compose_status or in
> "FRAME_XIC (f)" depending on whether that frame uses XIM/XIC), and
> subsequent "e" should in that same part of the code be turned into an
> "é" (so the Elisp code never even gets to know that this é was input
> as two separate key presses).

Thanks for the pointers.  I'll rebuild and poke around in gdb as you
suggest later in the week and report back any interesting findings.

Thanks for your assistance,

Ian Leroux

bug#7380: 23.2; Dead keys misinterpreted in gtk emacs

[Ian D. Leroux]

At Tue, 16 Nov 2010 11:57:05 -0500,
Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:
> Now, as to why this X11 key composition does not work for you.
> Could you maybe try to rebuild it and show us the output of "configure"?

The full config.log is up at:
http://web.mit.edu/~idleroux/Public/emacsbug7380/config.log

The summary printed at the end of configure was:

Configured for `x86_64--netbsd'.

  Where should the build process find the source code?    /pkg_comp/obj/pkgsrc/editors/emacs/default/emacs-23.2
  What operating system and machine description files should Emacs use?
        `s/netbsd.h' and `m/amdx86-64.h'
  What compiler should emacs be built with?               cc -O2 -pipe -I/usr/pkg/include -I/usr/include -I/usr/X11R7/include -I/usr/X11R7/include/freetype2
  Should Emacs use the GNU version of malloc?             yes
  Should Emacs use a relocating allocator for buffers?    yes
  Should Emacs use mmap(2) for buffer allocation?         no
  What window system should Emacs use?                    x11
  What toolkit should Emacs use?                          GTK
  Where do we find X Windows header files?                /usr/X11R7/include
  Where do we find X Windows libraries?                   /usr/X11R7/lib
  Does Emacs use -lXaw3d?                                 no
  Does Emacs use -lXpm?                                   yes
  Does Emacs use -ljpeg?                                  yes
  Does Emacs use -ltiff?                                  yes
  Does Emacs use a gif library?                           yes -lgif
  Does Emacs use -lpng?                                   yes
  Does Emacs use -lrsvg-2?                                yes
  Does Emacs use -lgpm?                                   no
  Does Emacs use -ldbus?                                  yes
  Does Emacs use -lgconf?                                 no
  Does Emacs use -lfreetype?                              yes
  Does Emacs use -lm17n-flt?                              yes
  Does Emacs use -lotf?                                   yes
  Does Emacs use -lxft?                                   yes
  Does Emacs use toolkit scroll bars?                     yes


D-Bus integration has been tested for GNU/Linux only.

configure: creating ./config.status
config.status: creating Makefile
config.status: creating lib-src/Makefile.c
config.status: creating oldXMenu/Makefile
config.status: creating doc/emacs/Makefile
config.status: creating doc/misc/Makefile
config.status: creating doc/lispintro/Makefile
config.status: creating doc/lispref/Makefile
config.status: creating src/Makefile.c
config.status: creating lwlib/Makefile
config.status: creating lisp/Makefile
config.status: creating leim/Makefile
config.status: creating src/config.h
config.status: executing default commands
creating src/epaths.h
creating lib-src/Makefile
creating src/Makefile
configure: WARNING: Unrecognized options: --without-libiconv-prefix, --without-libintl-prefix

> Not sure it'll help, tho.  We'll need either someone to be able to
> reproduce it, or you'll need to dig in the code, play with GDB to try
> and see what's going on there.  If you're up to it, you can try and
> place breakpoints near the call to XmbLookupString in xterm.c and single
> step there.  Normally, the dead-acute event should not escape from this
> part of the code: instead it should turn into "nothing" (just change
> some state somewhere either in compose_status or in "FRAME_XIC (f)"
> depending on whether that frame uses XIM/XIC),

I had a first crack at this.  XmbLookupString is never called because
FRAME_XIC (f) is NULL, so lines 6413--6441 are skipped and
XLookupString is called instead.  The result gets categorized as a
NON_ASCII_KEYSTROKE_EVENT.

Where is FRAME_XIC (f) supposed to be set?  My first impression is
that that is where the problem is (possibly a configuration problem?)

Any further guidance or tips will be appreciated.

Ian Leroux

bug#7380: Dead keys bug, additional info

[Ian D. Leroux]

This problem with dead keys in gtk emacs is still present in 23.3

It may be specific to UTF-8 locales; in an environment with

LANG=en_US.ISO8859-1

the dead keys work normally, while with

LANG=en_US.UTF-8

they misbehave as above, erroring out with "not defined" rather than
waiting for the following keystroke to complete the sequence.  I
suspect that this is why Stefan Monnier could not reproduce the bug
when he tried to assist me.  Hopefully this additional information will
allow a developer to reproduce the bug and/or guide me to produce more
useful information to track it down.

-- 
Ian D. Leroux

bug#7380: 23.2; Dead keys misinterpreted in gtk emacs

[Stefan Monnier]

>> Not sure it'll help, tho.  We'll need either someone to be able to
>> reproduce it, or you'll need to dig in the code, play with GDB to try
>> and see what's going on there.  If you're up to it, you can try and
>> place breakpoints near the call to XmbLookupString in xterm.c and single
>> step there.  Normally, the dead-acute event should not escape from this
>> part of the code: instead it should turn into "nothing" (just change
>> some state somewhere either in compose_status or in "FRAME_XIC (f)"
>> depending on whether that frame uses XIM/XIC),

> I had a first crack at this.  XmbLookupString is never called because
> FRAME_XIC (f) is NULL, so lines 6413--6441 are skipped and
> XLookupString is called instead.

Hmm... that's clearly the source of the problem (XLookupString can only
handle latin-1 locales, IIRC).

> Where is FRAME_XIC (f) supposed to be set?

In the x_window function, in src/xfns.c (there are 3 versions of this
function :-( depending on whether or not we're using Gtk or some other
toolkit or no toolkit at all).

Could it be that use_xim is 0, somehow?


        Stefan "still works for me, even with en_US.UTF-8"

bug#7380: 23.2; Dead keys misinterpreted in gtk emacs

[Ian D. Leroux]

On Friday, November 18, 2011 1:03 PM, "Stefan Monnier"
<monnier@iro.umontreal.ca> wrote:
> > Where is FRAME_XIC (f) supposed to be set?
>
> In the x_window function, in src/xfns.c (there are 3 versions of this
> function :-( depending on whether or not we're using Gtk or some other
> toolkit or no toolkit at all).
>
> Could it be that use_xim is 0, somehow?

use_xim is 1, but FRAME_X_XIM (f) is NULL when create_frame_xic is
called.  As a result, the "if(xim)" test at line 2315 in xfns.c fails,
xic does not get allocated, and FRAME_XIC(f) remains NULL at the end of
the function.

Grepping for FRAME_X_XIM doesn't show me where it is supposed to be set,
so I assume the relevant struct member gets assigned directly somewhere.
Where should that be happening?

xim_open_dpy never gets called.
xim_initialize does; dpyinfo->xim is NULL both at the beginning and at
the end of the function.

Any suggestions on where to look next?

Thanks for your time,

-- IDL

bug#7380: 23.2; Dead keys misinterpreted in gtk emacs

[Ian D. Leroux]

On Wed, Nov 23, 2011, at 11:13, Ian D. Leroux wrote:
> On Friday, November 18, 2011 1:03 PM, "Stefan Monnier"
> <monnier@iro.umontreal.ca> wrote:
> > > Where is FRAME_XIC (f) supposed to be set?
> >
> > In the x_window function, in src/xfns.c (there are 3 versions of
> > this function :-( depending on whether or not we're using Gtk or
> > some other toolkit or no toolkit at all).
> >
> > Could it be that use_xim is 0, somehow?
>
> use_xim is 1, but FRAME_X_XIM (f) is NULL when create_frame_xic is
> called.  As a result, the "if(xim)" test at line 2315 in xfns.c fails,
> xic does not get allocated, and FRAME_XIC(f) remains NULL at the end
> of the function.

The problem has now been identified: due to a configuration error in the
en_US.UTF-8 locale settings in NetBSD's default xorg installation,
libX11 was not loading the new callback based XIM apis.  Most software
was falling back to the older open/close api, and thus continuing to
support dead keys, while emacs gave up on XIM entirely and fell back to
XLookupString.  As far as I'm concerned, this bug can now be closed.

-- IDL

bug#7380: 23.2; Dead keys misinterpreted in gtk emacs

[Glenn Morris]

"Ian D. Leroux" wrote:

> The problem has now been identified: due to a configuration error in the
> en_US.UTF-8 locale settings in NetBSD's default xorg installation,
> libX11 was not loading the new callback based XIM apis.  Most software
> was falling back to the older open/close api, and thus continuing to
> support dead keys, while emacs gave up on XIM entirely and fell back to
> XLookupString.  As far as I'm concerned, this bug can now be closed.


Thanks for letting us know.