bug#11682: 23.3; meta key stops working if X modifier map becomes (temporarily) empty

bug#11682: 23.3; meta key stops working if X modifier map becomes (temporarily) empty

[Hans Meine]

Hi everybody,

I am running (K)Ubuntu 12.04, and I am suffering from a bug in VMWare Player 
that clears the X modifier map whenever it grabs the keyboard:
https://bugs.launchpad.net/ubuntu/+source/xkeyboard-config/+bug/195982
The workaround is to call 'setxkbmap' without any arguments to restore the 
current xkb map, effectively repopulating the modifier map.

The (new) problem with Emacs is that it loses its meaning of the meta key 
definition (this is meant to be a very superficial description from a user's 
perspective; I could not find out what exactly goes wrong), i.e. all key 
bindings including the meta key stop working.  For instance, I need to use 
"ESC x" instead of M-x.

I am using the Alt key for Meta, here's the output of 'xmodmap -pm':

xmodmap:  up to 4 keys per modifier, (keycodes in parentheses):

shift       Shift_L (0x32),  Shift_R (0x3e)
lock        Shift_L (0x32)
control     Control_L (0x25),  Control_R (0x69)
mod1        Alt_L (0x40),  Meta_L (0xcd)
mod2      
mod3        ISO_Level5_Shift (0x5e)
mod4        Super_L (0x85),  Super_R (0x86),  Super_L (0xce),  Hyper_L (0xcf)
mod5        ISO_Level3_Shift (0x33),  ISO_Level3_Shift (0x5c),  Mode_switch 
(0xcb)

This is the temporary state after I used VMWare, before restoring the working 
state using 'setxkbmap':

xmodmap:  up to 0 keys per modifier, (keycodes in parentheses):

shift     
lock      
control   
mod1      
mod2      
mod3      
mod4      
mod5      

(It /may/ not be irrelevant that I am using the German Neo layout.)

If I start a new Emacs, that one is taking meta all right, so I tried having 
both side-by-side, one Emacs that lost control over meta, and one fresh Emacs 
that has not yet seen an empty modifier map, and had a look at some lisp 
varibles, e.g. 'x-{meta,alt,...}-keysym' (which are all nil) and x-keysym-
table, but could not find any difference.

So my current assumption is that there is some code in Emacs that reacts to 
the modifier change, changes some internal state, and does not react to the 
restored modifier map in a way that the old internal state is restored.  I 
hope that this can be fixed, and I think the latter behavior (i.e. that the 
meta key auto-detection internal state depends only on the /current/ external 
state, and is independent of any intermediate states) would be desirable in 
general.

The following is unmodified output of M-x report-emacs-bug:

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/share/emacs/23.3/etc/DEBUG.


In GNU Emacs 23.3.1 (x86_64-pc-linux-gnu, GTK+ Version 2.24.10)
 of 2012-03-25 on allspice, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.11103000
configured using `configure  '--build' 'x86_64-linux-gnu' '--build' 'x86_64-
linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' 
'--localstatedir=/var/lib' '--infodir=/usr/share/info' '--
mandir=/usr/share/man' '--with-pop=yes' '--enable-
locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.3/site-
lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.3/site-
lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/23.3/leim' '--with-crt-
dir=/usr/lib/x86_64-linux-gnu' '--with-x=yes' '--with-x-toolkit=gtk' '--with-
toolkit-scroll-bars' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-DDEBIAN -g -O2' 
'LDFLAGS=-g' 'CPPFLAGS=-D_FORTIFY_SOURCE=2''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: de_DE.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Apropos

Minor modes in effect:
  delete-selection-mode: t
  shell-dirtrack-mode: t
  global-semantic-idle-summary-mode: t
  show-paren-mode: t
  recentf-mode: t
  pc-selection-mode: t
  global-semanticdb-minor-mode: t
  semantic-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x a p r o p o s <return> k e y s y m <return> C-x 
o C-x 1 <down> <down> <down> <down> <down> <down> <down> 
<return> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <return> <down> <down> <down> 
<return> <up> <up> <up> <up> <up> <up> <return> C-x 
C-g C-x o C-x 1 <C-end> <C-home> <next> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> <up> <up> <up> <up> <up> <up> <up> 
<up> <up> <up> <up> C-x C-g C-x C-b <down-mouse-1> 
<mouse-1> <help-echo> <down-mouse-2> <mouse-2> <help-echo> 
<help-echo> <help-echo> <down-mouse-2> <mouse-2> <help-echo> 
<help-echo> <help-echo> <down-mouse-2> <mouse-2> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<down-mouse-2> <mouse-2> <help-echo> <help-echo> <help-echo> 
<help-echo> <down-mouse-2> <mouse-2> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <down-mouse-2> <mouse-2> <help-echo> <help-echo> 
<help-echo> M-x <up> <return> m e t a <return> C-x 
1 <down> <down> <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> <down> <return> 
<help-echo> C-x 1 <C-home> M-x r e p o r t <tab> <
return>

Recent messages:
Loading paren...done
Loading /home/hmeine/.recentf...done
Cleaning up the recentf list...done (0 removed)
Ido mode enabled
Loading flymake...done
Loading /usr/share/emacs/23.3/site-lisp/tex-site.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Type C-x 1 to delete the help window.
line-move-visual: Beginning of buffer [11 times]
Type C-x 1 to delete the help window.

Load-path shadows:
/usr/share/emacs/23.3/site-lisp/auctex/tex-style hides /usr/share/emacs/site-
lisp/auctex/tex-style
/usr/share/emacs/23.3/site-lisp/auctex/tex-info hides /usr/share/emacs/site-
lisp/auctex/tex-info
/usr/share/emacs/23.3/site-lisp/auctex/bib-cite hides /usr/share/emacs/site-
lisp/auctex/bib-cite
/usr/share/emacs/23.3/site-lisp/auctex/tex-fold hides /usr/share/emacs/site-
lisp/auctex/tex-fold
/usr/share/emacs/23.3/site-lisp/auctex/tex-font hides /usr/share/emacs/site-
lisp/auctex/tex-font
/usr/share/emacs/23.3/site-lisp/auctex/font-latex hides /usr/share/emacs/site-
lisp/auctex/font-latex
/usr/share/emacs/23.3/site-lisp/auctex/context-nl hides /usr/share/emacs/site-
lisp/auctex/context-nl
/usr/share/emacs/23.3/site-lisp/auctex/tex-bar hides /usr/share/emacs/site-
lisp/auctex/tex-bar
/usr/share/emacs/23.3/site-lisp/auctex/multi-prompt hides 
/usr/share/emacs/site-lisp/auctex/multi-prompt
/usr/share/emacs/23.3/site-lisp/auctex/latex hides /usr/share/emacs/site-
lisp/auctex/latex
/usr/share/emacs/23.3/site-lisp/auctex/tex-mik hides /usr/share/emacs/site-
lisp/auctex/tex-mik
/usr/share/emacs/23.3/site-lisp/auctex/context hides /usr/share/emacs/site-
lisp/auctex/context
/usr/share/emacs/23.3/site-lisp/auctex/tex hides /usr/share/emacs/site-
lisp/auctex/tex
/usr/share/emacs/23.3/site-lisp/auctex/tex-fptex hides /usr/share/emacs/site-
lisp/auctex/tex-fptex
/usr/share/emacs/23.3/site-lisp/auctex/context-en hides /usr/share/emacs/site-
lisp/auctex/context-en
/usr/share/emacs/23.3/site-lisp/auctex/toolbar-x hides /usr/share/emacs/site-
lisp/auctex/toolbar-x
/usr/share/emacs/23.3/site-lisp/auctex/tex-buf hides /usr/share/emacs/site-
lisp/auctex/tex-buf
/usr/share/emacs/23.3/site-lisp/auctex/texmathp hides /usr/share/emacs/site-
lisp/auctex/texmathp
/usr/share/emacs/23.3/site-lisp/auctex/tex-jp hides /usr/share/emacs/site-
lisp/auctex/tex-jp
/usr/share/emacs/23.3/site-lisp/cmake-data/cmake-mode hides 
/usr/share/emacs/site-lisp/cmake-mode
/usr/share/emacs/23.3/site-lisp/debian-startup hides /usr/share/emacs/site-
lisp/debian-startup
/usr/share/emacs23/site-lisp/dictionaries-common/flyspell hides 
/usr/share/emacs/23.3/lisp/textmodes/flyspell
/usr/share/emacs/site-lisp/rst hides /usr/share/emacs/23.3/lisp/textmodes/rst
/usr/share/emacs23/site-lisp/dictionaries-common/ispell hides 
/usr/share/emacs/23.3/lisp/textmodes/ispell

Features:
(shadow sort mail-extr message sendmail ecomplete rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mailcap mail-parse rfc2231 rfc2047 rfc2045
qp ietf-drums mailabbrev nnheader mm-util mail-prsvr gmm-utils
mailheader canlock sha1 hex-util hashcash mail-utils emacsbug pp
help-mode view apropos server flymake gtags grin find-dired dired ffap
cus-edit MDL-mode cc-mode cc-fonts cc-menus cc-cmds cc-styles cc-align
cc-engine cc-vars cc-defs derived delsel tramp-imap tramp-gw tramp-fish
tramp-smb tramp-cache tramp-ftp tramp-cmds tramp regexp-opt auth-source
gnus-util netrc time-date shell password-cache format-spec tramp-compat
trampver saveplace semantic/idle semantic/decorate/mode
semantic/decorate pulse paren recentf tree-widget pc-select ido
cus-start cus-load ecb warnings edmacro kmacro ecb-symboldef ecb-analyse
ecb-compatibility ecb-winman-support ecb-autogen autoload ecb-tod
ecb-cycle ecb-eshell ecb-help ecb-jde ecb-method-browser hideshow
ecb-semantic ecb-file-browser ecb-speedbar ecb-layout compile comint
ecb-create-layout advice help-fns advice-preload ecb-compilation
ecb-common-browser speedbar sb-image dframe assoc ecb-navigate
ecb-mode-line ecb-face tree-buffer ecb-upgrade ecb-cedet-wrapper
semantic/db-mode semantic/db-find semantic/db-ref semantic/analyze
semantic/sort semantic/scope semantic/analyze/fcn semantic/db eieio-base
semantic/format ezimage semantic/tag-ls semantic/ctxt
semantic/util-modes easymenu semantic/util semantic semantic/tag
semantic/lex semantic/fw eieio byte-opt bytecomp byte-compile mode-local
cedet wid-edit ecb-util ring thingatpt preview-latex tex-site auto-loads
tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd
font-setting tool-bar dnd fontset image fringe lisp-mode register page
menu-bar rfn-eshadow timer select scroll-bar mldrag mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button
minibuffer faces cus-face files text-properties overlay md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind system-font-setting
font-render-setting gtk x-toolkit x multi-tty emacs)

-- 
Hans Meine • Fraunhofer MEVIS • Universitätsallee 29 • 28359 Bremen
Tel.: +49 421/218-59255  (Fax: +49 421/218-98 59255)

bug#11682: Problem remains

[Hans Meine]

Two additional pieces of information:

1) With "(new) problem with Emacs" above, I wanted to say that the problem 
appeared in recent versions of Emacs, but was not present e.g. 2 years ago.  
Unfortunately, I cannot (yet) more precisely say when the bug appeared.

2) I have added debug output to x_find_modifier_meanings and found that it 
gets called at program start, and when I enter VMWare Player, but not when I 
restore the modifier map by calling "setxkbmap" without arguments.  I would 
expect that ensuring that x_find_modifier_meanings gets called again here 
would fix the bug.

bug#11682: Correction about previous Emacs versions

[Hans Meine]

I have now tried to git-bisect the change, but I could not find a previous 
version of Emacs that worked.  So the change in behavior I observed must have 
to do something with either changes in VMWare, X11, or any other component of 
the (K)Ubuntu Linux distribution I am using (and I have upgraded in the 
meantime).

At least I have tried emacs-24.0.96, EMACS_23_4 („23.4.2“), and EMACS_23_2 
(„23.2.1“), and all suffered from the same problem.  (The further back I went, 
the harder it was to build, so I stopped testing older versions.)

bug#11682: I believe this is an X server bug

[Simon Tatham]

Hi,

I found this bug report a few months ago when I was looking to see if
anyone had had the same problem as me. I too found that if VMware
Player had input focus when emacs started up, then emacs would never
recognise the Alt key.

Debugging with xtruss suggested that the reason for this was that
emacs queried the modifier map at startup, found it empty, and
thereafter listened for MappingNotify in order to know when to reload
the mod map but for some reason it never received any MappingNotify
events. (But it did receive the similar XkbMapNotify when vmware lost
or gained focus.)

That was on Ubuntu 10.04. The machine where I was seeing the problem
has recently been upgraded to 12.04, and now the problem has stopped
happening - on the same version of emacs. Debugging with xtruss shows
that now emacs _is_ receiving MappingNotify, suggesting that the X
server has changed its behaviour.

I suspect, in fact, that the following X.Org commit is the fix:

  http://cgit.freedesktop.org/xorg/xserver/commit/?id=3231962db826f5efd431596a309c96e907a191d1

which appeared in xorg-server 1.11, which would indeed be consistent
with me having this problem in Ubuntu 10.04 but not 12.04 (which use
xorg-server versions 1.7.6 and 1.11.4 respectively).

So if anyone else still has this problem, it might be worth letting
them know to check the version of their X server.

Hope this is helpful!

Cheers,
Simon
-- 
import hashlib; print (lambda p,q,g,y,r,s,m: m if (lambda w:(pow(g,int(hashlib.
 sha1(m).hexdigest(),16)*w%q,p)*pow(y,r*w%q,p)%p)%q)(pow(s,q-2,q))==r else "!"
 )(0xb80b5dacabab6145, 0xf70027d345023, 0x7643bc4018957897, 0x11c2e5d9951130c9,
 0xa54d9cbe4e8ab, 0x746c50eaa1910, "Simon Tatham <anakin@pobox.com>")

bug#11682: I believe this is an X server bug

[Stefan Kangas]

Simon Tatham <anakin@pobox.com> writes:

> Hi,
>
> I found this bug report a few months ago when I was looking to see if
> anyone had had the same problem as me. I too found that if VMware
> Player had input focus when emacs started up, then emacs would never
> recognise the Alt key.
>
> Debugging with xtruss suggested that the reason for this was that
> emacs queried the modifier map at startup, found it empty, and
> thereafter listened for MappingNotify in order to know when to reload
> the mod map but for some reason it never received any MappingNotify
> events. (But it did receive the similar XkbMapNotify when vmware lost
> or gained focus.)
>
> That was on Ubuntu 10.04. The machine where I was seeing the problem
> has recently been upgraded to 12.04, and now the problem has stopped
> happening - on the same version of emacs. Debugging with xtruss shows
> that now emacs _is_ receiving MappingNotify, suggesting that the X
> server has changed its behaviour.
>
> I suspect, in fact, that the following X.Org commit is the fix:
>
>   http://cgit.freedesktop.org/xorg/xserver/commit/?id=3231962db826f5efd431596a309c96e907a191d1
>
> which appeared in xorg-server 1.11, which would indeed be consistent
> with me having this problem in Ubuntu 10.04 but not 12.04 (which use
> xorg-server versions 1.7.6 and 1.11.4 respectively).
>
> So if anyone else still has this problem, it might be worth letting
> them know to check the version of their X server.
>
> Hope this is helpful!

Thanks, that is helpful.  It seems like this is not a bug in Emacs then?
Hans, can you confirm if using a newer version of xorg-server fixes the
problem?

If I don't hear back from you within a couple of weeks, Ill just assume
that this has been fixed and close this bug.

Best regards,
Stefan Kangas

bug#11682: I believe this is an X server bug

[Meine, Hans]

Hi Stefan,

thanks for following up. I must admit that after this long time, my setup has changed too much to try to reproduce the problem.

I agree that it looks like this issue can be closed.

Have a nice day
  Hans

-- 
Hans Meine • Fraunhofer MEVIS • Am Fallturm 1 • 28359 Bremen
Tel.: +49 421/218-59255  (Fax: +49 421/218-98 59255)

> Am 01.11.2019 um 19:29 schrieb Stefan Kangas <stefan@marxist.se>:
> 
> Simon Tatham <anakin@pobox.com> writes:
> 
>> Hi,
>> 
>> I found this bug report a few months ago when I was looking to see if
>> anyone had had the same problem as me. I too found that if VMware
>> Player had input focus when emacs started up, then emacs would never
>> recognise the Alt key.
>> 
>> Debugging with xtruss suggested that the reason for this was that
>> emacs queried the modifier map at startup, found it empty, and
>> thereafter listened for MappingNotify in order to know when to reload
>> the mod map but for some reason it never received any MappingNotify
>> events. (But it did receive the similar XkbMapNotify when vmware lost
>> or gained focus.)
>> 
>> That was on Ubuntu 10.04. The machine where I was seeing the problem
>> has recently been upgraded to 12.04, and now the problem has stopped
>> happening - on the same version of emacs. Debugging with xtruss shows
>> that now emacs _is_ receiving MappingNotify, suggesting that the X
>> server has changed its behaviour.
>> 
>> I suspect, in fact, that the following X.Org commit is the fix:
>> 
>>  http://cgit.freedesktop.org/xorg/xserver/commit/?id=3231962db826f5efd431596a309c96e907a191d1
>> 
>> which appeared in xorg-server 1.11, which would indeed be consistent
>> with me having this problem in Ubuntu 10.04 but not 12.04 (which use
>> xorg-server versions 1.7.6 and 1.11.4 respectively).
>> 
>> So if anyone else still has this problem, it might be worth letting
>> them know to check the version of their X server.
>> 
>> Hope this is helpful!
> 
> Thanks, that is helpful.  It seems like this is not a bug in Emacs then?
> Hans, can you confirm if using a newer version of xorg-server fixes the
> problem?
> 
> If I don't hear back from you within a couple of weeks, Ill just assume
> that this has been fixed and close this bug.
> 
> Best regards,
> Stefan Kangas

bug#11682: I believe this is an X server bug

[Stefan Kangas]

"Meine, Hans" <hans.meine@mevis.fraunhofer.de> writes:

> thanks for following up. I must admit that after this long time, my
> setup has changed too much to try to reproduce the problem.
>
> I agree that it looks like this issue can be closed.

Thank you for reporting back.  Closing the bug now.

Best regards,
Stefan Kangas