bug#64300: 28.2; Can no longer input some tokens.

bug#64300: 28.2; Can no longer input some tokens.

[Christian Brolin]

Hi

I'm using a Swedish keyboard and running emacs in Kubuntu 23.04. From
today, I can no longer input, e.g. ~ or ^ by typing AltGr-X or Shift-X
where X is the key left to the Enter key, containing those symbols as
well as a horizontal colon.

When this was working, I pressed the X key in combination with a letter or
most often space. Now I just get a small pop-up with a gray background
containing the symbolunderlined. I don't know what to do with this
pop-up. It I press space or any other key it just disappear and nothing
goes into my buffer. I tried to click on it with the mouse but it
doesn't react. Also, if I first click on of the AltGr-X or Shift-X and
then the other the pop-up expands to show both symbols and then I can
add more of them even without alternate. Don't know how this is useful.

Btw, the ~ and ^ in this report I got by entering them in the terminal
can copied them from there.


In GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37, 
cairo version 1.16.0)
of 2023-03-16, modified by Debian built on lcy02-amd64-035
Windowing system distributor 'The X.Org Foundation', version 11.0.12101007
System Description: Ubuntu 23.04

Configured using:
'configure --build x86_64-linux-gnu --prefix=/usr
--sharedstatedir=/var/lib --libexecdir=/usr/libexec
--localstatedir=/var/lib --infodir=/usr/share/info
--mandir=/usr/share/man --with-libsystemd --with-pop=yes
--enable-locallisppath=/etc/emacs:/usr/local/share/emacs/28.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/28.2/site-lisp:/usr/share/emacs/site-lisp 

--with-sound=alsa --without-gconf --with-mailutils
--with-native-compilation --build x86_64-linux-gnu --prefix=/usr
--sharedstatedir=/var/lib --libexecdir=/usr/libexec
--localstatedir=/var/lib --infodir=/usr/share/info
--mandir=/usr/share/man --with-libsystemd --with-pop=yes
--enable-locallisppath=/etc/emacs:/usr/local/share/emacs/28.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/28.2/site-lisp:/usr/share/emacs/site-lisp 

--with-sound=alsa --without-gconf --with-mailutils
--with-native-compilation --with-cairo --with-x=yes
--with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
-ffile-prefix-map=/build/emacs-ezKr5Y/emacs-28.2+1=. 
-fstack-protector-strong
-Wformat -Werror=format-security
-fdebug-prefix-map=/build/emacs-ezKr5Y/emacs-28.2+1=/usr/src/emacs-1:28.2+1-13ubuntu3 

-Wall' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
value of $LC_MONETARY: sv_SE.UTF-8
value of $LC_NUMERIC: sv_SE.UTF-8
value of $LC_TIME: sv_SE.UTF-8
value of $LANG: en_SE.UTF-8
value of $XMODIFIERS: @im=ibus
locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
savehist-mode: t
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
column-number-mode: t
line-number-mode: t
transient-mark-mode: t

Load-path shadows:
/home/christian/.emacs.d/elpa/transient-20230213.1337/transient hides 
/usr/share/emacs/28.2/lisp/transient

Features:
(shadow sort mail-extr emacsbug message dired dired-loaddefs rfc822 mml
mml-sec epa gnus-util rmail rmail-loaddefs time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cus-edit pp
cus-start cus-load hs-lint derived lsp-mode comp comp-cstr warnings
cl-extra help-mode lsp-protocol xref project tree-widget wid-edit
spinner pcase network-stream puny nsm rmc markdown-mode color thingatpt
noutline outline easy-mmode lv inline imenu ht filenotify f s ewoc epg
rfc6068 epg-config dash compile text-property-search comint ansi-color
ring finder-inf edmacro kmacro savehist rx info package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 352755 20490)
(symbols 48 25468 1)
(strings 32 106133 5529)
(string-bytes 1 2877034)
(vectors 16 38081)
(vector-slots 8 661719 22711)
(floats 8 161 41)
(intervals 56 387 0)
(buffers 992 12))

bug#64300: 28.2; Can no longer input some tokens.

[Eli Zaretskii]

> Date: Mon, 26 Jun 2023 09:30:31 +0200
> From: Christian Brolin <cbrolin@gmail.com>
> 
> I'm using a Swedish keyboard and running emacs in Kubuntu 23.04. From
> today, I can no longer input, e.g. ~ or ^ by typing AltGr-X or Shift-X
> where X is the key left to the Enter key, containing those symbols as
> well as a horizontal colon.
> 
> When this was working, I pressed the X key in combination with a letter or
> most often space. Now I just get a small pop-up with a gray background
> containing the symbolunderlined. I don't know what to do with this
> pop-up. It I press space or any other key it just disappear and nothing
> goes into my buffer. I tried to click on it with the mouse but it
> doesn't react. Also, if I first click on of the AltGr-X or Shift-X and
> then the other the pop-up expands to show both symbols and then I can
> add more of them even without alternate. Don't know how this is useful.
> 
> Btw, the ~ and ^ in this report I got by entering them in the terminal
> can copied them from there.

Can someone please look into this?  I don't have access to GNU/Linux
systems localized that way.

bug#64300: 28.2; Can no longer input some tokens.

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 26 Jun 2023 09:30:31 +0200
>> From: Christian Brolin <cbrolin@gmail.com>
>> 
>> I'm using a Swedish keyboard and running emacs in Kubuntu 23.04. From
>> today, I can no longer input, e.g. ~ or ^ by typing AltGr-X or Shift-X
>> where X is the key left to the Enter key, containing those symbols as
>> well as a horizontal colon.
>> 
>> When this was working, I pressed the X key in combination with a letter or
>> most often space. Now I just get a small pop-up with a gray background
>> containing the symbolunderlined. I don't know what to do with this
>> pop-up. It I press space or any other key it just disappear and nothing
>> goes into my buffer. I tried to click on it with the mouse but it
>> doesn't react. Also, if I first click on of the AltGr-X or Shift-X and
>> then the other the pop-up expands to show both symbols and then I can
>> add more of them even without alternate. Don't know how this is useful.
>> 
>> Btw, the ~ and ^ in this report I got by entering them in the terminal
>> can copied them from there.
>
> Can someone please look into this?  I don't have access to GNU/Linux
> systems localized that way.

Thanks.  Christian, are you encountering this problem in Emacs 28.2, or
in the Emacs 29 pretests?

And finally, what was the last version of Emacs in which you remember
this working?

bug#64300: 28.2; Can no longer input some tokens.

[Eli Zaretskii]

Ping!  Christian, can you please answer Po Lu's questions?  I'd like
to continue investigating this issue, and in particular establish
whether this issue is present in the upcoming Emacs 29.

> From: Po Lu <luangruo@yahoo.com>
> Cc: Christian Brolin <cbrolin@gmail.com>,  64300@debbugs.gnu.org
> Date: Sat, 01 Jul 2023 17:41:24 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Date: Mon, 26 Jun 2023 09:30:31 +0200
> >> From: Christian Brolin <cbrolin@gmail.com>
> >> 
> >> I'm using a Swedish keyboard and running emacs in Kubuntu 23.04. From
> >> today, I can no longer input, e.g. ~ or ^ by typing AltGr-X or Shift-X
> >> where X is the key left to the Enter key, containing those symbols as
> >> well as a horizontal colon.
> >> 
> >> When this was working, I pressed the X key in combination with a letter or
> >> most often space. Now I just get a small pop-up with a gray background
> >> containing the symbolunderlined. I don't know what to do with this
> >> pop-up. It I press space or any other key it just disappear and nothing
> >> goes into my buffer. I tried to click on it with the mouse but it
> >> doesn't react. Also, if I first click on of the AltGr-X or Shift-X and
> >> then the other the pop-up expands to show both symbols and then I can
> >> add more of them even without alternate. Don't know how this is useful.
> >> 
> >> Btw, the ~ and ^ in this report I got by entering them in the terminal
> >> can copied them from there.
> >
> > Can someone please look into this?  I don't have access to GNU/Linux
> > systems localized that way.
> 
> Thanks.  Christian, are you encountering this problem in Emacs 28.2, or
> in the Emacs 29 pretests?
> 
> And finally, what was the last version of Emacs in which you remember
> this working?
> 

bug#64300: 28.2; Can no longer input some tokens.

[Christian Brolin]

Hi, sorry for late reply.

I'm currently using:
GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37, 
cairo version 1.16.0)
  of 2023-03-16, modified by Debian

I don't know how to find out my previous version, but since I accept 
updates regularly, I guess it was the latest Debian build before Jun 26.

The problem still persist, but now I found out, only when I start Emacs 
from the main menu, not when I start it from a terminal, so maybe some 
problem with the environment. I tried to add LANG=C.UTF-8 to the menu 
entry which helped! So this is no longer an issue for me. But I still 
get the gray Pop-up somewhere on the screen when I press a modifier, 
e.g. ~ until I press the second key. I don't remember to have seen this 
before, is this a new feature? And if so, maybe it is related to this 
issue. But the important thing is that I now can use Emacs again.

Thanks for your efforts.

On 2023-07-06 09:12, Eli Zaretskii wrote:
> Ping!  Christian, can you please answer Po Lu's questions?  I'd like
> to continue investigating this issue, and in particular establish
> whether this issue is present in the upcoming Emacs 29.
>
>> From: Po Lu <luangruo@yahoo.com>
>> Cc: Christian Brolin <cbrolin@gmail.com>,  64300@debbugs.gnu.org
>> Date: Sat, 01 Jul 2023 17:41:24 +0800
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> Date: Mon, 26 Jun 2023 09:30:31 +0200
>>>> From: Christian Brolin <cbrolin@gmail.com>
>>>>
>>>> I'm using a Swedish keyboard and running emacs in Kubuntu 23.04. From
>>>> today, I can no longer input, e.g. ~ or ^ by typing AltGr-X or Shift-X
>>>> where X is the key left to the Enter key, containing those symbols as
>>>> well as a horizontal colon.
>>>>
>>>> When this was working, I pressed the X key in combination with a letter or
>>>> most often space. Now I just get a small pop-up with a gray background
>>>> containing the symbolunderlined. I don't know what to do with this
>>>> pop-up. It I press space or any other key it just disappear and nothing
>>>> goes into my buffer. I tried to click on it with the mouse but it
>>>> doesn't react. Also, if I first click on of the AltGr-X or Shift-X and
>>>> then the other the pop-up expands to show both symbols and then I can
>>>> add more of them even without alternate. Don't know how this is useful.
>>>>
>>>> Btw, the ~ and ^ in this report I got by entering them in the terminal
>>>> can copied them from there.
>>> Can someone please look into this?  I don't have access to GNU/Linux
>>> systems localized that way.
>> Thanks.  Christian, are you encountering this problem in Emacs 28.2, or
>> in the Emacs 29 pretests?
>>
>> And finally, what was the last version of Emacs in which you remember
>> this working?
>>

bug#64300: 28.2; Can no longer input some tokens.

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Christian Brolin <cbrolin@gmail.com> writes:

> Hi, sorry for late reply.
>
> I'm currently using:
> GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37,
> cairo version 1.16.0)
>  of 2023-03-16, modified by Debian
>
> I don't know how to find out my previous version, but since I accept
> updates regularly, I guess it was the latest Debian build before Jun
> 26.

Thanks.  Then I guess it's unrelated to the misguided attempts to
work-around a bug in the Xlib XIM implementation that were installed in
Emacs 28.

> The problem still persist, but now I found out, only when I start
> Emacs from the main menu, not when I start it from a terminal, so
> maybe some problem with the environment. I tried to add LANG=C.UTF-8
> to the menu entry which helped! So this is no longer an issue for
> me. But I still get the gray Pop-up somewhere on the screen when I
> press a modifier, e.g. ~ until I press the second key. I don't
> remember to have seen this before, is this a new feature? And if so,
> maybe it is related to this issue. But the important thing is that I
> now can use Emacs again.

Xlib needs the system locale to be correctly defined in order to
negotiate text encodings with the input method.  Emacs also needs to see
a valid `locale-coding-system' in order to decode keyboard input.

In order to rule out any chance of this being a bug in Emacs, would you
please reply with the values of the environment variables $XMODIFIERS,
$LANG, and that of `locale-coding-system', in an Emacs session run
without any modifications to the menu entry?

TIA.

bug#64300: 28.2; Can no longer input some tokens.

[Christian Brolin]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]

On 2023-07-06 13:50, Po Lu wrote:
> $XMODIFIERS, $LANG, and that of `locale-coding-system',

Hi again,

I reverted my change to the menu item, started Emacs, verified that I 
can't type ~, and performed:

Shell command: echo "$XMODIFIERS, $LANG"
*@im=ibus, en_SE.UTF-8*

This LANG value is what I get by default, don't know where it comes from 
but seems correct because I mostly use English but live in Sweden. I 
also tried to explicitly specify LANG=en_*S**V*.UTF-8, doesn't work. As 
well as LANG=en_US.UTF-8 and LANG=sv_SE.UTF-8 which both work and as 
before LANG=C.UTF-8 which also works. I have not experienced any 
problems to type ~ in other applications. Seems to me that Emacs doesn't 
like when not using the main language for the chosen location... The 
value of XMODIFIER didn't change. Nor did the below variable:

Describe variable: locale-coding-system
locale-coding-system is a variable defined in ‘C source code’.

*Its value is ‘utf-8-unix’**
*
Coding system to use with system messages.
Also used for decoding keyboard input on X Window system, and for
encoding standard output and error streams.

   Probably introduced at or before Emacs version 21.1.

[-- Attachment #2: Type: text/html, Size: 2116 bytes --]

bug#64300: 28.2; Can no longer input some tokens.

[Christian Brolin]

[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]

Added a reference to *en_SE*

On 2023-07-06 13:50, Po Lu wrote:
> $XMODIFIERS, $LANG, and that of `locale-coding-system',

Hi again,

I reverted my change to the menu item, started Emacs, verified that I 
can't type ~, and performed:

Shell command: echo "$XMODIFIERS, $LANG"
*@im=ibus, en_SE.UTF-8*

This LANG value is what I get by default, don't know where it comes from 
but seems correct because I mostly use English but live in Sweden.

https://www.localeplanet.com/icu/en-SE/index.html

I also tried to explicitly specify LANG=en_*S**V*.UTF-8, doesn't work. 
As well as LANG=en_US.UTF-8 and LANG=sv_SE.UTF-8 which both work and as 
before LANG=C.UTF-8 which also works. I have not experienced any 
problems to type ~ in other applications. Seems to me that Emacs doesn't 
like when not using the main language for the chosen location... The 
value of XMODIFIER didn't change. Nor did the below variable:

Describe variable: locale-coding-system
locale-coding-system is a variable defined in ‘C source code’.

*Its value is ‘utf-8-unix’**
*
Coding system to use with system messages.
Also used for decoding keyboard input on X Window system, and for
encoding standard output and error streams.

   Probably introduced at or before Emacs version 21.1.

[-- Attachment #2: Type: text/html, Size: 2457 bytes --]

bug#64300: 28.2; Can no longer input some tokens.

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Christian Brolin <cbrolin@gmail.com> writes:

> Added a reference to en_SE
>
> On 2023-07-06 13:50, Po Lu wrote:
>
>  $XMODIFIERS, $LANG, and that of `locale-coding-system',
>
> Hi again,
>
> I reverted my change to the menu item, started Emacs, verified that I can't type ~, and performed:
>
> Shell command: echo "$XMODIFIERS, $LANG"
> @im=ibus, en_SE.UTF-8
>
> This LANG value is what I get by default, don't know where it comes from but seems correct because I mostly use English but live in Sweden. 
>
> https://www.localeplanet.com/icu/en-SE/index.html
>
> I also tried to explicitly specify LANG=en_SV.UTF-8, doesn't work. As well as LANG=en_US.UTF-8 and LANG=sv_SE.UTF-8 which both work and as
> before LANG=C.UTF-8 which also works. I have not experienced any problems to type ~ in other applications. Seems to me that Emacs doesn't
> like when not using the main language for the chosen location... The value of XMODIFIER didn't change. Nor did the below variable:
>
> Describe variable: locale-coding-system
> locale-coding-system is a variable defined in ‘C source code’.
>
> Its value is ‘utf-8-unix’
>
> Coding system to use with system messages.
> Also used for decoding keyboard input on X Window system, and for
> encoding standard output and error streams.
>
>   Probably introduced at or before Emacs version 21.1.

Thanks.  With this locale, I can reproduce this bug with only I-Bus on
my system as well.  It doesn't seem to be a problem in Emacs, so I'm
closing this bug.

bug#64300: 28.2; Can no longer input some tokens.

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  64300-done@debbugs.gnu.org
> Date: Fri, 07 Jul 2023 08:43:41 +0800
> 
> Thanks.  With this locale, I can reproduce this bug with only I-Bus on
> my system as well.  It doesn't seem to be a problem in Emacs, so I'm
> closing this bug.

If this is a bug in I-Bus, would you please suggest an entry for
etc/PROBLEMS, with the description of the buggy behavior and any
workarounds we know about?

Note that we already mention there several issues with fcitx, and we
suggest to switch to I-Bus as a workaround in at least one place...