23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

[Tassilo Horn]

Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the emacs-pretest-bug@gnu.org mailing
list.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

When I start emacs in a terminal with

  emacs -nw -Q

and perform an isearch for text containing Umlauts, they won't show up
in the minibuffer (or better, 3 times typing ö shows one ?¶ there) and
no matches will be found.

Isearch does work properly with X11 emacs.

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/usr/share/emacs/23.0.60/etc/DEBUG for instructions.


In GNU Emacs 23.0.60.1 (i686-pc-linux-gnu, GTK+ Version 2.12.8)
 of 2008-02-25 on localhost
configured using `configure  '--prefix=/usr' '--host=i686-pc-linux-gnu'
 '--mandir=/usr/share/man' '--infodir=/usr/share/info'
 '--datadir=/usr/share' '--\
sysconfdir=/etc' '--localstatedir=/var/lib' '--program-suffix=-emacs-23'
 '--infodir=/usr/share/info/emacs-23' '--without-carbon' '--with-sound'
 '--with-\
x' '--with-toolkit-scroll-bars' '--with-gif' '--with-jpeg' '--with-png'
 '--with-rsvg' '--with-tiff' '--with-xpm' '--enable-font-backend'
 '--with-freetyp\
e' '--with-xft' '--with-libotf' '--with-m17n-flt' '--with-x-toolkit=gtk'
 '--without-hesiod' '--with-kerberos' '--with-kerberos5' '--with-gpm'
 '--with-db\
us' '--build=i686-pc-linux-gnu' 'build_alias=i686-pc-linux-gnu'
 'host_alias=i686-pc-linux-gnu' 'CFLAGS=-march=i386 -O0 -g' 'LDFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
  outline-minor-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
C-s à ¶ à ¶ à ¶ C-g C-g C-g C-g ESC x r e p o TAB r
t TAB RET

Recent messages:
("emacs" "-Q" ".emacs")
For information about GNU Emacs and the GNU system, type C-h C-a.
Quit [3 times]
Making completion list...

Re: 23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

[Stefan Monnier]

> C-s à ¶ à ¶ à ¶ C-g C-g C-g C-g ESC x r e p o TAB r

Hmm... "Ã ¶" in latin-1 is "0xC3 0xB6" which is the utf-8 codes for ö,
so the C-h l output indicates your terminal sends utf-8 sequences and
Emacs gets them fine, but if you say that instead of ö you saw some ¶,
it may indicate that your keyboard-coding-system somehow is set to use
latin-1 coding system rather than utf-8, or maybe the binary codes get
treated as latin-1 by the unicode-charset.


        Stefan

Re: 23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ISO-2022-JP-2, Size: 1007 bytes --]

In article <jwvhcfvdrod.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > C-s ^[$(D**^[(B ^[$B"y^[(B ^[$(D**^[(B ^[$B"y^[(B ^[$(D**^[(B ^[$B"y^[(B C-g C-g C-g C-g ESC x r e p o TAB r
> Hmm... "^[$(D**^[(B ^[$B"y^[(B" in latin-1 is "0xC3 0xB6" which is the utf-8 codes for ^[$(D+S^[(B,
> so the C-h l output indicates your terminal sends utf-8 sequences and
> Emacs gets them fine, but if you say that instead of ^[$(D+S^[(B you saw some ^[$B"y^[(B,
> it may indicate that your keyboard-coding-system somehow is set to use
> latin-1 coding system rather than utf-8, or maybe the binary codes get
> treated as latin-1 by the unicode-charset.

It seems that something is broken in isearch-x.el (although
unicode merge didn't change it).  I'm now investigating.
But, perhaps, it's time to implement the decoding of
keyboard input by C.  After multi-tty merge, we have the
function tty_read_avail_input.  I think that is the right
place to do that decoding.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: 23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

[Stefan Monnier]

>> > C-s à ¶ à ¶ à ¶ C-g C-g C-g C-g ESC x r e p o TAB r
>> Hmm... "Ã ¶" in latin-1 is "0xC3 0xB6" which is the utf-8 codes for ö,
>> so the C-h l output indicates your terminal sends utf-8 sequences and
>> Emacs gets them fine, but if you say that instead of ö you saw some ¶,
>> it may indicate that your keyboard-coding-system somehow is set to use
>> latin-1 coding system rather than utf-8, or maybe the binary codes get
>> treated as latin-1 by the unicode-charset.

> It seems that something is broken in isearch-x.el (although
> unicode merge didn't change it).  I'm now investigating.

Thanks.

> But, perhaps, it's time to implement the decoding of
> keyboard input by C.  After multi-tty merge, we have the
> function tty_read_avail_input.  I think that is the right
> place to do that decoding.

Now that we've moved the keyboard decoding to input-event-map, I'm not
sure what would be the benefit.  Of course, if we can just reuse C code
and get rid of the encoded-kbd code, that's good.


        Stefan

Re: 23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

[Kenichi Handa]

In article <jwv8x162xdz.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> > It seems that something is broken in isearch-x.el (although
> > unicode merge didn't change it).  I'm now investigating.

I found that when isearch-printing-char is called,
last-command-char is already what decoded by
encoded-kbd-mode.  But, when I wrote isearch-x.el,
last-command-char was the first byte of utf-8 sequence.  So,
isearch-x.el pushed back that byte in unread-command-events
and re-read the whole utf-8 sequence.

It seems that something in handling keyboard input has been
changed.  I've just installed the attached change.  Tassilo,
could you please try again with the latest code?

> > But, perhaps, it's time to implement the decoding of
> > keyboard input by C.  After multi-tty merge, we have the
> > function tty_read_avail_input.  I think that is the right
> > place to do that decoding.

> Now that we've moved the keyboard decoding to input-event-map,

Ah!  Is that the change I wrote above?

> I'm not sure what would be the benefit.  Of course, if we
> can just reuse C code and get rid of the encoded-kbd code,
> that's good.

Yes.  I think it can be done by adding less than 100 lines
of C code (mostly for handling meta-key) in
tty_read_avail_input and removing most of encoded-kb.el (we
still need the code of calling set-input-mode property).

---
Kenichi Handa
handa@ni.aist.go.jp

2008-02-28  Kenichi Handa  <handa@ni.aist.go.jp>

	* isearch.el (isearch-printing-char): Don't check
	keyboard-coding-system.  Call
	isearch-process-search-multibyte-characters only when
	current-input-method is non-nil.

Index: isearch.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/isearch.el,v
retrieving revision 1.312
retrieving revision 1.313
diff -u -r1.312 -r1.313
--- isearch.el	25 Feb 2008 00:01:41 -0000	1.312
+++ isearch.el	28 Feb 2008 01:57:42 -0000	1.313
@@ -1842,15 +1842,9 @@
   (let ((char last-command-char))
     (if (= char ?\S-\ )
 	(setq char ?\s))
-    (if (and enable-multibyte-characters
-	     (>= char ?\200)
-	     (<= char ?\377))
-	(if (keyboard-coding-system)
-	    (isearch-process-search-multibyte-characters char)
-	  (isearch-process-search-char (unibyte-char-to-multibyte char)))
-      (if current-input-method
-	  (isearch-process-search-multibyte-characters char)
-	(isearch-process-search-char char)))))
+    (if current-input-method
+	(isearch-process-search-multibyte-characters char)
+      (isearch-process-search-char char))))
 
 (defun isearch-process-search-char (char)
   ;; * and ? are special in regexps when not preceded by \.

Re: 23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

[Stefan Monnier]

>> > It seems that something is broken in isearch-x.el (although
>> > unicode merge didn't change it).  I'm now investigating.

> I found that when isearch-printing-char is called,
> last-command-char is already what decoded by
> encoded-kbd-mode.

That's good, right?

> But, when I wrote isearch-x.el, last-command-char was the first byte
> of utf-8 sequence.  So, isearch-x.el pushed back that byte in
> unread-command-events and re-read the whole utf-8 sequence.

I see your patch doesn't remove this use of unread-command-events.
Is it still needed?

>> Now that we've moved the keyboard decoding to input-event-map,
> Ah!  Is that the change I wrote above?

I don't see why that change would have such an effect.
Actually, it seems that this "new" behavior is already present in the 22
branch, maybe even in Emacs-22.1.
So it looks like the

    (if (and enable-multibyte-characters
	     (>= char ?\200)
	     (<= char ?\377))

test has been harmful for a while, but in the non-unicode branch, the
o200-o377 range was never matched by "untranslated" data, whereas in the
unicode branch these are now perfectly valid latin-1 chars.

>> I'm not sure what would be the benefit.  Of course, if we
>> can just reuse C code and get rid of the encoded-kbd code,
>> that's good.

> Yes.  I think it can be done by adding less than 100 lines
> of C code (mostly for handling meta-key) in
> tty_read_avail_input and removing most of encoded-kb.el (we
> still need the code of calling set-input-mode property).

Might worth trying,


        Stefan

Re: 23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

[Kenichi Handa]

In article <jwvy79593wl.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> > It seems that something is broken in isearch-x.el (although
>>> > unicode merge didn't change it).  I'm now investigating.

> > I found that when isearch-printing-char is called,
> > last-command-char is already what decoded by
> > encoded-kbd-mode.

> That's good, right?

I hope so.

> > But, when I wrote isearch-x.el, last-command-char was the first byte
> > of utf-8 sequence.  So, isearch-x.el pushed back that byte in
> > unread-command-events and re-read the whole utf-8 sequence.

> I see your patch doesn't remove this use of unread-command-events.
> Is it still needed?

This part in isearch-process-search-multibyte-characters
(isearch-x.el) is still necessary because it seems that the
user key is not yet handled by an input method.

	(if isearch-input-method-function
	   ...)

but the following part is not necessary any more (but not
harmful even if it is there).

	(if (and (not str) (keyboard-coding-system))
	    (setq unread-command-events
		  (cons 'with-keyboard-coding
			(cons last-char unread-command-events))
		  str (read-string prompt nil 'junk-hist)))

I'll delete that part after confirming it is ok.

>>> Now that we've moved the keyboard decoding to input-event-map,
> > Ah!  Is that the change I wrote above?

> I don't see why that change would have such an effect.
> Actually, it seems that this "new" behavior is already present in the 22
> branch, maybe even in Emacs-22.1.

But, it was before the release of 21 when I modified
isearch-x last time.

> > Yes.  I think it can be done by adding less than 100 lines
> > of C code (mostly for handling meta-key) in
> > tty_read_avail_input and removing most of encoded-kb.el (we
> > still need the code of calling set-input-mode property).

> Might worth trying,

I'll work on it after the current problems of font selection
are all solved.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: 23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

[Tassilo Horn]

Kenichi Handa <handa@m17n.org> writes:

Hi!

> It seems that something in handling keyboard input has been changed.
> I've just installed the attached change.  Tassilo, could you please
> try again with the latest code?

Yes, it works again.  Thanks a lot.

Bye,
Tassilo

Re: 23.0.60; Cannot isearch for non-ascii chars with emacs -nw -Q

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> C-s à ¶ à ¶ à ¶ C-g C-g C-g C-g ESC x r e p o TAB r
>
> Hmm... "Ã ¶" in latin-1 is "0xC3 0xB6" which is the utf-8 codes for ö,
> so the C-h l output indicates your terminal sends utf-8 sequences and
> Emacs gets them fine, but if you say that instead of ö you saw some ¶,
> it may indicate that your keyboard-coding-system somehow is set to use
> latin-1 coding system rather than utf-8,

No, it's utf-8-unix.  I use LANG=en_US.utf8 and don't fiddle with coding
systems in any config file, so everything should use utf8.

Bye,
Tassilo