Interactive ucs-insert fails in HEAD

Interactive ucs-insert fails in HEAD

[Sascha Wilde]

emacs -Q -nw
M-x toggle-debug-on-error RET
C-x 8 RET sm TAB

results in:
Debugger entered--Lisp error: (wrong-type-argument symbolp 89000.0)
  signal(wrong-type-argument (symbolp 89000.0))
  completion--some(#[(style) "\b	\236A@\n\v\f
$\207" [style completion-styles-alist string table pred point] 5] (basic partial-completion emacs22))
  completion-try-completion("sm" (lambda (&rest --cl-rest--) (apply #[... "r\305 \x18\306\b!\203\x10\0\307\b!\202\x11\0p)q\210\310	\nJ\v!\v\f$)\207" [win action G33900 string pred minibuffer-selected-window window-live-p window-buffer complete-with-action] 5] (quote --fun--) --cl-rest--)) nil 2)
  completion--do-completion()
  minibuffer-complete()
  call-interactively(minibuffer-complete nil nil)
  completing-read("Unicode (name or hex): " (lambda (&rest --cl-rest--) (apply #[... "r\305 \x18\306\b!\203\x10\0\307\b!\202\x11\0p)q\210\310	\nJ\v!\v\f$)\207" [win action G33900 string pred minibuffer-selected-window window-live-p window-buffer complete-with-action] 5] (quote --fun--) --cl-rest--)))
  read-char-by-name("Unicode (name or hex): ")
  (list (read-char-by-name "Unicode (name or hex): ") (prefix-numeric-value current-prefix-arg) t)
  call-interactively(ucs-insert nil nil)

Actually the "sm" completion is just an arbitrary example, any use of
`C-x 8 RET' I tried fails with (wrong-type-argument symbolp 89000.0)...

cheers
sascha
-- 
Sascha Wilde : "Lies, was ich meine, nicht, was ich schreibe."
             : (Urs Traenkner in de.alt.admin)

Re: Interactive ucs-insert fails in HEAD

[Eli Zaretskii]

> From: Sascha Wilde <wilde@sha-bang.de>
> Date: Fri, 09 Oct 2009 11:57:03 +0200
> 
> Actually the "sm" completion is just an arbitrary example, any use of
> `C-x 8 RET' I tried fails with (wrong-type-argument symbolp 89000.0)...

More accurately, any use of "C-x 8 RET" except if you type a valid hex
code at the prompt.

Also, I don't get an error until I type RET.  IOW, this fails:

  C-x 8 RET sm RET

but this does not:

  C-x 8 RET sm TAB

Re: Interactive ucs-insert fails in HEAD

[Juanma Barranquero]

On Fri, Oct 9, 2009 at 13:32, Eli Zaretskii <eliz@gnu.org> wrote:

> More accurately, any use of "C-x 8 RET" except if you type a valid hex
> code at the prompt.

Could be related to the upgrade of the data files to Unicode 5.2.0,
though the format didn't change so it shouldn't...

I'll take a look.

    Juanma

Re: Interactive ucs-insert fails in HEAD

[Juanma Barranquero]

There's a new range of ideographs that must be excluded from loading
in ucs-names. It should work now (though you'll have to redump Emacs).

    Juanma

Re: Interactive ucs-insert fails in HEAD

[Sascha Wilde]

Juanma Barranquero <lekktu@gmail.com> wrote:

> There's a new range of ideographs that must be excluded from loading
> in ucs-names. It should work now (though you'll have to redump Emacs).

Indeed it's fixed.  Thanks!
☺ [0]

cheers
sascha

[0] C-x 8 WHITE SMILING FACE RET
-- 
Sascha Wilde

Lisp is the red pill.
  -- John Fraser, comp.lang.lisp

Re: Interactive ucs-insert fails in HEAD

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Fri, 9 Oct 2009 15:04:35 +0200
> Cc: emacs-devel@gnu.org
> 
> There's a new range of ideographs that must be excluded from loading
> in ucs-names.

Why is there reason to exclude any characters from ucs-names?

Anyway, if there are good reasons, they should be stated in a comment
near the exclusion code.  Also, I'd suggest to have some test run when
the Unicode database is updated, to reveal any additional characters
to be excluded, so that in the future such breakage could be avoided
with less pain.

Re: Interactive ucs-insert fails in HEAD

[Juanma Barranquero]

On Fri, Oct 9, 2009 at 18:21, Eli Zaretskii <eliz@gnu.org> wrote:

> Why is there reason to exclude any characters from ucs-names?
>
> Anyway, if there are good reasons, they should be stated in a comment
> near the exclusion code.  Also, I'd suggest to have some test run when
> the Unicode database is updated, to reveal any additional characters
> to be excluded, so that in the future such breakage could be avoided
> with less pain.

In fact, I think there's no reason to exclude names, except for the
ranges in UnicodeData.txt (Surrogates, Ideograph Extensions, etc.).

My patch is just a workaround so ucs-insert works again. I've already
asked Handa-san to please take a look at it.

    Juanma

Re: Interactive ucs-insert fails in HEAD

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Fri, 9 Oct 2009 21:47:59 +0200
> Cc: wilde@sha-bang.de, emacs-devel@gnu.org
> 
> On Fri, Oct 9, 2009 at 18:21, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Why is there reason to exclude any characters from ucs-names?
> >
> > Anyway, if there are good reasons, they should be stated in a comment
> > near the exclusion code.  Also, I'd suggest to have some test run when
> > the Unicode database is updated, to reveal any additional characters
> > to be excluded, so that in the future such breakage could be avoided
> > with less pain.
> 
> In fact, I think there's no reason to exclude names, except for the
> ranges in UnicodeData.txt (Surrogates, Ideograph Extensions, etc.).

If you mean entries like this one:

    20000;<CJK Ideograph Extension B, First>;Lo;0;L;;;;;N;;;;;
    2A6D6;<CJK Ideograph Extension B, Last>;Lo;0;L;;;;;N;;;;;

then I agree.  Still, I don't understand how including them could
screw completion in ucs-insert.  Can you explain?

Anyway, we now skip entries like this as well:

    1F210;SQUARED CJK UNIFIED IDEOGRAPH-624B

which I think we don't need to skip.

Re: Interactive ucs-insert fails in HEAD

[Juanma Barranquero]

On Sat, Oct 10, 2009 at 09:52, Eli Zaretskii <eliz@gnu.org> wrote:

> If you mean entries like this one:
>
>    20000;<CJK Ideograph Extension B, First>;Lo;0;L;;;;;N;;;;;
>    2A6D6;<CJK Ideograph Extension B, Last>;Lo;0;L;;;;;N;;;;;
>
> then I agree.  Still, I don't understand how including them could
> screw completion in ucs-insert.  Can you explain?

Is not that they screw completion, is that they provide no name for
completion. You still can use ucs-insert to insert them by codepoint
number.

> What I meant is to have something that detects in UnicodeData.txt the
> pattern of entries which need to be excluded.  For example, if entries
> ike this are the only ones:

    4DB5;<CJK Ideograph Extension A, Last>;Lo;0;L;;;;;N;;;;;

> then detecting bracketed text where a character name should be would
> suffice.  We could then run such a script as part of what
> admin/Makefile does for the unidata.txt file it creates, and display a
> proper message.

unidata-setup-list already detects these kind of ranges and deals with them.

> Anyway, we now skip entries like this as well:
>
>    1F210;SQUARED CJK UNIFIED IDEOGRAPH-624B
>
> which I think we don't need to skip.

As I said before, I agree: we *shouldn't* need to skip them. But
somehow, the process in unidata-gen.el that creates the chartables
with codepoint information is generating bad data exclusively for that
range: U+1F200..U+1F2FF (which are not the only characters added in
5.2.0, BTW). That's what triggered the ucs-insert error: getting the
name of one of those characters returns a number instead of a symbol.
I'm trying to determine why the bad data is generated in the first
place.

    Juanma

Re: Interactive ucs-insert fails in HEAD

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Sat, 10 Oct 2009 13:44:02 +0200
> Cc: emacs-devel@gnu.org
> 
> > Anyway, we now skip entries like this as well:
> >
> >    1F210;SQUARED CJK UNIFIED IDEOGRAPH-624B
> >
> > which I think we don't need to skip.
> 
> As I said before, I agree: we *shouldn't* need to skip them. But
> somehow, the process in unidata-gen.el that creates the chartables
> with codepoint information is generating bad data exclusively for that
> range: U+1F200..U+1F2FF (which are not the only characters added in
> 5.2.0, BTW). That's what triggered the ucs-insert error: getting the
> name of one of those characters returns a number instead of a symbol.
> I'm trying to determine why the bad data is generated in the first
> place.

OK, my bad: I misunderstood what you were saying.  Now it makes sense
to me.

Re: Interactive ucs-insert fails in HEAD

[Juanma Barranquero]

On Sat, Oct 10, 2009 at 15:14, Eli Zaretskii <eliz@gnu.org> wrote:

> OK, my bad: I misunderstood what you were saying.  Now it makes sense
> to me.

In fact, I didn't explain myself very well before. I should've made
clear from the start that the ucs-insert change was just a workaround.

    Juanma

Re: Interactive ucs-insert fails in HEAD

[Kenichi Handa]

In article <f7ccd24b0910091247u4d2f936ek36a00c2c973d636d@mail.gmail.com>, Juanma Barranquero <lekktu@gmail.com> writes:

> My patch is just a workaround so ucs-insert works again. I've already
> asked Handa-san to please take a look at it.

I found that the problem is caused by this name.

SQUARED CJK UNIFIED IDEOGRAPH-89E3

The Lisp reader reads both "89e3" and "89E3" as 89000.0, but
symbols of those names are printed differently:

(format "%S %S" '\89e3 '\89E3)
=> "\\89e3 89E3"

(1) "89E3" should be read as a symbol.
  OR
(2) '\89E3 should be printed as \89E3.

I'm not sure which is better.

Anyway, I've just committed uni-name.el that contains
"\89e3" (instead of "89e3"), so the current bug should be
fixed now.

The fix for (1) is this:

--- lread.c.~1.414.~	2009-09-25 21:43:00.000000000 +0900
+++ lread.c	2009-10-13 13:47:10.000000000 +0900
@@ -3272,7 +3272,7 @@
       while (*cp >= '0' && *cp <= '9')
 	cp++;
     }
-  if (*cp == 'e' || *cp == 'E')
+  if (*cp == 'e')
     {
       state |= E_CHAR;
       cp++;


And the fix for (2) is this:

--- print.c.~1.259.~	2009-09-25 21:43:00.000000000 +0900
+++ print.c	2009-10-13 13:40:36.000000000 +0900
@@ -1753,7 +1753,7 @@
 	  {
 	    while (p != end && ((*p >= '0' && *p <= '9')
 				/* Needed for \2e10.  */
-				|| *p == 'e'))
+				|| *p == 'e' || *p == 'E'))
 	      p++;
 	    confusing = (end == p);
 	  }

Stefan and Yidong, please install whichever you think better.

---
Kenichi Handa
handa@m17n.org

Re: Interactive ucs-insert fails in HEAD

[Stefan Monnier]

> And the fix for (2) is this:

> --- print.c.~1.259.~	2009-09-25 21:43:00.000000000 +0900
> +++ print.c	2009-10-13 13:40:36.000000000 +0900
> @@ -1753,7 +1753,7 @@
>  	  {
>  	    while (p != end && ((*p >= '0' && *p <= '9')
>  				/* Needed for \2e10.  */
> -				|| *p == 'e'))
> +				|| *p == 'e' || *p == 'E'))
>  	      p++;
>  	    confusing = (end == p);
>  	  }

That's the right fix, thank you,


        Stefan

Re: Interactive ucs-insert fails in HEAD

[Kenichi Handa]

In article <jwvpr8rdthx.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> > --- print.c.~1.259.~	2009-09-25 21:43:00.000000000 +0900
> > +++ print.c	2009-10-13 13:40:36.000000000 +0900
> > @@ -1753,7 +1753,7 @@
> >  	  {
> >  	    while (p != end && ((*p >= '0' && *p <= '9')
> >  				/* Needed for \2e10.  */
> > -				|| *p == 'e'))
> > +				|| *p == 'e' || *p == 'E'))
> >  	      p++;
> >  	    confusing = (end == p);
> >  	  }

> That's the right fix, thank you,

I've just installed that change.

---
Kenichi Handa
handa@m17n.org

Re: Interactive ucs-insert fails in HEAD

[Juanma Barranquero]

On Fri, Oct 9, 2009 at 18:21, Eli Zaretskii <eliz@gnu.org> wrote:

> Also, I'd suggest to have some test run when
> the Unicode database is updated, to reveal any additional characters
> to be excluded, so that in the future such breakage could be avoided
> with less pain.

I had the beta 5.2.0 files in my git checkout for weeks and did some
tests, BTW. I'm puzzled that I didn't fail to me before.

    Juanma

Re: Interactive ucs-insert fails in HEAD

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Fri, 9 Oct 2009 21:50:10 +0200
> Cc: wilde@sha-bang.de, emacs-devel@gnu.org
> 
> On Fri, Oct 9, 2009 at 18:21, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Also, I'd suggest to have some test run when
> > the Unicode database is updated, to reveal any additional characters
> > to be excluded, so that in the future such breakage could be avoided
> > with less pain.
> 
> I had the beta 5.2.0 files in my git checkout for weeks and did some
> tests, BTW. I'm puzzled that I didn't fail to me before.

What I meant is to have something that detects in UnicodeData.txt the
pattern of entries which need to be excluded.  For example, if entries
like this are the only ones:

     4DB5;<CJK Ideograph Extension A, Last>;Lo;0;L;;;;;N;;;;;

then detecting bracketed text where a character name should be would
suffice.  We could then run such a script as part of what
admin/Makefile does for the unidata.txt file it creates, and display a
proper message.