Re: 23.0.60; Segmentation fault loading auto-lang.el

Re: 23.0.60; Segmentation fault loading auto-lang.el

[Chong Yidong]

> - download http://www.marquardt-home.de/auto-lang.el to ~/.elisp/
> - run emacs -Q
> - M-x load-file
> - choose file ~/.elisp/auto-lang.el
> => Emacs segfaults (same result with emacs -Q -nw)

This is due to an infinite nesting depth in regexp-opt, which can be
tracked down to the following problem:

(let ((str (string-as-unibyte "ä")))
  (string-match (char-to-string (string-to-char str)) str))

evaluates to 0 in Emacs 22, and to nil in Emacs 23.  It turns out that
this screws up the use of all-completions in regexp-opt-group.

Anyone have any idea what's going on here?

Re: 23.0.60; Segmentation fault loading auto-lang.el

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ISO-2022-JP-2, Size: 1122 bytes --]

In article <87r6dg3oe2.fsf@stupidchicken.com>, Chong Yidong <cyd@stupidchicken.com> writes:

> > - download http://www.marquardt-home.de/auto-lang.el to ~/.elisp/
> > - run emacs -Q
> > - M-x load-file
> > - choose file ~/.elisp/auto-lang.el
> > => Emacs segfaults (same result with emacs -Q -nw)

> This is due to an infinite nesting depth in regexp-opt, which can be
> tracked down to the following problem:

> (let ((str (string-as-unibyte "^[$(D+#^[(B")))
>   (string-match (char-to-string (string-to-char str)) str))

> evaluates to 0 in Emacs 22, and to nil in Emacs 23.  It turns out that
> this screws up the use of all-completions in regexp-opt-group.

> Anyone have any idea what's going on here?

(string-as-unibyte "^[$(D+#^[(B") => "\303\244"
(string-to-char "\303\244") => 195 (because ?\303 == 195)
(char-to-string 195) => "^[$(D**^[(B" (because 195==0xC3 U+00C3=='^[$(D**^[(B')
(string-match "^[$(D**^[(B" "^[$(D+#^[(B") => nil (obvious)

Any Lisp program that depends on the result of
string-as-unibyte (thus Emacs' internal character
representation) won't work in Emacs 23.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: 23.0.60; Segmentation fault loading auto-lang.el

[Chong Yidong]

Kenichi Handa <handa@m17n.org> writes:

> In article <87r6dg3oe2.fsf@stupidchicken.com>, Chong Yidong <cyd@stupidchicken.com> writes:
>
>> > - download http://www.marquardt-home.de/auto-lang.el to ~/.elisp/
>> > - run emacs -Q
>> > - M-x load-file
>> > - choose file ~/.elisp/auto-lang.el
>> > => Emacs segfaults (same result with emacs -Q -nw)
>
>> This is due to an infinite nesting depth in regexp-opt, which can be
>> tracked down to the following problem:
>
>> (let ((str (string-as-unibyte "ä")))
>>   (string-match (char-to-string (string-to-char str)) str))
>
>> evaluates to 0 in Emacs 22, and to nil in Emacs 23.  It turns out that
>> this screws up the use of all-completions in regexp-opt-group.
>
>> Anyone have any idea what's going on here?
>
> (string-as-unibyte "ä") => "\303\244"
> (string-to-char "\303\244") => 195 (because ?\303 == 195)
> (char-to-string 195) => "Ã" (because 195==0xC3 U+00C3=='Ã')
> (string-match "Ã" "ä") => nil (obvious)
>
> Any Lisp program that depends on the result of
> string-as-unibyte (thus Emacs' internal character
> representation) won't work in Emacs 23.

I see.  However, maybe the following change to regexp-opt-group in
regexp-opt.el would make things a little more predictable.  What do you
think?

*** trunk/lisp/emacs-lisp/regexp-opt.el.~1.37.~	2008-03-14 17:17:34.000000000 -0400
--- trunk/lisp/emacs-lisp/regexp-opt.el	2008-04-08 12:46:49.000000000 -0400
***************
*** 226,232 ****
  
  	      ;; Otherwise, divide the list into those that start with a
  	      ;; particular letter and those that do not, and recurse on them.
! 	      (let* ((char (char-to-string (string-to-char (car strings))))
  		     (half1 (all-completions char strings))
  		     (half2 (nthcdr (length half1) strings)))
  		(concat open-group
--- 226,232 ----
  
  	      ;; Otherwise, divide the list into those that start with a
  	      ;; particular letter and those that do not, and recurse on them.
! 	      (let* ((char (substring (car strings) 0 1))
  		     (half1 (all-completions char strings))
  		     (half2 (nthcdr (length half1) strings)))
  		(concat open-group

Re: 23.0.60; Segmentation fault loading auto-lang.el

[Stefan Monnier]

>>> (let ((str (string-as-unibyte "ä")))
>>> (string-match (char-to-string (string-to-char str)) str))
>> 
>>> evaluates to 0 in Emacs 22, and to nil in Emacs 23.  It turns out that
>>> this screws up the use of all-completions in regexp-opt-group.
>> 
>>> Anyone have any idea what's going on here?
>> 
>> (string-as-unibyte "ä") => "\303\244"
>> (string-to-char "\303\244") => 195 (because ?\303 == 195)
>> (char-to-string 195) => "Ã" (because 195==0xC3 U+00C3=='Ã')
>> (string-match "Ã" "ä") => nil (obvious)
>> 
>> Any Lisp program that depends on the result of
>> string-as-unibyte (thus Emacs' internal character
>> representation) won't work in Emacs 23.

Notice that the problem is unrelated to string-as-unibyte:

   (string-match (char-to-string (string-to-char str)) str)

this should intuitively always return 0.  Of course, once you replace
`char-to-string' with just `string', you may be reminded that Emacs-23
introduced `unibyte-string', which leads you to the key, if `str' is
unibyte, you need to do

   (string-match (unibyte-string (string-to-char str)) str)

In Emacs-22, `string' used a heuristic to decide whether to build
a unibyte or multibyte string, and more importantly, the character
representing byte code 209 had code 209, whereas in Emacs-23, we have
the strange situation that byte 209 is character 4194257.

So an integer <256 needs to be accompagnied with some contextual info
that says whether it represents a char or a byte, otherwise you get
ambiguity that lead to bugs.  And string-to-char returns either a byte
or a char depending on whether the string was unibyte or multibyte.

> I see.  However, maybe the following change to regexp-opt-group in
> regexp-opt.el would make things a little more predictable.  What do you
> think?

Yes, it looks like a good fix.  Maybe "-no-properties" would be even
better.


        Stefan

Re: 23.0.60; Segmentation fault loading auto-lang.el

[Kenichi Handa]

In article <87skxwl29o.fsf@stupidchicken.com>, Chong Yidong <cyd@stupidchicken.com> writes:

> > Any Lisp program that depends on the result of
> > string-as-unibyte (thus Emacs' internal character
> > representation) won't work in Emacs 23.

> I see.  However, maybe the following change to regexp-opt-group in
> regexp-opt.el would make things a little more predictable.  What do you
> think?

I agree because that change will avoid a unibyte string
being changed to multibyte by accident.

But, I've just downloaded auto-lang.el and found that it
has codes something like this:

(string-as-multibyte
 (regexp-opt
  (mapcar 'string-as-unibyte
	  (append
	   al-german-common-words
	   al-german-8bit-words
	   nil))))

All of them should be changed to this simple form:
  (regexp-opt (append al-german-common-words al-german-8bit-words))

The above german case works just by chance, but
al-danish-common-words doesn't.  You'll see peculiar 8-bit
codes in it.

And, the file should have a coding tag.

---
Kenichi Handa
handa@ni.aist.go.jp

> *** trunk/lisp/emacs-lisp/regexp-opt.el.~1.37.~	2008-03-14 17:17:34.000000000 -0400
> --- trunk/lisp/emacs-lisp/regexp-opt.el	2008-04-08 12:46:49.000000000 -0400
> ***************
> *** 226,232 ****
  
>   	      ;; Otherwise, divide the list into those that start with a
>   	      ;; particular letter and those that do not, and recurse on them.
> ! 	      (let* ((char (char-to-string (string-to-char (car strings))))
>   		     (half1 (all-completions char strings))
>   		     (half2 (nthcdr (length half1) strings)))
>   		(concat open-group
> --- 226,232 ----
  
>   	      ;; Otherwise, divide the list into those that start with a
>   	      ;; particular letter and those that do not, and recurse on them.
> ! 	      (let* ((char (substring (car strings) 0 1))
>   		     (half1 (all-completions char strings))
>   		     (half2 (nthcdr (length half1) strings)))
>   		(concat open-group