regex and case-fold-search problem

[Kenichi Handa <handa@etl.go.jp>]

While working on emacs-unicode, I noticed a very difficult
problem which also exists in the current emacs.

(let ((case-fold-search nil))
  (string-match "[Þ-ß]" "Þ")) => 0
(let ((case-fold-search nil))
  (string-match "[Þß]" "Þ")) => 0

(let ((case-fold-search t))
  (string-match "[Þ-ß]" "Þ")) => nil !!!
(let ((case-fold-search t))
  (string-match "[Þß]" "Þ")) => 0

When you see the output of M-x list-charset-chars RET
latin-iso8859-1 RET,  you'll soon find what's going on.

The relevan character codes are as follows:
	Þ (#x8DE)
	ß (#x8DF)
	(downcase ?Þ) == ?þ (#x8FE)
	(downcase ?ß) == ?ß (#x8DF)

This problem is not specific to non-ASCII chars, it's just
rarer to face such a sitution in ASCII chars.

(let ((case-fold-search nil))
  (string-match "[A-_]" "A")) => 0
(let ((case-fold-search t))
  (string-match "[A-_]" "A")) => nil
(let ((case-fold-search t))
  (string-match "[A_]" "A")) => 0

In my opinion, specifying ranges by chars are nonsense
because there should be no semantics in the order of
characters codes.  But, anyway, we have to decide what to
do.

(1) Regard the above case as a bug, and fix it completely.
    As we don't support a range striding over different
    charsets by the current Emacs, I think the fix is
    difficult but not that much.  But, in emacs-unicode, we
    can't have such a restriction, and thus the fix is very
    difficult.

(2) Regard the above case as an (unpleasant) feature, and
    document it.

(3) Signal an error for such a regex (and of course document
    it).

---
Ken'ichi HANDA
handa@etl.go.jp