re-search bug?

re-search bug?

[Katsumi Yamaoka]

Hi,

With the latest Emacs CVS, the following form causes an error.

(with-temp-buffer
  (insert "Sender: ")
  (let ((pt (point)))
    (insert "ding-owner@lists.math.uh.edu")
    (re-search-backward "ding-owner@lists\\.math\\.uh\\.edu" pt)))

It's quite strange.  Because of this, all mails delivered to the
ding list are stored into the `bogus' group.  Please help!

Re: re-search bug?

[Kenichi Handa]

In article <b9yekfytpjy.fsf@jpl.org>, Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Hi,
> With the latest Emacs CVS, the following form causes an error.

> (with-temp-buffer
>   (insert "Sender: ")
>   (let ((pt (point)))
>     (insert "ding-owner@lists.math.uh.edu")
>     (re-search-backward "ding-owner@lists\\.math\\.uh\\.edu" pt)))

> It's quite strange.  Because of this, all mails delivered to the
> ding list are stored into the `bogus' group.  Please help!

This is due to my last change for handling cases for
I-WITH-DOT-ABOVE and DOTLESS-i.  I've just cancelled those
changes.

It seems that the current code in casefiddle.c has a serious
bug in supporting this kind of case-mapping.

i <--up/downcase--> I
DOTLESS-i --upcase-> I
I-WITH-DOT-ABOVE --downcase-> i

---
Ken'ichi HANDA
handa@m17n.org

Re: re-search bug?

[Katsumi Yamaoka]

>>>>> In <200502030223.LAA24558@etlken.m17n.org> Kenichi Handa wrote:

>> With the latest Emacs CVS, the following form causes an error.

>> (with-temp-buffer
>>   (insert "Sender: ")
>>   (let ((pt (point)))
>>     (insert "ding-owner@lists.math.uh.edu")
>>     (re-search-backward "ding-owner@lists\\.math\\.uh\\.edu" pt)))

> This is due to my last change for handling cases for
> I-WITH-DOT-ABOVE and DOTLESS-i.  I've just cancelled those
> changes.

I confirmed it has been fixed.  Thanks.  At just the time
Handa-san fixed it, I was looking for the cause of the problem. ;-)

> It seems that the current code in casefiddle.c has a serious
> bug in supporting this kind of case-mapping.

> i <--up/downcase--> I
> DOTLESS-i --upcase-> I
> I-WITH-DOT-ABOVE --downcase-> i

I see it is the case issue.  The form above did work if
`case-fold-search' was bound to nil.

Regards,

Re: re-search bug?

[Richard Stallman]

    It seems that the current code in casefiddle.c has a serious
    bug in supporting this kind of case-mapping.

    i <--up/downcase--> I
    DOTLESS-i --upcase-> I
    I-WITH-DOT-ABOVE --downcase-> i

Do you want to debug the serious bug?
If you send me a precise test case, I could try to debug it.

Re: re-search bug?

[Kenichi Handa]

In article <E1CwmQS-0003Eo-S0@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     It seems that the current code in casefiddle.c has a serious
>     bug in supporting this kind of case-mapping.

>     i <--up/downcase--> I
>     DOTLESS-i --upcase-> I
>     I-WITH-DOT-ABOVE --downcase-> i

> Do you want to debug the serious bug?

I can't work on it at the moment because I'll soon go abroad
and be off-line for a week.

Anyway, there's a possibility that the bug is not in
casefiddle.c but in case-fold searching and regexp matching
functions.  I'm not sure.

> If you send me a precise test case, I could try to debug it.

Please evaluate the attached code.  Then try these:

(test-regexp) => 1   ; correct
(test-search) => nil ; correct
(setup-case-table)
(test-regexp) => nil ; incorrect
(test-search) => nil ; incorrect
(recover-case-table)
(test-regexp) => 1   ; correct
(test-search) => nil ; correct

---
Ken'ichi HANDA
handa@m17n.org

(require 'case-table)

(defvar original-case-table (copy-case-table (standard-case-table)))
(defvar dotted-I (make-char 'latin-iso8859-9 ?\335))
(defvar dotless-i (make-char 'latin-iso8859-9 ?\375))

(defun setup-case-table ()
  (let ((tbl (standard-case-table)))
    (set-downcase-syntax dotted-I ?i tbl)
    (set-upcase-syntax ?I dotless-i tbl)))

(defun recover-case-table ()
  (set-standard-case-table (copy-case-table original-case-table)))

(defun test-regexp ()
  (with-temp-buffer
    (let ((case-fold-search t))
      (insert "i.")
      (re-search-backward "i\\." nil t))))

(defun test-search ()
  (with-temp-buffer
    (let ((case-fold-search t))
      (insert dotless-i)
      (search-backward "i" nil t))))

Re: re-search bug?

[Stefan Monnier]

>> i <--up/downcase--> I
>> DOTLESS-i --upcase-> I
>> I-WITH-DOT-ABOVE --downcase-> i

>> Do you want to debug the serious bug?

> I can't work on it at the moment because I'll soon go abroad
> and be off-line for a week.

> Anyway, there's a possibility that the bug is not in
> casefiddle.c but in case-fold searching and regexp matching
> functions.  I'm not sure.

That's what I would expect.  I think even the definition of "case fold
search" is not as obvious as it seems.  Does it mean (equal (upcase X)
(upcase Y)) or (equal (downcase X) (downcase Y)) or something yet different?

Thanks for the test case,


        Stefan

Re: re-search bug?

[Richard Stallman]

    That's what I would expect.  I think even the definition of "case fold
    search" is not as obvious as it seems.  Does it mean (equal (upcase X)
    (upcase Y)) or (equal (downcase X) (downcase Y)) or something yet different?

Neither of those.  It is supposed to find matches whenever the
characters are connected by any sequence of case conversions.  This
uses the canon table and the equiv table.  In search_buffer and its
subroutines, TRT refers to the canon table, and INVERSE_TRT refers to
the equiv table.  So the table passed to regex searching is the canon
table.

Definitions of these two tables:

CANONICALIZE maps each character to a canonical equivalent;
 any two characters that are related by case-conversion have the same
 canonical equivalent character...
EQUIVALENCES is a map that cyclicly permutes each equivalence class
 (of characters with the same canonical equivalent)...

A good first step would be to look at the canon and equiv table
contents, in the case that fails, and see if they correspond correctly
to the upcase and downcase tables.