Re: search-forward in emacs23 lisp

[rasmith@tamu.edu]

From: Eli Zaretskii <eliz@gnu.org>
Subject: Re: search-forward in emacs23 lisp
Date: Mon, 29 Mar 2010 09:51:07 +0300

>> From: bojohan@gnu.org (Johan =?utf-8?Q?Bockg=C3=A5rd?=)
>> Date: Mon, 29 Mar 2010 01:00:45 +0200
>> Cc: 
>> 
>> There does seem to be a bug regarding search in unibyte buffers,
> 
> Please report this ASAP to the Emacs bug-tracker.  Emacs 23.2 is in
> the last stages of pretest, and so we should not waste any time
> discussing bugs here, if we want them to be fixed in the next release.
> 

After further investigation, I'm not certain it's a bug: it may be an
intentional part of the modifications to accommodate utf-8.  Here are
the details;

In a multibyte-buffer (set-buffer-multibyte t), 
   
(search-forward (char-to-string ?\xff)) matches utf-8 "ÿ" (i.e. \303\277)
(search-forward (char-to-string ?\377)) matches utf-8 "ÿ"
(search-forward (unibyte-string ?\377)) matches byte \377

In a unibyte buffer (set-buffer-multibyte nil)

(search-forward (char-to-string ?\xff)) matches \231\277
(search-forward (char-to-string ?\377)) matches \231\277
(search-forward (unibyte-string ?\377)) matches \231\277

In other words, search-forward cannot find byte \377 when searching in
a *unibyte* buffer, but it can find that same byte if the buffer is
changed to multibyte.  The reason is that in a unibyte buffer,
search-forward apparently changes byte \377 to a two-byte
representation (but not to utf-8, which would be \303\277).  

The code I had a problem with can be fixed by using char-after
(or more elegantly, I've now learned, using skip-chars-forward),
However, there's probably other code out there that's now broken
because of this.  Is it a bug, or was it a mistake to expect
search-forward to find a single high byte in a multibyte buffer in the
first place?

Robin Smith