bug#5989: 23.1.96; bug 4209 not completely fixed (regression in re-search-forward)

bug#5989: 23.1.96; bug 4209 not completely fixed (regression in re-search-forward)

[Christopher J. Madsen]

I've discovered that bug #4209 was only partially fixed.  My original
recipe does not produce the bug in 23.1.96, but a slightly modified
one does:

;--- re-bug2.el starts here
(setq-default enable-multibyte-characters nil) ; this line is new

(set-buffer (get-buffer-create "*Test Buffer*"))

(insert "\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A")

(goto-char (point-min))

(message "looking-at: %s" (looking-at
"\\`\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A"))

(message "re-search-forward: %s"
         (re-search-forward "\\`\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A" 100 t))
;--- re-bug2.el ends here

At the command line, this is the expected output:

$ emacs --batch -Q -l re-bug2.el
looking-at: t
re-search-forward: 9

With emacs 23.1.96, the output I actually get is:
looking-at: t
re-search-forward: nil


This recipe is closer to what I'm actually doing:

;--- re-bug-file.el starts here
(find-file-literally "sample.file")

(goto-char (point-min))

(message "looking-at: %s" (looking-at
"\\`\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A"))

(message "re-search-forward: %s"
         (re-search-forward "\\`\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A" 100 t))
;--- re-bug-file.el ends here

It should have the same output as the previous recipe.  sample.file
must begin with those 8 bytes.  The following Perl command can create
a suitable file (just redirect its output to sample.file):

perl -e 'binmode STDOUT; print "\xC2\x4C\xEF\x77\xC6\x69\x8C\x0Aand more\n"'



The original patch proposed by Matthew Dempsky
(http://debbugs.gnu.org/cgi/bugreport.cgi?bug=4209#12) appears to fix
the bug in all its variations, but the patch that was actually applied
by Kenichi Handa only fixes the original recipe.

I apologize for not trying the official patch sooner.  I just never
got around to rebuilding Emacs.


In GNU Emacs 23.1.96.4 (i686-pc-linux-gnu)
 of 2010-04-20 on byte
configured using `configure  '--without-pop' '--without-sound'
'--without-x' '--with-x-toolkit=no' '--without-xpm' '--without-jpeg'
'--without-tiff' '--without-gif' '--without-png' '--without-rsvg'
'--without-xft' '--without-libotf' '--without-m17n-flt'
'--without-toolkit-scroll-bars' '--without-xaw3d' '--without-xim'
'--without-gpm' '--without-dbus' '--without-makeinfo''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr message sendmail regexp-opt ecomplete rfc822 mml
mml-sec password-cache mm-decode mm-bodies mm-encode mailcap mail-parse
rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader gnus-util
netrc time-date mm-util mail-prsvr gmm-utils wid-edit mailheader canlock
sha1 hex-util hashcash mail-utils emacsbug help-mode easymenu view
warnings ediff-hook vc-hooks lisp-float-type lisp-mode register page
menu-bar rfn-eshadow timer jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev loaddefs button minibuffer faces cus-face files
text-properties overlay md5 base64 format env code-pages mule custom
widget hashtable-print-readable backquote make-network-process multi-tty
emacs)

bug#5989: 23.1.96; bug 4209 not completely fixed (regression in re-search-forward)

[Noam Postavsky]

found 5989 23.4
found 5989 24.5
found 5989 25.0.94
tag 5989 + confirmed
quit

I can reproduce this on recent Emacs, with a small change to the
recipe since enable-multibyte-characters can no longer be set
directly:

(set-buffer (get-buffer-create "*Test Buffer*"))
(set-buffer-multibyte nil) ; instead of setting `enable-multibyte-characters'

(insert "\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A")

(goto-char (point-min))

(message "looking-at: %s" (looking-at
"\\`\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A"))

(message "re-search-forward: %s"
         (re-search-forward "\\`\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A" 100 t))

bug#5989: 23.1.96; bug 4209 not completely fixed (regression in re-search-forward)

[Eli Zaretskii]

> Date: Wed, 1 Jun 2016 21:29:36 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> 
> I can reproduce this on recent Emacs, with a small change to the
> recipe since enable-multibyte-characters can no longer be set
> directly:
> 
> (set-buffer (get-buffer-create "*Test Buffer*"))
> (set-buffer-multibyte nil) ; instead of setting `enable-multibyte-characters'
> 
> (insert "\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A")
> 
> (goto-char (point-min))
> 
> (message "looking-at: %s" (looking-at
> "\\`\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A"))
> 
> (message "re-search-forward: %s"
>          (re-search-forward "\\`\xC2\x4C\xEF\x77\xC6\x69\x8C\x0A" 100 t))

If you don't make the buffer unibyte, everything works as expected.
Using multibyte buffers even for raw bytes is what we recommend.  So I
see no problem here that we should work on: unibyte buffers are not
meant to process text.

Thanks.

bug#5989: 23.1.96; bug 4209 not completely fixed (regression in re-search-forward)

[Noam Postavsky]

tag 5989 + notabug
quit

On Sat, Jun 4, 2016 at 12:15 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> If you don't make the buffer unibyte, everything works as expected.
> Using multibyte buffers even for raw bytes is what we recommend.  So I
> see no problem here that we should work on: unibyte buffers are not
> meant to process text.


Makes sense. (adding OP back on CC; still getting used to debbugs, so
I had dropped them in my previous message)