Re: Questionable code in handling of wordend in the regexp engine in regex-emacs.c

[Alan Mackenzie <acm@muc.de>]

Hello, Eli.

On Fri, Mar 01, 2019 at 21:31:37 +0200, Eli Zaretskii wrote:
> > Date: Fri, 1 Mar 2019 19:16:07 +0000
> > Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > +  /* Check bytepos is not in the middle of a character. */
> > +  eassert (bytepos >= BUF_Z_BYTE (b)
> > +           || CHAR_HEAD_P (BUF_FETCH_BYTE (b, bytepos)));

> LGTM.

> > After configuring with --enable-checking and building, I tried make
> > check.  The tests errored out with this bytepos check three times.  In:

> >     src/coding-tests.log
> >     lisp/epg-tests.log
> >     lisp/emacs-lisp/package-tests.log

> > .  Quite possibly there is just one bug here, but there might be two or
> > three.  I think it would be best to track it/them down before committing
> > the change to marker.c.

> I agree.  Let me know if you need help with that.

On the coding tests, we fail in one of them when bytepos == 1, and the
first byte in the buffer is 0xa4.  The Lisp call stack at the time is

Lisp Backtrace:
"decode-coding-region" (0xc35b5cb0)
"progn" (0xc35b5d68)
"unwind-protect" (0xc35b5ea8)
"save-current-buffer" (0xc35b6018)
"let" (0xc35b6208)
0xd985d350 Lisp type 3
"ert--run-test-internal" (0xc35b69b0)
"ert-run-test" (0xc35b6ed8)
"ert-run-or-rerun-test" (0xc35b7410)
"ert-run-tests" (0xc35b7938)
"ert-run-tests-interactively" (0xc35b7f30)
"funcall-interactively" (0xc35b7f28)
"call-interactively" (0xc35b8310)
"command-execute" (0xc35b8838)
"execute-extended-command" (0xc35b8f20)
"funcall-interactively" (0xc35b8f18)
"call-interactively" (0xc35b9300)
"command-execute" (0xc35b9808)

, and the first few lines of the C backtrace are:

(gdb) backtrace
#0  terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:370
#1  0x000055a4d8603595 in die (msg=0x55a4d87637a8 "bytepos >= BUF_Z_BYTE (b) || CHAR_HEAD_P (BUF_FETCH_BYTE (b, bytepos))", file=0x55a4d8763728 "marker.c", line=327) at alloc.c:7442
#2  0x000055a4d85b42cf in buf_bytepos_to_charpos (b=0x55a4da051710, bytepos=1) at marker.c:326
#3  0x000055a4d85ab93f in move_gap_both (charpos=1, bytepos=1) at insdel.c:92
#4  0x000055a4d84ca5d9 in decode_coding_object (coding=0x7ffcc35b5950, src_object=XIL(0x55a4da051715), from=1, from_byte=1, to=5, to_byte=5, dst_object=XIL(0x55a4da051715)) at coding.c:8072
#5  0x000055a4d84cff9f in code_convert_region (start=make_number(1), end=make_number(5), coding_system=XIL(0x2a1906036aa0), dst_object=XIL(0x55a4da051715), encodep=false, norecord=false) at coding.c:9371
#6  0x000055a4d84d005a in Fdecode_coding_region (start=make_number(1), end=make_number(5), coding_system=XIL(0x2a1906036aa0), destination=XIL(0)) at coding.c:9401
#7  0x000055a4d86370e1 in eval_sub (form=XIL(0x55a4d981ded3)) at eval.c:2325

.  More precisely, the first few bytes of the buffer are:

    a4 a2 0d 0a 00 00 00 00

, and I suspect the buffer is just 4 bytes long (but I'm too lazy to
check it properly).

I have a suspicion that the CHAR_HEAD_P test isn't valid here.  I'm
guessing that we're converting an external coding, something like an
MS-DOS 8-bit coding, to internal UTF-8-like coding, so we can't use
CHAR_HEAD_P.

In fact, is buf_bytepos_to_charpos the Right Thing to use here?

-- 
Alan Mackenzie (Nuremberg, Germany).