bug#46933: Possible bugs in filepos-to-bufferpos / bufferpos-to-filepos

[handa <handa@gnu.org>]

In article <8335whowuj.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> Thanks.  The patch fixes the problem with the extra 6 bytes, so I
> installed it.

Thank you for the improved concise comment in the code.

> The results of filepos-to-bufferpos with the file attached by Gregory
> are better now, but there are still problems for some values of BYTE
> argument.  The problem is that ISO-2022 encoding (and others like it)
> include shift-in and shift-out sequences, used to switch between
> character sets.  As a trivial example, each CR+LF sequence has the
> "ESC ( B" sequence before it and "ESC $ B" sequence after it, to
> switch to ASCII before the newline, then switch to Japanese after it.
> And likewise whenever there's Latin text within Japanese (there are
> quite a lot of them in this particular file).  These shift-in and
> shift-out sequences consume bytes, but don't produce any characters.
> So if the BYTE argument of filepos-to-bufferpos specifies a byte in
> the middle of one of these shift sequences, the result will be
> incorrect, because decoding a partial sequence produces the bytes of
> that sequence verbatim, and the logic in filepos-to-bufferpos of using
> the length of the decoded text breaks.  We need special handling of
> this and other similar coding-systems to fix these corner use cases,
> similarly to what we do in filepos-to-bufferpos--dos.  Patches
> welcome.

How about something like this method:
1. Encode the buffer text one line by one until we get a longer byte
sequence than BYTE.
2. Delete the result of enoding the last line above.
3. Provided that the above last line has chars C1 C2 ... Cn, 
encode characters C1...Cn, C1...Cn-1, C1...Cn-2 until we get a shorter
byte sequence than BYTE.

The first step may be optimized by encode multiple lines instead of
single line.

---
K. Handa
handa@gnu.org