From: Eli Zaretskii <eliz@gnu.org>
To: handa <handa@gnu.org>
Cc: gregory@heytings.org, 46933@debbugs.gnu.org
Subject: bug#46933: Possible bugs in filepos-to-bufferpos / bufferpos-to-filepos
Date: Sat, 27 Mar 2021 10:54:28 +0300 [thread overview]
Message-ID: <8335whowuj.fsf@gnu.org> (raw)
In-Reply-To: <874kgxtatr.fsf@gnu.org> (message from handa on Sat, 27 Mar 2021 14:38:56 +0900)
> From: handa <handa@gnu.org>
> Cc: gregory@heytings.org, 46933@debbugs.gnu.org
> Date: Sat, 27 Mar 2021 14:38:56 +0900
>
> In article <83ft0obk7i.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
>
> > Kenichi, why are these 6 bytes inserted by encode-coding-region, but
> > not when we encode the same text as part of saving the buffer to its
> > file? And why does it happen near the end of the text, between those
> > 2 particular letters?
>
> There surely exists a bug. Could you please try the attached patch?
>
> The reason why that bug did not happen on file writing is that the code
> in write_region calls encoding routine repeatedly without
> CODING_MODE_LAST_BLOCK flag, and only in the case that flushing is
> required (e.g. the case of iso-2022-jp), just for flushing, it calls
> enoding routine again with CODING_MODE_LAST_BLOCK flag. In that case,
> carryover does not happen in encode_coding ().
Thanks. The patch fixes the problem with the extra 6 bytes, so I
installed it.
The results of filepos-to-bufferpos with the file attached by Gregory
are better now, but there are still problems for some values of BYTE
argument. The problem is that ISO-2022 encoding (and others like it)
include shift-in and shift-out sequences, used to switch between
character sets. As a trivial example, each CR+LF sequence has the
"ESC ( B" sequence before it and "ESC $ B" sequence after it, to
switch to ASCII before the newline, then switch to Japanese after it.
And likewise whenever there's Latin text within Japanese (there are
quite a lot of them in this particular file). These shift-in and
shift-out sequences consume bytes, but don't produce any characters.
So if the BYTE argument of filepos-to-bufferpos specifies a byte in
the middle of one of these shift sequences, the result will be
incorrect, because decoding a partial sequence produces the bytes of
that sequence verbatim, and the logic in filepos-to-bufferpos of using
the length of the decoded text breaks. We need special handling of
this and other similar coding-systems to fix these corner use cases,
similarly to what we do in filepos-to-bufferpos--dos. Patches
welcome.
I'm leaving this bug open because not all of the problem was fixed.
next prev parent reply other threads:[~2021-03-27 7:54 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-04 21:21 bug#46933: Possible bugs in filepos-to-bufferpos / bufferpos-to-filepos Gregory Heytings
2021-03-21 15:27 ` Eli Zaretskii
2021-03-27 5:38 ` handa
2021-03-27 7:54 ` Eli Zaretskii [this message]
2021-03-27 13:23 ` handa
2021-03-27 13:54 ` Eli Zaretskii
2021-03-28 14:29 ` handa
2021-03-28 14:51 ` Eli Zaretskii
2021-04-01 15:14 ` handa
2021-04-01 15:25 ` Eli Zaretskii
2021-04-01 15:32 ` Eli Zaretskii
2021-04-03 16:12 ` handa
2022-06-20 0:59 ` Lars Ingebrigtsen
2022-06-20 11:52 ` Eli Zaretskii
2022-06-21 10:40 ` Lars Ingebrigtsen
2022-06-21 12:14 ` Eli Zaretskii
2022-06-22 4:17 ` Lars Ingebrigtsen
2022-06-22 13:11 ` Eli Zaretskii
2021-03-27 14:24 ` Gregory Heytings
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/emacs/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8335whowuj.fsf@gnu.org \
--to=eliz@gnu.org \
--cc=46933@debbugs.gnu.org \
--cc=gregory@heytings.org \
--cc=handa@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).