From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#37580: 26.3; setting buffer as unibyte temporarily may change buffer contents Date: Sat, 05 Oct 2019 21:56:36 +0300 Message-ID: <83tv8n2f8b.fsf@gnu.org> References: <86ftkbscry.fsf@gmail.com> <83h84r89i4.fsf@gnu.org> <86lftz157z.fsf@gmail.com> Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="92392"; mail-complaints-to="usenet@blaine.gmane.org" Cc: 37580@debbugs.gnu.org To: ynyaaa@gmail.com Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Sat Oct 05 20:57:10 2019 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1iGpEr-000Nrq-PG for geb-bug-gnu-emacs@m.gmane.org; Sat, 05 Oct 2019 20:57:09 +0200 Original-Received: from localhost ([::1]:58424 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iGpEq-0005xK-Ih for geb-bug-gnu-emacs@m.gmane.org; Sat, 05 Oct 2019 14:57:08 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:33058) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iGpEk-0005xB-TR for bug-gnu-emacs@gnu.org; Sat, 05 Oct 2019 14:57:03 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iGpEj-0001oT-TG for bug-gnu-emacs@gnu.org; Sat, 05 Oct 2019 14:57:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:36417) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iGpEj-0001oN-Qj for bug-gnu-emacs@gnu.org; Sat, 05 Oct 2019 14:57:01 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1iGpEj-0005zD-Nh for bug-gnu-emacs@gnu.org; Sat, 05 Oct 2019 14:57:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 05 Oct 2019 18:57:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 37580 X-GNU-PR-Package: emacs Original-Received: via spool by 37580-submit@debbugs.gnu.org id=B37580.157030181422977 (code B ref 37580); Sat, 05 Oct 2019 18:57:01 +0000 Original-Received: (at 37580) by debbugs.gnu.org; 5 Oct 2019 18:56:54 +0000 Original-Received: from localhost ([127.0.0.1]:45236 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iGpEc-0005yW-BR for submit@debbugs.gnu.org; Sat, 05 Oct 2019 14:56:54 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:52216) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iGpEa-0005yI-MT for 37580@debbugs.gnu.org; Sat, 05 Oct 2019 14:56:52 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]:44911) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iGpEV-0001jY-G7; Sat, 05 Oct 2019 14:56:47 -0400 Original-Received: from [176.228.60.248] (port=2822 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1iGpEU-0006Tq-5e; Sat, 05 Oct 2019 14:56:47 -0400 In-reply-to: <86lftz157z.fsf@gmail.com> (ynyaaa@gmail.com) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:168394 Archived-At: > From: ynyaaa@gmail.com > Cc: 37580@debbugs.gnu.org > Date: Sun, 06 Oct 2019 02:18:08 +0900 > > Sometimes I find broken utf-8 texts on the Internet. > Some characters are split into surrogate pairs, and each surrogate > character is encoded as if it is a normal BMP character. > > utf-8 coding system does not decode such sequences. > Changing multibyte-ness converts them to surrogate characters. > And encode-decode process with utf-16be outputs the intended characeters. > > Suppose the character is #x10000, > the correspoding pair is (#xD800 #xDC00). > The miss-encoded sequence is: > (encode-coding-string "\xD800\xDC00" 'utf-8) > => "\355\240\200\355\260\200" > > It is not decoded with utf-8. > (decode-coding-string (encode-coding-string "\xD800\xDC00" 'utf-8) > 'utf-8) > => "\355\240\200\355\260\200" > > Changing multibyte-ness, the sequence is converted into surrogate > characters. > (with-temp-buffer > (insert (encode-coding-string "\xD800\xDC00" 'utf-8)) > (set-buffer-multibyte nil) > (set-buffer-multibyte t) > (buffer-string)) > => "\xD800\xDC00" > > The surrogate pair can be converted into the original character. > (decode-coding-string (encode-coding-string "\xD800\xDC00" 'utf-16be) > 'utf-16be) > => "\x10000" So where's the problem in all this? AFAIU, you describe a sequence of actions that successfully recovers text in an obscure situation. I think the problem is that you enable undo. So in that case, just don't do that.