From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#66760: 29.1; [BUG] GB18030 Incorrect Encoding Date: Sat, 04 Nov 2023 10:25:07 +0200 Message-ID: <83h6m253j0.fsf@gnu.org> References: <1015f5fcf69b9c0656d42932da193bd4@sics.ac.cn> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="11198"; mail-complaints-to="usenet@ciao.gmane.io" Cc: 66760@debbugs.gnu.org, yuruijie@sics.ac.cn To: Andreas Schwab Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Nov 04 09:26:39 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qzBzL-0002iL-Es for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 04 Nov 2023 09:26:39 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qzBzA-0005UG-Bc; Sat, 04 Nov 2023 04:26:28 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qzBz8-0005U7-M8 for bug-gnu-emacs@gnu.org; Sat, 04 Nov 2023 04:26:26 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qzBz8-0004NM-DZ for bug-gnu-emacs@gnu.org; Sat, 04 Nov 2023 04:26:26 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qzBzi-0007aS-E0 for bug-gnu-emacs@gnu.org; Sat, 04 Nov 2023 04:27:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 04 Nov 2023 08:27:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66760 X-GNU-PR-Package: emacs Original-Received: via spool by 66760-submit@debbugs.gnu.org id=B66760.169908636729083 (code B ref 66760); Sat, 04 Nov 2023 08:27:02 +0000 Original-Received: (at 66760) by debbugs.gnu.org; 4 Nov 2023 08:26:07 +0000 Original-Received: from localhost ([127.0.0.1]:60482 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qzByp-0007Z0-5l for submit@debbugs.gnu.org; Sat, 04 Nov 2023 04:26:07 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:34464) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qzByn-0007YX-Lc for 66760@debbugs.gnu.org; Sat, 04 Nov 2023 04:26:06 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qzBy6-0004BJ-FL; Sat, 04 Nov 2023 04:25:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=u54BpqatCQ96bNDs+ytnTslkub2Oo3YfvaUzI/y7BQU=; b=XKSo5x++ZXlh hoKijPy2DuJZoV6F1B6WjCVQsVa/RX9+ocRPzn3ifnomLl/u0ePsSLc5uKXZBTm1BmSIVXqv4wpvn 20pNDZ3v3k2GMcupGSfzI0MGM0sHjvgLYIUMdHEORxBMLPRso88fTZo7rcG5TlK5WamfMSllwUhP3 Gn9FJgLQ8knoUeJMXiWOHcU3GLE+PFtSY9CqCwJ4RgbiJnc4xaQL+ci0WxgBYD/UBDm6+xuyQfne5 Y7VjVbMcJbHLmEGV1aEWbqv8NveCAdPSPWaUmvakCE5GFv2/qa40eu0V4Spl+fk63aWiAT4Uz930g tutQklOAgkUfVhwGuB60sA==; In-Reply-To: (message from Andreas Schwab on Thu, 26 Oct 2023 16:20:59 +0200) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:273752 Archived-At: > Cc: 66760@debbugs.gnu.org > From: Andreas Schwab > Date: Thu, 26 Oct 2023 16:20:59 +0200 > > On Okt 26 2023, Ruijie Yu wrote: > > > I have noticed that in GB18030 encoding, certain ranges of characters > > have incorrect encodings. > > > > One example is U+217A (SMALL ROMAN NUMERAL ELEVEN). The expected > > encoding is 81 36 C5 30 (as can be seen from the GB18030 standard [1] > > and verified from other programs such as iconv and MySQL), whereas the > > observed encoding within Emacs is 81 36 C4 39, with a 1-codepoint > > offset. > > This is a bug in the generation of GB180304.map. The gb180303.awk > script assumes that the 4-byte encodings of GB18030 are filling the > holes in sequence of characters with a 2-byte encoding by Unicode > codepoint order, but there are some places where codepoints from the PUA > area are inserted into the sequence. For example, U+1E3E maps to 81 35 > F4 36, the next codepoint not mapped to a 2-byte code is U+1E40, but > that maps to 81 35 F4 38, whereas 81 35 F4 37 is the encoding of U+E7C7. > So the output gets out of sync. Thanks. I don't think I understand the issue well enough, so patches are welcome to fix this problem in the Awk script.