From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#63731: [PATCH] Support Emoji Variation Sequence 16 (FE0F) where appropriate Date: Thu, 01 Jun 2023 15:43:26 +0300 Message-ID: <83edmvcpzl.fsf@gnu.org> References: <87a5xrzsph.fsf@stebalien.com> <87mt1rmjjg.fsf@gmail.com> <83v8gfjnzj.fsf@gnu.org> <87edn3mbr3.fsf@gmail.com> <83mt1rjg69.fsf@gnu.org> <875y8fm7x7.fsf@gmail.com> <83lehbjdjd.fsf@gnu.org> <87wn0vkqn1.fsf@gmail.com> <83jzwvj94x.fsf@gnu.org> <87h6rw8y82.fsf@gmail.com> <83353gipww.fsf@gnu.org> <87edmzto0l.fsf@gmail.com> <83ilcbgrxo.fsf@gnu.org> <87zg5nb3m3.fsf@gmail.com> <83bki3gpai.fsf@gnu.org> <87sfbfazfp.fsf@gmail.com> <837csrgioe.fsf@gnu.org> <87jzwqb7r3.fsf@gmail.com> <83ttvuf29m.fsf@gnu.org> <87a5xmaqul.fsf@gmail.com> <83r0qxg4q0.fsf@gnu.org> <875y88bhvr.fsf@gmail.com> <83ilc8eapd.fsf@gnu.org> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="16749"; mail-complaints-to="usenet@ciao.gmane.io" Cc: 63731@debbugs.gnu.org, steven@stebalien.com To: rpluim@gmail.com Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Jun 01 14:43:14 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q4he6-0004Ag-J7 for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 01 Jun 2023 14:43:14 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q4hdw-0003Vr-Ix; Thu, 01 Jun 2023 08:43:04 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4hdu-0003VJ-OC for bug-gnu-emacs@gnu.org; Thu, 01 Jun 2023 08:43:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1q4hdu-0004J8-GS for bug-gnu-emacs@gnu.org; Thu, 01 Jun 2023 08:43:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1q4hdu-0003IP-C5 for bug-gnu-emacs@gnu.org; Thu, 01 Jun 2023 08:43:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 01 Jun 2023 12:43:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 63731 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 63731-submit@debbugs.gnu.org id=B63731.168562336912650 (code B ref 63731); Thu, 01 Jun 2023 12:43:02 +0000 Original-Received: (at 63731) by debbugs.gnu.org; 1 Jun 2023 12:42:49 +0000 Original-Received: from localhost ([127.0.0.1]:37224 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4hdg-0003Hv-Kc for submit@debbugs.gnu.org; Thu, 01 Jun 2023 08:42:49 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:52552) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q4hde-0003Hh-FQ for 63731@debbugs.gnu.org; Thu, 01 Jun 2023 08:42:47 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4hdY-0004GV-N1; Thu, 01 Jun 2023 08:42:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=x/ic3vCjty09id55d3pbUEvWHwVBQnRWzDr0NXp41JM=; b=ZbhP85P6WKFW 8BDRKiH7epElgJ0O3tgDo1axeFjuBFKlEx7cPt6f5EwpYcOlCD3QsANM8ieHJosPVbXcMVVT2HEuQ fJ8R5nqeG9jc184ATYTFjMzS7VMhK8b9j4whFMp5ZH3KWak4dFimMg0z1AOoj04VOv2WQbCq9e2w9 MxbWxp2H0Cj98Vp93sh53nGctkWurv6ENoH3g94XxxrBmMVOAALHZJ0h7cMbJaggj1kQPbbsqU7Il zz/mJMBKAYMim0f/kZi7u1cOlDqbFD/H/R52WR2ffNZrVYk84AHwUOcBeF4PlIoHXlGAdH/iEjJ8+ Vncnf+gOBCOV4FDxNqajtQ==; Original-Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4hdY-0002d3-7T; Thu, 01 Jun 2023 08:42:40 -0400 In-Reply-To: <83ilc8eapd.fsf@gnu.org> (message from Eli Zaretskii on Wed, 31 May 2023 19:18:22 +0300) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:262701 Archived-At: > Cc: 63731@debbugs.gnu.org, steven@stebalien.com > Date: Wed, 31 May 2023 19:18:22 +0300 > From: Eli Zaretskii > > > From: Robert Pluim > > Cc: 63731@debbugs.gnu.org, steven@stebalien.com > > Date: Wed, 31 May 2023 18:11:36 +0200 > > > > Eli> So there are two issues here: (a) why there's no composition in the > > Eli> first case, and (b) why does "C-u C-x =" says there is when there > > Eli> isn't. > > > > OK. I can poke around in gdb if you give me some idea of what I should > > be looking at. > > I don't really know. I plan to just step through the code in > composite.c tomorrow, unless you beat me to it. Once we understand > issue (a), I think we will also understand issue (b). OK, the issue is quite clear even without stepping with a debugger. Bottom line: we cannot support a situation where the same character can be composed by more than one slot in composition-function-table. If there are more than a single slot for the same character, one of them will be tried, and the rest will be ignored (not even tried). In particular, if a character CH has a "forward" composition rule that starts with itself, and also has a "backward" rule (one with non-zero look-back parameter) triggered by a different character (which should follow CH), the latter rule will never be tried. This is what happens in this case: the character #x1F44D has several rules that start with itself in emoji-zwj.el: (#x1F44D . ,(eval-when-compile (regexp-opt '( "\N{U+1F44D}\N{U+1F3FB}" "\N{U+1F44D}\N{U+1F3FC}" "\N{U+1F44D}\N{U+1F3FD}" "\N{U+1F44D}\N{U+1F3FE}" "\N{U+1F44D}\N{U+1F3FF}" )))) and it also has a "backward" rule: (set-char-table-range composition-function-table #xFE0F '(["\\c.\ufe0f" 1 font-shape-gstring])) The latter is triggered by #xFE0F and has a 1-character look-back, which will match #x1F44D, since its category is '.' (it's a "base character"). This latter rule is never tried. Why? because the former rules, anchored at #X1F44D, are tried first (Emacs redisplay examines characters in the order of their buffer positions), and fail to match. When those rules fail to match, due to how the composition-related functions called by the display engine are factored, we never again consider compositions triggered by a later character which "cover" also #x1F44D: once that position was examined and the attempted composition failed, we move to the next character. IOW, we assume that this first set of composition rules we find for a given character are the only ones that could possibly be relevant for that character. Which means that to have #xFE0F compose correctly with Emoji codepoints, we should include #xFE0F in the sequences in emoji-zwj.el. The reason why "C-u C-x =" lies to us saying there's a composition where really there isn't is because descr-text.el uses the find-composition primitive, whose implementation is parallel and separate from that of the display-engine routines, and is structured differently. So find-composition does succeed to detect the second rule, the one triggered by #xFE0F, which the display engine ignores. I will think whether this can be fixed, to avoid such false positives, but if we accept that there can be only one set of composition rules for a character, then we basically invoked undefined behavior here, and we got what we deserved. Thanks.