From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Stephen Berman via "Bug reports for GNU Emacs, the Swiss army knife of text editors" Newsgroups: gmane.emacs.bugs Subject: bug#69611: 30.0.50; Long bidi line with control characters freezes Emacs Date: Thu, 07 Mar 2024 18:52:20 +0100 Message-ID: <87h6hi6iaz.fsf@gmx.net> References: <87plw66tv6.fsf@gmx.net> <86v85y1217.fsf@gnu.org> Reply-To: Stephen Berman Mime-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="4742"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: 69611@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Mar 07 18:53:46 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1riHw9-00010j-GC for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 07 Mar 2024 18:53:45 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1riHvw-0007Ro-Oj; Thu, 07 Mar 2024 12:53:32 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1riHvu-0007RI-R3 for bug-gnu-emacs@gnu.org; Thu, 07 Mar 2024 12:53:30 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1riHvu-00079G-JM for bug-gnu-emacs@gnu.org; Thu, 07 Mar 2024 12:53:30 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1riHwP-0005zQ-Pr for bug-gnu-emacs@gnu.org; Thu, 07 Mar 2024 12:54:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Stephen Berman Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 07 Mar 2024 17:54:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69611 X-GNU-PR-Package: emacs Original-Received: via spool by 69611-submit@debbugs.gnu.org id=B69611.170983398322930 (code B ref 69611); Thu, 07 Mar 2024 17:54:01 +0000 Original-Received: (at 69611) by debbugs.gnu.org; 7 Mar 2024 17:53:03 +0000 Original-Received: from localhost ([127.0.0.1]:54841 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1riHvT-0005xm-3G for submit@debbugs.gnu.org; Thu, 07 Mar 2024 12:53:03 -0500 Original-Received: from mout.gmx.net ([212.227.15.18]:46579) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1riHvO-0005x2-BI for 69611@debbugs.gnu.org; Thu, 07 Mar 2024 12:53:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=s31663417; t=1709833941; x=1710438741; i=stephen.berman@gmx.net; bh=Ysx1ksWjQrgiqguMj1gBYcJsI8nK3f3ZzJSYRcJIYkg=; h=X-UI-Sender-Class:From:To:Cc:Subject:In-Reply-To:References: Date; b=ERZjA6NX4JLVBG1oGwpI1JIdoetL77SE9kWF9q/GWI6pK9nOkch7I5utrFISja27 7vWaUoDlnOVg2u3VUk5++f+ZPt6aPeIf6C4/lFj20ZBB1vZdZtXgovzTzZ8rTKDsd qmVmUpcbdOidUanHR1qc1tKJq9g6/yp5ZY5tE/snC3yG6z/g1oF8gDiCqhYEr0Psl C3QkVVVCRtPWUHy8OmiPIIJt09WziPQHt2OJb9cpZX9S92ljsNLJQnl5tLxzijyhA 63V+9kvJeGb/UiIoAsyiFONkv8alwFPm1WD30AxBpeYHo3VtkUQL4guy2pk2ihtCe aVN7NDZFcKmkF+MwBQ== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Original-Received: from strobelfs2 ([94.134.94.231]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N3bSj-1qiiz146zb-010fw1; Thu, 07 Mar 2024 18:52:21 +0100 In-Reply-To: <86v85y1217.fsf@gnu.org> (Eli Zaretskii's message of "Thu, 07 Mar 2024 17:42:44 +0200") X-Provags-ID: V03:K1:Ovq4HFP6PXQJFP3kITjR9p4wEV+cDmoyISBnKmQn+if0EkKeZ0F 5DVl66BviwnGPtYsEuHotzkaPOC5Jnv568S1JOsbdu37wCWFluvwCH0gJ80a9qycebRL3Aa 7egvOR90fwGFXxv/OC50HMYKfw/oCDWynAqWgveNT2+iFU/LRP+Jq8PpeI0a217PdiWpSnu tuGXCmyDFy8nGriK1VDPA== UI-OutboundReport: notjunk:1;M01:P0:uZRcCeWp990=;c52T2U77rd7GrGE4Jyyi+gnAO5v JMoTl3n81NhVpLLoqU953fc5B/E12BFPQOBiK5TS3W0A7fGYddOoYQiYIzheCXDA7D3Yojdwg +hDIkUCgK2JM+CyFViPJuTyPF0scNsdH5NXXpyyU6r1xLPC8cdYH1O95dMTCi+TDY2Z11+/OY 2VHbsXpVyXbYoF7riS/OsjuswdD+IXoJmOw7H8zOiTRa04XwuMCcGUHUhbnTL5+WPfKx1v+62 Dn1j6MYm9n8xAt80m8tkLvmlpcIstQlJwZ/Ti4FsZc0kI0SAfsomg1tJHIczxMLK8w9jPsKwq ibu01bWxlr7bXUJ7IShuNH3kbiJTpv7pXiJo+rzZstB/tRfPtfmiChsTZBLLjPY5GdzoKphN1 KWpLkr4JMvDEHAOPv2cBmTOtTHD+TH9ooq6Vo9u5VsTupPQFvfDbMkmSzG1vCx6d7myXc/mQ4 7XmgPW0Yn9BD9FDbRG0OwFmPrbMVAHjjhjvLaWNydYKEEfGWQAHpsd4PxhGx+zUSkZuEiOumw 12tqjgIKOQraAkWTGMprIfhqGCBld9xmGUYFvzxS5V7batIbg9aOJHOrPrh3jNG+zPBfVMccw bzDOFCPl0HExf6elbYKjtaOsqshsJs+lhuisoFfktzNX23RqT0O1Jsj9SnQUp5tGbcec3s54b HIL1OXkknpF3Th6nW3T4/r4NTT3DupX1TjPDP9FRMTb31X39ghf03JLsyi4NYj4EsyvXkPzwy EamMKwrZlDCi7ciD7MAzkLMFbcsEFeSZ+t9Ky9+1VcPlsd0UucaoAQXl3nftyMEhTCrCp7D/ X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:281183 Archived-At: On Thu, 07 Mar 2024 17:42:44 +0200 Eli Zaretskii wrote: >> Date: Thu, 07 Mar 2024 14:42:37 +0100 >> From: Stephen Berman via "Bug reports for GNU Emacs, >> the Swiss army knife of text editors" >> >> When I visited a certain elisp file generated by a program of mine and >> type `M-v', it took some time (see below for details) for the display t= o >> scroll to 4% from the top (according to the mode line) and then there >> was no further change and Emacs froze, using 100% of a CPU core. I >> found no way to unfreeze it within Emacs and after about 15 minutes >> terminated the emacs process from the shell. This is reliably >> reproducible with this file. >> >> The file in question is only about 50k bytes long, but it contains one >> line of more than 37k characters, consisting of a mix of ASCII and >> non-ASCII characters, including properly shaped Arabic script. The fil= e >> itself has base paragraph direction LTR. >> >> Most of the Arabic words in this file are enclosed in the bidirectional >> control characters POP DIRECTIONAL FORMATTING (#x202c) and RIGHT-TO-LEF= T >> EMBEDDING (#x202b). I did not add these characters, but I had >> copy-&-pasted most of the Arabic from a PDF file I did not create. I >> don't know if PDFs of Arabic text normally contain these control >> characters, but the consequences for Emacs were dramatic. When I simpl= y >> visited this file in Emacs (started with -Q) there was an immediate >> slowdown, and in top I could see Emacs using 100% of a CPU thread. I >> ran `M-: (benchmark-run nil (end-of-buffer))' on this file, and the >> result was: >> >> (27.962602113 2 0.0226042269999999977) > > This is a crazy file. UBA, the Unicode Bidirectional Algorithm, > allows the RLE..PDF embeddings to nest. The nesting is allowed to be > up to 125 deep(!), but I have never seen a text file using more than a > couple of nested embeddings. This file goes up to 111 nested > embedding levels! Moreover, quite a few embeddings are invalid: there > are 1021 RLE control characters in this file, but only 971 PDF > controls, so they don't pair as they should. This causes the > reordering algorithm to examine extremely long stretches of characters > each time we need to redisplay even a small portion of the window, > because reordering must always find where each nested level ends to do > its job. > > My suggestion is to remove all the RLE and PDF controls from the file. > They are not needed, not in Emacs anyway. I'm guessing the program > which created this file uses bidi controls because it wants to be > compatible with incomplete implementations of the UBA, which don't > support implicit embedding levels (those cause by bidirectional > properties of characters, as opposed to explicit bidi controls like > RLE and PDF). With full UBA implementations, the bidi controls are > needed only when the reordering using implicit levels produces wrong > results, which is quite rare. Indeed, I had already come to the conclusion that I don't need those controls before I decided to raise the problem I encountered with them. I've now checked a number of PDFs I have that contain Arabic script, and in all of those from which I was able to yank Arabic script from the PDF as Arabic script into Emacs (with some PDFs that wasn't possible), each Arabic word was enclosed in the control characters. So that appears to be standard or at least common with PDF. Being now aware of this, I can take care to remove any control characters from yanked text in future. In the case of the file I sent you, I may be to blame for the unbalanced control characters: after yanking the Arabic into Emacs, I did some editing of it and may well have unintentionally deleted some of the control characters. At the time I wasn't even aware of these; only after (re)reading the section on bidirectional display in the Elisp manual did I enable glyphless-display-mode and saw the characters, but I didn't bother to check if they paired up properly. >> The display of the benchmark result only appeared in the echo area afte= r >> more than a minute (I timed it with a stopwatch). At that point the >> mode line showed the buffer at 4% from the top, and the display remaine= d >> frozen afterwards. After several minutes during which Emacs consumed >> 100% CPU, and I had switched the focus away from the Emacs frame, the >> CPU consumption stopped, but as soon as I switch focus back to that >> frame, it went back to 100%. The display never changed from showing th= e >> buffer at 4%, apparently being in some kind of infinite loop. After >> about 15 minutes I started gdb, attached the Emacs process and produced >> a backtrace, which I've attached, in the hope it helps to diagnose the >> problem. > > The extremely deep nesting of embeddings in the file, coupled with the > fact that the first embedding starts near the beginning of the file, > but ends very near its end, causes the algorithm that finds where to > position the cursor to fail, because it cannot cope with the situation > where, after C-f or C-b, the position of point is very far outside of > the window. I guess this causes some infloop (even though I don't see > it here, I just see that the cursor doesn't move although point does > move). It could also be just a very long calculation, not an infloop, > because finding where to place the window-start point in this case is > also very expensive. Ok. But this is only an issue in conjunction with long lines, right? Because there is no slowdown or display issue with the file from which this elisp file was generated: that is the file into which I yanked the Arabic script from the PDF and subsequently edited, so it contains unpaired control characters, but only a few of its lines are longer than 80 characters, and I think none longer than 150 or so. >> Nevertheless, there seems to be something else besides the control >> characters involved in this issue, because as a further test, I created >> a buffer consisting of more than 1000 copies of the test string >> concatenating the Arabic example in etc/HELLO and "Hello" (see bug#6938= 5 >> for more on such test buffers), and manually enclosed each Arabic word >> in the above control characters, but the benchmark result in this buffe= r >> was not significantly different from the result without the control >> characters (and similar to the above result for the copy of the >> problematic file without the control characters), and the display did >> not freeze. > > Yes, because you never tried such deeply-nested embeddings, and didn't > make your embedding levels include so many characters long as this > file does. Indeed, I simply wrapped each Arabic word in the paired control characters, so there's no nesting at all. Now the difference makes sense. > This file is an interesting curiosity, as far as I'm concerned, but I > doubt whether I will find enough time and motivation to try to speed > up Emacs when such crazy files are visited. Given the special circumstances of this file's creation I think there's no need to spend any more time it, so unless you decide you do want to, as far as I'm concerned this bug can be closed. It might be beneficial to others to document the issue briefly, either in the Elisp manual under Bidirectional Display or just in etc/PROBLEMS, but maybe this is such an unusual case that even that isn't worth the effort. Thanks for looking into this and explaining it. Steve Berman