From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#56682: Fix the long lines font locking related slowdowns Date: Sun, 7 Aug 2022 01:58:06 +0300 Message-ID: <2c8d6755-cfe2-6559-3fde-3fa30ffb411e@yandex.ru> References: <8c7321f2f36494299e61@heytings.org> <8c7321f2f336523624e3@heytings.org> <83r1202meh.fsf@gnu.org> <6020ffaf-9069-0070-76cf-a13379ef01b5@yandex.ru> <83les71ilg.fsf@gnu.org> <06c5560d-3009-e5a5-3d33-fe5d2ec32d6b@yandex.ru> <74ddc877f17a06d8f120@heytings.org> <100d08b3-c25b-683f-6def-39052107ab59@yandex.ru> <83h72r16g1.fsf@gnu.org> <640c2e07-98e1-96d6-bb02-19f5f03f637f@yandex.ru> <834jyq29o1.fsf@gnu.org> <92da07bd028e3ede61a6@heytings.org> <47894c57-dd8b-5778-240a-3fa6540e4d37@yandex.ru> <92da07bd02941d5537e9@heytings.org> <5308e3b5-a160-17d7-77ee-b1d00acfa20d@yandex.ru> <92da07bd02a6cc861e1a@heytings.org> <837d3lzv8n.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="3087"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun Aug 07 00:59:45 2022 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oKSlk-0000e8-CZ for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 07 Aug 2022 00:59:44 +0200 Original-Received: from localhost ([::1]:38266 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oKSlj-0004YO-0h for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 06 Aug 2022 18:59:43 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:50484) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oKSl5-0003ww-ON for bug-gnu-emacs@gnu.org; Sat, 06 Aug 2022 18:59:06 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:45825) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oKSl5-00023h-GQ for bug-gnu-emacs@gnu.org; Sat, 06 Aug 2022 18:59:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oKSl5-00008A-7Y for bug-gnu-emacs@gnu.org; Sat, 06 Aug 2022 18:59:03 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 06 Aug 2022 22:59:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 56682 X-GNU-PR-Package: emacs Original-Received: via spool by 56682-submit@debbugs.gnu.org id=B56682.1659826699428 (code B ref 56682); Sat, 06 Aug 2022 22:59:03 +0000 Original-Received: (at 56682) by debbugs.gnu.org; 6 Aug 2022 22:58:19 +0000 Original-Received: from localhost ([127.0.0.1]:35569 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oKSkN-00006o-Ad for submit@debbugs.gnu.org; Sat, 06 Aug 2022 18:58:19 -0400 Original-Received: from mail-wr1-f41.google.com ([209.85.221.41]:37876) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oKSkJ-00006G-3r for 56682@debbugs.gnu.org; Sat, 06 Aug 2022 18:58:17 -0400 Original-Received: by mail-wr1-f41.google.com with SMTP id z17so6973774wrq.4 for <56682@debbugs.gnu.org>; Sat, 06 Aug 2022 15:58:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=Yzg6vyFflQEXTJ11F1nrcIb9105/63nxnZ91rmcBqFw=; b=Lof9QT24o75HlEG7i3YtfwlKpJRHjMURs9zbY+eb2my9zGfF4o5hqHVsyxmIGVxqlb WmF9fmL22HI8c/4Vy9t95MOj1WJJbGUa6jQRrfMtoGAmZaCTAPkSyb+fb6uDfaXPzsa7 AhcuWc3OhoSii9JVi0cnrJ5pZIDOUrA9Q508ynv8tD2KEFRoAQNIhC0XuMPeIsqVmOou I6h+jppgWklF0VUPrxw2C6/279tFbyvQtXTPW/I2OGRoHSBYX9bG1EEKBQ7PAgivUkJ+ nfoJgwLOZ2aOErTbGTuwPJi12yeEgL7Itj7ReavkSyc0lk/VBqPgMivZZz+M6o0sTHpB sIoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:message-id:date:mime-version:user-agent :subject:content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=Yzg6vyFflQEXTJ11F1nrcIb9105/63nxnZ91rmcBqFw=; b=ZiefUqaVbpHgRSZjVZCbiGVuaprqb8jtWy/5EThd2pvloYLcdBxyjv24CwVYc17e9Q oMVmrdCNQC60nhedNtM+L9u7qm78BQMPQpi2PhDCdPDcJe88zLdGyEumrED6wZNtHXOT GN+xOh6kqMsHEIDZKjXPa7ujwIbLaYBjc4ppkI2aPX2+Hcfo0NKhReP28egGEpyo3744 GGudwllKl1tJAhFvtI0pfLbeP2zsjAFmnxalFzPWlNuuZsG9+5V16IbN7nuK02VnjZK1 dCewIHpoJv8jeXD/uc8ly0TQeNbixgs+38r5Gd6OE/W8+0JCYrbOZdhyFdO9AStUPtdQ hmyQ== X-Gm-Message-State: ACgBeo3pVLxHhoG1zBJuKylvFQUX7nsvdUPRsQeWglHrd1STWGXdH9DH naGDo2aWWtxoRK4ibH57ROQ= X-Google-Smtp-Source: AA6agR7hshQVhkmjXgGtuwFD+3+JbPIxhCau9o/xjmvRxXnXfteOlUSxVMgD3h1y33Op2vtpevmTrg== X-Received: by 2002:a5d:638b:0:b0:220:6e1a:8794 with SMTP id p11-20020a5d638b000000b002206e1a8794mr7791716wru.193.1659826689178; Sat, 06 Aug 2022 15:58:09 -0700 (PDT) Original-Received: from [192.168.0.6] ([46.251.119.176]) by smtp.googlemail.com with ESMTPSA id bi17-20020a05600c3d9100b003a2d6f26babsm8857163wmb.3.2022.08.06.15.58.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 06 Aug 2022 15:58:08 -0700 (PDT) Content-Language: en-US In-Reply-To: <837d3lzv8n.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:239004 Archived-At: On 06.08.2022 10:28, Eli Zaretskii wrote: >> Date: Sat, 6 Aug 2022 01:38:05 +0300 >> Cc: 56682@debbugs.gnu.org, Eli Zaretskii , >> Stefan Monnier >> From: Dmitry Gutov >> >>> So what you prefer IIUC would be to call fontification-functions with a >>> locked narrowing to 1 MB if point is before that threshold, and to not >>> call fontification-functions at all after that threshold?  That might be >>> another doable approach. >> >> If we have to support huge files with max responsiveness, then that >> would be my preference, yes. >> >> I don't see the point of using a "locked" narrowing for this, though. >> Maybe not even a narrowing at all: just avoid calling >> fontification-functions with START > value_of(large_file_fontification_max). > > I don't understand why this would be a good idea. First, if we are > able to fontify the first 1MB of a file, why won't we be able to > fontify the 2nd, 3rd, ... Nth 1MB in a similarly fast fashion? Fontifying a window-ful of buffer near the end of 2N will take ~2x as long as doing that near the end of 1N, 3N - ~3x as long, etc. > Do you > assume that fontification must go to BOB, but will never go forward > towards EOB? That's my experience, yes. Language parsers either don't need to look forward at all, or only need to look forward a little. And especially since we regularly have to deal with unfinished code (and code is most often unfinished from the tail end, not from the front), our syntax-propertize and font-lock matchers have to account for that. > Why would we make such an assumption, especially with > long lines, where EOB could also be EOL? That's where syntax-wholeline-max comes into play: it won't extend the current region past additional 10000 chars. Even if that fails to work somehow, we could code up font-lock to apply narrowing past a certain point, to definitely limit the possible overhead. But I'd rather we not do that without a concrete problem case. > Next, why assume that your personal preferences in this case will be > shared by everyone else? I, for one, will find it very strange to > have font-lock suddenly disappear at an arbitrary point. Is it more strange than syntax highlighting starting to go crazy around the same arbitrary point? I'm assuming that people will quickly understand that very big files are treated in a special way, and I seem to recall that certain other editors use a similar approach (switch off some features past a certain file length). > If we don't > want to make the assumption it will be good for everyone, your > proposal should be one optional behavior, but not the only one. I'm happy for this to be customizable, and for everyone to be able to choose the tradeoffs they want. For that to be possible, this choice need to happen in Lisp. But I made my arguments for what seems like a good default behavior, both reasonable-looking and fast in (hopefully) all significant cases. The important part is for this to be customizable separately from long-line-threshold because these are different issues, and the slowdowns happen around different amounts of text. > And finally, there's no way to restrict fontification-functions from > accessing the buffer beyond the first 1MB, even if we want to, without > something like the locked narrowing. font-lock rules really shouldn't call 'widen', we've talked about this before (in mmm-mode threads), then went through the built-in major modes which did so, and fixed that. The sole (big) exception is CC Mode, but its internals aren't compatible with the current forced narrowing either, and Gregory said it's not important in the context of long lines or large files anyway. So as long as major modes behave, we won't need the locked narrowing in this area. And hopefully won't need narrowing at all. >> > emacs -Q >> > M-: (setq long-line-threshold nil syntax-wholeline-max most-positive- >> > fixnum) RET >> > C-x C-f dictionary.json RET y ;; takes 160 seconds >> > C-e ;; takes 200 seconds >> > >> > emacs -Q >> > M-: (setq long-line-threshold nil) RET >> > C-x C-f dictionary.json RET y ;; immediate >> > C-e ;; not finished after 1200 seconds (20 minutes), I killed Emacs >> >> I get what you're saying, but the approach does seem "right" to me >> because it works, letting me view and edit dictionary.json with good >> performance and behavior. > > On a Core i9 machine and with an optimized build, I presume? And with > "good" being your subjective judgment that places more importance on > fontification than on responsiveness? That's not necessarily a good > data point on which to base our decisions regarding the default > behavior. On a slower machine, you'll just need to find a correspondingly smaller file to see the same combination: the line is long enough for redisplay to stutter with (setq long-line-threshold nil), but the file is not big enough for font-lock to really become the bottleneck. Even with my "big and beefy" i9, I've been seeing redisplay slowdowns with lines containing 1000s of chars. Very noticeable around 40K. Whereas font-lock can remain fast with files up to 100x that size. And even then remain bearable so long as the user doesn't jump around too much, editing the file at BOB then going to EOB, time and time again. >> Either way, I believe the change is at the right level of abstraction, >> and if it has bugs, they should be solvable without major redesign. > > Alas, Mr. ShouldBe is not available for this project, and probably > won't be any time soon. No need to mock my request for a proper reproduction scenario.