bug#38407: 27.0.50; infinite loop with display of large file without newlines

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Pieter van Oostrum]

1) Load a large file, that contains no newlines. I used a 100KB json
file from my Firefox profile (but loaded as a text file)
2) set visual-line-mode aka Word wrap
3) scroll up (<next> scroll-up-command)

The scrolling is very slow, and the further in the file you come, the
slower it becomes. During that time, Emacs consumes close to 100% CPU.
At a certain moment it took me half an our for one scroll operation.
When I went to the end of the buffer (M-> end-of-buffer), Emacs looped
virtually forever. I decided to kill it after 12 hours.

In another experiment, I repeatedly did a scroll-up-command, followed by
the shell command 'date >> display.log'. I repeated this with a keyboard
macro.
The written times indicate that in the beginning the time for a
scroll-up goes up linearly until more than a minute, and then Emacs get
stuck.




In GNU Emacs 27.0.50 (build 1, i686-apple-darwin10.0.0, NS appkit-1561.61 Version 10.13.6 (Build 17G9016))
 of 2019-11-27 built on cochabamba.vanoostrum.org
Repository revision: 4eb7db5d4b84708912c63a77569c8adeeff6c640
Repository branch: master
Windowing system distributor 'Apple', version 10.3.1561
System Description:  Mac OS X 10.13.6

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
user-error: Beginning of history; no preceding item
 [Beginning of history; no preceding item]
Visual-Line mode enabled

Configured using:
 'configure --build i686-apple-darwin10.0.0 --without-dbus --with-ns
 build_alias=i686-apple-darwin10.0.0 'CFLAGS=-pipe -march=nocona'
 PKG_CONFIG_PATH=/usr/X11R6/pkgconfig/:/usr/local/lib/pkgconfig/:/usr/lib/pkgconfig/'

Configured features:
NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES
THREADS PDUMPER LCMS2

Important settings:
  value of $LC_CTYPE: UTF-8
  value of $LANG: en_NL.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  visual-line-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/ns-win ns-win ucs-normalize mule-util term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
tab-bar menu-bar rfn-eshadow isearch timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core term/tty-colors frame
minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite charscript charprop case-table epa-hook jka-cmpr-hook help
simple abbrev obarray cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
threads kqueue cocoa ns lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 47303 8427)
 (symbols 48 6678 1)
 (strings 32 16380 1383)
 (string-bytes 1 518147)
 (vectors 16 10228)
 (vector-slots 8 141557 8968)
 (floats 8 20 39)
 (intervals 56 193 0)
 (buffers 1000 13))

-- 
Pieter van Oostrum <pieter@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Phil Sainty]

See whether M-x global-so-long-mode makes things more tolerable?

Extremely long lines are a problem for Emacs, but so-long.el can
help significantly in some situations.

If you enable that, and then visit your file, it should switch to
so-long-mode.  It won't prevent the redisplay getting relatively
slower the deeper you navigate into a very long line, but it should
make navigating in the early stages of the file much faster, and
help to some extent throughout (but I would still expect displaying
the end of the line to be painful).

M-x so-long-commentary has all of the details.


-Phil

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Phil Sainty]

On 2019-11-28 10:52, Pieter van Oostrum wrote:
> I used a 100KB json file from my Firefox profile
> (but loaded as a text file)

I missed that bit, so I should add that global-so-long-mode doesn't
care about text-mode by default.  It mostly targets programming modes
(.json filenames included), but you can customize so-long-target-modes
to include text-mode if you think you're likely to encounter such
files in text-mode in practice.

However I wouldn't expect a 100K file in text-mode to cause anything
remotely like the degree of performance issue you've encountered.  I
have a 256K single-line .txt file which suffers perceivable slow-downs
at the end of the file (in text-mode with visual-line-mode enabled),
but nothing remotely so catastrophic as you're reporting.  On my
machine, I'd estimate less than a half-second delay when scrolling,
so the difference is several orders of magnitude.

Are you seeing that starting from "emacs -Q" ?


-Phil

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Phil Sainty]

On 2019-11-28 10:52, Pieter van Oostrum wrote:
> I used a 100KB json file from my Firefox profile

Out of interest I went looking for large JSON files in my own Firefox
profiles, and the largest one I could find was an "extensions.json"
which was also roughly 100K.  Curiously enough, everything was indeed
dramatically slower than normal in that buffer, even in so-long-mode.

I also observed that many different human languages were present in
the text, so I thought that bi-directional text processing might be a
factor -- and indeed setting the debug option bidi-display-reordering
to nil in that buffer made a *very* dramatic improvement to
performance.

Similarly, performance was good after using M-x find-file-literally
to visit the file.

I am guessing that this is also what is happening for you?


-Phil

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Pieter van Oostrum]

Phil Sainty wrote:

 > On 2019-11-28 10:52, Pieter van Oostrum wrote:
 > > I used a 100KB json file from my Firefox profile
 > 
 > Out of interest I went looking for large JSON files in my own Firefox
 > profiles, and the largest one I could find was an "extensions.json"
 > which was also roughly 100K.  Curiously enough, everything was indeed
 > dramatically slower than normal in that buffer, even in so-long-mode.
 > 
 > I also observed that many different human languages were present in
 > the text, so I thought that bi-directional text processing might be a
 > factor -- and indeed setting the debug option bidi-display-reordering
 > to nil in that buffer made a *very* dramatic improvement to
 > performance.
 > 
 > Similarly, performance was good after using M-x find-file-literally
 > to visit the file.
 > 
 > I am guessing that this is also what is happening for you?

It was exactly that file that caused the problem. I copied it to a .txt file to see if the problem might have been caused by the Javascript mode, and it didn't make any difference.

Normally I wouldn't open such a bizarre file, but I was doing an 'rgrep' on my home directory, and this was one of the files that came up in the grep results. When I was scrolling though the *grep* buffer, I got this problem. Then I decided to just open the file, to eliminate the possibility that grep-mode was one of the causes.

And yes, I checked it all with emacs -Q.

I forgot to mention that when I set debug-on-quit, a Quit indicated Emacs was in redisplay_internal.

-- 
Pieter van Oostrum <pieter@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Date: Thu, 28 Nov 2019 07:51:33 +0100
> From: Pieter van Oostrum <pieter@vanoostrum.org>
> Cc: 38407@debbugs.gnu.org
> 
> Normally I wouldn't open such a bizarre file, but I was doing an 'rgrep' on my home directory, and this was one of the files that came up in the grep results. When I was scrolling though the *grep* buffer, I got this problem. Then I decided to just open the file, to eliminate the possibility that grep-mode was one of the causes.
> 
> And yes, I checked it all with emacs -Q.

As an aside, using Text mode for such files, and visual-line-mode on
top of that, is exactly the opposite of what one should do to make
redisplay performance better.  Text mode resets
bidi-paragraph-direction to nil, which makes redisplay work harder
because it needs to find the beginning of the current paragraph each
time redisplay starts.  And visual-line-mode makes redisplay a bit
slower, because of the need to find a suitable point to break the
line.

So I suggest to use the default mode for visiting JSON files.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Pieter van Oostrum]

Eli Zaretskii wrote:

 > > Date: Thu, 28 Nov 2019 07:51:33 +0100
 > > From: Pieter van Oostrum <pieter@vanoostrum.org>
 > > Cc: 38407@debbugs.gnu.org
 > > 
 > > Normally I wouldn't open such a bizarre file, but I was doing an 'rgrep' on my home directory, and this was one of the files that came up in the grep results. When I was scrolling though the *grep* buffer, I got this problem. Then I decided to just open the file, to eliminate the possibility that grep-mode was one of the causes.
 > > 
 > > And yes, I checked it all with emacs -Q.
 > 
 > As an aside, using Text mode for such files, and visual-line-mode on
 > top of that, is exactly the opposite of what one should do to make
 > redisplay performance better.  Text mode resets
 > bidi-paragraph-direction to nil, which makes redisplay work harder
 > because it needs to find the beginning of the current paragraph each
 > time redisplay starts.  And visual-line-mode makes redisplay a bit
 > slower, because of the need to find a suitable point to break the
 > line.
 > 
 > So I suggest to use the default mode for visiting JSON files.

I did open it as .json, and also enabled global-so-long-mode, but still no joy.
The real culprit is visual-line-mode, because without it, it can scroll through the file, albeit slowly. But it does finish, whereas with visual-line-mode it keeps swallowing CPU-time, at least for hours.
The really bad thing is that it can't be interrupted. ^G does not really stop it. So when you encounter this problem the only way out is to externally abort Emacs, thereby risking to lose your work. And all the hassle to restart and set up what you were doing.

As I encounter this problem in real life with rgrep, I can't always avoid it. I am doing a massive cleanup in my home directory, and with rgrep I will occasionally encounter these nasty files. It happened a couple of times in the last weeks.

In Emacs 26, it is also slow, but the redisplay comes to an end, so I see this as a kind of regression. The timing with Emacs 26 is very similar to that in Emacs 27, except that it doesn't hang at the end. So my impression is that the real difference is when the scrolling reaches the end of the file. That also happens with M-> (end-of-buffer), which hangs indefinitely (or at least a long time that is virtually indistinguishable from indefinitely) in Emacs 27, but it does finish in Emacs 26.
-- 
Pieter van Oostrum <pieter@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Date: Thu, 28 Nov 2019 19:30:50 +0100
> From: Pieter van Oostrum <pieter@vanoostrum.org>
> Cc: psainty@orcon.net.nz,
>     38407@debbugs.gnu.org
> 
> In Emacs 26, it is also slow, but the redisplay comes to an end, so I see this as a kind of regression. The timing with Emacs 26 is very similar to that in Emacs 27, except that it doesn't hang at the end. So my impression is that the real difference is when the scrolling reaches the end of the file. That also happens with M-> (end-of-buffer), which hangs indefinitely (or at least a long time that is virtually indistinguishable from indefinitely) in Emacs 27, but it does finish in Emacs 26.

I don't think there's any difference in this aspect between Emacs 26
and 27.  I see the same problem in both, FWIW, and I definitely didn't
wait for hours for redisplay to finish, let alone saw it "hang
indefinitely", even in an unoptimized build of Emacs 27 on a very slow
machine.  In my case it took something like 15 minutes.  I don't
understand why you see such long times.  Of course, I tried this with
a different JSON file than you did, so maybe there's something in your
file that causes even more slowdown.

In any case, try the latest master after setting bidi-inhibit-bpa to
non-nil.  In my testing, this is as fast as Emacs 23, i.e. no bidi
reordering at all, give or take some 10%.

And if you want the fastest display possible, use Fundamental mode,
not Text.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Pieter van Oostrum]

Eli Zaretskii wrote:

 > > Date: Thu, 28 Nov 2019 19:30:50 +0100
 > > From: Pieter van Oostrum <pieter@vanoostrum.org>
 > > Cc: psainty@orcon.net.nz,
 > >     38407@debbugs.gnu.org
 > > 
 > > In Emacs 26, it is also slow, but the redisplay comes to an end, so I see this as a kind of regression. The timing with Emacs 26 is very similar to that in Emacs 27, except that it doesn't hang at the end. So my impression is that the real difference is when the scrolling reaches the end of the file. That also happens with M-> (end-of-buffer), which hangs indefinitely (or at least a long time that is virtually indistinguishable from indefinitely) in Emacs 27, but it does finish in Emacs 26.
 > 
 > I don't think there's any difference in this aspect between Emacs 26
 > and 27.  I see the same problem in both, FWIW, and I definitely didn't
 > wait for hours for redisplay to finish, let alone saw it "hang
 > indefinitely", even in an unoptimized build of Emacs 27 on a very slow
 > machine.  In my case it took something like 15 minutes.  I don't
 > understand why you see such long times.  Of course, I tried this with
 > a different JSON file than you did, so maybe there's something in your
 > file that causes even more slowdown.
 > 
 > In any case, try the latest master after setting bidi-inhibit-bpa to
 > non-nil.  In my testing, this is as fast as Emacs 23, i.e. no bidi
 > reordering at all, give or take some 10%.
 > 
 > And if you want the fastest display possible, use Fundamental mode,
 > not Text.
 > 
 > 
I have tried the latest version with so-long-mode, bidi-inhibit-bpa set to t, and the json file. Still hanging for a long time. Maybe it makes a difference that I am on MacOS?
-- 
Pieter van Oostrum <pieter@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Date: Thu, 28 Nov 2019 23:09:57 +0100
> From: Pieter van Oostrum <pieter@vanoostrum.org>
> Cc: psainty@orcon.net.nz,
>     38407@debbugs.gnu.org
> 
> I have tried the latest version with so-long-mode, bidi-inhibit-bpa
> set to t, and the json file. Still hanging for a long time.

You mean, you see no difference regardless of setting
bidi-inhibit-bpa?  Not even without so-long-mode in the picture?

> Maybe it makes a difference that I am on MacOS?

Unlikely.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Phil Sainty]

I've just pulled from master and recompiled, and in my own tests
setting bidi-inhibit-bpa to t improves performance greatly.

On my machine, with the particular extensions.json file I'm using:

* In js-mode, scrolling at the end of the file incurs a ~6s delay.

* In so-long-mode (without bidi-inhibit-bpa), the same thing incurs a
  ~5s delay -- not much of an improvement.

* After setting bidi-inhibit-bpa to t in so-long-variable-overrides
  and revisiting the file, the same thing incurs only a ~1s delay.

* Using M-: (setq-local bidi-inhibit-bpa t) in json-mode (i.e. without
  using so-long) likewise makes things a great deal faster (but the
  combination with so-long is better still).

Scrolling at the top of the file in so-long-mode with the new option
set is more or less instantaneous.  Without the new option that is
also quite slow (if not to the same extent as the end of the file).

In short, this enhancement seems very effective to me (thank you Eli),
so I will push the addition to so-long.el shortly.


-Phil

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Cc: Pieter van Oostrum <pieter@vanoostrum.org>, 38407@debbugs.gnu.org
> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Sat, 30 Nov 2019 00:48:22 +1300
> 
> In short, this enhancement seems very effective to me (thank you Eli),
> so I will push the addition to so-long.el shortly.

Thanks.  But note that setting bidi-inhibit-bpa non-nil will help
significantly only for files that are likely to have a lot of bracket
characters, especially if they are nested.  So maybe do that in
so-long.el only for modes where this is likely to happen, if not by
actually counting such characters.  E.g., I wouldn't recommend doing
that in Text mode, since there it might cause incorrect display
without any real benefits.

Characters that I call "bracket" for this purpose are listed in
admin/unidata/BidiBrackets.txt (a file we import from the UCD).
Significantly, the "<>" pair is not there, so files with lots of those
particular brackets will not be affected by this issue.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Dmitry Gutov]

On 29.11.2019 13:48, Phil Sainty wrote:
> * Using M-: (setq-local bidi-inhibit-bpa t) in json-mode (i.e. without
>    using so-long) likewise makes things a great deal faster (but the
>    combination with so-long is better still)

Interesting!

Should we consider setting this value in `prog-mode`, then? For all its 
derivatives.

And in compilation-mode, where it would make the most difference for me 
personally.

I think in all of these cases we're unlikely to see bidirectional text.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Cc: Pieter van Oostrum <pieter@vanoostrum.org>, 38407@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 29 Nov 2019 15:48:14 +0200
> 
> Should we consider setting this value in `prog-mode`, then? For all its 
> derivatives.

No, because programming-language modes include strings and comments,
where the correct rendering is important.

This variable should not be set non-nil unless the display engine is
abysmally slow.  In all other cases doing so is giving up correct
display while gaining nothing, because in "sensible" buffers the BPA
implementation is fast enough.

> I think in all of these cases we're unlikely to see bidirectional text.

I don't think "unlikely" is a good enough reason to do what you
suggest by default.  Emacs by default should produce correct display
of all scripts it supports, even in unlikely use cases.

Please don't make me regret I added this variable (or the bidi display
in general).

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Dmitry Gutov]

On 29.11.2019 16:26, Eli Zaretskii wrote:
> No, because programming-language modes include strings and comments,
> where the correct rendering is important.

What about just compilation modes, then?

This is where I have routinely seen display slowdown because of long lines.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Cc: psainty@orcon.net.nz, pieter@vanoostrum.org, 38407@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 29 Nov 2019 16:31:44 +0200
> 
> On 29.11.2019 16:26, Eli Zaretskii wrote:
> > No, because programming-language modes include strings and comments,
> > where the correct rendering is important.
> 
> What about just compilation modes, then?
> 
> This is where I have routinely seen display slowdown because of long lines.

Are there a lot of parentheses/braces there? are they nested?

This variable will only have any visible effect when both of the
following conditions are true:

  . a physical line is very long
  . the line includes many parentheses and/or braces, especially if
    they are nested

If these conditions are not true, the variable will have no tangible
effect on redisplay performance.

Compilation buffers show errors and warnings, and this can easily
include strings.  For example, a code line that calls 'message' in
Lisp or 'printf' in C usually includes a string.  We don't want those
strings to be displayed incorrectly, unless displaying them correctly
takes a preposterous amount of time, at least not by default.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Dmitry Gutov]

On 29.11.2019 17:03, Eli Zaretskii wrote:
> Are there a lot of parentheses/braces there? are they nested?

Almost never.

> This variable will only have any visible effect when both of the
> following conditions are true:
> 
>    . a physical line is very long
>    . the line includes many parentheses and/or braces, especially if
>      they are nested
> 
> If these conditions are not true, the variable will have no tangible
> effect on redisplay performance.

Guess I got over-excited, sorry. If I ever find performance benefits in 
compilation buffers with this variable on, will report back.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Cc: psainty@orcon.net.nz, pieter@vanoostrum.org, 38407@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 29 Nov 2019 18:53:16 +0200
> 
> On 29.11.2019 17:03, Eli Zaretskii wrote:
> > Are there a lot of parentheses/braces there? are they nested?
> 
> Almost never.

That's what I thought.

> Guess I got over-excited, sorry. If I ever find performance benefits in 
> compilation buffers with this variable on, will report back.

Thanks, that would help finding opportunities for redisplay
optimizations.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Pieter van Oostrum]

Yes, I mean no difference. I started it late in the evening, and the next morning it was still running. I haven’t tried without so-long-mode. I am a few days from home without my laptop, so I can’t do any more testing at the moment.

-- 
Pieter van Oostrum

Op 29 nov. 2019 om 08:12 heeft Eli Zaretskii <eliz@gnu.org> het volgende geschreven:

>> Date: Thu, 28 Nov 2019 23:09:57 +0100
>> From: Pieter van Oostrum <pieter@vanoostrum.org>
>> Cc: psainty@orcon.net.nz,
>>    38407@debbugs.gnu.org
>> 
>> I have tried the latest version with so-long-mode, bidi-inhibit-bpa
>> set to t, and the json file. Still hanging for a long time.
> 
> You mean, you see no difference regardless of setting
> bidi-inhibit-bpa?  Not even without so-long-mode in the picture?
> 
>> Maybe it makes a difference that I am on MacOS?
> 
> Unlikely.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> From: Pieter van Oostrum <pieter@vanoostrum.org>
> Date: Fri, 29 Nov 2019 18:24:07 +0100
> Cc: psainty@orcon.net.nz, 38407@debbugs.gnu.org
> 
> Yes, I mean no difference. I started it late in the evening, and the next morning it was still running. 

Then I think we must see that file, there's probably some other factor
at work here.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Pieter van Oostrum]

Op 29 nov. 2019 om 20:30 heeft Eli Zaretskii <eliz@gnu.org> het volgende geschreven:

>> From: Pieter van Oostrum <pieter@vanoostrum.org>
>> Date: Fri, 29 Nov 2019 18:24:07 +0100
>> Cc: psainty@orcon.net.nz, 38407@debbugs.gnu.org
>> 
>> Yes, I mean no difference. I started it late in the evening, and the next morning it was still running. 
> 
> Then I think we must see that file, there's probably some other factor
> at work here.

I’ll send it when I’m back home.

-- 
Pieter van Oostrum

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Pieter van Oostrum]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 458 bytes --]

Eli Zaretskii wrote:

 > > From: Pieter van Oostrum <pieter@vanoostrum.org>
 > > Date: Fri, 29 Nov 2019 18:24:07 +0100
 > > Cc: psainty@orcon.net.nz, 38407@debbugs.gnu.org
 > > 
 > > Yes, I mean no difference. I started it late in the evening, and the next morning it was still running. 
 > 
 > Then I think we must see that file, there's probably some other factor
 > at work here.
 
Here it is, compressed. I am loading the uncompressed file, of course.
 

[-- Attachment #2: extensions.json.gz --]
[-- Type: application/x-gzip, Size: 18675 bytes --]

[-- Attachment #3: .signature --]
[-- Type: text/plain, Size: 112 bytes --]


-- 
Pieter van Oostrum <pieter@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Phil Sainty]

On 1/12/19 8:23 PM, Pieter van Oostrum wrote:
> Here it is, compressed. I am loading the uncompressed file,
> of course.

Without bidi-inhibit-bpa, visual-line-mode is certainly brutal
with this file (I've been waiting several minutes for Emacs 27
to respond to `end-of-buffer').

However, for me, bidi-inhibit-bpa comprehensively deals to that.

Is it possible that you didn't recompile or reinstall Emacs in
the way you thought you had, after pulling Eli's changes?

Can you confirm that C-h v bidi-inhibit-bpa actually shows you
documentation about the variable?  If it doesn't, then setting
it will have no effect whatsoever.


-Phil


p.s. By default it should be telling you this:

bidi-inhibit-bpa is a variable defined in ‘C source code’.
Its value is nil

  Calls these functions when changed: (#<subr set-buffer-redisplay>)

Documentation:
Non-nil means inhibit the Bidirectional Parentheses Algorithm.
Disabling the BPA makes redisplay faster, but might produce incorrect
display reordering of bidirectional text with embedded parentheses and
other bracket characters whose ’paired-bracket’ Unicode property is
non-nil, see ‘get-char-code-property’.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Pieter van Oostrum]

Phil Sainty wrote:

 > On 1/12/19 8:23 PM, Pieter van Oostrum wrote:
 > > Here it is, compressed. I am loading the uncompressed file,
 > > of course.
 > 
 > Without bidi-inhibit-bpa, visual-line-mode is certainly brutal
 > with this file (I've been waiting several minutes for Emacs 27
 > to respond to `end-of-buffer').
 > 
 > However, for me, bidi-inhibit-bpa comprehensively deals to that.
 > 
 > Is it possible that you didn't recompile or reinstall Emacs in
 > the way you thought you had, after pulling Eli's changes?
 > 
 > Can you confirm that C-h v bidi-inhibit-bpa actually shows you
 > documentation about the variable?  If it doesn't, then setting
 > it will have no effect whatsoever.

This is what it says.

bidi-inhibit-bpa is a variable defined in ‘C source code’.
Its value is nil

  Calls these functions when changed: (#<subr set-buffer-redisplay>)

Documentation:
Non-nil means inhibit the Bidirectional Parentheses Algorithm.
Disabling the BPA makes redisplay faster, but might produce incorrect
display reordering of bidirectional text with embedded parentheses and
other bracket characters whose ’paired-bracket’ Unicode property is
non-nil, see ‘get-char-code-property’.

So yes, it is the right one.

Latest experiment:

Opened this version of Emacs. 
(setq bidi-inhibit-bpa t)
open the file extensions.json (i.e. without visual-line-mode)
go to the end of the buffer.
Emacs gets responsive again after about one minute.
Then I open a new frame (Cmd-N), the frame appears, so this proves that Emacs was responsive.
But after opening the frame, Emacs is unresponsive again with 100% CPU usage. After 30 minutes, I killed it.

-- 
Pieter van Oostrum <pieter@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Pieter van Oostrum]

Pieter van Oostrum wrote:

 > Phil Sainty wrote:
 > 
 >  > On 1/12/19 8:23 PM, Pieter van Oostrum wrote:
 >  > > Here it is, compressed. I am loading the uncompressed file,
 >  > > of course.
 >  > 
 >  > Without bidi-inhibit-bpa, visual-line-mode is certainly brutal
 >  > with this file (I've been waiting several minutes for Emacs 27
 >  > to respond to `end-of-buffer').
 >  > 
 >  > However, for me, bidi-inhibit-bpa comprehensively deals to that.
 >  > 
 >  > Is it possible that you didn't recompile or reinstall Emacs in
 >  > the way you thought you had, after pulling Eli's changes?
 >  > 
 >  > Can you confirm that C-h v bidi-inhibit-bpa actually shows you
 >  > documentation about the variable?  If it doesn't, then setting
 >  > it will have no effect whatsoever.
 > 
 > This is what it says.
 > 
 > bidi-inhibit-bpa is a variable defined in ‘C source code’.
 > Its value is nil
 > 
 >   Calls these functions when changed: (#<subr set-buffer-redisplay>)
 > 
 > Documentation:
 > Non-nil means inhibit the Bidirectional Parentheses Algorithm.
 > Disabling the BPA makes redisplay faster, but might produce incorrect
 > display reordering of bidirectional text with embedded parentheses and
 > other bracket characters whose ’paired-bracket’ Unicode property is
 > non-nil, see ‘get-char-code-property’.
 > 
 > So yes, it is the right one.
 > 
 > Latest experiment:
 > 
 > Opened this version of Emacs. 
 > (setq bidi-inhibit-bpa t)
 > open the file extensions.json (i.e. without visual-line-mode)
 > go to the end of the buffer.
 > Emacs gets responsive again after about one minute.
 > Then I open a new frame (Cmd-N), the frame appears, so this proves that Emacs was responsive.
 > But after opening the frame, Emacs is unresponsive again with 100% CPU usage. After 30 minutes, I killed it.

That was without so-long-mode. I retried with so-long-mode, and now everything works as it should. Even in grep-mode when the culprit file has a hit, it appears in the grep buffer and I can scroll over it. As soon as I set bidi-inhibit-bpa  to nil again, it fails.

I saw that bidi-inhibit-bpa is now part of so-long-mode, so I am compiling the latest master now.
Thanks everybody for all the help.
-- 
Pieter van Oostrum <pieter@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Date: Sun, 1 Dec 2019 17:35:35 +0100
> From: Pieter van Oostrum <pieter@vanoostrum.org>
> Cc: 38407@debbugs.gnu.org
> 
> Opened this version of Emacs. 
> (setq bidi-inhibit-bpa t)
> open the file extensions.json (i.e. without visual-line-mode)
> go to the end of the buffer.
> Emacs gets responsive again after about one minute.
> Then I open a new frame (Cmd-N), the frame appears, so this proves that Emacs was responsive.
> But after opening the frame, Emacs is unresponsive again with 100% CPU usage. After 30 minutes, I killed it.

When more than one frame is displayed, some redisplay cycles redraw
more than just the selected frame.  If you don't want the frame with
the "problematic" buffer ti get in the way, minimize it.

Although I don't understand why you see 30 minutes and longer delays,
you should see the same 1 minute as on the original frame.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, 38407@debbugs.gnu.org
> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Sun, 1 Dec 2019 23:37:46 +1300
> 
> On 1/12/19 8:23 PM, Pieter van Oostrum wrote:
> > Here it is, compressed. I am loading the uncompressed file,
> > of course.
> 
> Without bidi-inhibit-bpa, visual-line-mode is certainly brutal
> with this file (I've been waiting several minutes for Emacs 27
> to respond to `end-of-buffer').

8.5 minutes here, with an unoptimized build.

> However, for me, bidi-inhibit-bpa comprehensively deals to that.

Down to 1 min here.

One other observation is that the first time you do that in a session
after visiting the file, it takes much longer than the subsequent
times.  Here, the second M-> takes 5 min with bidi-inhibit-bpa nil and
just 17 sec with it non-nil.

So I think some of this time is spent doing something we do once,
maybe loading fonts or something (this file has a lot of different
scripts).

It would be interesting to run perf and compare its output with
bidi-inhibit-bpa nil and non-nil.  Maybe the performance hit due to
visual-line-mode is because of some code that could be optimized, or
maybe (gasp!) even a bug.  The difference between the first and
subsequent moves is also interesting to time.

I don't have access to a GUI system where I can run perf, so could
someone please do these measurements and post the resulting profiles?

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 4927 bytes --]

>>>>> On Sun, 01 Dec 2019 19:45:11 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> However, for me, bidi-inhibit-bpa comprehensively deals to that.

    Eli> Down to 1 min here.

    Eli> One other observation is that the first time you do that in a session
    Eli> after visiting the file, it takes much longer than the subsequent
    Eli> times.  Here, the second M-> takes 5 min with bidi-inhibit-bpa nil and
    Eli> just 17 sec with it non-nil.

    Eli> So I think some of this time is spent doing something we do once,
    Eli> maybe loading fonts or something (this file has a lot of different
    Eli> scripts).

I donʼt see any real difference between the first and second M-> with
bidi-inhibit-bpa t, so I didnʼt benchmark that, but I can if you want.

    Eli> It would be interesting to run perf and compare its output with
    Eli> bidi-inhibit-bpa nil and non-nil.  Maybe the performance hit due to
    Eli> visual-line-mode is because of some code that could be optimized, or
    Eli> maybe (gasp!) even a bug.  The difference between the first and
    Eli> subsequent moves is also interesting to time.

    Eli> I don't have access to a GUI system where I can run perf, so could
    Eli> someone please do these measurements and post the resulting profiles?

Iʼve done two runs, with visual-line-mode enabled, and bidi-inhibit-bpa
t/nil. Gzipped text versions of 'perf report' attached.

(Aside: why does char_table_ref check ASCII_CHAR_P again, when
CHAR_TABLE_REF already does that?)

bidi-inhibit-bpa t:

# Overhead  Command       Shared Object                  Symbol
# ........  ............  .............................  ............................................
#
    11.21%  emacs         emacs                          [.] composition_compute_stop_pos
     6.11%  emacs         emacs                          [.] CHAR_TABLE_REF
     5.65%  emacs         emacs                          [.] mark_object
     4.17%  emacs         emacs                          [.] re_match_2_internal
     2.85%  emacs         emacs                          [.] lookup_char_property
     1.82%  emacs         emacs                          [.] next_interval
     1.74%  emacs         emacs                          [.] Fassq
     1.65%  emacs         emacs                          [.] compute_stop_pos
     1.63%  emacs         emacs                          [.] find_interval
     1.19%  emacs         libc-2.29.so                   [.] _IO_getc
     1.15%  emacs         libc-2.29.so                   [.] __memmove_avx_unaligned_erms
     1.12%  emacs         emacs                          [.] exec_byte_code
     1.02%  emacs         emacs                          [.] move_it_in_display_line_to
     1.02%  emacs         emacs                          [.] sub_char_table_ref
     0.92%  emacs         ld-2.29.so                     [.] do_lookup_x
     0.90%  emacs         emacs                          [.] buf_bytepos_to_charpos

bidi-inhibit-bpa nil:

# Overhead  Command       Shared Object                  Symbol
# ........  ............  .............................  ..........................................................
#
    96.59%  emacs         libc-2.29.so                   [.] __memmove_avx_unaligned_erms
     0.22%  emacs         emacs                          [.] mark_object
     0.17%  emacs         emacs                          [.] composition_compute_stop_pos
     0.12%  emacs         emacs                          [.] re_match_2_internal
     0.12%  emacs         emacs                          [.] bidi_resolve_explicit
     0.09%  emacs         emacs                          [.] CHAR_TABLE_REF
     0.07%  emacs         emacs                          [.] lookup_char_property
     0.06%  emacs         emacs                          [.] bidi_fetch_char
     0.05%  emacs         emacs                          [.] find_interval
     0.05%  emacs         libc-2.29.so                   [.] _IO_getc
     0.04%  emacs         emacs                          [.] move_it_in_display_line_to
     0.04%  emacs         emacs                          [.] bidi_cache_search.constprop.0
     0.04%  emacs         emacs                          [.] exec_byte_code
     0.04%  emacs         emacs                          [.] next_interval
     0.04%  emacs         emacs                          [.] mark_char_table
     0.04%  emacs         emacs                          [.] pdumper_marked_p_implq
     0.04%  emacs         ld-2.29.so                     [.] do_lookup_x
     0.03%  emacs         libc-2.29.so                   [.] __strchr_avx2
     0.03%  emacs         emacs                          [.] rpl_re_search_2
     0.03%  emacs         emacs                          [.] sub_char_table_ref
     0.03%  emacs         emacs                          [.] bidi_cache_iterator_state


[-- Attachment #2: bidi-inhibit-bpa t --]
[-- Type: application/octet-stream, Size: 15335 bytes --]

[-- Attachment #3: bidi-inhibiit-bpa nil --]
[-- Type: application/octet-stream, Size: 14727 bytes --]

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Robert Pluim]

>>>>> On Mon, 02 Dec 2019 11:27:10 +0100, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Sun, 01 Dec 2019 19:45:11 +0200, Eli Zaretskii <eliz@gnu.org> said:
    Robert> bidi-inhibit-bpa nil:

    Robert> # Overhead  Command       Shared Object                  Symbol
    Robert> # ........  ............  .............................  ..........................................................
    Robert> #
    Robert>     96.59%  emacs         libc-2.29.so                   [.] __memmove_avx_unaligned_erms
    Robert>      0.22%  emacs         emacs                          [.] mark_object
    Robert>      0.17%  emacs         emacs                          [.] composition_compute_stop_pos
    Robert>      0.12%  emacs         emacs                          [.] re_match_2_internal

So further messing with perf leads me to the following code in
move_it_in_display_line_to:

	{
	  if (it->line_wrap == WORD_WRAP && it->area == TEXT_AREA)
	    {
	      if (IT_DISPLAYING_WHITESPACE (it))
		may_wrap = true;
	      else if (may_wrap)
		{
		  /* We have reached a glyph that follows one or more
		     whitespace characters.  If the position is
		     already found, we are done.  */
		  if (atpos_it.sp >= 0)
		    {
		      RESTORE_IT (it, &atpos_it, atpos_data);
		      result = MOVE_POS_MATCH_OR_ZV;
		      goto done;
		    }
		  if (atx_it.sp >= 0)
		    {
		      RESTORE_IT (it, &atx_it, atx_data);
		      result = MOVE_X_REACHED;
		      goto done;
		    }
		  /* Otherwise, we can wrap here.  */
		  SAVE_IT (wrap_it, *it, wrap_data);          <=====
		  may_wrap = false;
		}
	    }
	}

That SAVE_IT is what causes the __memmove_avx_unaligned_erms to show
up so high in the trace (via bidi_shelve_cache).

Robert

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Phil Sainty <psainty@orcon.net.nz>,  pieter@vanoostrum.org,
>   38407@debbugs.gnu.org
> Date: Tue, 03 Dec 2019 12:20:26 +0100
> 
> >>>>> On Mon, 02 Dec 2019 11:27:10 +0100, Robert Pluim <rpluim@gmail.com> said:
> 		  /* Otherwise, we can wrap here.  */
> 		  SAVE_IT (wrap_it, *it, wrap_data);          <=====
> 		  may_wrap = false;
> 		}
> 	    }
> 	}
> 
> That SAVE_IT is what causes the __memmove_avx_unaligned_erms to show
> up so high in the trace (via bidi_shelve_cache).

Yes, I was wondering about that myself.  But we need more details to
understand better what, if anything, can be done about this.

First, which part of SAVE_IT causes this?  I'm guessing it's this
part:

  #define SAVE_IT(ITCOPY, ITORIG, CACHE)	\
    do {					\
      if (CACHE)				\
	bidi_unshelve_cache (CACHE, true);	\
      ITCOPY = ITORIG;				\
      CACHE = bidi_shelve_cache ();		\  <<<<<<<<<<<<
    } while (false)

If that is true, then I think the offending part of bidi_shelve_cache
is this:

  alloc = (bidi_shelve_header_size
	   + bidi_cache_idx * sizeof (struct bidi_it));
  databuf = xmalloc (alloc);
  bidi_cache_total_alloc += alloc;

  memcpy (databuf, &bidi_cache_idx, sizeof (bidi_cache_idx));
  memcpy (databuf + sizeof (bidi_cache_idx),                      <<<<<<<
	  bidi_cache, bidi_cache_idx * sizeof (struct bidi_it));  <<<<<<<
  memcpy (databuf + sizeof (bidi_cache_idx)
	  + bidi_cache_idx * sizeof (struct bidi_it),
	  bidi_cache_start_stack, sizeof (bidi_cache_start_stack));

And if this guess is also true, then I think the problem is that
databuf + sizeof (bidi_cache_idx) is unaligned on 64-bit systems,
since bidi_cache_idx is an int.

Could you verify the above guesses?

Thanks.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Robert Pluim]

>>>>> On Tue, 03 Dec 2019 18:05:42 +0200, Eli Zaretskii <eliz@gnu.org> said:

    Eli> Yes, I was wondering about that myself.  But we need more details to
    Eli> understand better what, if anything, can be done about this.

    Eli> First, which part of SAVE_IT causes this?  I'm guessing it's this
    Eli> part:

    Eli>   #define SAVE_IT(ITCOPY, ITORIG, CACHE)	\
    Eli>     do {					\
    Eli>       if (CACHE)				\
    Eli> 	bidi_unshelve_cache (CACHE, true);	\
    Eli>       ITCOPY = ITORIG;				\
    Eli>       CACHE = bidi_shelve_cache ();		\  <<<<<<<<<<<<
    Eli>     } while (false)

Yes, itʼs bidi_shelve_cache

    Eli> If that is true, then I think the offending part of bidi_shelve_cache
    Eli> is this:

    Eli>   alloc = (bidi_shelve_header_size
    Eli> 	   + bidi_cache_idx * sizeof (struct bidi_it));
    Eli>   databuf = xmalloc (alloc);
    Eli>   bidi_cache_total_alloc += alloc;

    Eli>   memcpy (databuf, &bidi_cache_idx, sizeof (bidi_cache_idx));
    Eli>   memcpy (databuf + sizeof (bidi_cache_idx),                      <<<<<<<
    Eli> 	  bidi_cache, bidi_cache_idx * sizeof (struct bidi_it));  <<<<<<<
    Eli>   memcpy (databuf + sizeof (bidi_cache_idx)
    Eli> 	  + bidi_cache_idx * sizeof (struct bidi_it),
    Eli> 	  bidi_cache_start_stack, sizeof (bidi_cache_start_stack));

    Eli> And if this guess is also true, then I think the problem is that
    Eli> databuf + sizeof (bidi_cache_idx) is unaligned on 64-bit systems,
    Eli> since bidi_cache_idx is an int.

The '_unaligned_' bit of that memmove function name does not mean
thatʼs itʼs doing unoptimized unaligned copies: it means it accepts
unaligned pointers, and aligns them as necessary to enable fast
copying. Anyway, I made bidi_cache_idx an intptr_t, and it made no
difference.

I think that misalignment is vastly dwarfed by the following:

    (gdb) l bidi_shelve_cache
    973	{
    974	  unsigned char *databuf;
    975	  ptrdiff_t alloc;
    976
    977	  /* Empty cache.  */
    978	  if (bidi_cache_idx == 0)
    979	    return NULL;
    980
    981	  alloc = (bidi_shelve_header_size
    982		   + bidi_cache_idx * sizeof (struct bidi_it));
    (gdb) b 980
    Breakpoint 3 at 0x4b66aa: file bidi.c, line 981.
    (gdb) commands
    Type commands for breakpoint(s) 3, one per line.
    End with a line saying just "end".
    >p bidi_cache_idx
    >p bidi_cache_idx * sizeof(struct bidi_it)
    >end

    Thread 1 "emacs" hit Breakpoint 3, bidi_shelve_cache () at bidi.c:981
    981	  alloc = (bidi_shelve_header_size
    $25 = 30860
    $26 = 71842080

which means Emacs is copying 70MB of data every time bidi_shelve_cache
is called, and itʼs called *a lot* in this scenario. Could we not do
this shelving by pointer-swapping or similar rather than copying?

Robert

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: psainty@orcon.net.nz,  pieter@vanoostrum.org,  38407@debbugs.gnu.org
> Date: Wed, 04 Dec 2019 10:15:55 +0100
> 
>     Eli> First, which part of SAVE_IT causes this?  I'm guessing it's this
>     Eli> part:
> 
>     Eli>   #define SAVE_IT(ITCOPY, ITORIG, CACHE)	\
>     Eli>     do {					\
>     Eli>       if (CACHE)				\
>     Eli> 	bidi_unshelve_cache (CACHE, true);	\
>     Eli>       ITCOPY = ITORIG;				\
>     Eli>       CACHE = bidi_shelve_cache ();		\  <<<<<<<<<<<<
>     Eli>     } while (false)
> 
> Yes, itʼs bidi_shelve_cache

Thanks for verifying.

>     Eli> And if this guess is also true, then I think the problem is that
>     Eli> databuf + sizeof (bidi_cache_idx) is unaligned on 64-bit systems,
>     Eli> since bidi_cache_idx is an int.
> 
> The '_unaligned_' bit of that memmove function name does not mean
> thatʼs itʼs doing unoptimized unaligned copies: it means it accepts
> unaligned pointers, and aligns them as necessary to enable fast
> copying. Anyway, I made bidi_cache_idx an intptr_t, and it made no
> difference.

OK, thanks.

>     Thread 1 "emacs" hit Breakpoint 3, bidi_shelve_cache () at bidi.c:981
>     981	  alloc = (bidi_shelve_header_size
>     $25 = 30860
>     $26 = 71842080
> 
> which means Emacs is copying 70MB of data every time bidi_shelve_cache
> is called, and itʼs called *a lot* in this scenario. Could we not do
> this shelving by pointer-swapping or similar rather than copying?

Not sure I understand what kind of pointer-swapping you had in mind.
We don't swap between 2 buffers here, we save away a snapshot of the
iterator state each time we see a character where a line break can be
made, so that we could restore that state when we exhaust the window's
width.  We must restore the iterator state to continue to the next
visual line, and the bidi cache is an integral part of that state.

We could perhaps lower the cache size limit (see
BIDI_CACHE_MAX_ELTS_PER_SLOT in bidi.c), which would then
proportionally decrease the time for making a copy of the cache.  Or
we could make some non-trivial changes in the logic of
move_it_in_display_line_to (and similar changes in display_line) to
detect when the cache becomes too large, and use a backup algorithm
that doesn't copy it.  But I question the utility of such changes:
they will never get us a speedup like bidi-inhibit-bpa does, and for
the relatively rare use case like this one (extremely long lines in a
JSON file, with some bracketed parts containing R2L text, and the user
activating visual-line-mode on top of that) inhibiting the BPA,
whether via so-long or by the user or some other Lisp, sounds like an
okay solution to me.  If the JSON file has long lines, but no R2L
text, we already have an optimization in bidi.c to avoid having a
large cache; and if visual-line-mode is off, the cache doesn't need to
be copied so frequently.  So only the combination of the two causes
this tremendous slowdown, and bidi-inhibit-bpa solves it better than
any alternative.  WDYT?

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Robert Pluim]

>>>>> On Wed, 04 Dec 2019 17:45:40 +0200, Eli Zaretskii <eliz@gnu.org> said:
    >> Thread 1 "emacs" hit Breakpoint 3, bidi_shelve_cache () at bidi.c:981
    >> 981	  alloc = (bidi_shelve_header_size
    >> $25 = 30860
    >> $26 = 71842080
    >> 
    >> which means Emacs is copying 70MB of data every time bidi_shelve_cache
    >> is called, and itʼs called *a lot* in this scenario. Could we not do
    >> this shelving by pointer-swapping or similar rather than copying?

    Eli> Not sure I understand what kind of pointer-swapping you had in mind.
    Eli> We don't swap between 2 buffers here, we save away a snapshot of the
    Eli> iterator state each time we see a character where a line break can be
    Eli> made, so that we could restore that state when we exhaust the window's
    Eli> width.  We must restore the iterator state to continue to the next
    Eli> visual line, and the bidi cache is an integral part of that state.

Yes, I hadn't realized you needed to keep a copy of the data.

    Eli> We could perhaps lower the cache size limit (see
    Eli> BIDI_CACHE_MAX_ELTS_PER_SLOT in bidi.c), which would then
    Eli> proportionally decrease the time for making a copy of the
    Eli> cache.

I tried various values of that down to 5000, it improved matters by a
factor of 8 or so, but the resulting Emacs was still pretty laggy
(with visual-line-mode enabled).

    Eli> Or
    Eli> we could make some non-trivial changes in the logic of
    Eli> move_it_in_display_line_to (and similar changes in display_line) to
    Eli> detect when the cache becomes too large, and use a backup algorithm
    Eli> that doesn't copy it.  But I question the utility of such changes:
    Eli> they will never get us a speedup like bidi-inhibit-bpa does, and for
    Eli> the relatively rare use case like this one (extremely long lines in a
    Eli> JSON file, with some bracketed parts containing R2L text, and the user
    Eli> activating visual-line-mode on top of that) inhibiting the BPA,
    Eli> whether via so-long or by the user or some other Lisp, sounds like an
    Eli> okay solution to me.  If the JSON file has long lines, but no R2L
    Eli> text, we already have an optimization in bidi.c to avoid having a
    Eli> large cache; and if visual-line-mode is off, the cache doesn't need to
    Eli> be copied so frequently.  So only the combination of the two causes
    Eli> this tremendous slowdown, and bidi-inhibit-bpa solves it better than
    Eli> any alternative.  WDYT?

That sounds reasonable. I see global-so-long-mode is off by default,
do we want to enable it by default? It seems a dis-service to users to
make them have to find out about it by themselves.

Robert

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: psainty@orcon.net.nz,  pieter@vanoostrum.org,  38407@debbugs.gnu.org
> Date: Thu, 05 Dec 2019 08:27:00 +0100
> 
> I see global-so-long-mode is off by default, do we want to enable it
> by default? It seems a dis-service to users to make them have to
> find out about it by themselves.

If we want something like that to be enabled by default, it "needs
work", IMO.  For starters, the default line length beyond which it
kicks in, 250 characters, is waaay too low.  IME, problems start with
much longer lines, perhaps more than 10000 characters.  It would be
too radical, IMO, to start disabling important features when no
tangible slowdown happens yet.

Also, I think we should try to detect the special cases like the one
which triggered this bug report, because there the slowdown is
absolutely unbearable.  So maybe we should lower the default of
so-long-threshold when we detect an opening bracket character at BOB
(a clear sign of JSON or similar format), and maybe lower it yet more
if visual-line-mode is also enabled.

And finally, we could have a related global minor mode which just
_suggests_ turning on so-long in a buffer when it detects these
situations, but doesn't actually turn it on automatically.  Such a
global mode could be turned on by default much easier, and perhaps
could also use somewhat lower thresholds (but not 250).

WDYT?

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Phil Sainty]

On 6/12/19 4:01 AM, Eli Zaretskii wrote:
> If we want [so-long] to be enabled by default, it "needs work", IMO.
> For starters, the default line length beyond which it kicks in, 250
> characters, is waaay too low.  IME, problems start with much longer
> lines, perhaps more than 10000 characters.

Agreed.  There is a whole discussion pending (I intended to raise it
soon), asking people to test the library more widely, and to help to
figure out what the default settings should actually be.  The current
settings are almost entirely based on modes and features I have tried
personally, after all.

That 250 value is, as you note, incredibly small.  As with many things
in so-long it's something of a heuristic and trade-off.  The reasoning
was that not all files with extremely long lines are just one line --
newline chars may still occur in the text for reasons other than the
formatting of the file -- and so the question was: "given the context
of a buffer in some prog-mode derivative, what length is "long enough"
to indicate that we're not looking at a normal file, but something
which probably contains minified code?

In conjunction is the "how many lines do we test" variable.  It would
be nice to arrive at a decision without doing a lot of work, so we
don't want to inspect a really large number of lines; but we also
don't necessarily want to *require* that we've seen a line which will
definitely cause problems -- a shorter line which strongly suggests
that we're looking at minified code/data may be sufficient.

With the current conservative (*very* conservative) values we very
quickly reach a decision, but we're more likely to trigger in false-
positive situations (this has happened to me, but only very rarely;
and by default `so-long-revert' is a C-c C-c away).  With larger
values we may more effort to reach a decision.  With *much* larger
values, that extra effort might be noticeable (on slower systems
even if not on faster ones), so I think we want to be a bit careful
of that.

(Not that I've been testing large values for performance -- I've been
leaving that for the later discussion, and in the meantime I've been
content to have the small default values, in part because it seemed
more likely that people would discover potential problems with the
library if it was a bit more trigger-happy.)

As with most things in so-long, there's no single right solution;
but I'm keen to arrive at a set of defaults that people can agree
are a reasonable starting point for most users; and I'm certainly
not suggesting that I've done that already ("what should the default
settings be?" has been an open question from the outset, really.)


-Phil

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Cc: pieter@vanoostrum.org, 38407@debbugs.gnu.org
> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Fri, 6 Dec 2019 09:38:05 +1300
> 
> That 250 value is, as you note, incredibly small.

It is okay for a feature that is off by default.  I'm saying that
turning this on by default with such a low threshold would be wrong.

> As with many things in so-long it's something of a heuristic and
> trade-off.  The reasoning was that not all files with extremely long
> lines are just one line -- newline chars may still occur in the text
> for reasons other than the formatting of the file -- and so the
> question was: "given the context of a buffer in some prog-mode
> derivative, what length is "long enough" to indicate that we're not
> looking at a normal file, but something which probably contains
> minified code?

Did you try a different heuristic for detecting long lines, one that
isn't based on searching some portion of the buffer?  E.g., what about
going to a couple of random places and looking at the value returned
by current-column?

In general, searching for a newline is very fast in Emacs, so if you
use the right primitives, you should be able to scan quite a lot of
text before the time begins affecting responsiveness.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Phil Sainty]

On 6/12/19 9:04 PM, Eli Zaretskii wrote:
> Did you try a different heuristic for detecting long lines, one that
> isn't based on searching some portion of the buffer?  E.g., what
> about going to a couple of random places and looking at the value
> returned by current-column?

No, the library has essentially used the current approach from the
beginning.  However one of the changes I made for the new version was
to allow users to set a custom function for detecting long lines via
the `so-long-predicate' option, so it's easy to plug in alternatives.

I haven't planned to implement any alternatives myself, but if other
people do so I'd be interested to see how they compare in practice.


> In general, searching for a newline is very fast in Emacs, so if you
> use the right primitives, you should be able to scan quite a lot of
> text before the time begins affecting responsiveness.

`so-long-detected-long-line-p' is the provided predicate, and it
relies on `forward-line' (which I've assumed is as quick as I'm going
to get for finding newlines), along with narrowing to limit the amount
of text scanned to the maximum value that we care about (i.e. the
minimum length of a too-long line).

I've just done a bit of testing with much larger files and values, and
suspect that we could indeed make these values quite dramatically
bigger without any noticeable impact.

Files with lots of lines, yet no excessively-long ones, will ensure
that so-long scans the (customizable) maximum number of lines that
it's going to try before giving up.  I expect that will be a useful
test case for testing responsiveness on slow systems.


-Phil

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Cc: rpluim@gmail.com, pieter@vanoostrum.org, 38407@debbugs.gnu.org
> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Sat, 7 Dec 2019 14:28:37 +1300
> 
> > In general, searching for a newline is very fast in Emacs, so if you
> > use the right primitives, you should be able to scan quite a lot of
> > text before the time begins affecting responsiveness.
> 
> `so-long-detected-long-line-p' is the provided predicate, and it
> relies on `forward-line' (which I've assumed is as quick as I'm going
> to get for finding newlines), along with narrowing to limit the amount
> of text scanned to the maximum value that we care about (i.e. the
> minimum length of a too-long line).
> 
> I've just done a bit of testing with much larger files and values, and
> suspect that we could indeed make these values quite dramatically
> bigger without any noticeable impact.

Yes, I'd expect that.

Thanks.

bug#38407: 27.0.50; infinite loop with display of large file without newlines

[Eli Zaretskii]

> Date: Thu, 28 Nov 2019 14:22:07 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: 38407@debbugs.gnu.org
> 
> On 2019-11-28 10:52, Pieter van Oostrum wrote:
> > I used a 100KB json file from my Firefox profile
> 
> Out of interest I went looking for large JSON files in my own Firefox
> profiles, and the largest one I could find was an "extensions.json"
> which was also roughly 100K.  Curiously enough, everything was indeed
> dramatically slower than normal in that buffer, even in so-long-mode.
> 
> I also observed that many different human languages were present in
> the text, so I thought that bi-directional text processing might be a
> factor -- and indeed setting the debug option bidi-display-reordering
> to nil in that buffer made a *very* dramatic improvement to
> performance.

AFAICT, it isn't the bidi reordering in general, it's the BPA part of
the reordering -- that file has many embedded bracket characters.

I've added a new variable bidi-inhibit-bpa.  Try setting it non-nil,
and I think you will see performance very similar to disabling the
reordering entirely.  I guess that's another variable for so-long-mode
to futz with...

> Similarly, performance was good after using M-x find-file-literally
> to visit the file.

It makes very little sense to visit a text file with many non-ASCII
characters using find-file-literally, IMO.