bug#56682: Fix the long lines font locking related slowdowns

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

New bug number and thread to discuss the next chapter of the long lines 
slowdowns.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 21 Jul 2022 18:00:37 +0000
> From: Gregory Heytings <gregory@heytings.org>
> 
> New bug number and thread to discuss the next chapter of the long lines 
> slowdowns.

Thanks.

FTR, I repeat here the recipe from
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56393#425:

  emacs -Q
  C-x C-f long-line.xml RET

Now, do NOT disable font-lock, and wait for Emacs to say "Valid" in
the mode line (to get nXML mode out of the way).  Then:

  M-x toggle-truncate-lines RET

Now simple cursor motion commands that use redisplay optimizations are
fast, but commands that cause more thorough redisplay are as slow as
on master.  As a simple example, try just "M-x" and wait until the
"M-x" prompt appears in the minibuffer -- here it takes much longer,
basically as long as the version on master.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

The first step of the font locking related improvements was easier than I 
thought.

Two following commands:

1. C-x C-f dictionary.json RET y

2. C-e

on my laptop, with Emacs from master from a week ago, take respectively 
150 seconds and 40 seconds.  With the improvements on master, 1 is 
instantaneous but 2 still takes about 5 seconds.  Now, with the changes in 
the feature/long-lines-and-font-locking branch, both are instantaneous. 
The price of that speedup is that some portions of the buffer will be 
mis-highlighted, which is unavoidable.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 22 Jul 2022 10:16:52 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>
> 
> 
> The first step of the font locking related improvements was easier than I 
> thought.
> 
> Two following commands:
> 
> 1. C-x C-f dictionary.json RET y
> 
> 2. C-e
> 
> on my laptop, with Emacs from master from a week ago, take respectively 
> 150 seconds and 40 seconds.  With the improvements on master, 1 is 
> instantaneous but 2 still takes about 5 seconds.  Now, with the changes in 
> the feature/long-lines-and-font-locking branch, both are instantaneous. 
> The price of that speedup is that some portions of the buffer will be 
> mis-highlighted, which is unavoidable.

Thanks.  This is indeed the immediate idea for that aspect, but I
wonder what would developers of some major modes which widen the
buffer say about this.  Some of them were very much against such
restrictions in the past.

bug#56682: Fix the long lines font locking related slowdowns

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  This is indeed the immediate idea for that aspect, but I
> wonder what would developers of some major modes which widen the
> buffer say about this.  Some of them were very much against such
> restrictions in the past.

This reminds me of something I meant to mention -- Stefan M. once
proposed that there should be two kinds of narrowing (I think?).  The
first is the one that the user sets with `C-x n n', which says "the user
is only interested in this bit of the buffer", but programs are
"allowed" to remove that restriction when doing stuff (like font
locking).  The second type should be a strict one, where modes are not
allowed to widen the region.

Looking briefly at Gregory's new branch, it seems like that (sort of)
introduces this idea, but in a non-explicit way (i.e., by having an
inhibit-widen variable).

And, yes, some major mode authors were against this idea, but I think it
sounds like a sound idea.  And perhaps it'd make sense to implement it
like Stefan suggested, instead of `inhibit-widen'.

I've added Stefan to the CCs; perhaps he has some comments.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 22 Jul 2022 10:16:52 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>
> 
> 
> on my laptop, with Emacs from master from a week ago, take respectively 
> 150 seconds and 40 seconds.  With the improvements on master, 1 is 
> instantaneous but 2 still takes about 5 seconds.  Now, with the changes in 
> the feature/long-lines-and-font-locking branch, both are instantaneous. 
> The price of that speedup is that some portions of the buffer will be 
> mis-highlighted, which is unavoidable.

The assertion below can now be violated:

      if (it->narrowed_begv)
	{
	  record_unwind_protect (unwind_narrowed_begv, Fpoint_min ());
	  record_unwind_protect (unwind_narrowed_zv, Fpoint_max ());
	  SET_BUF_BEGV (current_buffer, it->narrowed_begv);
	  SET_BUF_ZV (current_buffer, it->narrowed_zv);
	  specbind (Qinhibit_widen, Qt);
	}

      val = Vfontification_functions;
      specbind (Qfontification_functions, Qnil);

      eassert (it->end_charpos == ZV);  <<<<<<<<<<<<<<<<<

because of the "narrowing".  (I actually saw this assertion violation
once on the branch, but I cannot reproduce it.)

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: 56682@debbugs.gnu.org
> Date: Fri, 22 Jul 2022 17:51:33 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Fri, 22 Jul 2022 10:16:52 +0000
> > From: Gregory Heytings <gregory@heytings.org>
> > cc: Eli Zaretskii <eliz@gnu.org>
> > 
> > 
> > on my laptop, with Emacs from master from a week ago, take respectively 
> > 150 seconds and 40 seconds.  With the improvements on master, 1 is 
> > instantaneous but 2 still takes about 5 seconds.  Now, with the changes in 
> > the feature/long-lines-and-font-locking branch, both are instantaneous. 
> > The price of that speedup is that some portions of the buffer will be 
> > mis-highlighted, which is unavoidable.
> 
> The assertion below can now be violated:
> 
>       if (it->narrowed_begv)
> 	{
> 	  record_unwind_protect (unwind_narrowed_begv, Fpoint_min ());
> 	  record_unwind_protect (unwind_narrowed_zv, Fpoint_max ());
> 	  SET_BUF_BEGV (current_buffer, it->narrowed_begv);
> 	  SET_BUF_ZV (current_buffer, it->narrowed_zv);
> 	  specbind (Qinhibit_widen, Qt);
> 	}
> 
>       val = Vfontification_functions;
>       specbind (Qfontification_functions, Qnil);
> 
>       eassert (it->end_charpos == ZV);  <<<<<<<<<<<<<<<<<
> 
> because of the "narrowing".  (I actually saw this assertion violation
> once on the branch, but I cannot reproduce it.)

Sorry, it's very easy to reproduce: visit dictionary.josn, and then
type C-v several times until it happens.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: 56682@debbugs.gnu.org
> Date: Fri, 22 Jul 2022 18:06:44 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > The assertion below can now be violated:
> > 
> >       if (it->narrowed_begv)
> > 	{
> > 	  record_unwind_protect (unwind_narrowed_begv, Fpoint_min ());
> > 	  record_unwind_protect (unwind_narrowed_zv, Fpoint_max ());
> > 	  SET_BUF_BEGV (current_buffer, it->narrowed_begv);
> > 	  SET_BUF_ZV (current_buffer, it->narrowed_zv);
> > 	  specbind (Qinhibit_widen, Qt);
> > 	}
> > 
> >       val = Vfontification_functions;
> >       specbind (Qfontification_functions, Qnil);
> > 
> >       eassert (it->end_charpos == ZV);  <<<<<<<<<<<<<<<<<
> > 
> > because of the "narrowing".  (I actually saw this assertion violation
> > once on the branch, but I cannot reproduce it.)
> 
> Sorry, it's very easy to reproduce: visit dictionary.josn, and then
> type C-v several times until it happens.

I installed a fix.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Before using a blunt tool like the forced-narrowing now in
`feature/long-lines-and-font-locking`, I think we should try and figure
out *why* the recipe below is so slow.

For example, the behavior surprises me because I can't see why `M-x`
should cause any font-locking at all in the `long-line.xml` file.

[ Other side note: IIRC `nxml-mode` uses font-lock and syntax-propertize
  in somewhat unusual ways (it started doing all of that work "by hand"
  in its own way, and only later was it coerced to try and play along
  with that "standard" infrastructure), so the problem may be specific to
  nxml-mode.  ]


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 22 Jul 2022 10:16:52 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>
> 
> on my laptop, with Emacs from master from a week ago, take respectively 
> 150 seconds and 40 seconds.  With the improvements on master, 1 is 
> instantaneous but 2 still takes about 5 seconds.  Now, with the changes in 
> the feature/long-lines-and-font-locking branch, both are instantaneous. 
> The price of that speedup is that some portions of the buffer will be 
> mis-highlighted, which is unavoidable.

The branch is much faster, indeed, but still font-lock imposes a
significant penalty on commands that needs to redisplay.  For example,
running the following simple benchmark:

  (defun scroll-up-benchmark ()
    (interactive)
    (let ((oldgc gcs-done)
	  (oldtime (float-time)))
      (condition-case nil (while t (scroll-up) (redisplay))
	(error (message "GCs: %d Elapsed time: %f seconds"
			(- gcs-done oldgc) (- (float-time) oldtime))))))

on long-line.xml produces a 15-fold slowdown with font-lock turned on
as compared to its being turned off (203 sec vs 13 sec).

This is an unoptimized build, so you will probably see times that are
4 times faster, but I'd be interested in the relative times on your
system.  Any explanations of the slowdown are also welcome.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Gregory Heytings <gregory@heytings.org>,  56682@debbugs.gnu.org
> Date: Fri, 22 Jul 2022 19:25:47 -0400
> 
> For example, the behavior surprises me because I can't see why `M-x`
> should cause any font-locking at all in the `long-line.xml` file.

Any command that enters the minibuffer causes a thorough redisplay of
the windows on that frame (because more than one window has to be
updated).  What does that have to do with font-lock is a separate
question (but you are the best person to answer it).

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> on long-line.xml produces a 15-fold slowdown with font-lock turned on
> as compared to its being turned off (203 sec vs 13 sec).
>
> This is an unoptimized build, so you will probably see times that are
> 4 times faster, but I'd be interested in the relative times on your
> system.  Any explanations of the slowdown are also welcome.

MacOS 12.5, M1 chip
Head: 792734a6e2cd5558debc8d9fe95d34cb3e809fa4 Improve efficiency of DND
tooltip movement
./configure --with-native-compilation

The long-lines.xml is 313295 bytes.  Hope that's the right one.

Font-lock       Output
------------------------------------------------------------
on              GCs: 14 Elapsed time: 7.880788 seconds
off             GCs: 2 Elapsed time: 0.885791 seconds

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Gregory Heytings <gregory@heytings.org>,  56682@debbugs.gnu.org,  Stefan
>  Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 23 Jul 2022 09:07:53 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > on long-line.xml produces a 15-fold slowdown with font-lock turned on
> > as compared to its being turned off (203 sec vs 13 sec).
> >
> > This is an unoptimized build, so you will probably see times that are
> > 4 times faster, but I'd be interested in the relative times on your
> > system.  Any explanations of the slowdown are also welcome.
> 
> MacOS 12.5, M1 chip
> Head: 792734a6e2cd5558debc8d9fe95d34cb3e809fa4 Improve efficiency of DND
> tooltip movement
> ./configure --with-native-compilation
> 
> The long-lines.xml is 313295 bytes.  Hope that's the right one.
> 
> Font-lock       Output
> ------------------------------------------------------------
> on              GCs: 14 Elapsed time: 7.880788 seconds
> off             GCs: 2 Elapsed time: 0.885791 seconds

Thanks.  This is still an order-of-magnitude slowdown, so the question
about the reasons is still relevant.

One thing we do under font-lock is merging faces, but that's supposed
to be very fast nowadays, given that faces are kept in a hash-table.
And what jit-lock does when the text is already fontified should be
negligible, right?

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Font-lock       Output
> ------------------------------------------------------------
> on              GCs: 14 Elapsed time: 7.880788 seconds
  on, and large gc-const-threshold
                  GCs: 0 Elapsed time: 7.720531 seconds
> off             GCs: 2 Elapsed time: 0.885791 seconds

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> And what jit-lock does when the text is already fontified should be
> negligible, right?

The iterator doesn't even stop and call jit-lock on regions with
'fontified' property, i.e. after font-lock.  I can't imagine ATM why
somone would have changed that.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Font-lock       Output
>> ------------------------------------------------------------
>> on              GCs: 14 Elapsed time: 7.880788 seconds
>   on, and large gc-const-threshold
>                   GCs: 0 Elapsed time: 7.720531 seconds
>> off             GCs: 2 Elapsed time: 0.885791 seconds

I've profiled this.  The result looks sensible, but I have to add that I
had mixed results with profiling optimized builds, so maybe...  Just
saying.

Sorry for posting an image.  I haven't found a way to get that info from
Instruments as text (pointers welcome).


[-- Attachment #2: Instrumens profile --]
[-- Type: image/png, Size: 209471 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>>> Font-lock       Output
>>> ------------------------------------------------------------
>>> on              GCs: 14 Elapsed time: 7.880788 seconds
>>   on, and large gc-const-threshold
>>                  GCs: 0 Elapsed time: 7.720531 seconds
    on, without scroll-bar
                    GCs: 15 Elapsed time: 5.574946 seconds
>>> off             GCs: 2 Elapsed time: 0.885791 seconds

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>>
>>>> Font-lock       Output
>>>> ------------------------------------------------------------
>>>> on              GCs: 14 Elapsed time: 7.880788 seconds
>>>   on, and large gc-const-threshold
>>>                  GCs: 0 Elapsed time: 7.720531 seconds
>     on, without scroll-bar
>                    GCs: 15 Elapsed time: 5.574946 seconds
      on without scroll-bar, auto-composition-mode off
                     GCs: 15 Elapsed time: 4.166101 seconds
>>>> off             GCs: 2 Elapsed time: 0.885791 seconds

Hm.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 56682@debbugs.gnu.org,  Gregory Heytings <gregory@heytings.org>,  Stefan
>  Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 23 Jul 2022 10:11:04 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> > Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> >
> >>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> >>>
> >>>> Font-lock       Output
> >>>> ------------------------------------------------------------
> >>>> on              GCs: 14 Elapsed time: 7.880788 seconds
> >>>   on, and large gc-const-threshold
> >>>                  GCs: 0 Elapsed time: 7.720531 seconds
> >     on, without scroll-bar
> >                    GCs: 15 Elapsed time: 5.574946 seconds
>       on without scroll-bar, auto-composition-mode off
>                      GCs: 15 Elapsed time: 4.166101 seconds
> >>>> off             GCs: 2 Elapsed time: 0.885791 seconds
> 
> Hm.

scroll-bar could be an issue only if we enter the new code that
doesn't use window_end_pos if it cannot be trusted.  Perhaps when
long-line optimizations are turned on for a buffer, we shouldn't try
to be holier than the Pope, and should use window_end_pos even if not
reliable?

And I've found a serious sink of CPU cycles under truncate-lines, and
installed a fix on the feature branch.  Gerd, if you have time to
eyeball the fix and comment on it, I'd appreciate.  It's commit
350e97d on the branch.  (I can post a more detailed explanation of
what I did and why, if that would help, because the code and the
functions it calls are somewhat tricky.)

After these changes, display of very long lines is quite reasonable,
when truncate-lines or truncate-partial-width-windows is in effect,
even without turning off font-lock and even in an unoptimized build.
Amusingly enough, show-paren-mode is now a serious performance killer
in these cases, because it puts overlays on buffer text, and that
disables a shortcut in the code that finds the next visible line
start, which is called a lot when lines are truncated.  Maybe we need
a smarter optimization there, one that doesn't immediately give up as
soon as it sees an overlay.  Hmm...

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Gregory Heytings <gregory@heytings.org>,  56682@debbugs.gnu.org
>> Date: Fri, 22 Jul 2022 19:25:47 -0400
>> 
>> For example, the behavior surprises me because I can't see why `M-x`
>> should cause any font-locking at all in the `long-line.xml` file.
>
> Any command that enters the minibuffer causes a thorough redisplay of
> the windows on that frame (because more than one window has to be
> updated).  What does that have to do with font-lock is a separate
> question (but you are the best person to answer it).

AFAIK if the buffer has not been modified (including things like
changing `window-start` or `point`), then a redisplay will just not
run jit-lock (and hence font-lock) at all, no matter how thorough.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> And I've found a serious sink of CPU cycles under truncate-lines, and
> installed a fix on the feature branch.  Gerd, if you have time to
> eyeball the fix and comment on it, I'd appreciate.  It's commit
> 350e97d on the branch.  (I can post a more detailed explanation of
> what I did and why, if that would help, because the code and the
> functions it calls are somewhat tricky.)

I'll look at it and come back.

BTW, does feature/long-lines-and-font-locking] build for you?  I'm getting

In toplevel form:
cedet/semantic/symref/list.el:35:2: Error: Wrong type argument: number-or-marker-p, nil
make[2]: *** [cedet/semantic/symref/list.elc] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [compile-main] Error 2
make: *** [lisp] Error 2

Configured --with-native-compilation.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> And I've found a serious sink of CPU cycles under truncate-lines, and
>> installed a fix on the feature branch.  Gerd, if you have time to
>> eyeball the fix and comment on it, I'd appreciate.  It's commit
>> 350e97d on the branch.  (I can post a more detailed explanation of
>> what I did and why, if that would help, because the code and the
>> functions it calls are somewhat tricky.)
>
> I'll look at it and come back.

modified   src/xdisp.c
@@ -7153,10 +7153,10 @@ forward_to_next_line_start (struct it *it, bool *skipped_p,
 	  || ((pos = Fnext_single_property_change (make_fixnum (start),
 						   Qdisplay, Qnil,
 						   make_fixnum (limit)),
-	       NILP (pos))
+	       (NILP (pos) || XFIXNAT (pos) == limit))
 	      && next_overlay_change (start) == ZV))
 	{
-	  if (!it->bidi_p)
+	  if (!it->bidi_p || !bidi_it_prev)
 	    {
 	      IT_CHARPOS (*it) = limit;
 	      IT_BYTEPOS (*it) = bytepos;

I understand the first diff, which makes a lot of sense, but I'm afraid
I don't know enough about bidi to be of any help.

bug#56682: Fix the long lines font locking related slowdowns

[Visuwesh]

[சனி ஜூலை 23, 2022] Gerd Möllmann wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> And I've found a serious sink of CPU cycles under truncate-lines, and
>> installed a fix on the feature branch.  Gerd, if you have time to
>> eyeball the fix and comment on it, I'd appreciate.  It's commit
>> 350e97d on the branch.  (I can post a more detailed explanation of
>> what I did and why, if that would help, because the code and the
>> functions it calls are somewhat tricky.)
>
> I'll look at it and come back.
>
> BTW, does feature/long-lines-and-font-locking] build for you?  I'm getting
>
> In toplevel form:
> cedet/semantic/symref/list.el:35:2: Error: Wrong type argument: number-or-marker-p, nil
> make[2]: *** [cedet/semantic/symref/list.elc] Error 1
> make[2]: *** Waiting for unfinished jobs....
> make[1]: *** [compile-main] Error 2
> make: *** [lisp] Error 2
>
> Configured --with-native-compilation.

I think Eli fixed it in the master branch in commit 4a4fcf628e1e4c8db47cd62fa5617b662fa8b5d6.

https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=4a4fcf628e1e4c8db47cd62fa5617b662fa8b5d6

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> BTW, does feature/long-lines-and-font-locking] build for you?  I'm getting
>
> In toplevel form:
> cedet/semantic/symref/list.el:35:2: Error: Wrong type argument: number-or-marker-p, nil
> make[2]: *** [cedet/semantic/symref/list.elc] Error 1
> make[2]: *** Waiting for unfinished jobs....
> make[1]: *** [compile-main] Error 2
> make: *** [lisp] Error 2
>
> Configured --with-native-compilation.

Same without native compilation.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Visuwesh <visuweshm@gmail.com> writes:

> I think Eli fixed it in the master branch in commit 4a4fcf628e1e4c8db47cd62fa5617b662fa8b5d6.

Yup that worked.  Thanks!

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

Another observation: any command that calls current_column, directly
or indirectly, will be very slow with long lines.  This includes
"C-x =" and column-number-mode, for example.

I installed a change to avoid calling current_column where possible,
but the problem is we have a bytecode which calls it, and it is
impossible to avoid the call because it could break some callers.
Perhaps we should expose the long-line-optimization flag to Lisp, so
that commands like what-cursor-position could refrain from calling
current-column.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

And the new lottery numbers on the feature branch are:

Font-lock Scroll-bar Composition Output
------------------------------------------------------------
on	  on	     on		 GCs: 14 Elapsed time: 7.276626 seconds
on	  off	     on		 GCs: 1 Elapsed time: 5.363002 seconds
on	  off	     off	 GCs: 1 Elapsed time: 3.967520 seconds

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> Perhaps when
> long-line optimizations are turned on for a buffer, we shouldn't try
> to be holier than the Pope, and should use window_end_pos even if not
> reliable?

I don't think anyone would complain.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gregory@heytings.org,  56682@debbugs.gnu.org
> Date: Sat, 23 Jul 2022 10:07:29 -0400
> 
> >> From: Stefan Monnier <monnier@iro.umontreal.ca>
> >> Cc: Gregory Heytings <gregory@heytings.org>,  56682@debbugs.gnu.org
> >> Date: Fri, 22 Jul 2022 19:25:47 -0400
> >> 
> >> For example, the behavior surprises me because I can't see why `M-x`
> >> should cause any font-locking at all in the `long-line.xml` file.
> >
> > Any command that enters the minibuffer causes a thorough redisplay of
> > the windows on that frame (because more than one window has to be
> > updated).  What does that have to do with font-lock is a separate
> > question (but you are the best person to answer it).
> 
> AFAIK if the buffer has not been modified (including things like
> changing `window-start` or `point`), then a redisplay will just not
> run jit-lock (and hence font-lock) at all, no matter how thorough.

But the fact is without font-lock the response is faster by a large
factor.  So something, somewhere, still depends on font-lock.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 56682@debbugs.gnu.org,  gregory@heytings.org,  monnier@iro.umontreal.ca
> Date: Sat, 23 Jul 2022 16:33:28 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> And I've found a serious sink of CPU cycles under truncate-lines, and
> >> installed a fix on the feature branch.  Gerd, if you have time to
> >> eyeball the fix and comment on it, I'd appreciate.  It's commit
> >> 350e97d on the branch.  (I can post a more detailed explanation of
> >> what I did and why, if that would help, because the code and the
> >> functions it calls are somewhat tricky.)
> >
> > I'll look at it and come back.
> 
> modified   src/xdisp.c
> @@ -7153,10 +7153,10 @@ forward_to_next_line_start (struct it *it, bool *skipped_p,
>  	  || ((pos = Fnext_single_property_change (make_fixnum (start),
>  						   Qdisplay, Qnil,
>  						   make_fixnum (limit)),
> -	       NILP (pos))
> +	       (NILP (pos) || XFIXNAT (pos) == limit))
>  	      && next_overlay_change (start) == ZV))
>  	{
> -	  if (!it->bidi_p)
> +	  if (!it->bidi_p || !bidi_it_prev)
>  	    {
>  	      IT_CHARPOS (*it) = limit;
>  	      IT_BYTEPOS (*it) = bytepos;
> 
> I understand the first diff, which makes a lot of sense

Thanks, this was the main part.  From what I see, this code never
really worked correctly, since Emacs 21, because
Fnext_single_property_change can never return nil when called like
that.  IOW, we never took the shortcut there, and probably didn't
notice it because 500-character lines are rare.

> but I'm afraid I don't know enough about bidi to be of any help.

The idea is that since the value of BIDI_IT_PREV is never used by the
caller when ON_NEWLINE_P is zero, we don't need to compute it when
ON_NEWLINE_P is zero, something that would have precluded us from
taking the shortcut.  Not taking the shortcut is very expensive when
we have very long lines, of course.

Btw, one quirk of the code in forward_to_next_line_start (which took
me some time to take in and convince myself it's okay) is that it
returns non-zero when it didn't find any new newlines till ZV.  This
is not very clean, but I guess we don't care in that case, since the
iterator will be moved to ZV, and from there there's not much we need
to do...

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> AFAIK if the buffer has not been modified (including things like
>> changing `window-start` or `point`), then a redisplay will just not
>> run jit-lock (and hence font-lock) at all, no matter how thorough.
> But the fact is without font-lock the response is faster by a large
> factor.  So something, somewhere, still depends on font-lock.

Yes, that's the part that we need to explore.
Maybe font-lock *is* run somehow?
Or maybe it's just the mere presence of text properties?  (Or overlays?)


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 56682@debbugs.gnu.org,  Eli Zaretskii <eliz@gnu.org>,
>   gregory@heytings.org,  monnier@iro.umontreal.ca
> Date: Sat, 23 Jul 2022 17:01:02 +0200
> 
> And the new lottery numbers on the feature branch are:
> 
> Font-lock Scroll-bar Composition Output
> ------------------------------------------------------------
> on	  on	     on		 GCs: 14 Elapsed time: 7.276626 seconds
> on	  off	     on		 GCs: 1 Elapsed time: 5.363002 seconds
> on	  off	     off	 GCs: 1 Elapsed time: 3.967520 seconds

Thanks.  How about now, after I installed the last change?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 56682@debbugs.gnu.org,  gregory@heytings.org,  monnier@iro.umontreal.ca
> Date: Sat, 23 Jul 2022 17:04:39 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Perhaps when
> > long-line optimizations are turned on for a buffer, we shouldn't try
> > to be holier than the Pope, and should use window_end_pos even if not
> > reliable?
> 
> I don't think anyone would complain.

I agree, and so I've now done so on the branch.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gregory@heytings.org,  56682@debbugs.gnu.org
> Date: Sat, 23 Jul 2022 11:46:37 -0400
> 
> >> AFAIK if the buffer has not been modified (including things like
> >> changing `window-start` or `point`), then a redisplay will just not
> >> run jit-lock (and hence font-lock) at all, no matter how thorough.
> > But the fact is without font-lock the response is faster by a large
> > factor.  So something, somewhere, still depends on font-lock.
> 
> Yes, that's the part that we need to explore.
> Maybe font-lock *is* run somehow?
> Or maybe it's just the mere presence of text properties?  (Or overlays?)

My bet is indeed on the mere presence of text properties, plus the
fact that we need to merge faces.  But I could well be wrong.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: 56682@debbugs.gnu.org, gregory@heytings.org
> Date: Sat, 23 Jul 2022 19:15:26 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: gregory@heytings.org,  56682@debbugs.gnu.org
> > Date: Sat, 23 Jul 2022 11:46:37 -0400
> > 
> > >> AFAIK if the buffer has not been modified (including things like
> > >> changing `window-start` or `point`), then a redisplay will just not
> > >> run jit-lock (and hence font-lock) at all, no matter how thorough.
> > > But the fact is without font-lock the response is faster by a large
> > > factor.  So something, somewhere, still depends on font-lock.
> > 
> > Yes, that's the part that we need to explore.
> > Maybe font-lock *is* run somehow?
> > Or maybe it's just the mere presence of text properties?  (Or overlays?)
> 
> My bet is indeed on the mere presence of text properties, plus the
> fact that we need to merge faces.  But I could well be wrong.

Btw, I think the best tool for determining this is run-time profiling,
such as with perf on GNU/Linux.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> Font-lock Scroll-bar Composition Output
>> ------------------------------------------------------------
>> on	  on	     on		 GCs: 14 Elapsed time: 7.276626 seconds
>> on	  off	     on		 GCs: 1 Elapsed time: 5.363002 seconds
>> on	  off	     off	 GCs: 1 Elapsed time: 3.967520 seconds
>
> Thanks.  How about now, after I installed the last change?

Font-lock Scroll-bar Composition Output
------------------------------------------------------------
on	  on	     on		 GCs: 16 Elapsed time: 5.496764 seconds
on	  off	     on		 GCs: 1 Elapsed time: 5.362916 seconds
on	  off	     off	 GCs: 1 Elapsed time: 3.947306 seconds

That's with
280b8c96cc origin/feature/long-lines-and-font-locking Improve display of columns on mode-lin

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 56682@debbugs.gnu.org,  gregory@heytings.org,  monnier@iro.umontreal.ca,
>   visuweshm@gmail.com
> Date: Sat, 23 Jul 2022 19:23:40 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Font-lock Scroll-bar Composition Output
> >> ------------------------------------------------------------
> >> on	  on	     on		 GCs: 14 Elapsed time: 7.276626 seconds
> >> on	  off	     on		 GCs: 1 Elapsed time: 5.363002 seconds
> >> on	  off	     off	 GCs: 1 Elapsed time: 3.967520 seconds
> >
> > Thanks.  How about now, after I installed the last change?
> 
> Font-lock Scroll-bar Composition Output
> ------------------------------------------------------------
> on	  on	     on		 GCs: 16 Elapsed time: 5.496764 seconds
> on	  off	     on		 GCs: 1 Elapsed time: 5.362916 seconds
> on	  off	     off	 GCs: 1 Elapsed time: 3.947306 seconds
> 
> That's with
> 280b8c96cc origin/feature/long-lines-and-font-locking Improve display of columns on mode-lin

Thanks.  So the scroll-bar effect is largely gone, and font-lock is
now just 25% slower than no-font-lock.  Which I think is reasonable,
given that there's a face change every 10 to 20 characters?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca,
>  visuweshm@gmail.com
> Date: Sat, 23 Jul 2022 20:44:43 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Font-lock Scroll-bar Composition Output
> > ------------------------------------------------------------
> > on	  on	     on		 GCs: 16 Elapsed time: 5.496764 seconds
> > on	  off	     on		 GCs: 1 Elapsed time: 5.362916 seconds
> > on	  off	     off	 GCs: 1 Elapsed time: 3.947306 seconds
> > 
> > That's with
> > 280b8c96cc origin/feature/long-lines-and-font-locking Improve display of columns on mode-lin
> 
> Thanks.  So the scroll-bar effect is largely gone, and font-lock is
> now just 25% slower than no-font-lock.  Which I think is reasonable,
> given that there's a face change every 10 to 20 characters?

Oops, I was confused: font-lock OFF is not in the table, so with it
we're still 4 to 5 times slower than without it.  It's
auto-composition mode that costs us 25% slowdown.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca,
>  visuweshm@gmail.com
> Date: Sat, 23 Jul 2022 20:49:39 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > > Font-lock Scroll-bar Composition Output
> > > ------------------------------------------------------------
> > > on	  on	     on		 GCs: 16 Elapsed time: 5.496764 seconds
> > > on	  off	     on		 GCs: 1 Elapsed time: 5.362916 seconds
> > > on	  off	     off	 GCs: 1 Elapsed time: 3.947306 seconds
> > > 
> > > That's with
> > > 280b8c96cc origin/feature/long-lines-and-font-locking Improve display of columns on mode-lin
> > 
> > Thanks.  So the scroll-bar effect is largely gone, and font-lock is
> > now just 25% slower than no-font-lock.  Which I think is reasonable,
> > given that there's a face change every 10 to 20 characters?
> 
> Oops, I was confused: font-lock OFF is not in the table, so with it
> we're still 4 to 5 times slower than without it.  It's
> auto-composition mode that costs us 25% slowdown.

Which seems to be similar to slowdown due to font-lock in other cases?
For example, scrolling with the same benchmark through xdisp.c takes
190 sec for the first time and 40 for the second time (when everything
is already fontified); whereas without font-lock it takes 20.  So it
sounds like font-lock generally slows down redisplay by such small
factors?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> My bet is indeed on the mere presence of text properties, plus the fact 
> that we need to merge faces.
>

That's precisely what I've been investigating during the last day.  I 
probably need two or three more days to reach a conclusion, so please tell 
me if you're doing that in parallel.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 23 Jul 2022 19:05:54 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, bug-gnu-emacs@gnu.org
> 
> > My bet is indeed on the mere presence of text properties, plus the fact 
> > that we need to merge faces.
> 
> That's precisely what I've been investigating during the last day.  I 
> probably need two or three more days to reach a conclusion, so please tell 
> me if you're doing that in parallel.

I'm not.  I did the measurements, and so did Gerd, but I'm not looking
into the font-lock influence any deeper than that.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 418 bytes --]


>> That's precisely what I've been investigating during the last day.  I 
>> probably need two or three more days to reach a conclusion, so please 
>> tell me if you're doing that in parallel.
>
> I'm not.  I did the measurements, and so did Gerd, but I'm not looking 
> into the font-lock influence any deeper than that.
>

Okay, thanks; I can continue without fearing that I might be wasting my 
time 😉

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> My bet is indeed on the mere presence of text properties, plus the
>> fact that we need to merge faces.  But I could well be wrong.

Can't say something about face merging, but "frequent" changes of faces
certainly have an effect on iterator performance.  It stops, looks up
properties again to determine the next stop pos, does what has to be
done for current properties...

> Btw, I think the best tool for determining this is run-time profiling,
> such as with perf on GNU/Linux.

Yes, I don't think there is something comparable on macOS.  Or I simply
can't find it.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> Which seems to be similar to slowdown due to font-lock in other cases?
> For example, scrolling with the same benchmark through xdisp.c takes
> 190 sec for the first time and 40 for the second time (when everything
> is already fontified); whereas without font-lock it takes 20.  So it
> sounds like font-lock generally slows down redisplay by such small
> factors?

Alas, I don't remember much about the exact figures I got when I
profiled this pre-21.  But a doubling of run time--after font-lock has
finished--doesn't appear to me to be entirely unplausible.

BTW, a memory that re-emerged right now: Interestingly, the number of
different text properties that the iterator checks, i.e. the number of
text property names like face, fontified, display, invisible, etc. that
the iterator checks, played on astonishing role back then.  And the
relation to performance wasn't linear either.  Which is why there's one
display property now, subsuming different subtypes.  Originally, the
subtypes were distinct properties, and display didn't exist.  Way before
Emacs 21.

Don't know if this is relevant for anything in this case.  I thought I
just mention that the interval tree might also have a potential for
improvement, if you will.  Amd another BTW: I was never 100% certain if
the interval tree is really always balanced because it didn't use an
algorithm that I knew and could recognize.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 56682@debbugs.gnu.org,  gregory@heytings.org,  monnier@iro.umontreal.ca,
>   visuweshm@gmail.com
> Date: Sun, 24 Jul 2022 08:16:33 +0200
> 
> And another BTW: I was never 100% certain if the interval tree is
> really always balanced because it didn't use an algorithm that I
> knew and could recognize.

AFAIU, we balance on-the-fly (see find_interval, which calls
balance_possible_root_interval).

> "frequent" changes of faces certainly have an effect on iterator
> performance.  It stops, looks up properties again to determine the
> next stop pos, does what has to be done for current properties...

To be more accurate: changes in text properties that are not relevant
to display basically affect the iterator performance in only one way:
they make the search for the next change in _relevant_ properties more
expensive.  See the last part of compute_stop_pos for the gory
details.  In a nutshell, we check one by one the intervals following
the interval of the current iterator position, until we find an
interval whose values for one or more of the properties of interest to
redisplay are different.  When we find one such interval, it is
guaranteed to have changes only in the text properties that are of
interest to redisplay, but the search could take more time if there
are many text properties that are not interesting, because there are
more intervals to check.

Btw, it might be interesting to measure the effect of enlarging
TEXT_PROP_DISTANCE_LIMIT, currently 100 character positions, on the
performance.  Looking at the code, it is not clear to me whether it
could affect the performance in any significant ways, but maybe I'm
wrong.

bug#56682: Interval tree balance (was: Fix the long lines font locking related slowdowns)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Don't know if this is relevant for anything in this case.  I thought I
> just mention that the interval tree might also have a potential for
> improvement, if you will.  And another BTW: I was never 100% certain if
> the interval tree is really always balanced because it didn't use an
> algorithm that I knew and could recognize.

I can answer this one: no it's not always balanced (tho in practice
it is most of the time).  We could make it use a more standard
algorithm, but I have not been able to measure any impact on performance
(I also played with a splay-tree alternative, under the assumption that
we mostly consult the tree "locally" (within the visible part of the
buffer, basically), so a splay-tree could turn the O(log N) into an O(n)
where `N` is the buffer-size and `n` is the distance between
window-start and window-end).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 24.07.2022 08:50, Gerd Möllmann wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>>> My bet is indeed on the mere presence of text properties, plus the
>>> fact that we need to merge faces.  But I could well be wrong.
> Can't say something about face merging, but "frequent" changes of faces
> certainly have an effect on iterator performance.  It stops, looks up
> properties again to determine the next stop pos, does what has to be
> done for current properties...

But the problem is contingent on having long lines, isn't it?

There must be some interplay between those circumstances. Not just 
having to look up faces (relatively) a lot.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> redisplay are different.  When we find one such interval, it is
> guaranteed to have changes only in the text properties that are of
> interest to redisplay, but the search could take more time if there
> are many text properties that are not interesting, because there are
> more intervals to check.

Not only more intervals, but also more elements in the `plist`s of
each interval.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 24 Jul 2022 17:35:19 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 24.07.2022 08:50, Gerd Möllmann wrote:
> > Eli Zaretskii<eliz@gnu.org>  writes:
> > 
> >>> My bet is indeed on the mere presence of text properties, plus the
> >>> fact that we need to merge faces.  But I could well be wrong.
> > Can't say something about face merging, but "frequent" changes of faces
> > certainly have an effect on iterator performance.  It stops, looks up
> > properties again to determine the next stop pos, does what has to be
> > done for current properties...
> 
> But the problem is contingent on having long lines, isn't it?

Not necessarily, see the times I measured scrolling through xdisp.c,
which I posted earlier.  It could be that with long lines font-lock
just makes it slower still, to the point where it becomes unbearable.

> There must be some interplay between those circumstances. Not just 
> having to look up faces (relatively) a lot.

What else did you have in mind?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>   56682@debbugs.gnu.org,
>   gregory@heytings.org,  visuweshm@gmail.com
> Date: Sun, 24 Jul 2022 10:36:41 -0400
> 
> > redisplay are different.  When we find one such interval, it is
> > guaranteed to have changes only in the text properties that are of
> > interest to redisplay, but the search could take more time if there
> > are many text properties that are not interesting, because there are
> > more intervals to check.
> 
> Not only more intervals, but also more elements in the `plist`s of
> each interval.

Could be, if there are many different properties.  But since we are
talking about font-lock, there's only one we are talking about:
'face'.

bug#56682: Interval tree balance

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> buffer, basically), so a splay-tree could turn the O(log N) into an O(n)
                                                                        ^
                                                                      log n

> where `N` is the buffer-size and `n` is the distance between
> window-start and window-end).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Not only more intervals, but also more elements in the `plist`s of
>> each interval.
> Could be, if there are many different properties.  But since we are
> talking about font-lock, there's only one we are talking about: 'face'.

`face` may be the one we care about, but there will be others
(especially in the case you were talking about, i.e. the impact of
non-display-related faces)


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gerd.moellmann@gmail.com,  56682@debbugs.gnu.org,  gregory@heytings.org,
>   visuweshm@gmail.com
> Date: Sun, 24 Jul 2022 11:48:35 -0400
> 
> >> Not only more intervals, but also more elements in the `plist`s of
> >> each interval.
> > Could be, if there are many different properties.  But since we are
> > talking about font-lock, there's only one we are talking about: 'face'.
> 
> `face` may be the one we care about, but there will be others
> (especially in the case you were talking about, i.e. the impact of
> non-display-related faces)

In general, sure.  But this particular discussion is about the
difference in performance between font-lock being on and off, and in
that case the only property that counts is 'face'.

If you are thinking about non-display-related properties, you are in
the wrong thread ;-)  That one was raised by João in another
discussion.

bug#56682: Fix the long lines font locking related slowdowns

[Lars Ingebrigtsen]

Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> `face` may be the one we care about, but there will be others
> (especially in the case you were talking about, i.e. the impact of
> non-display-related faces)

Random thought: Would it make sense to make `put-text-property' (and
friends) try to ensure that `face' (and `font-lock-face'?) are towards
the start of the plists?

That wouldn't help with intervals that don't have a `face' at all, but
should speed up the ones that have, and would be relatively low impact.

I don't think we guarantee anything about the order, so it might not
break anything.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  gerd.moellmann@gmail.com,
>   56682@debbugs.gnu.org,  gregory@heytings.org,  visuweshm@gmail.com
> Date: Sun, 24 Jul 2022 18:26:38 +0200
> 
> Random thought: Would it make sense to make `put-text-property' (and
> friends) try to ensure that `face' (and `font-lock-face'?) are towards
> the start of the plists?

The display code doesn't care only about 'face', it also cares for
'invisible' and 'display' properties.  And only one of them can be
"the first" ;-)

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Don't know if this is relevant for anything in this case.  I thought I
>> just mention that the interval tree might also have a potential for
>> improvement, if you will.  And another BTW: I was never 100% certain if
>> the interval tree is really always balanced because it didn't use an
>> algorithm that I knew and could recognize.
>
> I can answer this one: no it's not always balanced (tho in practice
> it is most of the time).  We could make it use a more standard
> algorithm, but I have not been able to measure any impact on performance
> (I also played with a splay-tree alternative, under the assumption that
> we mostly consult the tree "locally" (within the visible part of the
> buffer, basically), so a splay-tree could turn the O(log N) into an O(n)
> where `N` is the buffer-size and `n` is the distance between
> window-start and window-end).

Thanks.  And too bad.

But I'd really prefer a standard algorithm with formally proven
properties anyway.  Call me German.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

I pushed another improvement to the branch.  Now motion in a font-locked 
buffer is almost as fast as in a non font-locked one; I don't think it's 
possible to make further significant improvements.

It turns out that the motion slowdowns in font locked-buffers were caused 
only indirectly by font locking.  What happens is that in a font-locked 
buffer, get_next_display_element calls next_element_from_buffer which uses 
handle_stop (and therefore compute_stop_pos) to position the iterator on 
the next element.  In a file such as long-line.xml, there are many more 
stop positions when the buffer is font locked, roughly 100 times more. 
The solution is to make back_to_previous_visible_line_start do something 
closer to what it promises, namely "Set IT's current position to the 
previous visible line start."  Previously it moved IT to the beginning of 
the (temporarily narrowed) buffer, which was already much better than 
moving it to the beginning of the line, but not enough to make motion 
commands fast.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> This reminds me of something I meant to mention -- Stefan M. once 
> proposed that there should be two kinds of narrowing (I think?).  The 
> first is the one that the user sets with `C-x n n', which says "the user 
> is only interested in this bit of the buffer", but programs are 
> "allowed" to remove that restriction when doing stuff (like font 
> locking).  The second type should be a strict one, where modes are not 
> allowed to widen the region.
>
> Looking briefly at Gregory's new branch, it seems like that (sort of) 
> introduces this idea, but in a non-explicit way (i.e., by having an 
> inhibit-widen variable).
>

Thanks for the idea!  I think this could be useful in other contexts, so 
I'll try to implement that.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Before using a blunt tool like the forced-narrowing now in 
> `feature/long-lines-and-font-locking`, I think we should try and figure 
> out *why* the recipe below is so slow.
>

It's not a blunt tool, it's an appropriate tool to help making sure that 
Emacs remains responsive when large files are visited.  Think of it as 
POSIX's ulimits.  Allowing fontification-functions to search for 
arbitrarily complex regexpes in an arbitrarily large buffer, each and 
every time they are asked to to highlight a small chunk of the said 
buffer, is a recipe for disaster.  If for some reason modes really need to 
go through to the whole buffer to decide which highlighting to use, they 
should to do so outside of fontification-functions, and ideally once, for 
example, when the file is loaded.

(Note that at the moment that tool is enabled only when files with long 
lines are visited.)

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> And I've found a serious sink of CPU cycles under truncate-lines, and 
> installed a fix on the feature branch.  Gerd, if you have time to 
> eyeball the fix and comment on it, I'd appreciate.  It's commit 350e97d 
> on the branch.  (I can post a more detailed explanation of what I did 
> and why, if that would help, because the code and the functions it calls 
> are somewhat tricky.)
>

Hmmm...  After 350e97d78e, Isearch locks Emacs with toggle-truncate-lines. 
Recipe:

C-x C-f long-line.xml
C-x x t
C-s </

You have to kill Emacs, C-g does not work.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 24.07.2022 18:05, Eli Zaretskii wrote:
>> Date: Sun, 24 Jul 2022 17:35:19 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 24.07.2022 08:50, Gerd Möllmann wrote:
>>> Eli Zaretskii<eliz@gnu.org>  writes:
>>>
>>>>> My bet is indeed on the mere presence of text properties, plus the
>>>>> fact that we need to merge faces.  But I could well be wrong.
>>> Can't say something about face merging, but "frequent" changes of faces
>>> certainly have an effect on iterator performance.  It stops, looks up
>>> properties again to determine the next stop pos, does what has to be
>>> done for current properties...
>>
>> But the problem is contingent on having long lines, isn't it?
> 
> Not necessarily, see the times I measured scrolling through xdisp.c,
> which I posted earlier.  It could be that with long lines font-lock
> just makes it slower still, to the point where it becomes unbearable.

Why with long lines, though?

>> There must be some interplay between those circumstances. Not just
>> having to look up faces (relatively) a lot.
> 
> What else did you have in mind?

Some operation dependent on the length of the current line?

With font-lock, it seems to get progressively slower the farther you get 
along the current (long line).

E.g. you can have a long line spanning several screenfuls without line 
breaks. When the window is scrolled to the beginning, redisplay is 
relatively fast (I can press up/down arrows, and they seem responsive).

But if I scroll the window to the end of said long line, up/down 
commands become much less responsive.

Tested that with today's master and js-mode visiting a minified JS file.

Perhaps it's due to font-lock logic in that it has to match from the 
beginning of a line (not sure we'd want to abandon that promise, 
though). Or maybe something else.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Gregory Heytings <gregory@heytings.org> writes:

> I pushed another improvement to the branch.  Now motion in a
> font-locked buffer is almost as fast as in a non font-locked one; I
> don't think it's possible to make further significant improvements.

Font-lock Scroll-bar Composition Output
------------------------------------------------------------
on	  on	     on		 GCs: 13 Elapsed time: 1.632215 seconds
on	  off	     on		 GCs: 1 Elapsed time: 1.433515 seconds
on	  off	     off	 GCs: 2 Elapsed time: 1.209662 seconds
off	  off	     off	 GCs: 2 Elapsed time: 0.524913 seconds

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Gregory Heytings <gregory@heytings.org> writes:

> Hmmm...  After 350e97d78e, Isearch locks Emacs with
> toggle-truncate-lines. Recipe:
>
> C-x C-f long-line.xml
> C-x x t
> C-s </
>
> You have to kill Emacs, C-g does not work.

That's kind of funny :-):

(setq isearch-lazy-highlight nil)

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> With font-lock, it seems to get progressively slower the farther you get 
> along the current (long line).
>
> E.g. you can have a long line spanning several screenfuls without line 
> breaks. When the window is scrolled to the beginning, redisplay is 
> relatively fast (I can press up/down arrows, and they seem responsive).
>
> But if I scroll the window to the end of said long line, up/down 
> commands become much less responsive.
>
> Tested that with today's master and js-mode visiting a minified JS file.
>

You should have tried the feature/long-lines-and-font-locking branch 
instead, where that problem has been fixed a few hours ago.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> I pushed another improvement to the branch.  Now motion in a 
>> font-locked buffer is almost as fast as in a non font-locked one; I 
>> don't think it's possible to make further significant improvements.
>
> Font-lock Scroll-bar Composition Output
> ------------------------------------------------------------
> on	  on	     on		 GCs: 13 Elapsed time: 1.632215 seconds
> on	  off	     on		 GCs: 1 Elapsed time: 1.433515 seconds
> on	  off	     off	 GCs: 2 Elapsed time: 1.209662 seconds
> off	  off	     off	 GCs: 2 Elapsed time: 0.524913 seconds
>

Thanks, this confirms my "almost as fast" feeling and measurements.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gregory Heytings <gregory@heytings.org> writes:
>
>> Hmmm...  After 350e97d78e, Isearch locks Emacs with
>> toggle-truncate-lines. Recipe:
>>
>> C-x C-f long-line.xml
>> C-x x t
>> C-s </
>>
>> You have to kill Emacs, C-g does not work.
>
> That's kind of funny :-):
>
> (setq isearch-lazy-highlight nil)

I eventually finishes, sort of.  The many '<' in the XML lead to a stop
every few characters for drawing highlights.  Like a train that stops at
every trash can along the tracks.  That's more stops than with
font-lock.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 25 Jul 2022 21:47:55 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
>     56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> > And I've found a serious sink of CPU cycles under truncate-lines, and 
> > installed a fix on the feature branch.  Gerd, if you have time to 
> > eyeball the fix and comment on it, I'd appreciate.  It's commit 350e97d 
> > on the branch.  (I can post a more detailed explanation of what I did 
> > and why, if that would help, because the code and the functions it calls 
> > are somewhat tricky.)
> 
> Hmmm...  After 350e97d78e, Isearch locks Emacs with toggle-truncate-lines. 
> Recipe:
> 
> C-x C-f long-line.xml
> C-x x t
> C-s </
> 
> You have to kill Emacs, C-g does not work.

I see the same on master, where that change is not yet installed.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 26 Jul 2022 02:23:42 +0300
> Cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, gregory@heytings.org,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> But the problem is contingent on having long lines, isn't it?
> > 
> > Not necessarily, see the times I measured scrolling through xdisp.c,
> > which I posted earlier.  It could be that with long lines font-lock
> > just makes it slower still, to the point where it becomes unbearable.
> 
> Why with long lines, though?

Because they are long.  So there's more faces to encounter when going
from one line to another.

> >> There must be some interplay between those circumstances. Not just
> >> having to look up faces (relatively) a lot.
> > 
> > What else did you have in mind?
> 
> Some operation dependent on the length of the current line?
> 
> With font-lock, it seems to get progressively slower the farther you get 
> along the current (long line).
> 
> E.g. you can have a long line spanning several screenfuls without line 
> breaks. When the window is scrolled to the beginning, redisplay is 
> relatively fast (I can press up/down arrows, and they seem responsive).
> 
> But if I scroll the window to the end of said long line, up/down 
> commands become much less responsive.
> 
> Tested that with today's master and js-mode visiting a minified JS file.
> 
> Perhaps it's due to font-lock logic in that it has to match from the 
> beginning of a line (not sure we'd want to abandon that promise, 
> though). Or maybe something else.

It isn't font-lock, at least not in all major modes.  It's the display
engine itself that sometimes needs to go to the beginning of the line.
When it does, going back gets slower with font-lock than without.
This is why you see slower redisplay when you go deeper into a long
line.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Hmmm...  After 350e97d78e, Isearch locks Emacs with 
>> toggle-truncate-lines. Recipe:
>>
>> C-x C-f long-line.xml
>> C-x x t
>> C-s </
>>
>> You have to kill Emacs, C-g does not work.
>
> I see the same on master, where that change is not yet installed.
>

Are you sure?  I don't see the same at 304e2a3a05.  C-s is slow, but it 
does not hang Emacs, and the effect of C-s </ is visible within a couple 
of seconds.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 26 Jul 2022 11:53:28 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> 
> >> Hmmm...  After 350e97d78e, Isearch locks Emacs with 
> >> toggle-truncate-lines. Recipe:
> >>
> >> C-x C-f long-line.xml
> >> C-x x t
> >> C-s </
> >>
> >> You have to kill Emacs, C-g does not work.
> >
> > I see the same on master, where that change is not yet installed.
> 
> Are you sure?  I don't see the same at 304e2a3a05.  C-s is slow, but it 
> does not hang Emacs, and the effect of C-s </ is visible within a couple 
> of seconds.

I don't know the difference between "slow" and "hang".  Gerd says it
eventually finishes, so it isn't a "hang" at least on his machine.
And my build is unoptimized, so what is "slow" for you probably
qualifies as "hang" for me.

If you disable lazy-highlighting, does the problem go away?

And if you customize lazy-highlight-interval and
lazy-highlight-max-at-a-time to reasonable values, doesn't the "hang"
becomes much shorter?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  56682@debbugs.gnu.org,
>   monnier@iro.umontreal.ca
> Date: Tue, 26 Jul 2022 08:51:54 +0200
> 
> Gregory Heytings <gregory@heytings.org> writes:
> 
> > Hmmm...  After 350e97d78e, Isearch locks Emacs with
> > toggle-truncate-lines. Recipe:
> >
> > C-x C-f long-line.xml
> > C-x x t
> > C-s </
> >
> > You have to kill Emacs, C-g does not work.
> 
> That's kind of funny :-):
> 
> (setq isearch-lazy-highlight nil)

Did you try this on master or on the feature branch?

And I found that customizing lazy-highlight-max-at-a-time and
lazy-highlight-interval can alleviate the problem to some extent.

The reason seems to be that lazy-highlighting tries to highlight every
match "in th window", but its interpretation of "in the window" seems
to be "buffer position before window-end position".  Which of course
includes a lot of stuff in a window with truncate-lines.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

This is on Gregory's branch.

On Tue, 26 Jul 2022, 14:12 Eli Zaretskii, <eliz@gnu.org> wrote:

> > From: Gerd Möllmann <gerd.moellmann@gmail.com>
> > Cc: Eli Zaretskii <eliz@gnu.org>,  56682@debbugs.gnu.org,
> >   monnier@iro.umontreal.ca
> > Date: Tue, 26 Jul 2022 08:51:54 +0200
> >
> > Gregory Heytings <gregory@heytings.org> writes:
> >
> > > Hmmm...  After 350e97d78e, Isearch locks Emacs with
> > > toggle-truncate-lines. Recipe:
> > >
> > > C-x C-f long-line.xml
> > > C-x x t
> > > C-s </
> > >
> > > You have to kill Emacs, C-g does not work.
> >
> > That's kind of funny :-):
> >
> > (setq isearch-lazy-highlight nil)
>
> Did you try this on master or on the feature branch?
>
> And I found that customizing lazy-highlight-max-at-a-time and
> lazy-highlight-interval can alleviate the problem to some extent.
>
> The reason seems to be that lazy-highlighting tries to highlight every
> match "in th window", but its interpretation of "in the window" seems
> to be "buffer position before window-end position".  Which of course
> includes a lot of stuff in a window with truncate-lines.
>

[-- Attachment #2: Type: text/html, Size: 1968 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I don't know the difference between "slow" and "hang".  Gerd says it 
> eventually finishes, so it isn't a "hang" at least on his machine. And 
> my build is unoptimized, so what is "slow" for you probably qualifies as 
> "hang" for me.
>

I just tried again, and on my machine with an optimized build C-s </ RET 
takes 220 seconds to finish (and cannot be interrupted).  That's what I 
call "hang".  With 350e97d78e, C-s </ RET takes about 1 second.  That's 
what I call "slow" (without truncate-lines C-s </ RET is instantaneous).

>
> If you disable lazy-highlighting, does the problem go away?
>

Yes, as I said to Gerd, with (setq isearch-lazy-highlight nil) the problem 
goes away (and C-s </ RET is instantaneous).

>
> And if you customize lazy-highlight-interval and 
> lazy-highlight-max-at-a-time to reasonable values, doesn't the "hang" 
> becomes much shorter?
>

With (setq lazy-highlight-interval 5) the problem does not go away.  With 
(setq lazy-highlight-max-at-a-time 10) the problem goes away, C-s </ RET 
takes about 1 second.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 26 Jul 2022 12:34:30 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> > I don't know the difference between "slow" and "hang".  Gerd says it 
> > eventually finishes, so it isn't a "hang" at least on his machine. And 
> > my build is unoptimized, so what is "slow" for you probably qualifies as 
> > "hang" for me.
> 
> I just tried again, and on my machine with an optimized build C-s </ RET 
> takes 220 seconds to finish (and cannot be interrupted).  That's what I 
> call "hang".  With 350e97d78e, C-s </ RET takes about 1 second.  That's 
> what I call "slow" (without truncate-lines C-s </ RET is instantaneous).

OK, I will take a look soon.  That change makes us use a branch of
code that AFAIU was never used since it was written, so some kind of
trouble can be expected.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

What about the following idea, in general.  Sorry if that is in the
wrong sub-thread, I'm beginning to loose orientation a bit.

While iterating,

- remmember where we started, say it's 'start_pos'.

- when reaching start_pos + X, without reaching a line end, refuse to do
  any complex stuff, no faces, invisible text and so on.

I could think of numerous variations of that theme.

I just loaded long-lines.xml in VSCode, and it seems to do something
like that.  It says that "tokenization has been disabled" in the long
line.  The long line doesn't have the highlighting that the shorter ones
have, and VSCode also wraps the long line instead of truncating it.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: Gregory Heytings <gregory@heytings.org>,  56682@debbugs.gnu.org,
>   monnier@iro.umontreal.ca
> Date: Tue, 26 Jul 2022 15:08:18 +0200
> 
> What about the following idea, in general.  Sorry if that is in the
> wrong sub-thread, I'm beginning to loose orientation a bit.
> 
> While iterating,
> 
> - remmember where we started, say it's 'start_pos'.
> 
> - when reaching start_pos + X, without reaching a line end, refuse to do
>   any complex stuff, no faces, invisible text and so on.
> 
> I could think of numerous variations of that theme.
> 
> I just loaded long-lines.xml in VSCode, and it seems to do something
> like that.  It says that "tokenization has been disabled" in the long
> line.  The long line doesn't have the highlighting that the shorter ones
> have, and VSCode also wraps the long line instead of truncating it.

We keep this possibility in mind all the time, you can see it in, for
example, the discussion whether to turn off line truncation.  But, as
I explained a minute ago, I think we haven't yet exhausted less
drastic measures, and there could still be opportunities for speedup
in the basic display code or thereabouts.  Our display is different
from that of VSCode, so the trade-offs are different as well.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> Date: Tue, 26 Jul 2022 15:41:49 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > I just tried again, and on my machine with an optimized build C-s </ RET 
> > takes 220 seconds to finish (and cannot be interrupted).  That's what I 
> > call "hang".  With 350e97d78e, C-s </ RET takes about 1 second.  That's 
> > what I call "slow" (without truncate-lines C-s </ RET is instantaneous).
> 
> OK, I will take a look soon.  That change makes us use a branch of
> code that AFAIU was never used since it was written, so some kind of
> trouble can be expected.

Heh, it's a case of "better is worse".  There's nothing wrong with the
change in commit 350e97d78e: it does indeed make redisplay of
long-and-truncated lines significantly faster.  And that speedup is
what's causing the problem: because the initial redisplay of the
buffer, before the results of C-s are displayed, is much faster, the
lazy-highlighting starts very soon, and it inserts thousands of
overlays into the buffer (because it by default highlights all the
matches till window-end, which is EOB in this case).  And once it
inserts enough overlays, we are dead in the water: the shortcut in
forward_to_next_line_start can no longer be taken, and we instead
iterate the slow way, one character at a time, all the way to EOB, and
on top of that need on the way to examine every one of the overlays
inserted by lazy-highlighting.

If I revert commit 350e97d78e, redisplay of long-and-truncated lines
becomes much slower, but because of that, lazy-highlighting doesn't
have time to highlight anything before you hit C-s for repeated
search, and thus there are no overlays in the buffer, and on the
average the responses are faster.

I tried to limit the lazy-highlighting by long-line-threshold, but the
speedup due to that is not significant enough (because even a single
overlay in the long line disables the shortcut in
forward_to_next_line_start).  Only limiting
lazy-highlight-initial-delay to, say, 1 sec and
lazy-highlight-max-at-a-time to a small number (like 2 or 5) makes the
problem go away because lazy-highlighting doesn't have enough time to
highlight too much.  And even then once you go deep enough into the
file, the problem comes back.

At this point, I think the only way to produce reasonable performance
from C-s in this case is to turn off lazy-highlighting when lines are
truncated and the buffer has long lines.  Which means we need to
expose the value of long_line_optimizations_p to Lisp, via an accessor
function.  I already have one other use for this: "C-x =", which
attempts to report the column of the character, something that is very
slow in a long-and-truncated line.  And I think we will see more cases
where Lisp code needs to know about this in order to adapt itself to
long lines.

Comments?

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 26.07.2022 14:45, Eli Zaretskii wrote:
>> Perhaps it's due to font-lock logic in that it has to match from the
>> beginning of a line (not sure we'd want to abandon that promise,
>> though). Or maybe something else.
> It isn't font-lock, at least not in all major modes.  It's the display
> engine itself that sometimes needs to go to the beginning of the line.
> When it does, going back gets slower with font-lock than without.
> This is why you see slower redisplay when you go deeper into a long
> line.

Makese sense: the text only has to be fontified once, but all redisplays 
are slowed down (on master), not just the first one.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Heh, it's a case of "better is worse".  There's nothing wrong with the 
> change in commit 350e97d78e: it does indeed make redisplay of 
> long-and-truncated lines significantly faster.  And that speedup is 
> what's causing the problem: because the initial redisplay of the buffer, 
> before the results of C-s are displayed, is much faster, the 
> lazy-highlighting starts very soon, and it inserts thousands of overlays 
> into the buffer (because it by default highlights all the matches till 
> window-end, which is EOB in this case).  And once it inserts enough 
> overlays, we are dead in the water: the shortcut in 
> forward_to_next_line_start can no longer be taken, and we instead 
> iterate the slow way, one character at a time, all the way to EOB, and 
> on top of that need on the way to examine every one of the overlays 
> inserted by lazy-highlighting.
>

Thanks; as far as I can see your analysis is correct.  At 304e2a3a05, if I 
press C-s </ and wait long enough (a few seconds are enough), 
lazy-highlighting has enough time to put enough overlays in the buffer to 
hang Emacs.

>
> At this point, I think the only way to produce reasonable performance 
> from C-s in this case is to turn off lazy-highlighting when lines are 
> truncated and the buffer has long lines.  Which means we need to expose 
> the value of long_line_optimizations_p to Lisp, via an accessor 
> function.  I already have one other use for this: "C-x =", which 
> attempts to report the column of the character, something that is very 
> slow in a long-and-truncated line.  And I think we will see more cases 
> where Lisp code needs to know about this in order to adapt itself to 
> long lines.
>

My conclusion is different: we will see more such cases, but only with 
truncate-lines enabled, and that means that truncate-lines should be 
disabled in such buffers.

The fundamental problem is that with truncate-lines we cannot really 
narrow the buffer to a smaller (contiguous) portion without creating 
problems.  What would be necessary would be some kind of non-contiguous 
narrowing.  If we are on line 1, on column 10000, of a buffer with 80 
lines each of which is 20000 characters wide, what we see on screen are 
characters 9960-10040, 29960-30040, 49960-50040, 69960-70040, and so 
forth.  Without such a non-continguous narrowing, all kinds of problems 
like the C-s and C-x = one you identified will appear.  These problems can 
indeed, in principle, be solved by adding local fixes depending on 
(long-line-optimizations-p) whenever we encounter such a bug, but doing 
that would be most regrettable, as the simplicity of these optimizations 
would be largely lost.  Doing that while knowing that, as I said, we'll 
hit another ceil very soon is clearly (to me at least) not TRT.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-07-25 21:23:55] wrote:
>> Before using a blunt tool like the forced-narrowing now in
>> `feature/long-lines-and-font-locking`, I think we should try and figure
>> out *why* the recipe below is so slow.
> It's not a blunt tool, it's an appropriate tool to help making sure that
> Emacs remains responsive when large files are visited.

I'm not opposed to reducing the size of the text that's considered, but
doing it via narrowing is a blunt tool.

> Think of it as POSIX's ulimits.

That's also a blunt tool.

> Allowing fontification-functions to search for arbitrarily complex
> regexpes in an arbitrarily large buffer, each and every time they are
> asked to to highlight a small chunk of the said buffer, is a recipe
> for disaster.

`font-lock.el` could enforce a smaller scope in a more discerning way
that narrowing can.

> If for some reason modes really need to go through to the whole
> buffer to decide which highlighting to use, they should to do so outside of
> fontification-functions, and ideally once, for example, when the file
> is loaded.

With the current narrowing they can't even know why the buffer is
narrowed and hence can't make an informed decision whether they should
maybe widen to look elsewhere.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> Date: Tue, 26 Jul 2022 20:46:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> At this point, I think the only way to produce reasonable performance
> from C-s in this case is to turn off lazy-highlighting when lines are
> truncated and the buffer has long lines.  Which means we need to
> expose the value of long_line_optimizations_p to Lisp, via an accessor
> function.  I already have one other use for this: "C-x =", which
> attempts to report the column of the character, something that is very
> slow in a long-and-truncated line.  And I think we will see more cases
> where Lisp code needs to know about this in order to adapt itself to
> long lines.

Actually, I might have one more idea that could perhaps resolve this
more nicely.

Gerd, the search for display properties and overlays in
forward_to_next_line_start, which disables the 'reseat' shortcut, is
because these could have strings as values, and those strings could
have embedded newlines, right?  Or are there some other reasons?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 26 Jul 2022 20:55:05 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> My conclusion is different: we will see more such cases, but only with 
> truncate-lines enabled, and that means that truncate-lines should be 
> disabled in such buffers.

I think these two issues are almost orthogonal.  Even if we decide to
disable truncate-lines in such buffers, we won't prevent users from
enabling it.  And if and when they enable it, we'd still like to give
them the best performance we can.  So speeding up the performance when
truncate-lines is enabled is still a worthy goal, although if that is
disabled in long-line buffers, that goal becomes less important.

> The fundamental problem is that with truncate-lines we cannot really 
> narrow the buffer to a smaller (contiguous) portion without creating 
> problems.

Yes, we need to find speedup opportunities that aren't solved by
narrowing.  But what I see for now is that the main bottleneck when
lines are truncated is reseat_at_next_visible_line_start, because we
call it each time we reach the right edge of a window, and need to
decide where to display the next screen line.  I've sped that up with
recent commits on the branch, but it is still very slow in some
situations, such as the one with isearch-lazy-highlight.  Another
situation which slows it down tremendously is when show-paren-mode
(which is ON by default nowadays) has a highlighted parenthesis
somewhere in the portion of the buffer outside of the viewport, the
part that reseat_at_next_visible_line_start needs to traverse to get
to the next newline.

I'm trying to speed up these situations as much as possible, and I
think it will serve us well even if we decide to turn off
truncate-lines in long-line buffers.

bug#56682: Fix the long lines font locking related slowdowns

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> Gerd, the search for display properties and overlays in
> forward_to_next_line_start, which disables the 'reseat' shortcut, is
> because these could have strings as values, and those strings could
> have embedded newlines, right?  Or are there some other reasons?

Correct, and I don't think there are other reasons.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> Before using a blunt tool like the forced-narrowing now in 
>>> `feature/long-lines-and-font-locking`, I think we should try and 
>>> figure out *why* the recipe below is so slow.
>>
>> It's not a blunt tool, it's an appropriate tool to help making sure 
>> that Emacs remains responsive when large files are visited.
>
> I'm not opposed to reducing the size of the text that's considered, but 
> doing it via narrowing is a blunt tool.
>

It isn't.  The only way to make sure that the size of the text is small 
enough is a forced narrowing from which fontification-functions cannot 
escape.

But let's try to be constructive.  You tell me that you're not opposed to 
reducing the size of the text, and that font-lock.el could enforce a 
smaller scope.  So could you please design a (Elisp) function (in 
font-lock.el) which, given a (beg end) with beg <= end in a buffer, 
returns a (beg' end') with beg <= beg' <= end' <= end that are better 
starting end end points for the forced narrowing?  That function would be 
run in handle_fontified_prop, before fontification-functions, and would 
have access to the whole buffer.

>> Think of it as POSIX's ulimits.
>
> That's also a blunt tool.
>

It isn't either.  It's a practical way to limit what a single process can 
do to make sure that what it does doesn't impact other running processes. 
At $job, each time I've seen a ulimit reached (usually the limit on open 
file descriptors), it was because of a bug.  I think I've seen a single 
exception, with a program that sometimes really needed to open more 
(temporary) files.  And even in that case the solution was not to remove 
the limit, but to raise it (from 1024 to 4096 IIRC).

>
> With the current narrowing they can't even know why the buffer is 
> narrowed and hence can't make an informed decision whether they should 
> maybe widen to look elsewhere.
>

There is no point to make such a decision, because they can't escape the 
narrowing.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> But I'm seeing incorrect fontification. Is this one expected?
>

Yes, occasional mis-fontification is expected.  It's a compromise between 
"no fontification" and "slow fontification".

>
> Perhaps something to do with the number 40000?
>

There is no magical number 40000 in the implementation, the buffer limits 
to which fontification-functions are constrained are determined 
dynamically, depending on the width and height of the window.

My guess in this specific case is that the first instance of 
"Downloadify.Container" was fontified by the previous call to 
fontification-functions, and that the next chunk of text in which the two 
other instances of "Downloadify.Container" are contained was fontified by 
the next call to fontification-functions, which did not have access 
anymore to the place where Downloadify.Container is defined.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Even if we decide to disable truncate-lines in such buffers, we won't 
> prevent users from enabling it.
>

That's correct, but if they consciously decide to shoot themselves in the 
foot, that's not Emacs' responsibility anymore.

>
> And if and when they enable it, we'd still like to give them the best 
> performance we can.  So speeding up the performance when truncate-lines 
> is enabled is still a worthy goal, although if that is disabled in 
> long-line buffers, that goal becomes less important.
>

I fully agree with that.

>
> I'm trying to speed up these situations as much as possible, and I think 
> it will serve us well even if we decide to turn off truncate-lines in 
> long-line buffers.
>

Thanks!

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  56682@debbugs.gnu.org
> Date: Tue, 26 Jul 2022 17:17:29 -0400
> 
> > Allowing fontification-functions to search for arbitrarily complex
> > regexpes in an arbitrarily large buffer, each and every time they are
> > asked to to highlight a small chunk of the said buffer, is a recipe
> > for disaster.
> 
> `font-lock.el` could enforce a smaller scope in a more discerning way
> that narrowing can.
> 
> > If for some reason modes really need to go through to the whole
> > buffer to decide which highlighting to use, they should to do so outside of
> > fontification-functions, and ideally once, for example, when the file
> > is loaded.
> 
> With the current narrowing they can't even know why the buffer is
> narrowed and hence can't make an informed decision whether they should
> maybe widen to look elsewhere.

Feel free to suggest better ways of handling these issues, or even
ways to solve this entirely inside font-lock.  If and when such
suggestions materialize, I'm sure we will be glad to use them instead
of less elegant/more direct solutions.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: gregory@heytings.org,  56682@debbugs.gnu.org,  monnier@iro.umontreal.ca
> Date: Wed, 27 Jul 2022 08:24:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Gerd, the search for display properties and overlays in
> > forward_to_next_line_start, which disables the 'reseat' shortcut, is
> > because these could have strings as values, and those strings could
> > have embedded newlines, right?  Or are there some other reasons?
> 
> Correct, and I don't think there are other reasons.

Thanks, I installed an optimization, which makes Isearch faster in
such cases.

Turning off isearch-lazy-highlight in these cases is still a good
idea, though.

The next problem with long-and-truncated lines is that in many places
as part of redisplay we begin from the line's beginning and then go to
the position of point or the first visible X coordinate or somesuch.
This becomes slow when point is very far to the right, like near the
end of the line.  A typical place in the code where this happens is at
beginning of display_line, where we call move_it_in_display_line_to to
get to it->first_visible_x.  This could take seconds if there are many
overlays on the line, which is what happens, for example, if we are in
the middle of Isearch and isearch-lazy-highlight is turned ON.
(Searching for "</" in long-line.xml produces 9K overlays!)

I'll try to think how to speed up these move_it_* calls in those
cases.  Ideas welcome.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 27.07.2022 09:55, Gregory Heytings wrote:
> 
>>
>> But I'm seeing incorrect fontification. Is this one expected?
>>
> 
> Yes, occasional mis-fontification is expected.  It's a compromise 
> between "no fontification" and "slow fontification".

I wonder now if the majority of the slowdown was caused by the 
redisplay, whereas font-lock (which only has to run once per screenful) 
was actually "fast enough".

>> Perhaps something to do with the number 40000?
>>
> 
> There is no magical number 40000 in the implementation, the buffer 
> limits to which fontification-functions are constrained are determined 
> dynamically, depending on the width and height of the window.
> 
> My guess in this specific case is that the first instance of 
> "Downloadify.Container" was fontified by the previous call to 
> fontification-functions, and that the next chunk of text in which the 
> two other instances of "Downloadify.Container" are contained was 
> fontified by the next call to fontification-functions, which did not 
> have access anymore to the place where Downloadify.Container is defined.

Could you clarify what you mean by "access ... to the place where ... is 
defined"? "new Downloadify.Container" is highlighted by a regular regexp 
matcher, not some custom elisp code which has to visit the position 
where the identifier is defined.

Same goes for the tokens like "null", "function" and "return", but those 
do get fontified after position 40000 in this example.

And the way those rules get applied doesn't seem particularly different, 
it's just that the keyword matcher goes before the class instantiation 
matcher inside js--font-lock-keywords-3.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 28 Jul 2022 00:38:09 +0300
> Cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Yes, occasional mis-fontification is expected.  It's a compromise 
> > between "no fontification" and "slow fontification".
> 
> I wonder now if the majority of the slowdown was caused by the 
> redisplay, whereas font-lock (which only has to run once per screenful) 
> was actually "fast enough".

To establish that, you can run font-lock-fontify-buffer, wait until it
finishes, and then try moving around and comparing that with a buffer
in which font-lock was turned off.  (Do this on master, of course,
where the changes we are discussing are not yet installed.)

In general, even after a full fontification, you should see at least
some slowdown due to the faces.  How much slowdown, quantitatively, I
don't know, but perhaps Gregory measured that as part of his work.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]


>> Yes, occasional mis-fontification is expected.  It's a compromise 
>> between "no fontification" and "slow fontification".
>
> I wonder now if the majority of the slowdown was caused by the 
> redisplay, whereas font-lock (which only has to run once per screenful) 
> was actually "fast enough".
>

Those two statements are not mutually exclusive.  The majority of the 
slowdown was indeed caused by redisplay, but font-lock was not fast 
enough.  Try to open a sufficiently large file (e.g. the dictionary.json 
one) with the code on master, and type M->.  You'll see that Emacs needs 
about five seconds (on my laptop) to display the end of the buffer.  Now 
compare that with the feature branch, with which the end of the buffer is 
displayed instantaneously.  That five seconds delay is caused by 
fontification-functions.

>
> Could you clarify what you mean by "access ... to the place where ... is 
> defined"? "new Downloadify.Container" is highlighted by a regular regexp 
> matcher, not some custom elisp code which has to visit the position 
> where the identifier is defined.
>

Sorry, I cannot be more precise, I don't have the "downloadify.js" file 
here.  It was just a guess, based on what I saw on the screenshot, that 
one function called by fontification-functions collects all class 
definitions and highlights their identifiers elsewhere in the buffer with 
a specific face.  When the buffer is narrowed, that function may not see 
the Downloadify.Container definition (which is, I guess, placed near the 
beginning of the file) anymore.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Thanks, I installed an optimization, which makes Isearch faster in such 
> cases.
>

Thanks.  Alas, as far as I can see, we're still quite far from a usable 
Emacs with long-and-truncated lines.  A recipe:

emacs -Q
C-x C-f long-line.xml RET
M-g g 12
C-u 12 M-x duplicate-line
C-x C-s multiple-long-lines.xml RET
C-x C-f multiple-long-lines.xml RET y
C-x x t
C-s </ RET

This last step is not yet finished after 10 minutes (600 seconds), and you 
cannot abort it.

>
> Turning off isearch-lazy-highlight in these cases is still a good idea, 
> though.
>

Indeed, (setq isearch-lazy-highlight nil) "fixes" the above recipe (but 
see below).

>
> The next problem with long-and-truncated lines is that in many places as 
> part of redisplay we begin from the line's beginning and then go to the 
> position of point or the first visible X coordinate or somesuch. This 
> becomes slow when point is very far to the right, like near the end of 
> the line.  A typical place in the code where this happens is at 
> beginning of display_line, where we call move_it_in_display_line_to to 
> get to it->first_visible_x.  This could take seconds if there are many 
> overlays on the line, which is what happens, for example, if we are in 
> the middle of Isearch and isearch-lazy-highlight is turned ON.
>

Indeed, Emacs isn't really usable.  For example, with 
multiple-long-lines.xml, after M-g g 20 C-e (which already takes several 
seconds to complete) each motion, search or insertion command still takes 
several seconds (even with find-file-literally).  For example (without 
isearch-lazy-highlight), C-s </releases> RET takes about 10 seconds. 
After that, C-s C-s takes another 10 seconds, an additional C-s also takes 
10 seconds, and C-g to abort isearch takes about 90 seconds.  And, even 
when you don't do anything, Emacs uses 100% of the CPU.  All this is, in 
the same file, instantaneous without truncate-lines.

>
> (Searching for "</" in long-line.xml produces 9K overlays!)
>

That's (another sign of) the fundamental problem I mentioned earlier. 
Without truncate-lines the visible portion of the buffer remains small, 
and it is thus possible to reduce the portion of the buffer that display 
routines will see.  With truncate-lines it can easily become huge.  For 
example, with multiple-long-lines.xml, (- (window-end) (window-start)) is 
7744464 with truncate-lines and 2454 without truncate-lines.  And 
multiple-long-lines.xml is still a relatively "small" file, it's only 15 
MB.

>
> I'll try to think how to speed up these move_it_* calls in those cases. 
> Ideas welcome.
>

I cannot claim I understand the display code enough, but I see no way to 
make the portion of the buffer that is considered by display routines 
smaller without introducing the kind of rectangular narrowing I mentioned 
earlier.  As far as I understand, everything else is just a band-aid, that 
will make Emacs behave a bit faster only in some cases, and with which 
Emacs will remain unresponsive in too many cases.

Perfect is the enemy of good.  I strongly suggest we just admit that Emacs 
can't cope with long lines and truncate-lines together, which is true 
anyway given the DISP_INFINITY limit.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> emacs -Q
> C-x C-f long-line.xml RET
> M-g g 12
> C-u 12 M-x duplicate-line
> C-x C-s multiple-long-lines.xml RET
> C-x C-f multiple-long-lines.xml RET y
> C-x x t
> C-s </ RET
>

Sorry, there's a typo in that recipe, it should be:

C-u 50 M-x duplicate-line

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 28 Jul 2022 16:29:44 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
>     56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> Indeed, Emacs isn't really usable.  For example, with 
> multiple-long-lines.xml, after M-g g 20 C-e (which already takes several 
> seconds to complete) each motion, search or insertion command still takes 
> several seconds (even with find-file-literally).  For example (without 
> isearch-lazy-highlight), C-s </releases> RET takes about 10 seconds. 
> After that, C-s C-s takes another 10 seconds, an additional C-s also takes 
> 10 seconds, and C-g to abort isearch takes about 90 seconds.  And, even 
> when you don't do anything, Emacs uses 100% of the CPU.  All this is, in 
> the same file, instantaneous without truncate-lines.

I think such long lines happen in files that have very little except
the long line.  So duplicating it makes the use case much less
important.  And in any case, we shouldn't try dealing with such files
before we have a good solution for a single-line files.

> > (Searching for "</" in long-line.xml produces 9K overlays!)
> 
> That's (another sign of) the fundamental problem I mentioned earlier. 
> Without truncate-lines the visible portion of the buffer remains small, 
> and it is thus possible to reduce the portion of the buffer that display 
> routines will see.  With truncate-lines it can easily become huge.  For 
> example, with multiple-long-lines.xml, (- (window-end) (window-start)) is 
> 7744464 with truncate-lines and 2454 without truncate-lines.  And 
> multiple-long-lines.xml is still a relatively "small" file, it's only 15 
> MB.

This is actually a bug in isearch.el: using window-end in a buffer
under truncate-lines is simply wrong.  I'm going to file a bug about
that soon.

But that's not a fundamental problem, or at least I don't yet see why
it would be fundamental.  It's something specific to isearch.el and to
how isearch-lazy-highlight is implemented.  The fix I installed
prevented us from walking all that long line till the end of the
buffer, so in effect the display code doesn't see most of the buffer
text, they just jump over it.

> > I'll try to think how to speed up these move_it_* calls in those cases. 
> > Ideas welcome.
> 
> I cannot claim I understand the display code enough, but I see no way to 
> make the portion of the buffer that is considered by display routines 
> smaller without introducing the kind of rectangular narrowing I mentioned 
> earlier.

I still have one or two ideas to try, and they don't involve anything
as complex as some new kind of narrowing.

> Perfect is the enemy of good.  I strongly suggest we just admit that Emacs 
> can't cope with long lines and truncate-lines together, which is true 
> anyway given the DISP_INFINITY limit.

Feel free to give up on it and stop trying to make that case faster,
but I'm not ready to give up yet.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I think such long lines happen in files that have very little except the 
> long line.  So duplicating it makes the use case much less important.
>

That's not correct.  Some files have indeed have a single long line, but 
others have more than one long line.  Duplicating the line in that file 
was just a quick way to generate a file with more long lines.  For 
example, a database dump will typically contain one long line for each 
table.  I have a database dump here with ~100 lines that are each about 1 
MB long, each of which is a single "insert into" statement.

>
> This is actually a bug in isearch.el: using window-end in a buffer under 
> truncate-lines is simply wrong.
>

The measurements I gave were with isearch, but as I said you see the same 
kind of slowdowns with motion and insertion commands.

>
> But that's not a fundamental problem, or at least I don't yet see why it 
> would be fundamental.  It's something specific to isearch.el and to how 
> isearch-lazy-highlight is implemented.
>

No, the measurements I gave were with isearch-lazy-highlight turned OFF. 
And it is the fundamental problem, because it means that all display 
routines have to deal with a very large amount of data.

>
> I still have one or two ideas to try, and they don't involve anything as 
> complex as some new kind of narrowing.
>

Okay, so I'll wait a bit more.  I'd like to reach a conclusion as to 
whether truncate-lines should be turned off when long_line_optimizations_p 
is on before merging the branch into master.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 28 Jul 2022 17:16:32 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> > This is actually a bug in isearch.el: using window-end in a buffer under 
> > truncate-lines is simply wrong.
> 
> The measurements I gave were with isearch, but as I said you see the same 
> kind of slowdowns with motion and insertion commands.

No, that's not what I see.  Without the thousands of Isearch overlays
Emacs works much faster, almost 2 orders of magnitude faster.

> No, the measurements I gave were with isearch-lazy-highlight turned OFF. 
> And it is the fundamental problem, because it means that all display 
> routines have to deal with a very large amount of data.

By multiplying a very long and truncated line enough times you can
always make Emacs useless.  The speedups I have in mind scale linearly
with the number of such lines, so eventually, with enough such lines,
Emacs will always become very slow at some point, especially if the
window is hscrolled very far to the right.  That doesn't mean we
shouldn't try speeding up the code: someone just told me that the
perfect is the enemy of the good.

> > I still have one or two ideas to try, and they don't involve anything as 
> > complex as some new kind of narrowing.
> 
> Okay, so I'll wait a bit more.  I'd like to reach a conclusion as to 
> whether truncate-lines should be turned off when long_line_optimizations_p 
> is on before merging the branch into master.

That's unrelated.  The branch was created for your work on font-lock,
and if you are done with that, feel free to land it on master.  I can
continue working on master, and/or will create a feature branch if I
feel it's justified.

Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> And it is the fundamental problem, because it means that all display 
>> routines have to deal with a very large amount of data.
>
> By multiplying a very long and truncated line enough times you can 
> always make Emacs useless.  The speedups I have in mind scale linearly 
> with the number of such lines, so eventually, with enough such lines, 
> Emacs will always become very slow at some point, especially if the 
> window is hscrolled very far to the right.
>

That's exactly my point: with long and truncated lines Emacs can become 
unusable with a 20 MB file, with long lines Emacs does not become unusable 
even with a 1 GB file.  And what I (and everybody I guess) wants is a 
"full and complete" solution.

>> Okay, so I'll wait a bit more.  I'd like to reach a conclusion as to 
>> whether truncate-lines should be turned off when 
>> long_line_optimizations_p is on before merging the branch into master.
>
> That's unrelated.  The branch was created for your work on font-lock, 
> and if you are done with that, feel free to land it on master.  I can 
> continue working on master, and/or will create a feature branch if I 
> feel it's justified.
>

It's not unrelated, at least not in my mind.  The branch was initially 
created to fix the remaining font-lock related issues, but this thread 
discusses, and the branch contains, fixes to the other remaining issues. 
I don't want to close this bug without a "full and complete" solution, and 
currently someone who has (setq truncate-lines t) in their init file or 
who presses C-x x t will see Emacs become unusable with a file with long 
lines.

What about the following course of action: I add a "disable truncate-lines 
in buffers with long lines" feature, and you remove it/disable it/make it 
optional later if/when you consider that your fixes make Emacs fast enough 
with long-and-truncated lines.  And we can/should open a new bug report 
and branch to discuss these long-and-truncated lines issues and solution 
attempts.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 28 Jul 2022 18:40:26 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> > By multiplying a very long and truncated line enough times you can 
> > always make Emacs useless.  The speedups I have in mind scale linearly 
> > with the number of such lines, so eventually, with enough such lines, 
> > Emacs will always become very slow at some point, especially if the 
> > window is hscrolled very far to the right.
> 
> That's exactly my point: with long and truncated lines Emacs can become 
> unusable with a 20 MB file, with long lines Emacs does not become unusable 
> even with a 1 GB file.  And what I (and everybody I guess) wants is a 
> "full and complete" solution.

Feel free to want it and feel free to wait for it, or work on it.  But
that doesn't mean I or others cannot meanwhile install less "full and
complete" solutions that improve some use cases.  And any user can
always turn off truncate-lines in such buffers, if some particular use
case makes Emacs unusable for them, it's not like this possibility is
taken from them.

> > That's unrelated.  The branch was created for your work on font-lock, 
> > and if you are done with that, feel free to land it on master.  I can 
> > continue working on master, and/or will create a feature branch if I 
> > feel it's justified.
> 
> It's not unrelated, at least not in my mind.  The branch was initially 
> created to fix the remaining font-lock related issues, but this thread 
> discusses, and the branch contains, fixes to the other remaining issues. 

I worked on truncated-lines case on the branch because without the
improvements you installed it was impossible to do anything with that:
Emacs was too slow to allow reasonable debugging and measurements.
That is the only relation between the two.

I don't see any reason to delay landing the important improvements we
have on the branch.  It will allow more people to use those
improvements, and thus will contribute both to their stability and to
further progress in this direction.

> I don't want to close this bug without a "full and complete" solution, and 
> currently someone who has (setq truncate-lines t) in their init file or 
> who presses C-x x t will see Emacs become unusable with a file with long 
> lines.

The bug can be kept open, if you don't want to close it.  I don't
mind.

> What about the following course of action: I add a "disable truncate-lines 
> in buffers with long lines" feature, and you remove it/disable it/make it 
> optional later if/when you consider that your fixes make Emacs fast enough 
> with long-and-truncated lines.

No.  Please stop pressuring me into making a decision I'm not yet
ready to make.  There's absolutely no rush.

> And we can/should open a new bug report and branch to discuss these
> long-and-truncated lines issues and solution attempts.

Opening a new bug about that is fine by me.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I don't see any reason to delay landing the important improvements we 
> have on the branch.  It will allow more people to use those 
> improvements, and thus will contribute both to their stability and to 
> further progress in this direction.
>

Okay, I pushed two final (?) improvements: a new optional argument "lock" 
to narrow-to-region that Lars (added in Cc) suggested and which is now 
used to lock the restriction around fontification-functions, and two 
updates to the documentation.

Could you please tell me if it's ready to merge?

>> And we can/should open a new bug report and branch to discuss these 
>> long-and-truncated lines issues and solution attempts.
>
> Opening a new bug about that is fine by me.
>

Actually there's no need to open a new bug report, as bug#56683 already 
exists.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 28 Jul 2022 21:31:30 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Lars Ingebrigtsen <larsi@gnus.org>, gerd.moellmann@gmail.com, 
>     56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> Okay, I pushed two final (?) improvements: a new optional argument "lock" 
> to narrow-to-region that Lars (added in Cc) suggested and which is now 
> used to lock the restriction around fontification-functions, and two 
> updates to the documentation.

Thanks, I made a few minor documentation changes there.

> Could you please tell me if it's ready to merge?

Yes, I think it's ready, thanks for all your hard work on this.

> >> And we can/should open a new bug report and branch to discuss these 
> >> long-and-truncated lines issues and solution attempts.
> >
> > Opening a new bug about that is fine by me.
> >
> 
> Actually there's no need to open a new bug report, as bug#56683 already 
> exists.

Right.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Could you please tell me if it's ready to merge?
>
> Yes, I think it's ready, thanks for all your hard work on this.
>

Now done, and closing this bug.  I made a few further documentation 
changes, in particular "arbitrarily long" is important to me, factually 
correct, and correct English (see e.g. 
https://en.wikipedia.org/wiki/Arbitrarily_large ).

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 08:33:38 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682-done@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> >> Could you please tell me if it's ready to merge?
> >
> > Yes, I think it's ready, thanks for all your hard work on this.
> >
> 
> Now done, and closing this bug.

Thanks.

> I made a few further documentation changes, in particular
> "arbitrarily long" is important to me, factually correct, and
> correct English (see e.g.
> https://en.wikipedia.org/wiki/Arbitrarily_large ).

Please always assume that I gave these aspects and your perspective
due consideration before making my changes, and please never revert
them without discussing first.

In this case, "arbitrarily large" contradicts the text that follows,
which describes the circumstances where that might not be true.  Other
minor changes are due to simple humility: we shouldn't boast
achievements that have yet to see serious real-world testing.

In any case, the final responsibility for what the documentation says
and how is Lars's and mine, as the current project maintainers.

Thanks again for all the efforts invested in these important
improvements.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Please always assume that I gave these aspects and your perspective due 
> consideration before making my changes, and please never revert them 
> without discussing first.
>

Okay, I'll keep that in mind in the future.

>
> In this case, "arbitrarily large" contradicts the text that follows, 
> which describes the circumstances where that might not be true.
>

I've carefully chosen the words of the title, and it doesn't contradict 
what follows, as far as I understand.  It says "Emacs is now capable of 
editing files with arbitrarily long lines", in which "capable" means that 
it can do it, but will not always do it.  The circumstances that are 
described in the text that follows tell the reader that the remaining 
cases in which Emacs would choke on such files are outside of Emacs' 
responsibility, they are the responsibility of major and minor mode 
writers.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> In this case, "arbitrarily large" contradicts the text that follows, 
>> which describes the circumstances where that might not be true.
>
> I've carefully chosen the words of the title, and it doesn't contradict 
> what follows, as far as I understand.  It says "Emacs is now capable of 
> editing files with arbitrarily long lines", in which "capable" means 
> that it can do it, but will not always do it.  The circumstances that 
> are described in the text that follows tell the reader that the 
> remaining cases in which Emacs would choke on such files are outside of 
> Emacs' responsibility, they are the responsibility of major and minor 
> mode writers.
>

I see that you've already reverted.  Sigh.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 10:44:27 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> > Please always assume that I gave these aspects and your perspective due 
> > consideration before making my changes, and please never revert them 
> > without discussing first.
> 
> Okay, I'll keep that in mind in the future.

Thank you.

> > In this case, "arbitrarily large" contradicts the text that follows, 
> > which describes the circumstances where that might not be true.
> 
> I've carefully chosen the words of the title, and it doesn't contradict 
> what follows, as far as I understand.  It says "Emacs is now capable of 
> editing files with arbitrarily long lines", in which "capable" means that 
> it can do it, but will not always do it.  The circumstances that are 
> described in the text that follows tell the reader that the remaining 
> cases in which Emacs would choke on such files are outside of Emacs' 
> responsibility, they are the responsibility of major and minor mode 
> writers.

The usual interpretation of "capable" in Emacs is that we do it unless
the user tells us not to.  Otherwise users will ask why not do it
whenever possible, definitely for a feature like this one.  So that is
the contradiction which I had in mind.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> The usual interpretation of "capable" in Emacs is that we do it unless 
> the user tells us not to.  Otherwise users will ask why not do it 
> whenever possible, definitely for a feature like this one.  So that is 
> the contradiction which I had in mind.
>

I understand.  Nonetheless, the OED tells me "capable: adj. 1 (capable of 
doing something) having the ability or quality necessary to do something."

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 10:50:02 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> 
> >> In this case, "arbitrarily large" contradicts the text that follows, 
> >> which describes the circumstances where that might not be true.
> >
> > I've carefully chosen the words of the title, and it doesn't contradict 
> > what follows, as far as I understand.  It says "Emacs is now capable of 
> > editing files with arbitrarily long lines", in which "capable" means 
> > that it can do it, but will not always do it.  The circumstances that 
> > are described in the text that follows tell the reader that the 
> > remaining cases in which Emacs would choke on such files are outside of 
> > Emacs' responsibility, they are the responsibility of major and minor 
> > mode writers.
> 
> I see that you've already reverted.  Sigh.

I didn't revert.  I've changed back only some parts of the changes you
made, and left others as you changed them.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>>> In this case, "arbitrarily large" contradicts the text that follows, 
>>>> which describes the circumstances where that might not be true.
>>>
>>> I've carefully chosen the words of the title, and it doesn't 
>>> contradict what follows, as far as I understand.  It says "Emacs is 
>>> now capable of editing files with arbitrarily long lines", in which 
>>> "capable" means that it can do it, but will not always do it.  The 
>>> circumstances that are described in the text that follows tell the 
>>> reader that the remaining cases in which Emacs would choke on such 
>>> files are outside of Emacs' responsibility, they are the 
>>> responsibility of major and minor mode writers.
>>
>> I see that you've already reverted.  Sigh.
>
> I didn't revert.  I've changed back only some parts of the changes you 
> made, and left others as you changed them.
>

You did revert the only change of which I said it was important to me, and 
without discussing it: the NEWS title "Emacs is now capable of editing 
files with arbitrarily long lines."  Which isn't boasting about an 
achievement, but an accurate statement.  Adding "unlike all other editors 
out there" would have been boasting about an achievement (but an accurate 
statement, too).

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 12:05:16 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> > I didn't revert.  I've changed back only some parts of the changes you 
> > made, and left others as you changed them.
> 
> You did revert the only change of which I said it was important to me

I understand that it's important to you, and understood that the first
time, but I cannot let our documentation say things that I'm unable to
defend in good faith.  When the dust settles and all is said and done,
it is only us the maintainers who are left to defend the decisions we
made as a project.  Please try to see this from our POV as well.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 08:33:38 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682-done@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> Now done, and closing this bug.

Hmm...  I'm bothered by this code in handle_fontified_prop:

      if (it->narrowed_begv)
	Fnarrow_to_region (make_fixnum (it->narrowed_begv),
			   make_fixnum (it->narrowed_zv), Qt);

This narrows the buffer around window's point position (since this is
how narrowed_begv and narrowed_zv are computed), but the display
iterator can be called for position outside this range.  This is
unlikely to happen when the function is called as part of actual
redisplay of a window, but it can easily happen when the display code
is used by other primitives, for example vertical-motion or
pos-visible-in-window-p.  What happens then is that
fontification-functions are called with the argument POS that is
outside of the restriction, and that can cause errors.  (jit-lock
simply does nothing in that case, AFAICT.)

Is this intended?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: gerd.moellmann@gmail.com, 56682-done@debbugs.gnu.org, larsi@gnus.org,
>  monnier@iro.umontreal.ca
> Date: Fri, 29 Jul 2022 16:27:50 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Hmm...  I'm bothered by this code in handle_fontified_prop:
> 
>       if (it->narrowed_begv)
> 	Fnarrow_to_region (make_fixnum (it->narrowed_begv),
> 			   make_fixnum (it->narrowed_zv), Qt);
> 

And another thing: the condition on it->narrowed_begv being non-zero
means that as long as we are close enough to the beginning of a
buffer, we don't restrict fontification-functions from going as far as
they want into the buffer.

So I think the condition should be the long_line_optimizations_p flag
of the buffer, and we should narrow the buffer even when we are at
BOB, to prevent fontification-functions from going too far.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Hmm...  I'm bothered by this code in handle_fontified_prop:
>
> if (it->narrowed_begv)
>   Fnarrow_to_region (make_fixnum (it->narrowed_begv),
>                      make_fixnum (it->narrowed_zv), Qt);
>

Hmmm... that code has been on the branch for a week, and I didn't see 
particular bugs.  Which doesn't mean there are none, of course.

>
> This narrows the buffer around window's point position (since this is 
> how narrowed_begv and narrowed_zv are computed), but the display 
> iterator can be called for position outside this range. This is unlikely 
> to happen when the function is called as part of actual redisplay of a 
> window, but it can easily happen when the display code is used by other 
> primitives, for example vertical-motion or pos-visible-in-window-p. 
> What happens then is that fontification-functions are called with the 
> argument POS that is outside of the restriction, and that can cause 
> errors.  (jit-lock simply does nothing in that case, AFAICT.)
>
> Is this intended?
>

I'm not sure I understand how this could happen.  Can a non-visible part 
of the buffer be fontified by fontification-functions when for example 
pos-visible-in-window-p is called and eventually returns nil?  At least if 
I do (pos-visible-in-window-p (point-max)), they are not: 
handle_fontified_prop is not even called with it at point-max.  Even with 
(pos-visible-in-window-p (1+ (window-end))) fontification-functions are 
not called.

Should we perhaps be extra careful and add not apply the narrowing when 
IT_CHARPOS is not between narrowed_begv and narrowed_zv?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 15:19:49 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682-done@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> > Hmm...  I'm bothered by this code in handle_fontified_prop:
> >
> > if (it->narrowed_begv)
> >   Fnarrow_to_region (make_fixnum (it->narrowed_begv),
> >                      make_fixnum (it->narrowed_zv), Qt);
> >
> 
> Hmmm... that code has been on the branch for a week, and I didn't see 
> particular bugs.  Which doesn't mean there are none, of course.

It depends on what kind of bugs we expect.  I've seen in a debugger
that fontification-functions are called with POS outside the narrowed
region.  But since jit-lock simply does nothing in that case, this is
silently ignored.

> > This narrows the buffer around window's point position (since this is 
> > how narrowed_begv and narrowed_zv are computed), but the display 
> > iterator can be called for position outside this range. This is unlikely 
> > to happen when the function is called as part of actual redisplay of a 
> > window, but it can easily happen when the display code is used by other 
> > primitives, for example vertical-motion or pos-visible-in-window-p. 
> > What happens then is that fontification-functions are called with the 
> > argument POS that is outside of the restriction, and that can cause 
> > errors.  (jit-lock simply does nothing in that case, AFAICT.)
> >
> > Is this intended?
> 
> I'm not sure I understand how this could happen.  Can a non-visible part 
> of the buffer be fontified by fontification-functions when for example 
> pos-visible-in-window-p is called and eventually returns nil?

Yes.  Whenever the display code is called, it fontifies the portions
of the buffer that it moves through.  And that is in general
justified: if we didn't fontify, we could produce wrong results from
pos-visible-in-window-p, due to faces that change the font height.

> At least if I do (pos-visible-in-window-p (point-max)), they are
> not: handle_fontified_prop is not even called with it at point-max.
> Even with (pos-visible-in-window-p (1+ (window-end)))
> fontification-functions are not called.

Try with vertical-motion.  Visit long-line.xml, go to position 20000,
and then do "C-u 200 C-n" or "M-: (vertical-motion 200) RET.  Sooner
or later you will see that it->current in handle_fontified_prop will
be outside of the narrowing.

> Should we perhaps be extra careful and add not apply the narrowing when 
> IT_CHARPOS is not between narrowed_begv and narrowed_zv?

I'd rather narrow around IT_CHARPOS in that case.  That would be also
consistent with what the doc string of fontification-functions now
says.

Perhaps we should also change what init_iterator does: if the start
position with which it's called is outside of the restriction,
recompute the restriction using the start point instead of the
window's point position.  WDYT?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> And another thing: the condition on it->narrowed_begv being non-zero 
> means that as long as we are close enough to the beginning of a buffer, 
> we don't restrict fontification-functions from going as far as they want 
> into the buffer.
>
> So I think the condition should be the long_line_optimizations_p flag of 
> the buffer, and we should narrow the buffer even when we are at BOB, to 
> prevent fontification-functions from going too far.
>

That's correct, indeed, thanks.  Now fixed on master.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> At least if I do (pos-visible-in-window-p (point-max)), they are not: 
>> handle_fontified_prop is not even called with it at point-max. Even 
>> with (pos-visible-in-window-p (1+ (window-end))) 
>> fontification-functions are not called.
>
> Try with vertical-motion.  Visit long-line.xml, go to position 20000, 
> and then do "C-u 200 C-n" or "M-: (vertical-motion 200) RET.  Sooner or 
> later you will see that it->current in handle_fontified_prop will be 
> outside of the narrowing.
>

Thanks, I was able to reproduce the bug with that recipe.

>> Should we perhaps be extra careful and add not apply the narrowing when 
>> IT_CHARPOS is not between narrowed_begv and narrowed_zv?
>
> I'd rather narrow around IT_CHARPOS in that case.  That would be also 
> consistent with what the doc string of fontification-functions now says.
>
> Perhaps we should also change what init_iterator does: if the start 
> position with which it's called is outside of the restriction, recompute 
> the restriction using the start point instead of the window's point 
> position.  WDYT?
>

Doing it in init_iterator is too early alas, with the above recipe at 
least init_iterator is called with charpos inside the narrowing bounds, 
after which the iterator moves outside the narrowing bounds.  So I fixed 
the bug in handle_fontified_prop.

I don't know yet if it's necessary to add another similar recomputation 
inside init_iterator.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 16:37:14 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> 
> >> At least if I do (pos-visible-in-window-p (point-max)), they are not: 
> >> handle_fontified_prop is not even called with it at point-max. Even 
> >> with (pos-visible-in-window-p (1+ (window-end))) 
> >> fontification-functions are not called.
> >
> > Try with vertical-motion.  Visit long-line.xml, go to position 20000, 
> > and then do "C-u 200 C-n" or "M-: (vertical-motion 200) RET.  Sooner or 
> > later you will see that it->current in handle_fontified_prop will be 
> > outside of the narrowing.
> >
> 
> Thanks, I was able to reproduce the bug with that recipe.
> 
> >> Should we perhaps be extra careful and add not apply the narrowing when 
> >> IT_CHARPOS is not between narrowed_begv and narrowed_zv?
> >
> > I'd rather narrow around IT_CHARPOS in that case.  That would be also 
> > consistent with what the doc string of fontification-functions now says.
> >
> > Perhaps we should also change what init_iterator does: if the start 
> > position with which it's called is outside of the restriction, recompute 
> > the restriction using the start point instead of the window's point 
> > position.  WDYT?
> >
> 
> Doing it in init_iterator is too early alas, with the above recipe at 
> least init_iterator is called with charpos inside the narrowing bounds, 
> after which the iterator moves outside the narrowing bounds.  So I fixed 
> the bug in handle_fontified_prop.

Thanks.

> I don't know yet if it's necessary to add another similar recomputation 
> inside init_iterator.

I'll play with other callers of init_iterator and start_display, and
see if they can do similar things.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> I don't know yet if it's necessary to add another similar recomputation 
>> inside init_iterator.
>
> I'll play with other callers of init_iterator and start_display, and see 
> if they can do similar things.
>

In case this helps, it is easy to enter the following conditional, for 
example with C-s, or with your previous M-g c 20000 and M-: 
(vertical-motion 200) RET recipe (in both steps).  It's not clear to me if 
updating the narrowing bounds there has an actual impact.  At least 
applying that change does not seem to have negative effects.

diff --git a/src/xdisp.c b/src/xdisp.c
index b1ee7889d4..e415320a52 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -3429,6 +3429,12 @@ init_iterator (struct it *it, struct window *w,
      {
        it->narrowed_begv = get_narrowed_begv (w, window_point (w));
        it->narrowed_zv = get_narrowed_zv (w, window_point (w));
+      if (charpos >= 0
+         && (charpos < it->narrowed_begv || charpos > it->narrowed_zv))
+       {
+         it->narrowed_begv = get_narrowed_begv (w, charpos);
+         it->narrowed_zv = get_narrowed_zv (w, charpos);
+       }
      }

    /* If a buffer position was specified, set the iterator there,

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Doing it in init_iterator is too early alas, with the above recipe at 
> least init_iterator is called with charpos inside the narrowing bounds, 
> after which the iterator moves outside the narrowing bounds.  So I fixed 
> the bug in handle_fontified_prop.
>

Actually that doesn't work correctly.  A recipe:

emacs -Q
M-: (progn (set-frame-width nil 119) (set-frame-height nil 38)) RET
C-x C-f dictionary.json RET y
C-s aan SPC

Now you'll see that the last line at the bottom of the window, which does 
not contain "aan ", is highlighted.  Is the following okay from your point 
of view?

diff --git a/src/xdisp.c b/src/xdisp.c
index b1ee7889d4..8c62f088b8 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4412,13 +4412,8 @@ handle_fontified_prop (struct it *it)
           ptrdiff_t begv = it->narrowed_begv ? it->narrowed_begv : BEGV;
           ptrdiff_t zv = it->narrowed_zv;
           ptrdiff_t charpos = IT_CHARPOS (*it);
-         if (charpos < begv || charpos > zv)
-           {
-             begv = get_narrowed_begv (it->w, charpos);
-             if (!begv) begv = BEGV;
-             zv = get_narrowed_zv (it->w, charpos);
-           }
-         Fnarrow_to_region (make_fixnum (begv), make_fixnum (zv), Qt);
+         if (begv <= charpos && charpos <= zv)
+           Fnarrow_to_region (make_fixnum (begv), make_fixnum (zv), Qt);
         }

        /* Don't allow Lisp that runs from 'fontification-functions'

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> At least applying that change does not seem to have negative effects.
>

Actually it does, with the same recipe:

emacs -Q
M-: (progn (set-frame-width nil 119) (set-frame-height nil 38)) RET
C-x C-f dictionary.json RET y
C-s aan SPC

So you can forget this patch.

>
> diff --git a/src/xdisp.c b/src/xdisp.c
> index b1ee7889d4..e415320a52 100644
> --- a/src/xdisp.c
> +++ b/src/xdisp.c
> @@ -3429,6 +3429,12 @@ init_iterator (struct it *it, struct window *w,
>     {
>       it->narrowed_begv = get_narrowed_begv (w, window_point (w));
>       it->narrowed_zv = get_narrowed_zv (w, window_point (w));
> +      if (charpos >= 0
> +         && (charpos < it->narrowed_begv || charpos > it->narrowed_zv))
> +       {
> +         it->narrowed_begv = get_narrowed_begv (w, charpos);
> +         it->narrowed_zv = get_narrowed_zv (w, charpos);
> +       }
>     }
>
>   /* If a buffer position was specified, set the iterator there,
>

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I'm not opposed to reducing the size of the text that's considered, but
>> doing it via narrowing is a blunt tool.
> It isn't.

There's no point arguing about it.  I find it to be and you don't and
that's that.  The reason why I find it to be is because it removes all
possibilities of making different choices for different elements
depending on the cost of those elements and the amount of
available information.

> The only way to make sure that the size of the text is small
> enough is a forced narrowing from which fontification-functions
> cannot escape.

It's clearly not the only way, nor is it an absolute guarantee.
I agree that it's a convenient way, tho.

> But let's try to be constructive.  You tell me that you're not opposed to
>  reducing the size of the text, and that font-lock.el could enforce
>  a smaller scope.  So could you please design a (Elisp) function (in
>  font-lock.el) which, given a (beg end) with beg <= end in a buffer, returns
>  a (beg' end') with beg <= beg' <= end' <= end that are better starting end
>  end points for the forced narrowing?  That function would be run in
>  handle_fontified_prop, before fontification-functions, and would have
> access to the whole buffer.

That can't be done.  What I think would be a better option is to
(somehow) pass the `beg..end` "limit" to jit-lock which can then pass it
on to its own clients (e.g. font-lock) so they each can make their
own choices.

E.g. the syntax-ppss part of the job performed by font-lock is heavily
cached, does not depends on lines, is theoretically always computed from
BOB but with a cache which makes it fast even when working near EOB (tho
it can still be somewhat slow when jumping from BOB to EOB, but that
depends on the size of the buffer, not the size of lines).  This part
*should* ignore your limits, which will make sure comments and strings are
recognized correctly at least in simple cases (i.e. cases which don't
depend on `syntax-propertize-function`).

>>> Think of it as POSIX's ulimits.
>> That's also a blunt tool.
> It isn't either.

Maybe we don't use the same meaning for "blunt".  What I mean is that
it's a tool whose effect cannot be fine-tuned for specific cases.
E.g. limiting the amount of memory used to store images, or the amount
of time spent in a particular operation, rather than applying those
limits to the whole process (even all its children as well).

> It's a practical way to limit what a single process can do

Yup, I use it too, but it's a one-size fits all.

Eli wrote:
> Feel free to suggest better ways of handling these issues, or even
> ways to solve this entirely inside font-lock.  If and when such
> suggestions materialize, I'm sure we will be glad to use them instead
> of less elegant/more direct solutions.

I'd suggest to keep things mostly as they are but move the decision to
ELisp: i.e. pass the beg..end limits to jit-lock and let jit-lock do
the narrowing.  This way it's easy to later refine the mechanism.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  56682@debbugs.gnu.org
> Date: Sat, 30 Jul 2022 03:16:29 -0400
> 
> > Feel free to suggest better ways of handling these issues, or even
> > ways to solve this entirely inside font-lock.  If and when such
> > suggestions materialize, I'm sure we will be glad to use them instead
> > of less elegant/more direct solutions.
> 
> I'd suggest to keep things mostly as they are but move the decision to
> ELisp: i.e. pass the beg..end limits to jit-lock and let jit-lock do
> the narrowing.  This way it's easy to later refine the mechanism.

That's already happening: code called via fontification-functions can
access the restriction via point-min and point-max.  If you or someone
else can come up with efficient methods of using that information so
as not to go too far forward and back, we could consider removing the
lock from the narrowing.  But we'd need to see the code first and
assess the resulting performance with long lines.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 20:02:47 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> > Doing it in init_iterator is too early alas, with the above recipe at 
> > least init_iterator is called with charpos inside the narrowing bounds, 
> > after which the iterator moves outside the narrowing bounds.  So I fixed 
> > the bug in handle_fontified_prop.
> 
> Actually that doesn't work correctly.  A recipe:
> 
> emacs -Q
> M-: (progn (set-frame-width nil 119) (set-frame-height nil 38)) RET
> C-x C-f dictionary.json RET y
> C-s aan SPC
> 
> Now you'll see that the last line at the bottom of the window, which does 
> not contain "aan ", is highlighted.  Is the following okay from your point 
> of view?

I see the problem, but I don't understand how it could be related to
handle_fontified_prop.  The highlight in this case is the isearch
highlighting of matches, and those are shown via overlays, not via
face text properties, so AFAIK handle_fontified_prop doesn't handle
them.

Do you understand the relation of this to handle_fontified_prop?

The immediate reason for the wrong highlighting seems to be an overlay
whose end position is the match for "aan ", and whose start position
is much earlier in the buffer (about 600K characters earlier).  I
don't yet understand why and how this overlay comes into existence.
Hmm...

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> Feel free to suggest better ways of handling these issues, or even 
>>> ways to solve this entirely inside font-lock.  If and when such 
>>> suggestions materialize, I'm sure we will be glad to use them instead 
>>> of less elegant/more direct solutions.
>>
>> I'd suggest to keep things mostly as they are but move the decision to 
>> ELisp: i.e. pass the beg..end limits to jit-lock and let jit-lock do 
>> the narrowing.  This way it's easy to later refine the mechanism.
>
> That's already happening: code called via fontification-functions can 
> access the restriction via point-min and point-max.  If you or someone 
> else can come up with efficient methods of using that information so as 
> not to go too far forward and back, we could consider removing the lock 
> from the narrowing.  But we'd need to see the code first and assess the 
> resulting performance with long lines.
>

IIUC, what Stefan suggests is the following, which seems (almost) fine to 
me.  The only problem I see is that jit-lock-function is not the only user 
of fontication-functions.  It has at least two other users: in ELPA 
multi-mode.el sets fontification-functions to multi-fontify, and in MELPA 
poly-lock.el sets fontification-functions to poly-lock-function.

diff --git a/lisp/jit-lock.el b/lisp/jit-lock.el
index be26ca55f0..aa26b990bc 100644
--- a/lisp/jit-lock.el
+++ b/lisp/jit-lock.el
@@ -370,27 +370,36 @@ jit-lock-refontify

  ;;; On demand fontification.

+(defun jit-lock-function--internal (start)
+  "Internal function called by `jit-lock-function'."
+  (if (not (and jit-lock-defer-timer
+                (or (not (eq jit-lock-defer-time 0))
+                    (input-pending-p))))
+      ;; No deferral.
+      (jit-lock-fontify-now start (+ start jit-lock-chunk-size))
+    ;; Record the buffer for later fontification.
+    (unless (memq (current-buffer) jit-lock-defer-buffers)
+      (push (current-buffer) jit-lock-defer-buffers))
+    ;; Mark the area as defer-fontified so that the redisplay engine
+    ;; is happy and so that the idle timer can find the places to fontify.
+    (with-buffer-prepared-for-jit-lock
+     (put-text-property start
+			(next-single-property-change
+			 start 'fontified nil
+			 (min (point-max) (+ start jit-lock-chunk-size)))
+			'fontified 'defer))))
+
  (defun jit-lock-function (start)
    "Fontify current buffer starting at position START.
  This function is added to `fontification-functions' when `jit-lock-mode'
  is active."
    (when (and jit-lock-mode (not memory-full))
-    (if (not (and jit-lock-defer-timer
-                  (or (not (eq jit-lock-defer-time 0))
-                      (input-pending-p))))
-	;; No deferral.
-	(jit-lock-fontify-now start (+ start jit-lock-chunk-size))
-      ;; Record the buffer for later fontification.
-      (unless (memq (current-buffer) jit-lock-defer-buffers)
-	(push (current-buffer) jit-lock-defer-buffers))
-      ;; Mark the area as defer-fontified so that the redisplay engine
-      ;; is happy and so that the idle timer can find the places to fontify.
-      (with-buffer-prepared-for-jit-lock
-       (put-text-property start
-			  (next-single-property-change
-			   start 'fontified nil
-			   (min (point-max) (+ start jit-lock-chunk-size)))
-			  'fontified 'defer)))))
+    (if (not fontification-functions-restriction)
+        (jit-lock-function--internal start)
+      (narrow-to-region (car fontification-functions-restriction)
+                        (cdr fontification-functions-restriction)
+                        t)
+      (jit-lock-function--internal start))))

  (defun jit-lock--run-functions (beg end)
    (let ((tight-beg nil) (tight-end nil)
diff --git a/src/xdisp.c b/src/xdisp.c
index 0fdb1922e5..726e77b8eb 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4412,9 +4412,9 @@ handle_fontified_prop (struct it *it)
  	  ptrdiff_t begv = it->narrowed_begv ? it->narrowed_begv : BEGV;
  	  ptrdiff_t zv = it->narrowed_zv;
  	  ptrdiff_t charpos = IT_CHARPOS (*it);
  	  if (begv <= charpos && charpos <= zv)
-	    Fnarrow_to_region (make_fixnum (begv), make_fixnum (zv), Qt);
+	    specbind (Qfontification_functions_restriction,
+		      Fcons (make_fixnum (begv), make_fixnum (zv)));
  	}

        /* Don't allow Lisp that runs from 'fontification-functions'
@@ -36673,6 +36673,13 @@ syms_of_xdisp (void)
    Vfontification_functions = Qnil;
    Fmake_variable_buffer_local (Qfontification_functions);

+  DEFSYM (Qfontification_functions_restriction,
+	  "fontification-functions-restriction");
+  DEFVAR_LISP ("fontification-functions-restriction",
+	       Vfontification_functions_restriction,
+	       doc: /* TODO  */);
+  Vfontification_functions_restriction = Qnil;
+
    DEFVAR_BOOL ("unibyte-display-via-language-environment",
                 unibyte_display_via_language_environment,
      doc: /* Non-nil means display unibyte text according to language environment.

By the way, while trying the above, it became clear that I forgot to 
properly handle the new optional argument to narrow-to-region in 
byte-compiled code.  But I don't know how to do that:

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index b4954eee9f..1ecd77f751 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -767,7 +767,7 @@ 121
  (byte-defop 122  0 byte-char-syntax)
  (byte-defop 123 -1 byte-buffer-substring)
  (byte-defop 124 -1 byte-delete-region)
-(byte-defop 125 -1 byte-narrow-to-region)
+(byte-defop 125 -2 byte-narrow-to-region)
  (byte-defop 126  1 byte-widen)
  (byte-defop 127  0 byte-end-of-line)

@@ -3833,7 +3833,7 @@ setcar
  (byte-defop-compiler setcdr            2)
  (byte-defop-compiler buffer-substring  2)
  (byte-defop-compiler delete-region     2)
-(byte-defop-compiler narrow-to-region  2)
+(byte-defop-compiler narrow-to-region  2-3)
  (byte-defop-compiler (% byte-rem)      2)
  (byte-defop-compiler aset              3)

is apparently not enough, because "2-3" seems to install an 
integer-or-marker-p check on the third argument, which raises a 
(wrong-type-argument integer-or-marker-p nil) or (wrong-type-argument 
integer-or-marker-p t) error when narrow-to-region is called from 
byte-compiled code.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 30 Jul 2022 10:52:48 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, 56682@debbugs.gnu.org
> 
> >> I'd suggest to keep things mostly as they are but move the decision to 
> >> ELisp: i.e. pass the beg..end limits to jit-lock and let jit-lock do 
> >> the narrowing.  This way it's easy to later refine the mechanism.
> >
> > That's already happening: code called via fontification-functions can 
> > access the restriction via point-min and point-max.  If you or someone 
> > else can come up with efficient methods of using that information so as 
> > not to go too far forward and back, we could consider removing the lock 
> > from the narrowing.  But we'd need to see the code first and assess the 
> > resulting performance with long lines.
> >
> 
> IIUC, what Stefan suggests is the following, which seems (almost) fine to 
> me.

I fail to see the difference, sorry.  Instead of doing the narrowing
from C we leave it to jit-lock, but tell it the limits to narrow to?
Or am I missing something?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> IIUC, what Stefan suggests is the following, which seems (almost) fine 
>> to me.
>
> I fail to see the difference, sorry.  Instead of doing the narrowing 
> from C we leave it to jit-lock, but tell it the limits to narrow to? Or 
> am I missing something?
>

TBH, I don't see a real difference either.  Stefan says that "this way 
it's easy to later refine the mechanism", I guess because it is possible 
to "refine the mechanism" without recompiling Emacs.  But as I said the 
problem is that there are other users of fontification-functions besides 
jit-lock-function.

Could you (or Stefan) please have a look at the bytecomp.el problem I 
mentioned at the end of my previous post?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 30 Jul 2022 10:52:48 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, 56682@debbugs.gnu.org
> 
> By the way, while trying the above, it became clear that I forgot to 
> properly handle the new optional argument to narrow-to-region in 
> byte-compiled code.  But I don't know how to do that:
> 
> diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
> index b4954eee9f..1ecd77f751 100644
> --- a/lisp/emacs-lisp/bytecomp.el
> +++ b/lisp/emacs-lisp/bytecomp.el
> @@ -767,7 +767,7 @@ 121
>   (byte-defop 122  0 byte-char-syntax)
>   (byte-defop 123 -1 byte-buffer-substring)
>   (byte-defop 124 -1 byte-delete-region)
> -(byte-defop 125 -1 byte-narrow-to-region)
> +(byte-defop 125 -2 byte-narrow-to-region)
>   (byte-defop 126  1 byte-widen)
>   (byte-defop 127  0 byte-end-of-line)
> 
> @@ -3833,7 +3833,7 @@ setcar
>   (byte-defop-compiler setcdr            2)
>   (byte-defop-compiler buffer-substring  2)
>   (byte-defop-compiler delete-region     2)
> -(byte-defop-compiler narrow-to-region  2)
> +(byte-defop-compiler narrow-to-region  2-3)
>   (byte-defop-compiler (% byte-rem)      2)
>   (byte-defop-compiler aset              3)
> 
> is apparently not enough, because "2-3" seems to install an 
> integer-or-marker-p check on the third argument, which raises a 
> (wrong-type-argument integer-or-marker-p nil) or (wrong-type-argument 
> integer-or-marker-p t) error when narrow-to-region is called from 
> byte-compiled code.

Where's the integer-or-marker-p test installed and/or called from?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Actually that doesn't work correctly.  A recipe:
>>
>> emacs -Q
>> M-: (progn (set-frame-width nil 119) (set-frame-height nil 38)) RET
>> C-x C-f dictionary.json RET y
>> C-s aan SPC
>>
>> Now you'll see that the last line at the bottom of the window, which 
>> does not contain "aan ", is highlighted.  Is the following okay from 
>> your point of view?
>
> I see the problem, but I don't understand how it could be related to 
> handle_fontified_prop.  The highlight in this case is the isearch 
> highlighting of matches, and those are shown via overlays, not via face 
> text properties, so AFAIK handle_fontified_prop doesn't handle them.
>
> Do you understand the relation of this to handle_fontified_prop?
>

I cannot claim that fully understand what happens, but see below.  What I 
do know is that that problem wasn't present before 9c12c3b7c5, and that it 
disappears with:

diff --git a/src/xdisp.c b/src/xdisp.c
index b1ee7889d4..8c62f088b8 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4412,13 +4412,8 @@ handle_fontified_prop (struct it *it)
           ptrdiff_t begv = it->narrowed_begv ? it->narrowed_begv : BEGV;
           ptrdiff_t zv = it->narrowed_zv;
           ptrdiff_t charpos = IT_CHARPOS (*it);
-         if (charpos < begv || charpos > zv)
-           {
-             begv = get_narrowed_begv (it->w, charpos);
-             if (!begv) begv = BEGV;
-             zv = get_narrowed_zv (it->w, charpos);
-           }
-         Fnarrow_to_region (make_fixnum (begv), make_fixnum (zv), Qt);
+         if (begv <= charpos && charpos <= zv)
+           Fnarrow_to_region (make_fixnum (begv), make_fixnum (zv), Qt);
         }

As far as I understand, what happens is this:

C-s aan SPC RET positions point at 730072.  When typing C-s aan SPC in a 
not yet fontified buffer, it->narrowed_begv and it->narrowed_zv are 
correctly positioned around that position, at 706860 and 732564 
respectively.  But the iterator is late, and is still at the position at 
which it was after C-s aan (without SPC); the first occurrence of "aan" is 
at 152274, the corresponding narrowing bounds are 128520 and 154224, which 
means that it has stopped at 154224.  So if we "reseat" the narrowing for 
fontification-functions around the position 154224 of the iterator, the 
narrowing becomes 141372-167076, which is well before the position of "aan 
", namely 730072.  At that point, isearch has found a match, and puts the 
match overlay at 167076 (the last possible position of the narrowed 
portion) and beyond, instead of putting it at 730072.

In short, it seems to me that using the position of the iterator is a too 
fragile solution, and that it is better to not apply a narrowing when the 
iterator is outside of the narrowing bounds.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> is apparently not enough, because "2-3" seems to install an 
>> integer-or-marker-p check on the third argument, which raises a 
>> (wrong-type-argument integer-or-marker-p nil) or (wrong-type-argument 
>> integer-or-marker-p t) error when narrow-to-region is called from 
>> byte-compiled code.
>
> Where's the integer-or-marker-p test installed and/or called from?
>

It is called when narrow-to-region, which has its own opcode, is called 
from byte-compiled code.  But I have no idea where it is installed.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> is apparently not enough, because "2-3" seems to install an 
>>> integer-or-marker-p check on the third argument, which raises a 
>>> (wrong-type-argument integer-or-marker-p nil) or (wrong-type-argument 
>>> integer-or-marker-p t) error when narrow-to-region is called from 
>>> byte-compiled code.
>> 
>> Where's the integer-or-marker-p test installed and/or called from?
>
> It is called when narrow-to-region, which has its own opcode, is called 
> from byte-compiled code.  But I have no idea where it is installed.
>

Got it, fixed on master.  make bootstrap is probably necessary.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> I'm not opposed to reducing the size of the text that's considered, but
>>> doing it via narrowing is a blunt tool.
>>
>> It isn't.
>
> There's no point arguing about it.  I find it to be and you don't and 
> that's that.
>

Actually, with your explanation below that "blunt" means for you "cannot 
be fine-tuned", I agree with you.

>
> The reason why I find it to be is because it removes all possibilities 
> of making different choices for different elements depending on the cost 
> of those elements and the amount of available information.
>

With that I also agree.  It would be better to have a refined decision 
method, but at the moment it doesn't exist.

>
> What I think would be a better option is to (somehow) pass the 
> `beg..end` "limit" to jit-lock which can then pass it on to its own 
> clients (e.g. font-lock) so they each can make their own choices.
>

I sent a patch a few hours ago in this thread.

The limit of what you suggest is that its effect would depend on the 
goodwill of major and minor mode authors, which could decide to ignore 
these beg..end recommendations altogether.  Whereas the point of that 
feature is more or less to protect Emacs users from major and minor modes, 
to make sure that Emacs remains responsive when the buffer contains long 
lines.

Also, there are other users of fontification-functions besides 
jit-lock-function.

(And for some reason the patch I sent does not give the results I would 
have expected, font-locking is still too slow, but it's perhaps a bug in 
the patch.)

>
> E.g. the syntax-ppss part of the job performed by font-lock is heavily 
> cached, does not depends on lines, is theoretically always computed from 
> BOB but with a cache which makes it fast even when working near EOB (tho 
> it can still be somewhat slow when jumping from BOB to EOB, but that 
> depends on the size of the buffer, not the size of lines).  This part 
> *should* ignore your limits, which will make sure comments and strings 
> are recognized correctly at least in simple cases (i.e. cases which 
> don't depend on `syntax-propertize-function`).
>

I see.  Do you see a way to somehow extract the syntax-ppss part out of 
font-lock?  Would that be feasible?

And another question: can syntax-ppss not be used to determine a "good 
starting position" for the narrowing, outside of any comments or strings 
(if possible)?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 30 Jul 2022 11:34:03 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> C-s aan SPC RET positions point at 730072.  When typing C-s aan SPC in a 
> not yet fontified buffer, it->narrowed_begv and it->narrowed_zv are 
> correctly positioned around that position, at 706860 and 732564 
> respectively.  But the iterator is late, and is still at the position at 
> which it was after C-s aan (without SPC); the first occurrence of "aan" is 
> at 152274, the corresponding narrowing bounds are 128520 and 154224, which 
> means that it has stopped at 154224.  So if we "reseat" the narrowing for 
> fontification-functions around the position 154224 of the iterator, the 
> narrowing becomes 141372-167076, which is well before the position of "aan 
> ", namely 730072.  At that point, isearch has found a match, and puts the 
> match overlay at 167076 (the last possible position of the narrowed 
> portion) and beyond, instead of putting it at 730072.

What I see is that isearch puts the match overlay on text that starts
at 167076 and ends at 730068 (the latter is the beginning of the
match), instead of on text between 730068 and 730072.

> In short, it seems to me that using the position of the iterator is a too 
> fragile solution, and that it is better to not apply a narrowing when the 
> iterator is outside of the narrowing bounds.

But this means we give up on the narrowing, which means the display
could be very slow.

And I've found the culprit: we weren't restoring point after lifting
the locked narrowing.  narrow-to-region can move point if the new
restriction puts point outside of the region.  So what was happening
is that isearch-update was calling pos-visible-in-window-group-p to
see whether the match is visible, and that call would move point from
under the feet of isearch-update, because pos-visible-in-window-p
calls display routines.  So any subsequent uses of point would use a
completely wrong value of point.

I've now made narrow-to-region preserve point across locked narrowing,
and the problem went away.

Ugh! this one was a bitch to debug!

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]


>
> And I've found the culprit: we weren't restoring point after lifting the 
> locked narrowing.  narrow-to-region can move point if the new 
> restriction puts point outside of the region.  So what was happening is 
> that isearch-update was calling pos-visible-in-window-group-p to see 
> whether the match is visible, and that call would move point from under 
> the feet of isearch-update, because pos-visible-in-window-p calls 
> display routines.  So any subsequent uses of point would use a 
> completely wrong value of point.
>
> I've now made narrow-to-region preserve point across locked narrowing, 
> and the problem went away.
>
> Ugh! this one was a bitch to debug!
>

😉  Thanks, that's even better!

So the only remaining question is whether it is necessary to recompute 
narrowed_begv and narrowed_zv in init_iterator:

diff --git a/src/xdisp.c b/src/xdisp.c
index b1ee7889d4..e415320a52 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -3429,6 +3429,12 @@ init_iterator (struct it *it, struct window *w,
     {
       it->narrowed_begv = get_narrowed_begv (w, window_point (w));
       it->narrowed_zv = get_narrowed_zv (w, window_point (w));
+      if (charpos >= 0
+         && (charpos < it->narrowed_begv || charpos > it->narrowed_zv))
+       {
+         it->narrowed_begv = get_narrowed_begv (w, charpos);
+         it->narrowed_zv = get_narrowed_zv (w, charpos);
+       }
     }

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 30 Jul 2022 13:31:42 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> So the only remaining question is whether it is necessary to recompute 
> narrowed_begv and narrowed_zv in init_iterator:

I tend to think we should, but let me think about this some more.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> So the only remaining question is whether it is necessary to recompute 
>> narrowed_begv and narrowed_zv in init_iterator:
>
> I tend to think we should, but let me think about this some more.
>

Okay, I'll wait for your feedback.

Wouldn't it make sense to also limit the portion of the buffer to which 
pre-/post-command-hook have access (see below)?

With that patch, I was able to open and edit a file with a single 50 GB 
(!) line, in js-mode.  Does that still not qualify as "arbitrarily large"?

I compared that with a 50 GB JSON file with 80 character wide lines. 
With that file it was necessary to disable font-lock-mode, which took too 
much time.  Apart from that, I did not see any significant performance 
differences while editing the file, compared to the single line one.

diff --git a/src/keyboard.c b/src/keyboard.c
index 2863058d63..ce529222a3 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -1461,7 +1461,22 @@ command_loop_1 (void)
        }
        Vthis_command = cmd;
        Vreal_this_command = cmd;
-      safe_run_hooks (Qpre_command_hook);
+
+      if (current_buffer->long_line_optimizations_p)
+	{
+	  specpdl_ref count = SPECPDL_INDEX ();
+	  struct window *w = XWINDOW (selected_window);
+	  ptrdiff_t begv = get_narrowed_begv (w, PT);
+	  ptrdiff_t zv = get_narrowed_zv (w, PT);
+	  if (!begv) begv = BEGV;
+	  Fnarrow_to_region (make_fixnum (begv), make_fixnum (zv), Qt);
+	  safe_run_hooks (Qpre_command_hook);
+	  unbind_to (count, Qnil);
+	}
+      else
+	{
+	  safe_run_hooks (Qpre_command_hook);
+	}

        already_adjusted = 0;

@@ -1513,7 +1528,21 @@ command_loop_1 (void)
            }
        kset_last_prefix_arg (current_kboard, Vcurrent_prefix_arg);

-      safe_run_hooks (Qpost_command_hook);
+      if (current_buffer->long_line_optimizations_p)
+	{
+	  specpdl_ref count = SPECPDL_INDEX ();
+	  struct window *w = XWINDOW (selected_window);
+	  ptrdiff_t begv = get_narrowed_begv (w, PT);
+	  ptrdiff_t zv = get_narrowed_zv (w, PT);
+	  if (!begv) begv = BEGV;
+	  Fnarrow_to_region (make_fixnum (begv), make_fixnum (zv), Qt);
+	  safe_run_hooks (Qpost_command_hook);
+	  unbind_to (count, Qnil);
+	}
+      else
+	{
+	  safe_run_hooks (Qpost_command_hook);
+	}

        /* If displaying a message, resize the echo area window to fit
  	 that message's size exactly.  Do this only if the echo area

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 30 Jul 2022 18:13:18 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> Wouldn't it make sense to also limit the portion of the buffer to which 
> pre-/post-command-hook have access (see below)?

Those generally don't belong to the display department, so I'd
hesitate doing so.  Which pre-/post-command-hook functions did you
find that cause slowdown because of long lines.

Before considering these hooks, I'd consider window-scroll-functions
and window-*-change-functions, which _are_ run by redisplay.

> With that patch, I was able to open and edit a file with a single 50 GB 
> (!) line, in js-mode.  Does that still not qualify as "arbitrarily large"?

We don't even claim to be able to edit _files_ of arbitrary size
(because we are limited by fixnums).

> I compared that with a 50 GB JSON file with 80 character wide lines. 
> With that file it was necessary to disable font-lock-mode, which took too 
> much time.

How so?  We now restrict font-lock to a small region, so why does it
matter how much more stuff is there outside of the viewport?  What
other aspects of the line size still affect performance?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Wouldn't it make sense to also limit the portion of the buffer to which 
>> pre-/post-command-hook have access (see below)?
>
> Those generally don't belong to the display department, so I'd hesitate 
> doing so.  Which pre-/post-command-hook functions did you find that 
> cause slowdown because of long lines.
>

jit-lock--antiblink-post-command

>> With that patch, I was able to open and edit a file with a single 50 GB 
>> (!) line, in js-mode.  Does that still not qualify as "arbitrarily 
>> large"?
>
> We don't even claim to be able to edit _files_ of arbitrary size 
> (because we are limited by fixnums).
>

That's theory, isn't it?  With 64-bit builds we are limited to files that 
are less than 2047 Po.  No computer on this planet has that much RAM.

>> I compared that with a 50 GB JSON file with 80 character wide lines. 
>> With that file it was necessary to disable font-lock-mode, which took 
>> too much time.
>
> How so?  We now restrict font-lock to a small region, so why does it 
> matter how much more stuff is there outside of the viewport?  What other 
> aspects of the line size still affect performance?
>

We do not restrict font-lock in large files _without long lines_, hence 
the difference.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 30 Jul 2022 18:47:04 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> >> Wouldn't it make sense to also limit the portion of the buffer to which 
> >> pre-/post-command-hook have access (see below)?
> >
> > Those generally don't belong to the display department, so I'd hesitate 
> > doing so.  Which pre-/post-command-hook functions did you find that 
> > cause slowdown because of long lines.
> 
> jit-lock--antiblink-post-command

OK, but is it TRT to "punish" every one of these hooks for the
"crimes" of the few?  Maybe we should instead handle the problematic
ones locally, by exposing the long_line_optimizations_p flag to Lisp
(through an accessor), and then modifying those that misbehave to
"behave"?

> >> With that patch, I was able to open and edit a file with a single 50 GB 
> >> (!) line, in js-mode.  Does that still not qualify as "arbitrarily 
> >> large"?
> >
> > We don't even claim to be able to edit _files_ of arbitrary size 
> > (because we are limited by fixnums).
> 
> That's theory, isn't it?  With 64-bit builds we are limited to files that 
> are less than 2047 Po.  No computer on this planet has that much RAM.

You forget that we are talking about VM.

But let's not restart that argument, okay?

> >> I compared that with a 50 GB JSON file with 80 character wide lines. 
> >> With that file it was necessary to disable font-lock-mode, which took 
> >> too much time.
> >
> > How so?  We now restrict font-lock to a small region, so why does it 
> > matter how much more stuff is there outside of the viewport?  What other 
> > aspects of the line size still affect performance?
> >
> 
> We do not restrict font-lock in large files _without long lines_, hence 
> the difference.

Sorry, I thought you were talking about a single-line file.

If JS mode wants to access the entire buffer for fontifications, then
IMO the problem is in JS mode, and should be fixed there.
narrow-to-region is available to Lisp programs as well ;-)

IOW, it isn't an infrastructure problem that needs to be fixed in
display code.  (It is even possible that tree-sitter integration will
fix this, or at least alleviate it, as a side effect.)

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]


>>>> Wouldn't it make sense to also limit the portion of the buffer to 
>>>> which pre-/post-command-hook have access (see below)?
>>>
>>> Those generally don't belong to the display department, so I'd 
>>> hesitate doing so.  Which pre-/post-command-hook functions did you 
>>> find that cause slowdown because of long lines.
>>
>> jit-lock--antiblink-post-command
>
> OK, but is it TRT to "punish" every one of these hooks for the "crimes" 
> of the few?  Maybe we should instead handle the problematic ones 
> locally, by exposing the long_line_optimizations_p flag to Lisp (through 
> an accessor), and then modifying those that misbehave to "behave"?
>

It's the same problem than with fontification-functions.  We cannot know 
what all these hooks that are installed by major and minor modes will do, 
we cannot hope to fix them one by one, so it seems to me that with 
long_line_optimizations_p, which is an unusual case anyway, it makes sense 
to "punish" them all in the same way.

>> That's theory, isn't it?  With 64-bit builds we are limited to files 
>> that are less than 2047 Po.  No computer on this planet has that much 
>> RAM.
>
> You forget that we are talking about VM.
>
> But let's not restart that argument, okay?
>

Hmmm... okay 😉

>
> If JS mode wants to access the entire buffer for fontifications, then 
> IMO the problem is in JS mode, and should be fixed there. 
> narrow-to-region is available to Lisp programs as well ;-)
>
> IOW, it isn't an infrastructure problem that needs to be fixed in 
> display code.  (It is even possible that tree-sitter integration will 
> fix this, or at least alleviate it, as a side effect.)
>

Agreed.  My point was only that Emacs now behaves a bit better when 
editing a single-line very large file compared to a multi-line very large 
file.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 30 Jul 2022 19:11:57 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> >> jit-lock--antiblink-post-command
> >
> > OK, but is it TRT to "punish" every one of these hooks for the "crimes" 
> > of the few?  Maybe we should instead handle the problematic ones 
> > locally, by exposing the long_line_optimizations_p flag to Lisp (through 
> > an accessor), and then modifying those that misbehave to "behave"?
> 
> It's the same problem than with fontification-functions.  We cannot know 
> what all these hooks that are installed by major and minor modes will do, 
> we cannot hope to fix them one by one, so it seems to me that with 
> long_line_optimizations_p, which is an unusual case anyway, it makes sense 
> to "punish" them all in the same way.

It sounds...too drastic.  Lars, WDYT?

I don't find it problematic to have to fix any such hooks in the core
that we discover as misbehaving with long lines.  How many of those
could we have?  And those in 3rd party packages can follow suit if
they want (and if the respective modes are relevant to files with long
lines, I expect to see pressure on their developers to do so).

Once again, IME it is impossible to fix such problems only in
low-level C infrastructure.  There will always be left-overs and
fallouts that should be fixed locally in Lisp where they happen.
There's no problem here, and I don't expect us to be able to fix
everything by a small number of quick fixes, and declare a victory
once and for all.

> Agreed.  My point was only that Emacs now behaves a bit better when 
> editing a single-line very large file compared to a multi-line very large 
> file.

Well, WDYT about a similar feature for very large files?  IOW, when
the buffer's size is above some threshold, turn on the
long_line_optimizations_p flag (which should perhaps be renamed to
better reflect its purpose) even if no long lines are seen?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org,
>  monnier@iro.umontreal.ca
> Date: Sat, 30 Jul 2022 18:23:12 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Sat, 30 Jul 2022 13:31:42 +0000
> > From: Gregory Heytings <gregory@heytings.org>
> > cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
> >     monnier@iro.umontreal.ca
> > 
> > So the only remaining question is whether it is necessary to recompute 
> > narrowed_begv and narrowed_zv in init_iterator:
> 
> I tend to think we should, but let me think about this some more.

Here are my thoughts.

First, I think the setting of narrowed_begv and narrowed_zv should be
done in 'reseat', not in init_iterator.  The latter always calls the
former when invoked to start iteration of buffer text, but we also
call 'reseat' from other places, when we "jump" the iterator to a new
place, potentially far from the last.  If nothing else, this should
help with truncate-lines, where using window_point is basically right
only for the point's line.  And currently, init_iterator computes and
sets the narrowing even when we iterate on strings, which is unneeded
and incorrect.

Whether always to correct narrowed_begv and narrowed_zv if we are
reseating to a position outside the narrowing, is a more complicated
question.  The basic problem here is that we don't have an easy way of
restoring the previous narrowing (except by unwind_protect), and the
display code sometimes calls init_iterator or start_display using the
iterator that already has these members set by previous code, a
situation which we currently cannot easily detect.  However, when this
code runs as part of redisplay, we generally don't expect the original
narrowing to be insufficient, except perhaps in the truncate-line
case.

So I think we should correct narrowed_begv and narrowed_zv only if
either the 'redisplaying_p' flag is reset (meaning the display code is
being invoked outside of redisplay) or it->line_wrap == TRUNCATE.

Comments? thoughts?

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> IIUC, what Stefan suggests is the following, which seems (almost) fine to
> me.  The only problem I see is that jit-lock-function is not the only user
> of fontication-functions.  It has at least two other users: in ELPA
> multi-mode.el sets fontification-functions to multi-fontify, and in MELPA
> poly-lock.el sets fontification-functions to poly-lock-function.

Good point.

> @@ -4412,9 +4412,9 @@ handle_fontified_prop (struct it *it)
>  	  ptrdiff_t begv = it->narrowed_begv ? it->narrowed_begv : BEGV;
>  	  ptrdiff_t zv = it->narrowed_zv;
>  	  ptrdiff_t charpos = IT_CHARPOS (*it);
>  	  if (begv <= charpos && charpos <= zv)
> -	    Fnarrow_to_region (make_fixnum (begv), make_fixnum (zv), Qt);
> +	    specbind (Qfontification_functions_restriction,
> +		      Fcons (make_fixnum (begv), make_fixnum (zv)));
>  	}

Hpw 'bout we do the reverse then: set the narrowing, but let-bind
a variable to indicate that we're inside a line-length-induced
narrowing, together with the previous narrowing bounds, so jit-lock or
its clients can undo the narrowing when needed?


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  56682@debbugs.gnu.org
> Date: Sun, 31 Jul 2022 03:25:46 -0400
> 
> How 'bout we do the reverse then: set the narrowing, but let-bind
> a variable to indicate that we're inside a line-length-induced
> narrowing, together with the previous narrowing bounds, so jit-lock or
> its clients can undo the narrowing when needed?

I'd rather prefer an extension of 'widen' that would be able to undo
the locked narrowing "when needed".  That way, jit-lock clients should
actually do something to request that, instead of letting them keep
the current code that widens whenever they feel like it.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> How 'bout we do the reverse then: set the narrowing, but let-bind a 
> variable to indicate that we're inside a line-length-induced narrowing, 
> together with the previous narrowing bounds, so jit-lock or its clients 
> can undo the narrowing when needed?
>

That's not possible: the narrowing is (really) locked (with un uninterned 
symbol), it cannot be undone.  What would be possible would be to add an 
optional "unlock" argument to widen.  But somehow I don't think that would 
be TRT, as mode authors who now do a (widen) would simply take the habit 
to write (widen t) instead, and the same problems would surface again.

BTW, my tests show that syntax-ppss can be rather slow, when the file is 
large enough (say 1 GB).  I didn't look at what it does, but is it not 
possible to design a version of syntax-ppss that would approximate, with 
some heuristics, what syntax-ppss does, but on a smaller chunk of the 
buffer?  For example, I'd guess that '"' immediately followed by an 
alphanumeric character most likely starts a string, and '"' immediately 
preceded by an alphanumeric character most likely ends a string.

bug#56682: Fix the long lines font locking related slowdowns

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> It's the same problem than with fontification-functions.  We cannot know 
>> what all these hooks that are installed by major and minor modes will do, 
>> we cannot hope to fix them one by one, so it seems to me that with 
>> long_line_optimizations_p, which is an unusual case anyway, it makes sense 
>> to "punish" them all in the same way.
>
> It sounds...too drastic.  Lars, WDYT?

I agree with Gregory that it makes sense to disable (some) fontification
stuff in long-line buffers.  It much more important to be able to view
and edit these files than getting fontification details completely
correct.

Of course, it'd be better if everything just worked perfectly, but
that's very ambitious. 

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> It sounds...too drastic.
>

Are you sure?  The docstring already says "It is a bad idea to use this 
hook for expensive processing."  And Emacs already removes a function from 
the hook when it misbhaves.  Adding something like "In a too large buffer 
or in a buffer with long lines, the functions in this hook will only have 
access to a small portion of the buffer" seems coherent, at least to me.

>
> Once again, IME it is impossible to fix such problems only in low-level 
> C infrastructure.  There will always be left-overs and fallouts that 
> should be fixed locally in Lisp where they happen. There's no problem 
> here, and I don't expect us to be able to fix everything by a small 
> number of quick fixes, and declare a victory once and for all.
>

I both agree and disagree with that.  It is true that it is, strictly 
speaking, impossible to fix _all_ such problems _only_ in low-level C 
intrastructure, and that there will always be left-overs.  But it is 
possible to fix _most_ of these problems only in low-level C 
infrastructure, and we should do so, just like an operating system kernel 
in which everything is done to avoid crashing the system/leaving it in an 
unusable state (which includes killing a mis-behaving process when 
necessary).  And we should do so even more when the amount of code to do 
so in the low-level C infrastructure remains small.

>
> Well, WDYT about a similar feature for very large files?  IOW, when the 
> buffer's size is above some threshold, turn on the 
> long_line_optimizations_p flag (which should perhaps be renamed to 
> better reflect its purpose) even if no long lines are seen?
>

I was thinking about such a feature indeed.  But it would be separate from 
the long_line_optimizations_p one, because the optimizations to activate 
in both cases are different, and their thresholds are different, too.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Gregory Heytings <gregory@heytings.org>,  gerd.moellmann@gmail.com,
>   56682@debbugs.gnu.org,  monnier@iro.umontreal.ca
> Date: Sun, 31 Jul 2022 10:22:40 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> It's the same problem than with fontification-functions.  We cannot know 
> >> what all these hooks that are installed by major and minor modes will do, 
> >> we cannot hope to fix them one by one, so it seems to me that with 
> >> long_line_optimizations_p, which is an unusual case anyway, it makes sense 
> >> to "punish" them all in the same way.
> >
> > It sounds...too drastic.  Lars, WDYT?
> 
> I agree with Gregory that it makes sense to disable (some) fontification
> stuff in long-line buffers.  It much more important to be able to view
> and edit these files than getting fontification details completely
> correct.

No disagreement here, but I was asking about the proposal to make the
locked narrowing in effect when any pre-command-hook or
post-command-hook runs.  These are usually unrelated to
fontifications, although Gregory found an example where it is.

IOW, the issue for which I wanted to hear your opinion was whether you
think it's okay to preclude, in a buffer with long lines, all
pre/post-command-hooks from accessing the entire buffer.

bug#56682: Fix the long lines font locking related slowdowns

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> IOW, the issue for which I wanted to hear your opinion was whether you
> think it's okay to preclude, in a buffer with long lines, all
> pre/post-command-hooks from accessing the entire buffer.

Yes, I think that's reasonable.  There will inevitably be many modes out
there that'll wedge themselves in the presence of gargantuan lines, and
limiting these hooks is a reasonable line of defence.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 31 Jul 2022 08:30:14 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> > It sounds...too drastic.
> 
> Are you sure?  The docstring already says "It is a bad idea to use this 
> hook for expensive processing."  And Emacs already removes a function from 
> the hook when it misbhaves.  Adding something like "In a too large buffer 
> or in a buffer with long lines, the functions in this hook will only have 
> access to a small portion of the buffer" seems coherent, at least to me.

But that assumes these hooks will _always_ be "too expensive" in such
buffers.  Which is not necessarily true: I can think of many things a
hook can do in a long-line buffer without being expensive.

And since we already remove expensive hook functions, maybe that is
enough?  Or maybe we should use a different threshold for "expensive"
in buffers with long lines?

> > Once again, IME it is impossible to fix such problems only in low-level 
> > C infrastructure.  There will always be left-overs and fallouts that 
> > should be fixed locally in Lisp where they happen. There's no problem 
> > here, and I don't expect us to be able to fix everything by a small 
> > number of quick fixes, and declare a victory once and for all.
> 
> I both agree and disagree with that.  It is true that it is, strictly 
> speaking, impossible to fix _all_ such problems _only_ in low-level C 
> intrastructure, and that there will always be left-overs.  But it is 
> possible to fix _most_ of these problems only in low-level C 
> infrastructure, and we should do so, just like an operating system kernel 
> in which everything is done to avoid crashing the system/leaving it in an 
> unusable state (which includes killing a mis-behaving process when 
> necessary).  And we should do so even more when the amount of code to do 
> so in the low-level C infrastructure remains small.

I think we are in agreement: my point was that solving such problems
locally is not unthinkable.

> > Well, WDYT about a similar feature for very large files?  IOW, when the 
> > buffer's size is above some threshold, turn on the 
> > long_line_optimizations_p flag (which should perhaps be renamed to 
> > better reflect its purpose) even if no long lines are seen?
> >
> 
> I was thinking about such a feature indeed.  But it would be separate from 
> the long_line_optimizations_p one, because the optimizations to activate 
> in both cases are different, and their thresholds are different, too.

Different thresholds are easy to reconcile: the optimizations should
be turned on if either of the two thresholds is exceeded.  But why do
you say the optimizations will be different? what's wrong with using
the same optimizations, i.e. restrict the display code from accessing
the entire buffer?

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> That's not possible: the narrowing is (really) locked (with un uninterned
> symbol), it cannot be undone.  What would be possible would be to add an
> optional "unlock" argument to widen.  But somehow I don't think that would
> be TRT, as mode authors who now do a (widen) would simply take the habit to
> write (widen t) instead, and the same problems would surface again.

Emacs is not in the business of preventing people from shooting
themselves in the foot.  If we need this narrowing to be enforced
because Emacs would otherwise crash, then it's OK, but if not, then we
*should* provide a way to undo it.

> BTW, my tests show that syntax-ppss can be rather slow, when the file is
>  large enough (say 1 GB).

No doubt.  But it's no slower with long lines than with short lines.
[ Note that it calls `syntax-propertize` internally, which would need
  to be considered separately since `syntax-propertize` does work
  line-by-line, and should hence obey the narrowing.  ]


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Emacs is not in the business of preventing people from shooting 
> themselves in the foot.  If we need this narrowing to be enforced 
> because Emacs would otherwise crash, then it's OK, but if not, then we 
> *should* provide a way to undo it.
>

And how do you define "crash"?  Is Emacs becoming unresponsive because an 
operation takes say two minutes to complete and cannot be interrupted a 
"crash"?  Or is a "crash" only a segfault?

>> BTW, my tests show that syntax-ppss can be rather slow, when the file 
>> is large enough (say 1 GB).
>
> No doubt.  But it's no slower with long lines than with short lines.
>

Yes, I wasn't clear enough, I should have written "when the file is large 
enough (say 1 GB), even without long lines".

But you didn't answer my question: is it not possible to design a version 
of syntax-ppss that would approximate, with some heuristics, what 
syntax-ppss does, but on a smaller chunk of the buffer?  With such a 
syntax-ppss-approximate function, we could do something like

(defun syntax-ppss (args)
   (if (narrow-to-region-locked)
       (syntax-ppss-approximage args)
     (syntax-ppss-accurate args)))

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> And since we already remove expensive hook functions, maybe that is 
> enough?  Or maybe we should use a different threshold for "expensive" in 
> buffers with long lines?
>

Do we?  As far as I can see, there is no time limit in safe_run_hooks. 
It could make sense to add one (or at least to clear out hook functions 
that have taken too long, as we already do it with functions that return 
an error... but doing that after the hook function returns is already too 
late, if it has taken say 30 seconds).

>>> Well, WDYT about a similar feature for very large files?  IOW, when 
>>> the buffer's size is above some threshold, turn on the 
>>> long_line_optimizations_p flag (which should perhaps be renamed to 
>>> better reflect its purpose) even if no long lines are seen?
>>
>> I was thinking about such a feature indeed.  But it would be separate 
>> from the long_line_optimizations_p one, because the optimizations to 
>> activate in both cases are different, and their thresholds are 
>> different, too.
>
> Different thresholds are easy to reconcile: the optimizations should be 
> turned on if either of the two thresholds is exceeded.  But why do you 
> say the optimizations will be different? what's wrong with using the 
> same optimizations, i.e. restrict the display code from accessing the 
> entire buffer?
>

I don't know exactly yet.  It seems to me that some of the optimizations 
for large buffers would be similar to the ones for long lines, and that 
many of the specific optimizations for long lines are not necessary for 
large buffers.  I think it would be better/safer to only enable the 
optimizations that are really necessary in each case.  But let's start 
thinking in more detail about the large buffer optimizations once the long 
lines optimizations are done, okay?

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-07-31 10:50:18] wrote:
>> Emacs is not in the business of preventing people from shooting themselves
>> in the foot.  If we need this narrowing to be enforced because Emacs would
>> otherwise crash, then it's OK, but if not, then we *should* provide a way
>> to undo it.
> And how do you define "crash"?

Core dump.

> Is Emacs becoming unresponsive because an operation takes say two
> minutes to complete and cannot be interrupted a "crash"?  Or is
> a "crash" only a segfault?

Try `M-: (use-global-map (make-keymap)) RET`
Should we prevent users from doing that?

Let's focus on making it easy to make it work well, rather than making
it impossible to make it work poorly.

>>> BTW, my tests show that syntax-ppss can be rather slow, when the file is
>>> large enough (say 1 GB).
>> No doubt.  But it's no slower with long lines than with short lines.
>
> Yes, I wasn't clear enough, I should have written "when the file is large
> enough (say 1 GB), even without long lines".
>
> But you didn't answer my question: is it not possible to design a version of
> syntax-ppss that would approximate, with some heuristics, what syntax-ppss
> does, but on a smaller chunk of the buffer?

The answer is basically "no" but even before getting there, I have to
remind the reader that it hasn't really been requested.

In order to know if POS is within a string (which is one of the main
uses of `syntax-ppss`), you basically need to know if there's an odd or
even number of quotes before POS, which fundamentally needs to look at
all the chars between POS and BOB.  Of course we use a cache to try and
avoid looking at them over and over again, but the cache can't be of any
use the first time around.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 31.07.2022 13:50, Gregory Heytings wrote:
>>> BTW, my tests show that syntax-ppss can be rather slow, when the file 
>>> is large enough (say 1 GB).
>>
>> No doubt.  But it's no slower with long lines than with short lines.
>>
> 
> Yes, I wasn't clear enough, I should have written "when the file is 
> large enough (say 1 GB), even without long lines".

What kind of scenario are you thinking of that would exhibit this 
slowness in a 1 GB file?

If we're talking about syntax-ppss only, regular editing operations 
(typing and deleting code) limited to, say, one screen should trigger 
only a rescan of a limited area inside such buffer.

Both when the editing happens near the beginning or near the end of the 
buffer. Or in the middle -- no difference.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 2452 bytes --]


>>> Emacs is not in the business of preventing people from shooting 
>>> themselves in the foot.  If we need this narrowing to be enforced 
>>> because Emacs would otherwise crash, then it's OK, but if not, then we 
>>> *should* provide a way to undo it.
>>
>> And how do you define "crash"?
>
> Core dump.
>

Aha, interesting!  So an infinite loop is not a crash, according to that 
definition?

>> Is Emacs becoming unresponsive because an operation takes say two 
>> minutes to complete and cannot be interrupted a "crash"?  Or is a 
>> "crash" only a segfault?
>
> Try `M-: (use-global-map (make-keymap)) RET`
>
> Should we prevent users from doing that?
>

It's a misleading question.  No "user" would ever do that.  Sure, it's a 
nice example, but only an Elisp hacker would do that, in the middle of a 
debugging session, and they would do that on purpose (although perhaps 
without knowing the effect in advance).  Which has nothing to do with a 
regular user who just opens a file.

>
> Let's focus on making it easy to make it work well, rather than making 
> it impossible to make it work poorly.
>

You lost me here.  I've read that sentence twenty times, and cannot 
understand what you mean.

>> But you didn't answer my question: is it not possible to design a 
>> version of syntax-ppss that would approximate, with some heuristics, 
>> what syntax-ppss does, but on a smaller chunk of the buffer?
>
> The answer is basically "no" but even before getting there, I have to 
> remind the reader that it hasn't really been requested.
>

It has, now 😉  Not "requested", however.  I respectfully, with all due 
respect, ask whether doing such a thing would be possible.

>
> In order to know if POS is within a string (which is one of the main 
> uses of `syntax-ppss`), you basically need to know if there's an odd or 
> even number of quotes before POS, which fundamentally needs to look at 
> all the chars between POS and BOB.  Of course we use a cache to try and 
> avoid looking at them over and over again, but the cache can't be of any 
> use the first time around.
>

But if you use heuristics, as I said, you don't need to look at all the 
chars between BOB and POS.  You try your best to guess, on a small (a few 
kilobytes) portion of the buffer, where the strings most likely start and 
stop.  And if you're only right in 95% of the cases, that's more than 
fine.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Yes, I wasn't clear enough, I should have written "when the file is 
>> large enough (say 1 GB), even without long lines".
>
> What kind of scenario are you thinking of that would exhibit this 
> slowness in a 1 GB file?
>
> If we're talking about syntax-ppss only, regular editing operations 
> (typing and deleting code) limited to, say, one screen should trigger 
> only a rescan of a limited area inside such buffer.
>

That's true, but with such big files, the initial scan is slow.  So the 
scenario is simple: you open a big enough file, type M->, and C-p.  M-> 
will be instantaneous, and C-p will take a while, because of syntax-ppss.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 01:23, Gregory Heytings wrote:
> 
>>> Yes, I wasn't clear enough, I should have written "when the file is 
>>> large enough (say 1 GB), even without long lines".
>>
>> What kind of scenario are you thinking of that would exhibit this 
>> slowness in a 1 GB file?
>>
>> If we're talking about syntax-ppss only, regular editing operations 
>> (typing and deleting code) limited to, say, one screen should trigger 
>> only a rescan of a limited area inside such buffer.
>>
> 
> That's true, but with such big files, the initial scan is slow.  So the 
> scenario is simple: you open a big enough file, type M->, and C-p.  M-> 
> will be instantaneous, and C-p will take a while, because of syntax-ppss.

Yeah, ok. If you are going to visit EOB, a single full scan seems 
unavoidable. I don't think 'M->' should be instantaneous, though: to 
display the last page, you need to fontify it, and font-lock depends on 
syntax-ppss.

But one big slow scan (and how slow it is actually depends on a 
particular major mode) followed by responsive editing sounds much better 
than what we've had before.

So I would recommend against trying to solve this part right now. And 
yes, some faster approximations of syntax-propertize-rules are possible, 
especially if we ask individual language modes to provide "simpler" 
syntax rules.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Try `M-: (use-global-map (make-keymap)) RET`
>> Should we prevent users from doing that?
> It's a misleading question.  No "user" would ever do that.  Sure, it's
> a nice example, but only an Elisp hacker would do that, in the middle of
> a debugging session, and they would do that on purpose (although perhaps
> without knowing the effect in advance).  Which has nothing to do with
> a regular user who just opens a file.

FWIW, the above is my standard example because I ended up doing exactly
that by accident, locking myself out of the session I was trying to
debug, so in a sense, I'd have been happy (that one time) if Emacs had
prevented me from doing it.

>> Let's focus on making it easy to make it work well, rather than making it
>> impossible to make it work poorly.
> You lost me here.  I've read that sentence twenty times, and cannot
> understand what you mean.

Your current code makes it impossible for a major mode to make Emacs
slow by widening in a too-long-line.  I'd prefer if we made it easy
(i.e. the default) for Emacs to work well in that case, without making
it impossible for the major mode to mess things up.

E.g. use narrowing (and arrange for the known widening culprit to be
disabled) so that the default behavior is sane, but sllow an ELisp
package from re-widening (possibly using a specific call to do that) if
it thinks it's a good idea (even if it may turn out not to be so).

> But if you use heuristics, as I said, you don't need to look at all the
> chars between BOB and POS.  You try your best to guess, on a small (a few
> kilobytes) portion of the buffer, where the strings most likely start and
> stop.  And if you're only right in 95% of the cases, that's more than fine.

For specific languages, you can use various heuristics to guess which
quotes start and which quotes end a string (for some languages you can
even do it reliably), but `syntax-ppss` handles all kinds of languages
(and doesn't have access to such heuristics currently), such as ELisp
where it's hard to do it well.

I'd prefer to first see concrete examples where speeding up the
"syntax-ppss in a 1GB buffer" would make a significant difference to the
end-user's experience.  Then we can think about what's the better way to
solve the problem (which may be to just give up on font-lock altogether,
or maybe to refine the `syntax.el` code (maybe move some of it to C), or
to speed up `parse-partial-sexp`, or maybe let major modes provide
those heuristics to find a "safe point" again (these used to exist, see
`syntax-begin-function`, for example, but they tended to suck)).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> IOW, the issue for which I wanted to hear your opinion was whether you 
>> think it's okay to preclude, in a buffer with long lines, all 
>> pre/post-command-hooks from accessing the entire buffer.
>
> Yes, I think that's reasonable.  There will inevitably be many modes out 
> there that'll wedge themselves in the presence of gargantuan lines, and 
> limiting these hooks is a reasonable line of defence.
>

I've now done so, in a new branch feature/long-lines-improvements (to 
avoid breaking master).  A nice bonus is that it is not necessary anymore 
to disable flyspell-mode or flymake-mode for example, even in huge files.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> That's true, but with such big files, the initial scan is slow.  So the
> scenario is simple: you open a big enough file, type M->, and C-p.  M-> will
> be instantaneous, and C-p will take a while, because of syntax-ppss.

Really?  I'd expect that `M->` is slow because of `syntax-ppss`
(called by font-lock) and then `C-p` is instantaneous.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> But one big slow scan (and how slow it is actually depends on a 
> particular major mode) followed by responsive editing sounds much better 
> than what we've had before.
>

Indeed.  But then the question is: is it possible to do that scan while 
opening the file, before it becomes editable?  It is way better to wait a 
few seconds more while the file is being opened than to wait before two 
basic motion commands when the file is already opened.

>
> So I would recommend against trying to solve this part right now.
>

It doesn't only solve the syntax-ppss problem, it also makes flyspell-mode 
usable in such files, for example.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Here are my thoughts.
>

Thanks!

>
> First, I think the setting of narrowed_begv and narrowed_zv should be 
> done in 'reseat', not in init_iterator.  The latter always calls the 
> former when invoked to start iteration of buffer text, but we also call 
> 'reseat' from other places, when we "jump" the iterator to a new place, 
> potentially far from the last.  If nothing else, this should help with 
> truncate-lines, where using window_point is basically right only for the 
> point's line.  And currently, init_iterator computes and sets the 
> narrowing even when we iterate on strings, which is unneeded and 
> incorrect.
>
> Whether always to correct narrowed_begv and narrowed_zv if we are 
> reseating to a position outside the narrowing, is a more complicated 
> question.  The basic problem here is that we don't have an easy way of 
> restoring the previous narrowing (except by unwind_protect), and the 
> display code sometimes calls init_iterator or start_display using the 
> iterator that already has these members set by previous code, a 
> situation which we currently cannot easily detect.  However, when this 
> code runs as part of redisplay, we generally don't expect the original 
> narrowing to be insufficient, except perhaps in the truncate-line case.
>
> So I think we should correct narrowed_begv and narrowed_zv only if 
> either the 'redisplaying_p' flag is reset (meaning the display code is 
> being invoked outside of redisplay) or it->line_wrap == TRUNCATE.
>

I admit I do not really understand your last two paragraphs, but I tried 
to do what you suggested, and it doesn't seem to introduce regressions, so 
I pushed it to the new feature branch.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> But one big slow scan (and how slow it is actually depends on a particular
> major mode)

Indeed, I think the `syntax-ppss` part itself should be fast enough even
if very large files.  But the `syntax-propertize` part (which is called
by `syntax-ppss`) can take a long time in some major modes.

In those major modes where that's a problem (i.e. major modes that have
a complex `syntax-propertize-function` and that also happen to be used
in very large files) maybe it would be worth (re)introducing some sort
of `syntax(-propertize)-begin-function`.  But these kinds of heuristics
have proved problematic over the years (and they'd introduce extra
complexity since we won't be able to just rely on
a `syntax-propertize-done` high-watermark to know what's been
propertized and what hasn't, combined with the interaction with the
`syntax-ppss` cache), so we'd have to try a few different approaches.

In any case this is not a long-lines problem.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> Let's focus on making it easy to make it work well, rather than making 
>>> it impossible to make it work poorly.
>>
>> You lost me here.  I've read that sentence twenty times, and cannot 
>> understand what you mean.
>
> Your current code makes it impossible for a major mode to make Emacs 
> slow by widening in a too-long-line.
>

Which is a good thing, isn't it?  Or do you think that it's okay for Emacs 
to become unresponsive just because it is busy highlighting characters in 
the buffer?

>
> I'd prefer if we made it easy (i.e. the default) for Emacs to work well 
> in that case, without making it impossible for the major mode to mess 
> things up.
>

Sure, I would also welcome a better solution.  Until it materializes, the 
only reasonable way is to use a less optimal solution.

>
> For specific languages, you can use various heuristics to guess which 
> quotes start and which quotes end a string (for some languages you can 
> even do it reliably), but `syntax-ppss` handles all kinds of languages 
> (and doesn't have access to such heuristics currently), such as ELisp 
> where it's hard to do it well.
>

Don't worry, I've not yet seen an Elisp file with long lines.  If using 
various heuristics is sometimes or often feasible, that's already a good 
thing.

>
> I'd prefer to first see concrete examples where speeding up the 
> "syntax-ppss in a 1GB buffer" would make a significant difference to the 
> end-user's experience.
>

I just sent one such example to Dmitry.  And I pointed to another possible 
solution, namely to scan the whole buffer while opening it (instead of 
scanning it lazily, which is IIUC what currently happens).  From a user 
viewpoint, it's understandable that opening a big file takes some time.

>
> Then we can think about what's the better way to solve the problem 
> (which may be to just give up on font-lock altogether,
>

That would be regrettable, given the amount of effort that has been put 
into making font-lock work "as much as possible".

>
> or maybe to refine the `syntax.el` code (maybe move some of it to C), or 
> to speed up `parse-partial-sexp`, or maybe let major modes provide those 
> heuristics to find a "safe point" again (these used to exist, see 
> `syntax-begin-function`, for example, but they tended to suck)).
>

All this is possible.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> That's true, but with such big files, the initial scan is slow.  So the 
>> scenario is simple: you open a big enough file, type M->, and C-p. 
>> M-> will be instantaneous, and C-p will take a while, because of 
>> syntax-ppss.
>
> Really?  I'd expect that `M->` is slow because of `syntax-ppss` (called 
> by font-lock) and then `C-p` is instantaneous.
>

Yes, really.  M-> is fast because syntax-ppss is called inside 
fontification-functions, which are evaluated in a small portion of the 
buffer (with locked narrowing).  And C-p is slow because post-command-hook 
is (or rather was) not subjected to the same locked narrowing.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> But one big slow scan (and how slow it is actually depends on a 
>> particular major mode) followed by responsive editing sounds much 
>> better than what we've had before.
>
> Indeed.  But then the question is: is it possible to do that scan while 
> opening the file, before it becomes editable?  It is way better to wait 
> a few seconds more while the file is being opened than to wait before 
> two basic motion commands when the file is already opened.
>

Sorry, I meant "to wait _between_ two basic motion commands".

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 01:50, Gregory Heytings wrote:
> 
>>
>> But one big slow scan (and how slow it is actually depends on a 
>> particular major mode) followed by responsive editing sounds much 
>> better than what we've had before.
>>
> 
> Indeed.  But then the question is: is it possible to do that scan while 
> opening the file, before it becomes editable?

IIUC this state of affairs is caused by your chosen approach to speeding 
up font-lock (hard narrowing while it is called), which makes the 
initial call to syntax-ppss happen inside that narrowing as well.

The alternative being that font-lock would call syntax-ppss right away 
with no restriction, but then only apply highlighting to limited parts 
of the buffer.

 > It is way better to wait
 > a few seconds more while the file is being opened than to wait before
 > two basic motion commands when the file is already opened.

I agree, yes.

>> So I would recommend against trying to solve this part right now.
>>
> 
> It doesn't only solve the syntax-ppss problem, it also makes 
> flyspell-mode usable in such files, for example.

Does flyspell-mode always scan the full buffer?

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> That's true, but with such big files, the initial scan is slow.  So the
>>> scenario is simple: you open a big enough file, type M->, and C-p. M->
>>> will be instantaneous, and C-p will take a while, because of syntax-ppss.
>> Really?  I'd expect that `M->` is slow because of `syntax-ppss` (called by
>> font-lock) and then `C-p` is instantaneous.
> Yes, really.  M-> is fast because syntax-ppss is called inside
> fontification-functions, which are evaluated in a small portion of the
> buffer (with locked narrowing).

Ah, you're talking about a file with a long line.  I was thinking of just
a normal large file.

> And C-p is slow because post-command-hook is (or rather was) not
> subjected to the same locked narrowing.

I wonder what `C-p` has to do with `post-command-hook`.
After all, that same `post-command-hook` is also run after `M->` as well.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Your current code makes it impossible for a major mode to make Emacs slow
>> by widening in a too-long-line.
> Which is a good thing, isn't it?

No: we don't want to prevent people from shooting themselves in the foot.
Basically if it says "impossible" there's a good chance it's not a good thing.

> Or do you think that it's okay for Emacs to become unresponsive just
> because it is busy highlighting characters in the buffer?

Widening in a too long line will not necessarily lead to an unresponsive
Emacs, so by preventing it you're also preventing useful cases.

>> I'd prefer if we made it easy (i.e. the default) for Emacs to work well in
>> that case, without making it impossible for the major mode to mess
>> things up.
> Sure, I would also welcome a better solution.  Until it materializes, the
> only reasonable way is to use a less optimal solution.

It's not difficult to make it possible to re-widen after your narrowing.

> I just sent one such example to Dmitry.  And I pointed to another possible
> solution, namely to scan the whole buffer while opening it (instead of
> scanning it lazily, which is IIUC what currently happens).  From a user
> viewpoint, it's understandable that opening a big file takes some time.

We used to scan eagerly in the background with `jit-lock-stealth`, but that was
not very popular (eats up your battery for fairly little benefit).
We also have "lazier" highlighting via `jit-lock-defer`, but that hasn't
been adapted to `syntax-ppss`.  It might be with investigating.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1976 bytes --]


>>> Your current code makes it impossible for a major mode to make Emacs 
>>> slow by widening in a too-long-line.
>>
>> Which is a good thing, isn't it?
>
> No: we don't want to prevent people from shooting themselves in the 
> foot. Basically if it says "impossible" there's a good chance it's not a 
> good thing.
>

Apparently you don't understand what I'm doing.  I'm trying to make Emacs 
handle exceptional cases gracefully, by:

1. adding as few hard limits as possible, and

2. keeping as many functionalities as possible (my task would have been _a 
lot_ easier if I had just stopped after saying "just open these files with 
M-x find-file-literally").

If you think there's anything wrong with that approach, I'll have to 
conclude that we live on different planets.  On Earth and on Theory, 
maybe?

And no, it's not even "impossible" for Emacs uses to shoot themselves in 
the foot, so your argument collapses.  Those who prefer the "Traditional 
Emacs Approach"™ when it encounters such exceptional cases, that is, 
those who:

1. do not want to add any limits whatsoever in any circumstance, and

2. agree to take the risk of making Emacs completely unfonctional,

can simply add (setq long-line-threshold nil) in their init file.

>> I just sent one such example to Dmitry.  And I pointed to another 
>> possible solution, namely to scan the whole buffer while opening it 
>> (instead of scanning it lazily, which is IIUC what currently happens). 
>> From a user viewpoint, it's understandable that opening a big file 
>> takes some time.
>
> We used to scan eagerly in the background with `jit-lock-stealth`, but 
> that was not very popular (eats up your battery for fairly little 
> benefit). We also have "lazier" highlighting via `jit-lock-defer`, but 
> that hasn't been adapted to `syntax-ppss`.  It might be worth 
> investigating.
>

It might be, indeed.  But I will most probably not do that myself.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>>> That's true, but with such big files, the initial scan is slow.  So 
>>>> the scenario is simple: you open a big enough file, type M->, and 
>>>> C-p. M-> will be instantaneous, and C-p will take a while, because of 
>>>> syntax-ppss.
>>>
>>> Really?  I'd expect that `M->` is slow because of `syntax-ppss` 
>>> (called by font-lock) and then `C-p` is instantaneous.
>>
>> Yes, really.  M-> is fast because syntax-ppss is called inside 
>> fontification-functions, which are evaluated in a small portion of the 
>> buffer (with locked narrowing).
>>
>> [...]
>> 
>> And C-p is slow because post-command-hook is (or rather was) not 
>> subjected to the same locked narrowing.
>
> I wonder what `C-p` has to do with `post-command-hook`.
>
> After all, that same `post-command-hook` is also run after `M->` as 
> well.
>

You're right here.  Technically it's M-> that is slow.  But it's not what 
the user sees (hence my own confusion): the effect of M-> is immediately 
visible.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

Given your and Dmitry's feedback, I just tried to add an explicit call to 
(syntax-ppss (point-max)), without narrowing, when the buffer is opened 
(see below).

The problem is that this is, as I said, slow.  On my laptop, opening a 1 
GB file takes about 6 seconds.  The call to syntax-ppss adds 70 seconds, 
so opening a large file becomes an order of magnitude slower (13 times 
slower).  Which I think is too much for the added benefit.

diff --git a/src/buffer.c b/src/buffer.c
index a07194aef7..bff6dce1d7 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -986,6 +986,7 @@ reset_buffer (register struct buffer *b)
    b->clip_changed = 0;
    b->prevent_redisplay_optimizations_p = 1;
    b->long_line_optimizations_p = 0;
+  b->long_line_syntax_ppss_done_p = 0;
    bset_backed_up (b, Qnil);
    bset_local_minor_modes (b, Qnil);
    BUF_AUTOSAVE_MODIFF (b) = 0;
@@ -2448,6 +2449,7 @@ #define swapfield_(field, type) \
    current_buffer->prevent_redisplay_optimizations_p = 1;
    other_buffer->prevent_redisplay_optimizations_p = 1;
    swapfield (long_line_optimizations_p, bool_bf);
+  swapfield (long_line_syntax_ppss_done_p, bool_bf);
    swapfield (overlays_before, struct Lisp_Overlay *);
    swapfield (overlays_after, struct Lisp_Overlay *);
    swapfield (overlay_center, ptrdiff_t);
diff --git a/src/buffer.h b/src/buffer.h
index 47b4bdf749..3e020f1953 100644
--- a/src/buffer.h
+++ b/src/buffer.h
@@ -686,6 +686,8 @@ #define BVAR(buf, field) ((buf)->field ## _)
       display optimizations must be used.  */
    bool_bf long_line_optimizations_p : 1;

+  bool_bf long_line_syntax_ppss_done_p : 1;
+
    /* List of overlays that end at or before the current center,
       in order of end-position.  */
    struct Lisp_Overlay *overlays_before;
diff --git a/src/xdisp.c b/src/xdisp.c
index 8a19b3bda9..d70ab6f9c1 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4391,6 +4391,13 @@ handle_fontified_prop (struct it *it)

        if (current_buffer->long_line_optimizations_p)
  	{
+	  if (!current_buffer->long_line_syntax_ppss_done_p)
+	    {
+	      current_buffer->long_line_syntax_ppss_done_p = 1;
+	      specbind (Qinhibit_quit, Qt);
+	      CALLN (Ffuncall, intern ("syntax-ppss"), Fpoint_max ());
+	      unbind_to (count, Qnil);
+	    }
  	  ptrdiff_t begv = it->narrowed_begv ? it->narrowed_begv : BEGV;
  	  ptrdiff_t zv = it->narrowed_zv;
  	  ptrdiff_t charpos = IT_CHARPOS (*it);

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 12:34, Gregory Heytings wrote:
> 
> Given your and Dmitry's feedback, I just tried to add an explicit call 
> to (syntax-ppss (point-max)), without narrowing, when the buffer is 
> opened (see below).
> 
> The problem is that this is, as I said, slow.  On my laptop, opening a 1 
> GB file takes about 6 seconds.  The call to syntax-ppss adds 70 seconds, 
> so opening a large file becomes an order of magnitude slower (13 times 
> slower).  Which I think is too much for the added benefit.

But that only has to happen when the buffer is scrolled to the bottom, 
right?

And syntax-ppss's speed depends on the rules applied by the particular 
major mode. Those could be sped up. Some optimization of this function's 
speed is not out of the question either.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]


>> Given your and Dmitry's feedback, I just tried to add an explicit call 
>> to (syntax-ppss (point-max)), without narrowing, when the buffer is 
>> opened (see below).
>> 
>> The problem is that this is, as I said, slow.  On my laptop, opening a 
>> 1 GB file takes about 6 seconds.  The call to syntax-ppss adds 70 
>> seconds, so opening a large file becomes an order of magnitude slower 
>> (13 times slower).  Which I think is too much for the added benefit.
>
> But that only has to happen when the buffer is scrolled to the bottom, 
> right?
>

No, it happens when the buffer is opened.  Given the importance that you 
and Stefan seem to give to that function, it is, with the patch I sent in 
my previous post, called once on the whole buffer (without any narrowing) 
when the file is opened.  Later calls (inside fontification-functions or 
post-command-hook) are subject to a forced narrowing.

>
> And syntax-ppss's speed depends on the rules applied by the particular 
> major mode. Those could be sped up. Some optimization of this function's 
> speed is not out of the question either.
>

They would be more than welcome.  In fact, without such optimizations, it 
would be unreasonable to do what the patch does.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> And syntax-ppss's speed depends on the rules applied by the particular 
>> major mode. Those could be sped up. Some optimization of this 
>> function's speed is not out of the question either.
>
> They would be more than welcome.  In fact, without such optimizations, 
> it would be unreasonable to do what the patch does.
>

To clarify: it should become at least six times faster.  That would mean 
that opening a large file would take 3N seconds instead of N seconds, 
which would be okay.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 12:56, Gregory Heytings wrote:
> 
>>> Given your and Dmitry's feedback, I just tried to add an explicit 
>>> call to (syntax-ppss (point-max)), without narrowing, when the buffer 
>>> is opened (see below).
>>>
>>> The problem is that this is, as I said, slow.  On my laptop, opening 
>>> a 1 GB file takes about 6 seconds.  The call to syntax-ppss adds 70 
>>> seconds, so opening a large file becomes an order of magnitude slower 
>>> (13 times slower).  Which I think is too much for the added benefit.
>>
>> But that only has to happen when the buffer is scrolled to the bottom, 
>> right?
>>
> 
> No, it happens when the buffer is opened.  Given the importance that you 
> and Stefan seem to give to that function, it is, with the patch I sent 
> in my previous post, called once on the whole buffer (without any 
> narrowing) when the file is opened.

But if the buffer is not scrolled to the end, shouldn't it be called 
with a position that's close to the beginning?

That shouldn't force the full buffer scan, meaning this call should 
complete quickly.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1610 bytes --]


>> No, it happens when the buffer is opened.  Given the importance that 
>> you and Stefan seem to give to that function, it is, with the patch I 
>> sent in my previous post, called once on the whole buffer (without any 
>> narrowing) when the file is opened.
>
> But if the buffer is not scrolled to the end, shouldn't it be called 
> with a position that's close to the beginning?
>
> That shouldn't force the full buffer scan, meaning this call should 
> complete quickly.
>

I'm trying to reconcile two conflicting constraints.

It is necessary to add a locked narrowing around fontification-functions 
and pre/post-command-hook to ensure that Emacs remains responsive.

At the same time, you and Stefan tell me that syntax-ppss does an 
important job and will not do it correctly with such a locked narrowing, 
IOW, that at least syntax-ppss should be called without a locked 
narrowing.  But you also tell me that its result is cached so that a full 
buffer scan isn't necessary anymore when it has happened at least once.

So what I'm suggesting is to do a full buffer scan immediately, when the 
file is opened, without any narrowing.  If that happens, later calls to 
syntax-ppss inside fontification-functions and pre/post-command-hook will 
use the cached result of the initial scan, and will do their job correctly 
even with a locked narrowing.

Unless I misunderstand something, I think (and my tests seem to confirm) 
that that would be a workable solution, provided that the initial scan is 
reasonably fast, that is, at least six times faster than it is now.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The problem is that this is, as I said, slow.  On my laptop, opening a 1 GB
> file takes about 6 seconds.  The call to syntax-ppss adds 70 seconds, so
> opening a large file becomes an order of magnitude slower (13 times slower).

It's meaningless to talk about the time taken by `syntax-ppss` without specifying
the major mode that was in use.
You might also want to compare to the time to run

    (parse-partial-sexp (point-min) (point-max))

which is a kind of "speed of light" for `syntax-ppss`.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> The problem is that this is, as I said, slow.  On my laptop, opening a 
>> 1 GB file takes about 6 seconds.  The call to syntax-ppss adds 70 
>> seconds, so opening a large file becomes an order of magnitude slower 
>> (13 times slower).
>
> It's meaningless to talk about the time taken by `syntax-ppss` without 
> specifying the major mode that was in use.
>

It isn't.  The benchmark above was with a JSON file (js-mode), but you'll 
see the same ratio with an Elisp file for example:

for I in $(seq 1 2500); do cat lisp/simple.el; done > complex.el

That file opens in about 5 seconds, and (benchmark-run 1 (syntax-ppss 
(point-max))) takes about 45 seconds.

Sure, there are perhaps modes that are slower, but my tests seem to 
indicate that the 1/10 ratio is correct, or IOW that syntax-ppss is an 
order of magnitude slower than opening the file.

>
> You might also want to compare to the time to run
>
> (parse-partial-sexp (point-min) (point-max))
>
> which is a kind of "speed of light" for `syntax-ppss`.
>

What do you mean?  With the above file (benchmark-run 1 
(parse-partial-sexp (point-min) (point-max))) takes 55 seconds.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  56682@debbugs.gnu.org
> Date: Sun, 31 Jul 2022 18:45:16 -0400
> 
> Your current code makes it impossible for a major mode to make Emacs
> slow by widening in a too-long-line.  I'd prefer if we made it easy
> (i.e. the default) for Emacs to work well in that case, without making
> it impossible for the major mode to mess things up.
> 
> E.g. use narrowing (and arrange for the known widening culprit to be
> disabled) so that the default behavior is sane, but sllow an ELisp
> package from re-widening (possibly using a specific call to do that) if
> it thinks it's a good idea (even if it may turn out not to be so).

The problem is that too many popular major modes, notably including
those whose files tend to like having very long lines, are slow in
their fontifications.  In fact, I'd challenge you to find a major mode
that doesn't present such a degradation in behavior with long lines
(you should be able to measure it by comparing performance and
responsiveness in a buffer with and without font-lock).

Given this situation, it sounds reasonable to start by restricting
font-lock.

As I wrote elsewhere, I'm okay with extending 'widen' so that it could
"unlock" the locked narrowing, which could then be used in major modes
that convince us their performance is adequate (or clearly announce in
their docs that they don't care about files with long lines ;-).

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 31 Jul 2022 22:50:23 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>, 
>     Stefan Monnier <monnier@iro.umontreal.ca>
> 
> It is way better to wait a few seconds more while the file is being
> opened than to wait before two basic motion commands when the file
> is already opened.

I don't think I agree.  Having to wait for the initial display is
pretty annoying.  In fact, this was how the original font-lock worked:
it was triggered by find-file, and would fontify the entire buffer
before showing any of it.  That led very quickly to the likes of
lazy-lock and fast-lock, and was eventually fixed by jit-lock.

Try visiting a very large file encoded in something of the ISO-2022
family, and you will see what I mean.  It's an annoyance.

I have jit-lock-stealth enabled because I don't like the initial wait,
but don't want any subsequent waits, either.  We also have the (little
used, for some reason) jit-lock-defer feature: if you set
jit-lock-defer-time to a large enough value, M-> followed by C-p will
not be as slow as they are now, I think.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 1 Aug 2022 04:23:21 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> IIUC this state of affairs is caused by your chosen approach to speeding 
> up font-lock (hard narrowing while it is called), which makes the 
> initial call to syntax-ppss happen inside that narrowing as well.
> 
> The alternative being that font-lock would call syntax-ppss right away 
> with no restriction, but then only apply highlighting to limited parts 
> of the buffer.

AFAIU, this seems to assume that highlighting is much faster than
syntax-ppss.  Is that a given?  If not, I don't think I understand how
this could help.

>  > It is way better to wait
>  > a few seconds more while the file is being opened than to wait before
>  > two basic motion commands when the file is already opened.
> 
> I agree, yes.

I don't.

> > It doesn't only solve the syntax-ppss problem, it also makes 
> > flyspell-mode usable in such files, for example.
> 
> Does flyspell-mode always scan the full buffer?

flyspell-mode doesn't scan anything but the text you type, as you type
it.  Anything else must be explicitly invoked by the user.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  56682@debbugs.gnu.org
> Date: Mon, 01 Aug 2022 03:11:43 -0400
> 
> >> Your current code makes it impossible for a major mode to make Emacs slow
> >> by widening in a too-long-line.
> > Which is a good thing, isn't it?
> 
> No: we don't want to prevent people from shooting themselves in the foot.

But the problem here is that it isn't "people shooting themselves in
the foot", it's that "major modes shoot their users in the foot".
IOW, the ones who shoot and the ones who get shot are not the same
people.

What do you want a user to do when he/she is faced with a mode which
makes Emacs very slow?  Such a user cannot blame his/herself; in many
cases the use doesn't even know enough to realize it's the major mode
and its fontifications that are the culprit.

> Widening in a too long line will not necessarily lead to an unresponsive
> Emacs, so by preventing it you're also preventing useful cases.

The experience up till now says otherwise.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> It is way better to wait a few seconds more while the file is being 
>> opened than to wait before two basic motion commands when the file is 
>> already opened.
>
> I don't think I agree.  Having to wait for the initial display is pretty 
> annoying.
>

It is, indeed.  But here we are talking about big files (say 1 GB).  In 
that case (and in that case only) it is much better, from a user 
viewpoint, to wait say 20 seconds before the file is opened and being at 
that point able to freely move through the file, instead of waiting only 6 
seconds, and then having to wait another 10 seconds between two motion 
commands like M-> C-p.  In fact, no user expects that a 1 GB file would 
open instantaneously.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 31 Jul 2022 22:54:00 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> > Whether always to correct narrowed_begv and narrowed_zv if we are 
> > reseating to a position outside the narrowing, is a more complicated 
> > question.  The basic problem here is that we don't have an easy way of 
> > restoring the previous narrowing (except by unwind_protect), and the 
> > display code sometimes calls init_iterator or start_display using the 
> > iterator that already has these members set by previous code, a 
> > situation which we currently cannot easily detect.  However, when this 
> > code runs as part of redisplay, we generally don't expect the original 
> > narrowing to be insufficient, except perhaps in the truncate-line case.
> >
> > So I think we should correct narrowed_begv and narrowed_zv only if 
> > either the 'redisplaying_p' flag is reset (meaning the display code is 
> > being invoked outside of redisplay) or it->line_wrap == TRUNCATE.
> >
> 
> I admit I do not really understand your last two paragraphs, but I tried 
> to do what you suggested, and it doesn't seem to introduce regressions, so 
> I pushed it to the new feature branch.

Thanks.  I will try to explain some more; feel free to ask questions
if something's still unclear.

You can see in the sources that both init_iterator and start_display
are many times called with 'struct it' that was used by the caller, so
it could already have the narrowed_begv and narrowed_zv members
initialized by the caller.  If we discover that we want to recalculate
these values, we'd then need to restore the previous value before we
return, so that the caller will see the same values it used before the
call.  But we have no easy way of doing that, and moreover, cannot
even detect that these members were initialized.  The inability to
detect that they were initialized is due to the fact that we don't
initialize 'struct it' before we call init_iterator, and so these
fields can originally have any arbitrary value.  Which means, for
example, that tests like this:

  if (current_buffer->long_line_optimizations_p)
    {
      if (!it->narrowed_begv  <<<<<<<<<<<<<<<<<<<<<<<<<<
         || ((pos.charpos < it->narrowed_begv || pos.charpos > it->narrowed_zv)
             && (!redisplaying_p || it->line_wrap == TRUNCATE)))

are not necessarily reliable, because we never initialize
narrowed_begv to zero, AFAICT.  Right?

The other part of my reasoning is that callers of the display code
which are outside redisplay could legitimately move the iterator far
from point: think about pos-visible-in-window-p and its ilk.  So, when
we are not called by redisplay, I think it would be preferable to
update the narrowing due to these considerations.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> You can see in the sources that both init_iterator and start_display are 
> many times called with 'struct it' that was used by the caller, so it 
> could already have the narrowed_begv and narrowed_zv members initialized 
> by the caller.  If we discover that we want to recalculate these values, 
> we'd then need to restore the previous value before we return, so that 
> the caller will see the same values it used before the call.  But we 
> have no easy way of doing that, and moreover, cannot even detect that 
> these members were initialized.  The inability to detect that they were 
> initialized is due to the fact that we don't initialize 'struct it' 
> before we call init_iterator, and so these fields can originally have 
> any arbitrary value.
>

Thanks, it's much clearer now.

>
> Which means, for example, that tests like this:
>
>  if (current_buffer->long_line_optimizations_p)
>    {
>      if (!it->narrowed_begv  <<<<<<<<<<<<<<<<<<<<<<<<<<
>         || ((pos.charpos < it->narrowed_begv || pos.charpos > it->narrowed_zv)
>             && (!redisplaying_p || it->line_wrap == TRUNCATE)))
>
> are not necessarily reliable, because we never initialize narrowed_begv 
> to zero, AFAICT.  Right?
>

Indeed.  That wasn't a problem with the previous code in which 
narrowed_begv was unconditionally assigned.  Now it is.  I think the 
following change should be enough to fix this.  Agreed?

diff --git a/src/xdisp.c b/src/xdisp.c
index 8a19b3bda9..9574d06bd5 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -3472,6 +3472,9 @@ init_iterator (struct it *it, struct window *w,
                         &it->bidi_it);
         }

+      if (current_buffer->long_line_optimizations_p)
+       it->narrowed_begv = 0;
+
        /* Compute faces etc.  */
        reseat (it, it->current.pos, true);
      }

>
> The other part of my reasoning is that callers of the display code which 
> are outside redisplay could legitimately move the iterator far from 
> point: think about pos-visible-in-window-p and its ilk.  So, when we are 
> not called by redisplay, I think it would be preferable to update the 
> narrowing due to these considerations.
>

Thanks, this too clarifies what you meant.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 01 Aug 2022 12:20:39 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: dgutov@yandex.ru, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> > Having to wait for the initial display is pretty annoying.
> 
> It is, indeed.  But here we are talking about big files (say 1 GB).  In 
> that case (and in that case only) it is much better, from a user 
> viewpoint, to wait say 20 seconds before the file is opened and being at 
> that point able to freely move through the file, instead of waiting only 6 
> seconds, and then having to wait another 10 seconds between two motion 
> commands like M-> C-p.  In fact, no user expects that a 1 GB file would 
> open instantaneously.

You can maybe have that for C-p that follows M->, but wouldn't the
wait return, with a vengeance, if you insert a single character
(because then the buffer needs to be re-scanned)?  If so, we've gained
nothing, really.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 01 Aug 2022 12:51:47 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> > Which means, for example, that tests like this:
> >
> >  if (current_buffer->long_line_optimizations_p)
> >    {
> >      if (!it->narrowed_begv  <<<<<<<<<<<<<<<<<<<<<<<<<<
> >         || ((pos.charpos < it->narrowed_begv || pos.charpos > it->narrowed_zv)
> >             && (!redisplaying_p || it->line_wrap == TRUNCATE)))
> >
> > are not necessarily reliable, because we never initialize narrowed_begv 
> > to zero, AFAICT.  Right?
> >
> 
> Indeed.  That wasn't a problem with the previous code in which 
> narrowed_begv was unconditionally assigned.  Now it is.  I think the 
> following change should be enough to fix this.  Agreed?

Yes, of course.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> It is, indeed.  But here we are talking about big files (say 1 GB). 
>> In that case (and in that case only) it is much better, from a user 
>> viewpoint, to wait say 20 seconds before the file is opened and being 
>> at that point able to freely move through the file, instead of waiting 
>> only 6 seconds, and then having to wait another 10 seconds between two 
>> motion commands like M-> C-p.  In fact, no user expects that a 1 GB 
>> file would open instantaneously.
>
> You can maybe have that for C-p that follows M->, but wouldn't the wait 
> return, with a vengeance, if you insert a single character (because then 
> the buffer needs to be re-scanned)?  If so, we've gained nothing, 
> really.
>

Fortunately no: the buffer doesn't need to be rescanned, syntax-ppss 
caches its result, to avoid having to rescan the whole buffer again and 
again.  At least that's what Stefan and Dmitry told me, and that's what I 
see in my tests.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 01 Aug 2022 13:14:20 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: dgutov@yandex.ru, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> > You can maybe have that for C-p that follows M->, but wouldn't the wait 
> > return, with a vengeance, if you insert a single character (because then 
> > the buffer needs to be re-scanned)?  If so, we've gained nothing, 
> > really.
> 
> Fortunately no: the buffer doesn't need to be rescanned, syntax-ppss 
> caches its result, to avoid having to rescan the whole buffer again and 
> again.

But the buffer has changed, so the cache is not necessarily valid,
right?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 01 Aug 2022 12:51:47 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> diff --git a/src/xdisp.c b/src/xdisp.c
> index 8a19b3bda9..9574d06bd5 100644
> --- a/src/xdisp.c
> +++ b/src/xdisp.c
> @@ -3472,6 +3472,9 @@ init_iterator (struct it *it, struct window *w,
>                          &it->bidi_it);
>          }
> 
> +      if (current_buffer->long_line_optimizations_p)
> +       it->narrowed_begv = 0;
> +

Sorry, I wrote that this is OK, but it isn't: if init_iterator is
called with 'struct it' that was already initialized by a previous
call to 'reseat', the above will nuke the narrowing.

So we need something more complicated.  ATM I don't see how to solve
this without manually initializing narrowed_begv before the first call
to init_iterator or start_display.  Hmm...

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> Which means, for example, that tests like this:
>>>
>>>  if (current_buffer->long_line_optimizations_p)
>>>    {
>>>      if (!it->narrowed_begv  <<<<<<<<<<<<<<<<<<<<<<<<<<
>>>         || ((pos.charpos < it->narrowed_begv || pos.charpos > it->narrowed_zv)
>>>             && (!redisplaying_p || it->line_wrap == TRUNCATE)))
>>>
>>> are not necessarily reliable, because we never initialize 
>>> narrowed_begv to zero, AFAICT.  Right?
>>
>> Indeed.  That wasn't a problem with the previous code in which 
>> narrowed_begv was unconditionally assigned.  Now it is.  I think the 
>> following change should be enough to fix this.  Agreed?
>
> Yes, of course.
>

Thanks; now pushed.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> You can maybe have that for C-p that follows M->, but wouldn't the 
>>> wait return, with a vengeance, if you insert a single character 
>>> (because then the buffer needs to be re-scanned)?  If so, we've gained 
>>> nothing, really.
>>
>> Fortunately no: the buffer doesn't need to be rescanned, syntax-ppss 
>> caches its result, to avoid having to rescan the whole buffer again and 
>> again.
>
> But the buffer has changed, so the cache is not necessarily valid, 
> right?
>

I didn't look at the internals of syntax-ppss, but I'd guess (and again my 
tests seem to confirm) that it was designed well enough, and doesn't need 
to rescan the whole buffer whenever a single character is inserted.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]


>> diff --git a/src/xdisp.c b/src/xdisp.c
>> index 8a19b3bda9..9574d06bd5 100644
>> --- a/src/xdisp.c
>> +++ b/src/xdisp.c
>> @@ -3472,6 +3472,9 @@ init_iterator (struct it *it, struct window *w,
>>                          &it->bidi_it);
>>          }
>>
>> +      if (current_buffer->long_line_optimizations_p)
>> +       it->narrowed_begv = 0;
>> +
>
> Sorry, I wrote that this is OK, but it isn't: if init_iterator is called 
> with 'struct it' that was already initialized by a previous call to 
> 'reseat', the above will nuke the narrowing.
>
> So we need something more complicated.  ATM I don't see how to solve 
> this without manually initializing narrowed_begv before the first call 
> to init_iterator or start_display.  Hmm...
>

Hmmm...  So I wasn't completely wrong when I wasn't sure it was TRT 😉 
It's a chicken-and-egg problem.  On the other hand, even if we nuke the 
narrowing there, it will be recomputed two lines later, which should be 
okay.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 01 Aug 2022 16:24:00 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org,
>  monnier@iro.umontreal.ca
> 
> > diff --git a/src/xdisp.c b/src/xdisp.c
> > index 8a19b3bda9..9574d06bd5 100644
> > --- a/src/xdisp.c
> > +++ b/src/xdisp.c
> > @@ -3472,6 +3472,9 @@ init_iterator (struct it *it, struct window *w,
> >                          &it->bidi_it);
> >          }
> > 
> > +      if (current_buffer->long_line_optimizations_p)
> > +       it->narrowed_begv = 0;
> > +
> 
> Sorry, I wrote that this is OK, but it isn't: if init_iterator is
> called with 'struct it' that was already initialized by a previous
> call to 'reseat', the above will nuke the narrowing.
> 
> So we need something more complicated.  ATM I don't see how to solve
> this without manually initializing narrowed_begv before the first call
> to init_iterator or start_display.  Hmm...

I think we should simply unconditionally recompute the narrowing in
'reseat'.  At least I couldn't think of a situation where that would
cause trouble, and 'reseat' is called rarely enough not to make this
expensive.  Am I missing something?

And another nit:

  if (current_buffer->long_line_optimizations_p)
    {
      if (!it->narrowed_begv
	  || ((pos.charpos < it->narrowed_begv || pos.charpos > it->narrowed_zv)
	      && (!redisplaying_p || it->line_wrap == TRUNCATE)))
	{
	  it->narrowed_begv = get_narrowed_begv (it->w, window_point (it->w));
	  it->narrowed_zv = get_narrowed_zv (it->w, window_point (it->w));
	}
    }

I think this should pass pos.charpos as the 2nd argument to
get_narrowed_begv and get_narrowed_zv, otherwise it might not really
correct anything, right?  In particular, when lines are truncated,
that will definitely happen when we display any line but the very
first.

Or perhaps we should check that using window-point indeed brings
pos.charpos into the narrowed region, and only use pos.charpos if it
doesn't?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I think we should simply unconditionally recompute the narrowing in 
> 'reseat'.  At least I couldn't think of a situation where that would 
> cause trouble, and 'reseat' is called rarely enough not to make this 
> expensive.  Am I missing something?
>

I don't think you are missing something, and that's what I suggested too. 
So there is nothing to change here, right?

>
> And another nit:
>
>  if (current_buffer->long_line_optimizations_p)
>    {
>      if (!it->narrowed_begv
> 	  || ((pos.charpos < it->narrowed_begv || pos.charpos > it->narrowed_zv)
> 	      && (!redisplaying_p || it->line_wrap == TRUNCATE)))
> 	{
> 	  it->narrowed_begv = get_narrowed_begv (it->w, window_point (it->w));
> 	  it->narrowed_zv = get_narrowed_zv (it->w, window_point (it->w));
> 	}
>    }
>
> I think this should pass pos.charpos as the 2nd argument to 
> get_narrowed_begv and get_narrowed_zv, otherwise it might not really 
> correct anything, right?  In particular, when lines are truncated, that 
> will definitely happen when we display any line but the very first.
>
> Or perhaps we should check that using window-point indeed brings 
> pos.charpos into the narrowed region, and only use pos.charpos if it 
> doesn't?
>

I changed this into:

   if (current_buffer->long_line_optimizations_p)
     {
       if (!it->narrowed_begv)
         {
           it->narrowed_begv = get_narrowed_begv (it->w, window_point (it->w));
           it->narrowed_zv = get_narrowed_zv (it->w, window_point (it->w));
         }
       else if ((pos.charpos < it->narrowed_begv || pos.charpos > it->narrowed_zv)
                 && (!redisplaying_p || it->line_wrap == TRUNCATE))
         {
           it->narrowed_begv = get_narrowed_begv (it->w, pos.charpos);
           it->narrowed_zv = get_narrowed_zv (it->w, pos.charpos);
         }
     }

which seems better indeed.  Is that okay from your point of view?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 01 Aug 2022 15:08:42 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> > Or perhaps we should check that using window-point indeed brings 
> > pos.charpos into the narrowed region, and only use pos.charpos if it 
> > doesn't?
> >
> 
> I changed this into:
> 
>    if (current_buffer->long_line_optimizations_p)
>      {
>        if (!it->narrowed_begv)
>          {
>            it->narrowed_begv = get_narrowed_begv (it->w, window_point (it->w));
>            it->narrowed_zv = get_narrowed_zv (it->w, window_point (it->w));
>          }
>        else if ((pos.charpos < it->narrowed_begv || pos.charpos > it->narrowed_zv)
>                  && (!redisplaying_p || it->line_wrap == TRUNCATE))
>          {
>            it->narrowed_begv = get_narrowed_begv (it->w, pos.charpos);
>            it->narrowed_zv = get_narrowed_zv (it->w, pos.charpos);
>          }
>      }
> 
> which seems better indeed.  Is that okay from your point of view?

Yes, thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

This hunk:

> @@ -8894,7 +8891,7 @@ get_visually_first_element (struct it *it)
>  				find_newline_no_quit (IT_CHARPOS (*it),
>  						      IT_BYTEPOS (*it), -1,
>  						      &it->bidi_it.bytepos),
> -				it->narrowed_begv);
> +				get_closer_narrowed_begv (it->w, IT_CHARPOS (*it)));
>        bidi_paragraph_init (it->paragraph_embedding, &it->bidi_it, true);
>        do
>  	{

isn't quite right, I think.

Can you explain which, if any, problems you saw or had in mind, and
what and how did you want to fix them with this particular hunk?  I'd
then suggest a better change (or agree with your change, in case I
missed something).

Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Can you explain which, if any, problems you saw or had in mind, and what 
> and how did you want to fix them with this particular hunk?  I'd then 
> suggest a better change (or agree with your change, in case I missed 
> something).
>

You're correct, now that I think again about this, it was just wrong. 
Fixed.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 01 Aug 2022 16:49:55 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, larsi@gnus.org, 
>     monnier@iro.umontreal.ca
> 
> > Can you explain which, if any, problems you saw or had in mind, and what 
> > and how did you want to fix them with this particular hunk?  I'd then 
> > suggest a better change (or agree with your change, in case I missed 
> > something).
> 
> You're correct, now that I think again about this, it was just wrong. 
> Fixed.

OK, thanks.  But anyway: the possibly slow part there is not
find_newline_no_quit, it's the loop that calls
bidi_move_to_visually_next afterwards, in case the newline is very far
back.  So if we ever want to make that part faster, we should force
that loop to start from narrowed_begv, not from where we find the
previous newline.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]


>>> Try `M-: (use-global-map (make-keymap)) RET`
>>>
>>> Should we prevent users from doing that?
>>
>> It's a misleading question.  No "user" would ever do that.  Sure, it's 
>> a nice example, but only an Elisp hacker would do that, in the middle 
>> of a debugging session, and they would do that on purpose (although 
>> perhaps without knowing the effect in advance).  Which has nothing to 
>> do with a regular user who just opens a file.
>
> FWIW, the above is my standard example because I ended up doing exactly 
> that by accident, locking myself out of the session I was trying to 
> debug, so in a sense, I'd have been happy (that one time) if Emacs had 
> prevented me from doing it.
>

So... what do you think of the following? 😉

diff --git a/src/keymap.c b/src/keymap.c
index 506b755e5d..e42bc64717 100644
--- a/src/keymap.c
+++ b/src/keymap.c
@@ -1881,6 +1881,17 @@ DEFUN ("use-global-map", Fuse_global_map, 
Suse_global_map, 1, 1, 0,
    (Lisp_Object keymap)
  {
    keymap = get_keymap (keymap, 1, 1);
+
+  /* Prevent locking Emacs if someone inadvertently evaluates
+     (use-global-map (make-keymap)) */
+  if (EQ (Fequal (keymap, Fmake_keymap (Qnil)), Qt))
+    {
+      Lisp_Object meta_colon_key = CALLN (Fvector, make_fixnum (134217786));
+      Lisp_Object default_key = CALLN (Fvector, Qt);
+      Fdefine_key (keymap, meta_colon_key, intern ("eval-expression"), Qnil);
+      Fdefine_key (keymap, default_key, intern ("self-insert-command"), Qnil);
+    }
+
    current_global_map = keymap;

    return Qnil;

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 16:19, Eli Zaretskii wrote:
>> Date: Mon, 01 Aug 2022 13:14:20 +0000
>> From: Gregory Heytings<gregory@heytings.org>
>> cc:dgutov@yandex.ru,56682@debbugs.gnu.org,monnier@iro.umontreal.ca
>>
>>> You can maybe have that for C-p that follows M->, but wouldn't the wait
>>> return, with a vengeance, if you insert a single character (because then
>>> the buffer needs to be re-scanned)?  If so, we've gained nothing,
>>> really.
>> Fortunately no: the buffer doesn't need to be rescanned, syntax-ppss
>> caches its result, to avoid having to rescan the whole buffer again and
>> again.
> But the buffer has changed, so the cache is not necessarily valid,
> right?

syntax-ppss cache is a list of checkpoints spread along the buffer.

After a modification, only the checkpoints below it are invalidated (to 
be recomputed on-demand later).

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 14:23, Gregory Heytings wrote:
>> You might also want to compare to the time to run
>>
>> (parse-partial-sexp (point-min) (point-max))
>>
>> which is a kind of "speed of light" for `syntax-ppss`.
>>
> 
> What do you mean?  With the above file (benchmark-run 1 
> (parse-partial-sexp (point-min) (point-max))) takes 55 seconds.

Then this is the base price of knowing (with confidence) whether the EOB 
position is inside a paren, inside a string, or inside a comment, all of 
which may be positioned anywhere inside the buffer.

Perhaps someone will come up with further optimizations for the 
parse-partial-sexp implementation (without changing its behavior). 
Likely not by an order of a magnitude, but who knows.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 15:12, Eli Zaretskii wrote:
> But the problem here is that it isn't "people shooting themselves in
> the foot", it's that "major modes shoot their users in the foot".
> IOW, the ones who shoot and the ones who get shot are not the same
> people.
> 
> What do you want a user to do when he/she is faced with a mode which
> makes Emacs very slow?  Such a user cannot blame his/herself; in many
> cases the use doesn't even know enough to realize it's the major mode
> and its fontifications that are the culprit.

Just like we do in such cases where an Emacs feature is not optimized 
enough for a given use case: wait for the user to realize the situation 
can and should be improved, and file a bug report/feature request.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 15:08, Eli Zaretskii wrote:
>> Date: Mon, 1 Aug 2022 04:23:21 +0300
>> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>>   Stefan Monnier <monnier@iro.umontreal.ca>
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> IIUC this state of affairs is caused by your chosen approach to speeding
>> up font-lock (hard narrowing while it is called), which makes the
>> initial call to syntax-ppss happen inside that narrowing as well.
>>
>> The alternative being that font-lock would call syntax-ppss right away
>> with no restriction, but then only apply highlighting to limited parts
>> of the buffer.
> 
> AFAIU, this seems to assume that highlighting is much faster than
> syntax-ppss.  Is that a given?  If not, I don't think I understand how
> this could help.

I don't have the concrete numbers at hand, but from experience I'd say:

- syntax-ppss over the whole buffer is fast-ish. But it takes O(N) time 
of course, and the bigger the buffer is, the longer it'll take. Not much 
we can do about it.
- font-lock has to do more work, so over the whole buffer it will take 
an order of a magnitude more time than syntax-ppss.

Further:

- syntax-ppss is also important for correctness: for commands to 
understand whether the point is inside a string, comments, etc. So it's 
better to avoid applying narrowing when calling it. Unless you're in a 
multiple-major-modes situation.
- font-lock calls syntax-ppss.

So ideally font-lock is either called with undo-able narrowing, or is 
simply passed a range of positions, and shouldn't fontify too far from them.

The latter seems to be the case already (if you open xdisp.c and press 
M->, only top and bottom of the buffer are fontified), with the caveat 
that font-lock always tries to backtrack to BOL when fontifying the 
current hunk. Which makes sense, of course, but could be tweaked for 
long lines to avoid re-fontifying the whole buffer again and again.

IOW, IIUC the fix for font-lock performance could be better implemented 
inside font-lock itself, as long as all the info about whether the 
current line is "long" is available to Lisp.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 2 Aug 2022 00:50:10 +0300
> Cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 01.08.2022 16:19, Eli Zaretskii wrote:
> >> Date: Mon, 01 Aug 2022 13:14:20 +0000
> >> From: Gregory Heytings<gregory@heytings.org>
> >> cc:dgutov@yandex.ru,56682@debbugs.gnu.org,monnier@iro.umontreal.ca
> >>
> >>> You can maybe have that for C-p that follows M->, but wouldn't the wait
> >>> return, with a vengeance, if you insert a single character (because then
> >>> the buffer needs to be re-scanned)?  If so, we've gained nothing,
> >>> really.
> >> Fortunately no: the buffer doesn't need to be rescanned, syntax-ppss
> >> caches its result, to avoid having to rescan the whole buffer again and
> >> again.
> > But the buffer has changed, so the cache is not necessarily valid,
> > right?
> 
> syntax-ppss cache is a list of checkpoints spread along the buffer.
> 
> After a modification, only the checkpoints below it are invalidated (to 
> be recomputed on-demand later).

So a suitably-concocted replace command will still invalidate a lot of
that cache, right?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 2 Aug 2022 00:54:55 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 01.08.2022 15:12, Eli Zaretskii wrote:
> > But the problem here is that it isn't "people shooting themselves in
> > the foot", it's that "major modes shoot their users in the foot".
> > IOW, the ones who shoot and the ones who get shot are not the same
> > people.
> > 
> > What do you want a user to do when he/she is faced with a mode which
> > makes Emacs very slow?  Such a user cannot blame his/herself; in many
> > cases the use doesn't even know enough to realize it's the major mode
> > and its fontifications that are the culprit.
> 
> Just like we do in such cases where an Emacs feature is not optimized 
> enough for a given use case: wait for the user to realize the situation 
> can and should be improved, and file a bug report/feature request.

The intent of this activity is to make Emacs reasonably performant and
responsive in the relevant use cases without asking them to wait for
something that likely won't happen.

IOW, in this case the Emacs developers, due to long-standing bug
reports about this situation, recognized that it _can_ be improved,
albeit in slightly unorthodox ways, and have taken the measures to
optimize Emacs for the users.

bug#56682: Fix the long lines font locking related slowdowns

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings <gregory@heytings.org> writes:

>
> +  bool_bf long_line_syntax_ppss_done_p : 1;

Please write a comment for every bitfield in that struct.

> +	      current_buffer->long_line_syntax_ppss_done_p = 1;

Why not write "true" instead of "1"?

> +	      CALLN (Ffuncall, intern ("syntax-ppss"), Fpoint_max ());

Why not write this instead:

  call2 (Qsyntax_ppss, Fpoint_max ());

of course, with the appropriate DEFSYM added to the right file?
I think calling "intern" with a static string in C code is an example of
lazy programming.  At the very least, it unduly wastes cycles.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Perhaps someone will come up with further optimizations for the 
> parse-partial-sexp implementation (without changing its behavior). 
> Likely not by an order of a magnitude, but who knows.
>

Do I understand correctly that you do not plan to work on this?  If so, 
I'll add it to my TODO list.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> used, for some reason) jit-lock-defer feature: if you set
> jit-lock-defer-time to a large enough value, M-> followed by C-p will
> not be as slow as they are now, I think.

I think currently, with such a setting, in Gregory's example, the 50s of
`syntax-ppss` will still bite the users because they're uninterruptible,
although if the users happen to leave Emacs idle at the right time they
may not notice.

We could/should make sure the 50s are done "in the background" and can
be interrupted at any time.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> It is, indeed.  But here we are talking about big files (say 1 GB).  In that
> case (and in that case only) it is much better, from a user viewpoint, to
> wait say 20 seconds before the file is opened and being at that point able
> to freely move through the file, instead of waiting only 6 seconds, and then
> having to wait another 10 seconds between two motion commands like M-> C-p.
> In fact, no user expects that a 1 GB file would open instantaneously.

The problem will re-occur if you change something at the beginning of
the buffer and then go back to the end of the buffer.

I think something like `font-lock-defer` which displays the text
unfontified until we had time to apply the fontification should make the
1GB case much more usable (tho currently, it's probably not quite good
enough).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> only top and bottom of the buffer are fontified), with the caveat that
> font-lock always tries to backtrack to BOL when fontifying the current
> hunk. Which makes sense, of course, but could be tweaked for long lines to
> avoid re-fontifying the whole buffer again and again.

It has already been (recently) tweaked for this on `master`.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Given this situation, it sounds reasonable to start by restricting
> font-lock.

I fully agree.  The current behavior is the right default.

> As I wrote elsewhere, I'm okay with extending 'widen' so that it could
> "unlock" the locked narrowing, which could then be used in major modes
> that convince us their performance is adequate (or clearly announce in
> their docs that they don't care about files with long lines ;-).

`post-command-hook` also has been abused in all kinds of ways that suck
for the user if they have too-large buffers, or too many buffers, or too
many frames, or ...
That has never prompted calls to ban it completely.

So, I think it's important to provide this option to "unlock the
widening".  Even for purely philosophical reasons.

We can even add a user-option to "re-lock" the widening which would
prevent the "unlock the widening" from working, so that users can
override a poorly-thought-out use of widening which makes their large
file unusable (tho I'd argue that you can get the same result with an
`advice-add`).

Also, let's not forget that the speed impact of large buffers is not
limited to the redisplay, so trying to work extra-hard to eliminate all
possible cases of the redisplay spending too much time in large buffers
won't prevent "apparent lockups" where the time is spent in the command
(or some hook run at that occasion) rather than in the redisplay itself.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> So... what do you think of the following? 😉

:-)


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> `post-command-hook` also has been abused in all kinds of ways that suck 
> for the user if they have too-large buffers, or too many buffers, or too 
> many frames, or ...
>

Indeed.  Which is why you'll see on the feature branch that in buffers 
with long lines post-command-hook is now also subjected to a locked 
narrowing.

>
> We can even add a user-option to "re-lock" the widening which would 
> prevent the "unlock the widening" from working, so that users can 
> override a poorly-thought-out use of widening which makes their large 
> file unusable (tho I'd argue that you can get the same result with an 
> `advice-add`).
>

I don't understand what you have in mind here.  How would such a user 
option be different from (setq long-line-threshold nil)?  Do you mean that 
we should make it possible for users to fine-tune each and every aspect of 
the optimizations, with a bunch of user configurable options?

>
> Also, let's not forget that the speed impact of large buffers is not 
> limited to the redisplay, so trying to work extra-hard to eliminate all 
> possible cases of the redisplay spending too much time in large buffers 
> won't prevent "apparent lockups" where the time is spent in the command 
> (or some hook run at that occasion) rather than in the redisplay itself.
>

It is not limited to redisplay only, but by far the largest fraction of 
the speed impact is (or rather was) in redisplay, and asymptotically so. 
Commands that used to take minutes on a reasonably recent computer now 
take a fraction of a second, only because redisplay is now faster.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> `post-command-hook` also has been abused in all kinds of ways that suck
>> for the user if they have too-large buffers, or too many buffers, or too
>> many frames, or ...
> Indeed.  Which is why you'll see on the feature branch that in buffers with
> long lines post-command-hook is now also subjected to a locked narrowing.

But the problems can also affect buffers without long lines (just with
many lines).  Or the case of having a thousand buffers opened.  Or ...

>> We can even add a user-option to "re-lock" the widening which would
>> prevent the "unlock the widening" from working, so that users can override
>> a poorly-thought-out use of widening which makes their large file unusable
>> (tho I'd argue that you can get the same result with an `advice-add`).
> I don't understand what you have in mind here.

If the major mode overrides the locked narrowing which causes the users
experience to be unbearable in long buffers, the users could set this
variable to override the override (while sending a bug report to the
major mode's maintainers and waiting for the bug to be fixed).

> It is not limited to redisplay only, but by far the largest fraction of the
> speed impact is (or rather was) in redisplay, and asymptotically
> so. Commands that used to take minutes on a reasonably recent computer now
> take a fraction of a second, only because redisplay is now faster.

Yes, your work is *really* appreciated in this area.  I'm just pointing
out that there's no point trying to make it technically *impossible* for
users to shoot themselves in the foot making redisplay too slow,
because there are still plenty of other ways they can shoot themselves
in the foot, and because every time we make something undesirable
impossible, we *also* make it impossible to do some desirable things
(even if we can't yet imagine what those might be).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> But the problems can also affect buffers without long lines (just with 
> many lines).
>

Yes, and that's what sits next on my list: "too large" buffers.

>
> Or the case of having a thousand buffers opened.  Or ...
>

I don't think the "too many buffers" case is worth investigating. 
Because it won't lock Emacs suddenly, like what we had.  It could make 
Emacs gradually slower, in which case the user can take some appropriate 
measures.

>
> If the major mode overrides the locked narrowing which causes the users 
> experience to be unbearable in long buffers, the users could set this 
> variable to override the override (while sending a bug report to the 
> major mode's maintainers and waiting for the bug to be fixed).
>

I see what you mean now.  But I don't think it would work.  What I want is 
to take reasonable measures to ensure that Emacs remains responsive "in 
spite of" mode maintainers.

>
> Yes, your work is *really* appreciated in this area.  I'm just pointing 
> out that there's no point trying to make it technically *impossible* for 
> users to shoot themselves in the foot making redisplay too slow,
>

Yes, with that I agree.  But as Eli said, it's not users who shoot 
themselves in the foot, it's innocent users that are shooted in the foot 
by syntax-ppss and/or by mode maintainers who are not careful enough. 
And as I said its not technically impossible for this to happen, users can 
freely unset long-line-threshold (although that's not recommended of 
course).

>
> because there are still plenty of other ways they can shoot themselves 
> in the foot,
>

I'm curious, are there other ways for a regular user (*not* an Elisp 
hacker!) to make Emacs completely unresponsive with regular editing 
commands, starting with emacs -Q?

>
> and because every time we make something undesirable impossible, we 
> *also* make it impossible to do some desirable things (even if we can't 
> yet imagine what those might be).
>

I'm curious again, because I cannot imagine what that could be either.

Also note that you can blame yourself if you don't like the locked 
narrowing idea.  See the following three lines which you added ten years 
ago in eval.c:

/* Don't export this variable to Elisp, so no one can mess with it
    (Just imagine if someone makes it buffer-local).  */
Funintern (Qinternal_interpreter_environment, Qnil);

and which make it technically impossible to do something, incidentally 
without providing a way for Elisp hackers to escape that impossibility.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 02.08.2022 10:34, Gregory Heytings wrote:
>>
>> Perhaps someone will come up with further optimizations for the 
>> parse-partial-sexp implementation (without changing its behavior). 
>> Likely not by an order of a magnitude, but who knows.
>>
> 
> Do I understand correctly that you do not plan to work on this?  If so, 
> I'll add it to my TODO list.

Please go ahead. The C part of Emacs is far from my area of expertise.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 2 Aug 2022 04:05:57 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> The alternative being that font-lock would call syntax-ppss right away
> >> with no restriction, but then only apply highlighting to limited parts
> >> of the buffer.
> > 
> > AFAIU, this seems to assume that highlighting is much faster than
> > syntax-ppss.  Is that a given?  If not, I don't think I understand how
> > this could help.
> 
> I don't have the concrete numbers at hand, but from experience I'd say:
> 
> - syntax-ppss over the whole buffer is fast-ish. But it takes O(N) time 
> of course, and the bigger the buffer is, the longer it'll take. Not much 
> we can do about it.
> - font-lock has to do more work, so over the whole buffer it will take 
> an order of a magnitude more time than syntax-ppss.
> 
> Further:
> 
> - syntax-ppss is also important for correctness: for commands to 
> understand whether the point is inside a string, comments, etc. So it's 
> better to avoid applying narrowing when calling it. Unless you're in a 
> multiple-major-modes situation.
> - font-lock calls syntax-ppss.

I believe I was talking about syntax-ppss being called from font-lock,
indeed.  Before Gregory's changes, if you visit a large file with very
long lines, and interrupt Emacs while it is non-responsive, you will
in many/most cases find yourself in syntax-propertize or its
subroutines, and you will see that it is almost always called to
traverse the entire long line.

> So ideally font-lock is either called with undo-able narrowing, or is 
> simply passed a range of positions, and shouldn't fontify too far from them.

Many major-modes do widen the buffer, though.

> The latter seems to be the case already (if you open xdisp.c and press 
> M->, only top and bottom of the buffer are fontified)

It is not enough to look for faces in order to realize how much of the
buffer was scanned.

> with the caveat that font-lock always tries to backtrack to BOL when
> fontifying the current hunk. Which makes sense, of course, but could
> be tweaked for long lines to avoid re-fontifying the whole buffer
> again and again.

"Tweaked" how?

> IOW, IIUC the fix for font-lock performance could be better implemented 
> inside font-lock itself, as long as all the info about whether the 
> current line is "long" is available to Lisp.

No one will object to making font-lock faster.  But the experts who
can do that are few and far in-between, and seem to have other itches
to scratch, since these issues are known for a long time, and several
times were even discussed at length.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 02.08.2022 10:55, Stefan Monnier wrote:
>> only top and bottom of the buffer are fontified), with the caveat that
>> font-lock always tries to backtrack to BOL when fontifying the current
>> hunk. Which makes sense, of course, but could be tweaked for long lines to
>> avoid re-fontifying the whole buffer again and again.
> It has already been (recently) tweaked for this on `master`.

Sorry I missed that. Commit 15b2138719b34083967001c3903e7560d5e0947c, right?

Then, as long as redisplay avoids narrowing when running 
fontification-functions, everything should work well.

Or at least much better than before.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 02.08.2022 05:27, Eli Zaretskii wrote:
>> syntax-ppss cache is a list of checkpoints spread along the buffer.
>>
>> After a modification, only the checkpoints below it are invalidated (to
>> be recomputed on-demand later).
> So a suitably-concocted replace command will still invalidate a lot of
> that cache, right?

For any cache, one can invent an operation that would result in 
thrashing it repeatedly.

A regular search-replace should work well enough, though. Because when 
the buffer is long, the user is likely, on average, to spend a lot more 
time examining the occurrences and deciding whether to replace each one. 
And since the operation goes from top to bottom, this will likely 
invalidate the list of caches once, and then rebuild it from the 
beginning (or from wherever the first replacement was).

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 02.08.2022 05:31, Eli Zaretskii wrote:
>> Just like we do in such cases where an Emacs feature is not optimized
>> enough for a given use case: wait for the user to realize the situation
>> can and should be improved, and file a bug report/feature request.
> The intent of this activity is to make Emacs reasonably performant and
> responsive in the relevant use cases without asking them to wait for
> something that likely won't happen.
> 
> IOW, in this case the Emacs developers, due to long-standing bug
> reports about this situation, recognized that it_can_  be improved,
> albeit in slightly unorthodox ways, and have taken the measures to
> optimize Emacs for the users.

It would be a shame to have the better-behaving (faster) major modes 
exhibit worse behavior that they could have because of the approach we 
choose to solve the long-lines problem.

Regarding the long-standing bug reports, we did solve a bunch of issues 
already. One major one, IIUC, was redisplay of already fontified text on 
long lines. Another piece of the puzzle was added by Stefan in 
15b2138719b340.

So perhaps we should re-evaluate the testing scenario to see where the 
current bottlenecks are. If we current main issue is the 55s spent in 
syntax-ppss, a more constructive approach would be to look into 
optimizing parse-partial-sexp. Or even give up on certain scenarios, 
admitting that waiting 55s once to visit the end of a 1 GB buffer is not 
so bad (and that could part could also be sped up by setting 
syntax-propertize-function to nil and using a very simple syntax table, 
for instance).

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 02.08.2022 15:35, Eli Zaretskii wrote:

>> - syntax-ppss is also important for correctness: for commands to
>> understand whether the point is inside a string, comments, etc. So it's
>> better to avoid applying narrowing when calling it. Unless you're in a
>> multiple-major-modes situation.
>> - font-lock calls syntax-ppss.
> 
> I believe I was talking about syntax-ppss being called from font-lock,
> indeed.  Before Gregory's changes, if you visit a large file with very
> long lines, and interrupt Emacs while it is non-responsive, you will
> in many/most cases find yourself in syntax-propertize or its
> subroutines, and you will see that it is almost always called to
> traverse the entire long line.

Interrupt it right after pressing 'M->'? Or at any time during editing 
the buffer later?

The latter really shouldn't happen. If it does, perhaps it's the result 
of narrowing during redisplay, which might blow syntax-ppss's caches.

In any case, if you could point to a scenario and the revision to test 
it on, I'll be sure to take a look.

>> So ideally font-lock is either called with undo-able narrowing, or is
>> simply passed a range of positions, and shouldn't fontify too far from them.
> 
> Many major-modes do widen the buffer, though.

Whether they do or not, font-lock widens by default, see 
font-lock-dont-widen.

>> The latter seems to be the case already (if you open xdisp.c and press
>> M->, only top and bottom of the buffer are fontified)
> 
> It is not enough to look for faces in order to realize how much of the
> buffer was scanned.

I evaluated (next-single-property-change 1 'fontified), when near BOB 
and when near EOB.

>> with the caveat that font-lock always tries to backtrack to BOL when
>> fontifying the current hunk. Which makes sense, of course, but could
>> be tweaked for long lines to avoid re-fontifying the whole buffer
>> again and again.
> 
> "Tweaked" how?

15b2138719b34083 is one example.

>> IOW, IIUC the fix for font-lock performance could be better implemented
>> inside font-lock itself, as long as all the info about whether the
>> current line is "long" is available to Lisp.
> 
> No one will object to making font-lock faster.  But the experts who
> can do that are few and far in-between, and seem to have other itches
> to scratch, since these issues are known for a long time, and several
> times were even discussed at length.

The fact that we have +1 contributor to the C part of Emacs (the display 
engine, etc), and a successful one at that, does nothing about the fact 
that Lisp is easier to write and debug.

If we're able to demonstrate that the remaining bottlenecks are inside 
font-lock, it should be easier to improve there.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 14:01, Gregory Heytings wrote:
> 
>>> No, it happens when the buffer is opened.  Given the importance that 
>>> you and Stefan seem to give to that function, it is, with the patch I 
>>> sent in my previous post, called once on the whole buffer (without 
>>> any narrowing) when the file is opened.
>>
>> But if the buffer is not scrolled to the end, shouldn't it be called 
>> with a position that's close to the beginning?
>>
>> That shouldn't force the full buffer scan, meaning this call should 
>> complete quickly.
>>
> 
> I'm trying to reconcile two conflicting constraints.
> 
> It is necessary to add a locked narrowing around fontification-functions 
> and pre/post-command-hook to ensure that Emacs remains responsive.

Perhaps tweaking the value of syntax-wholeline-max could have a similar 
enough effect?

> At the same time, you and Stefan tell me that syntax-ppss does an 
> important job and will not do it correctly with such a locked narrowing, 
> IOW, that at least syntax-ppss should be called without a locked 
> narrowing.  But you also tell me that its result is cached so that a 
> full buffer scan isn't necessary anymore when it has happened at least 
> once.
> 
> So what I'm suggesting is to do a full buffer scan immediately, when the 
> file is opened, without any narrowing.  If that happens, later calls to 
> syntax-ppss inside fontification-functions and pre/post-command-hook 
> will use the cached result of the initial scan, and will do their job 
> correctly even with a locked narrowing.

syntax-ppss has two caches: one for the widened buffer (when point-min 
equals to 1), and another for narrowings. The latter cache is always 
cleared when point-min changes.

The latter cache works okay-ish for multiple-major-mode scenario, but if 
redisplay narrows with different limits, that would blow cache often. 
And if the whole chunk is big enough (i.e. bigger than the average 
distance between the cached syntax-ppss positions), that could actually 
result in syntax-ppss working slower than it could.

> Unless I misunderstand something, I think (and my tests seem to confirm) 
> that that would be a workable solution, provided that the initial scan 
> is reasonably fast, that is, at least six times faster than it is now.

I'm afraid this approach is too blunt to be a solution.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Regarding the long-standing bug reports, we did solve a bunch of issues 
> already. One major one, IIUC, was redisplay of already fontified text on 
> long lines.
>

Try to open the dictionary.json with Emacs on master a month ago.  It's a 
small file (only 18 MB).  On my computer just opening the file with emacs 
-Q takes 220 seconds.  220 seconds during which Emacs is completely 
locked, because of font-lock mode.  If you're not convinced, turn 
font-lock mode off, open the file, and turn font-lock mode on.

>
> Another piece of the puzzle was added by Stefan in 15b2138719b340.
>

That looked promising, but sadly it had only a very limited effect.

>
> So perhaps we should re-evaluate the testing scenario to see where the 
> current bottlenecks are. If we current main issue is the 55s spent in 
> syntax-ppss, a more constructive approach would be to look into 
> optimizing parse-partial-sexp. Or even give up on certain scenarios, 
> admitting that waiting 55s once to visit the end of a 1 GB buffer is not 
> so bad (and that could part could also be sped up by setting 
> syntax-propertize-function to nil and using a very simple syntax table, 
> for instance).
>

It is bad, especially now that it became clear that in fact it's not 
"waiting 55s once" but "waiting 55s each time the buffer is modified and 
you move to another position in the buffer".

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Whether they do or not, font-lock widens by default, see 
> font-lock-dont-widen.
>

And even if one sets font-lock-dont-widen to nil, mode authors are 
completely free to ignore that setting (assuming they're aware it exists).

>
> If we're able to demonstrate that the remaining bottlenecks are inside 
> font-lock, it should be easier to improve there.
>

It has been demonstrated, as far as I can see, and repeatedly so.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> It is necessary to add a locked narrowing around 
>> fontification-functions and pre/post-command-hook to ensure that Emacs 
>> remains responsive.
>
> Perhaps tweaking the value of syntax-wholeline-max could have a similar 
> enough effect?
>

It doesn't, alas.

>> Unless I misunderstand something, I think (and my tests seem to 
>> confirm) that that would be a workable solution, provided that the 
>> initial scan is reasonably fast, that is, at least six times faster 
>> than it is now.
>
> I'm afraid this approach is too blunt to be a solution.
>

Indeed, that's my conclusion too.  So until syntax-ppss (at least) is made 
an order of magnitude faster, the right thing to do is to use the forced 
narrowing method.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 2 Aug 2022 17:10:53 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 02.08.2022 05:27, Eli Zaretskii wrote:
> >> syntax-ppss cache is a list of checkpoints spread along the buffer.
> >>
> >> After a modification, only the checkpoints below it are invalidated (to
> >> be recomputed on-demand later).
> > So a suitably-concocted replace command will still invalidate a lot of
> > that cache, right?
> 
> For any cache, one can invent an operation that would result in 
> thrashing it repeatedly.

Yes, but when thrashing causes delays of dozens of seconds, the result
is not just a rare delay, the result is simply unacceptable.

> A regular search-replace should work well enough, though. Because when 
> the buffer is long, the user is likely, on average, to spend a lot more 
> time examining the occurrences and deciding whether to replace each one. 
> And since the operation goes from top to bottom, this will likely 
> invalidate the list of caches once, and then rebuild it from the 
> beginning (or from wherever the first replacement was).

We want every basic operation in such buffers to perform reasonably
well.  That's the goal of this activity.  Because partial solutions
that sometimes work we already have: there's so-long-mode, there's
longlines.el, and a couple of other trick up our sleeve.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 2 Aug 2022 17:29:47 +0300
> Cc: monnier@iro.umontreal.ca, 56682@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > IOW, in this case the Emacs developers, due to long-standing bug
> > reports about this situation, recognized that it_can_  be improved,
> > albeit in slightly unorthodox ways, and have taken the measures to
> > optimize Emacs for the users.
> 
> It would be a shame to have the better-behaving (faster) major modes 
> exhibit worse behavior that they could have because of the approach we 
> choose to solve the long-lines problem.

Like I said: I'm okay with extending 'widen' to allow it to optionally
break the lock on the narrowing.  Then those modes which will indeed
show reasonable performance in these cases can use that optional
behavior.  Otherwise, I think we've waited for such improvements in
the modes long enough.

> Regarding the long-standing bug reports, we did solve a bunch of issues 
> already. One major one, IIUC, was redisplay of already fontified text on 
> long lines. Another piece of the puzzle was added by Stefan in 
> 15b2138719b340.

I invite you and Stefan to show that the improvement is performant
enough in the cases we are using as examples of long lines.

> So perhaps we should re-evaluate the testing scenario to see where the 
> current bottlenecks are. If we current main issue is the 55s spent in 
> syntax-ppss, a more constructive approach would be to look into 
> optimizing parse-partial-sexp. Or even give up on certain scenarios, 
> admitting that waiting 55s once to visit the end of a 1 GB buffer is not 
> so bad (and that could part could also be sped up by setting 
> syntax-propertize-function to nil and using a very simple syntax table, 
> for instance).

I reported the problem with syntax-propertize to Stefan 1.5 month ago,
see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45898#92.  There was
an improvement in that department, but AFAICT the net result is still
unsatisfactory if you disable the recent long-line optimizations
(e.g., by setting long-line-threshold to nil).

So, unless I'm mistaken, the same scenario described there is still
relevant, and doesn't yet need to be reevaluated.  But more data
points are always welcome, so feel free (you and everyone else) to
test the performance with and without this feature, on different files
with different major modes, and report back what you have found.  At
least I don't think the work on this is completed, we still have a lot
of turf to cover and probably deal with fallout we haven't yet heard
about.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 2 Aug 2022 17:47:12 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > I believe I was talking about syntax-ppss being called from font-lock,
> > indeed.  Before Gregory's changes, if you visit a large file with very
> > long lines, and interrupt Emacs while it is non-responsive, you will
> > in many/most cases find yourself in syntax-propertize or its
> > subroutines, and you will see that it is almost always called to
> > traverse the entire long line.
> 
> Interrupt it right after pressing 'M->'? Or at any time during editing 
> the buffer later?

Just after "C-x C-f" and answering YES to the question whether to
visit it normally.

> The latter really shouldn't happen. If it does, perhaps it's the result 
> of narrowing during redisplay, which might blow syntax-ppss's caches.

I'm talking about running with narrowing disabled.

> In any case, if you could point to a scenario and the revision to test 
> it on, I'll be sure to take a look.

See my other message I sent a few minutes ago.  And I believe Gregory
also pointed to it.

> > Many major-modes do widen the buffer, though.
> 
> Whether they do or not, font-lock widens by default, see 
> font-lock-dont-widen.

Which is a breach of contract with jit-lock, if you ask me.  It's no
accident that jit-lock invokes fontification-functions only on a small
region of the buffer: that's how this feature was designed and
implemented, and its entirely consistent with the overall design of
the basic principle of the Emacs display engine: examine only as much
of the buffer as needed to be displayed.

> >> The latter seems to be the case already (if you open xdisp.c and press
> >> M->, only top and bottom of the buffer are fontified)
> > 
> > It is not enough to look for faces in order to realize how much of the
> > buffer was scanned.
> 
> I evaluated (next-single-property-change 1 'fontified), when near BOB 
> and when near EOB.

That's not enough either.  Try running under a debugger with a
watchpoint on the position point, while fontification-functions run,
and be amazed how many time it moves point waaay out of the region
that is about to be displayed.

> >> with the caveat that font-lock always tries to backtrack to BOL when
> >> fontifying the current hunk. Which makes sense, of course, but could
> >> be tweaked for long lines to avoid re-fontifying the whole buffer
> >> again and again.
> > 
> > "Tweaked" how?
> 
> 15b2138719b34083 is one example.

It's a good improvement, but much more is needed.

Again, why don't you try this yourself, after disabling the recently
added optimizations?

> > No one will object to making font-lock faster.  But the experts who
> > can do that are few and far in-between, and seem to have other itches
> > to scratch, since these issues are known for a long time, and several
> > times were even discussed at length.
> 
> The fact that we have +1 contributor to the C part of Emacs (the display 
> engine, etc), and a successful one at that, does nothing about the fact 
> that Lisp is easier to write and debug.
> 
> If we're able to demonstrate that the remaining bottlenecks are inside 
> font-lock, it should be easier to improve there.

And it will be very easy to disable these optimizations by default
when we decide that they are no longer needed.  Meanwhile, Emacs users
get to edit long lines with reasonable performance.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 02 Aug 2022 14:57:25 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, 56682@debbugs.gnu.org, 
>     monnier@iro.umontreal.ca
> 
> > Regarding the long-standing bug reports, we did solve a bunch of issues 
> > already. One major one, IIUC, was redisplay of already fontified text on 
> > long lines.
> 
> Try to open the dictionary.json with Emacs on master a month ago.

I believe it can also be done with the current master, just after
setting long-line-threshold to the nil value.  Right?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Try to open the dictionary.json with Emacs on master a month ago.
>
> I believe it can also be done with the current master, just after 
> setting long-line-threshold to the nil value.  Right?
>

Indeed.  With master from one month ago it's even more crystal-clear that 
you see the statu quo ante.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> It is necessary to add a locked narrowing around
> fontification-functions and pre/post-command-hook to ensure that Emacs
> remains responsive.

It's neither necessary (it's perfectly possible to do something quickly
that just needs to look at the first few lines of the buffer to decide
in which way to parse the nearby surrounding bytes, for example) nor
sufficient (it's easy to spend minutes wasting time running in circles
because of a bug, e.g. a bug triggered by the fact that the "unusual"
nature of the visible part of the buffer after an arbitrary narrowing).

It's very useful to get closer to this goal like your code does, but
let's keep things in perspective.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Indeed, that's my conclusion too.  So until syntax-ppss (at least) is made
> an order of magnitude faster, the right thing to do is to use the forced
> narrowing method.

By tying this to "long lines" is wrong, since it has nothing to do with
long lines, only with large buffers.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> It is necessary to add a locked narrowing around 
>> fontification-functions and pre/post-command-hook to ensure that Emacs 
>> remains responsive.
>
> It's neither necessary (it's perfectly possible to do something quickly 
> that just needs to look at the first few lines of the buffer to decide 
> in which way to parse the nearby surrounding bytes, for example) nor 
> sufficient (it's easy to spend minutes wasting time running in circles 
> because of a bug, e.g. a bug triggered by the fact that the "unusual" 
> nature of the visible part of the buffer after an arbitrary narrowing).
>

You're a mathematician, aren't you?  I'm not speaking of mathematical 
(i.e. absolute) necessity here, but of a practical necessity.  And I did 
not say (or thought) that it is sufficient.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-08-01 11:23:52] wrote:

>>> The problem is that this is, as I said, slow.  On my laptop, opening
>>> a 1 GB file takes about 6 seconds.  The call to syntax-ppss adds 70
>>> seconds, so opening a large file becomes an order of magnitude slower (13
>>> times slower).
>>
>> It's meaningless to talk about the time taken by `syntax-ppss` without
>> specifying the major mode that was in use.
>>
>
> It isn't.  The benchmark above was with a JSON file (js-mode), but you'll
> see the same ratio with an Elisp file for example:

Hmmm was that using the GNU ELPA `json-mode` package?
If so, then indeed there's no `syntax-propertize-function` setup there
and `syntax-ppss` should be ideally not much slower than
`parse-partial-sexp`.

> for I in $(seq 1 2500); do cat lisp/simple.el; done > complex.el
>
> That file opens in about 5 seconds, and (benchmark-run 1 (syntax-ppss
> (point-max))) takes about 45 seconds.
>
> Sure, there are perhaps modes that are slower, but my tests seem to indicate
> that the 1/10 ratio is correct, or IOW that syntax-ppss is an order of
> magnitude slower than opening the file.

You might be right.
But there are still significant differences between different major modes:

    LISP> (benchmark-run 1 (fundamental-mode) (parse-partial-sexp (point-min) (point-max)))
    (0.276774213 0 0.0)
    
    LISP> (benchmark-run 1 (fundamental-mode) (syntax-ppss (point-max)))
    (0.329234636 0 0.0)
    
    ELISP> (benchmark-run 1 (emacs-lisp-mode) (syntax-ppss (point-max)))
    (0.392759479 0 0.0)
    
    ELISP> (benchmark-run 1 (js-mode) (syntax-ppss (point-max)))
    (1.036089104 7 0.20054423700000001)
    
    ELISP> (benchmark-run 1 (nxml-mode) (syntax-ppss (point-max)))
    (1.169055192 7 0.15886504199999996)
    
    ELISP> (benchmark-run 1 (cperl-mode) (syntax-ppss (point-max)))
    (1.857638439 9 0.19724271499999996)

(this was in a 5MB buffer).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I see what you mean now.  But I don't think it would work.  What I want is
> to take reasonable measures to ensure that Emacs remains responsive "in
> spite of" mode maintainers.

"Reasonable measures" is good.
But once you step into "impossible to circumvent", you start going
directly against the goals of Free Software (and Emacs) to empower the user.

>> because there are still plenty of other ways they can shoot themselves in
>> the foot,
> I'm curious, are there other ways for a regular user (*not* an Elisp
> hacker!) to make Emacs completely unresponsive with regular editing
> commands, starting with emacs -Q?

We've had enough bug reports over the years (and I experienced such
things many times as well), yes.  Usually linked to bugs in some
package, of course, but in many case it wasn't a clear-cut bug, just
some bad interaction of various elements.

> I'm curious again, because I cannot imagine what that could be either.

I suffer from the same lack of imagination, as I sure many others here
do.  To make up for that, we've learned to follow a philosophy of
empowering the users (and not imposing arbitrary limits just because we
couldn't think of good reasons to go beyond them).

> Also note that you can blame yourself if you don't like the locked narrowing
> idea.  See the following three lines which you added ten years ago in
> eval.c:
>
> /* Don't export this variable to Elisp, so no one can mess with it
>    (Just imagine if someone makes it buffer-local).  */
> Funintern (Qinternal_interpreter_environment, Qnil);
>
> and which make it technically impossible to do something, incidentally
> without providing a way for Elisp hackers to escape that impossibility.

:-)


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Hmmm was that using the GNU ELPA `json-mode` package?
>

No, it's with emacs -Q.

>> Sure, there are perhaps modes that are slower, but my tests seem to 
>> indicate that the 1/10 ratio is correct, or IOW that syntax-ppss is an 
>> order of magnitude slower than opening the file.
>
> You might be right.
>
> But there are still significant differences between different major 
> modes:
>
> LISP> (benchmark-run 1 (fundamental-mode) (parse-partial-sexp (point-min) (point-max)))
> (0.276774213 0 0.0)
>
> LISP> (benchmark-run 1 (fundamental-mode) (syntax-ppss (point-max)))
> (0.329234636 0 0.0)
>
> ELISP> (benchmark-run 1 (emacs-lisp-mode) (syntax-ppss (point-max)))
> (0.392759479 0 0.0)
>
> ELISP> (benchmark-run 1 (js-mode) (syntax-ppss (point-max)))
> (1.036089104 7 0.20054423700000001)
>
> ELISP> (benchmark-run 1 (nxml-mode) (syntax-ppss (point-max)))
> (1.169055192 7 0.15886504199999996)
>
> ELISP> (benchmark-run 1 (cperl-mode) (syntax-ppss (point-max)))
> (1.857638439 9 0.19724271499999996)
>
> (this was in a 5MB buffer).
>

Yes, that's correct.  (But did you test each mode with the same 5 MB 
buffer?  If so, that's perhaps not representative of what happens in 
reality.)  The general idea is that syntax-ppss is currently an order of 
magnitude too slow for "too large" buffers.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> So ideally font-lock is either called with undo-able narrowing, or is 
>> simply passed a range of positions, and shouldn't fontify too far from them.
> Many major-modes do widen the buffer, though.

Actually, IIUC this should be considered a bug (it breaks the use of
that major in mmm-mode and friends).

[ And `grep` suggests that less than half of progmodes/*.el use `widen`.  ]


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> No one will object to making font-lock faster.  But the experts who
> can do that are few and far in-between, and seem to have other itches
> to scratch, since these issues are known for a long time, and several
> times were even discussed at length.

With a locked narrowing that can't be circumvented, this itch can't be
scratched any more :-(


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> And even if one sets font-lock-dont-widen to nil, mode authors are
> completely free to ignore that setting (assuming they're aware it exists).

Hmm... `font-lock-dont-widen` is not about telling the major modes's
font-lock code to refrain from narrowing.  It's there so a major mode
(like mmm-mode) can tell font-lock.el to refrain from widening.

The major-mode's own font-lock rules should never need to widen
(because font-lock itself does it, normally).  If they do, it's a bug.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> With a locked narrowing that can't be circumvented, this itch can't be 
> scratched any more :-(
>

Once again, it can!  Just M-: (setq long-line-threshold nil) RET.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Another piece of the puzzle was added by Stefan in 15b2138719b340.
> That looked promising, but sadly it had only a very limited effect.

FWIW, that was also my experience :-(

>> So perhaps we should re-evaluate the testing scenario to see where the
>> current bottlenecks are. If we current main issue is the 55s spent in
>> syntax-ppss, a more constructive approach would be to look into optimizing
>> parse-partial-sexp. Or even give up on certain scenarios, admitting that
>> waiting 55s once to visit the end of a 1 GB buffer is not so bad (and that
>> could part could also be sped up by setting syntax-propertize-function to
>> nil and using a very simple syntax table, for instance).

Those 55s were with a `syntax-propertize-function` set to nil already.

We could do a bit better in some modes, tho, e.g. in fundamental mode no
char has string or comment syntax, IIRC, so we could arguably make
`syntax-ppss` return a value without parsing anything at all (except
that `syntax-ppss` is also supposed to count parentheses, which *are*
present in fundamental-mode).

> It is bad, especially now that it became clear that in fact it's not
> "waiting 55s once" but "waiting 55s each time the buffer is modified and you
> move to another position in the buffer".

Actually, if 55s is the time it takes from BOB to EOB, then the time for
"change at POS1 plus move to POS2" should be approximately

    55s * (POS2 - POS1) / (buffer-size)


-- Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> I see what you mean now.  But I don't think it would work.  What I want 
>> is to take reasonable measures to ensure that Emacs remains responsive 
>> "in spite of" mode maintainers.
>
> "Reasonable measures" is good.
>
> But once you step into "impossible to circumvent", you start going 
> directly against the goals of Free Software (and Emacs) to empower the 
> user.
>

But it is *not* impossible to circumvent.  Just add (setq 
long-line-threshold nil) in your init file, or evaluate it with M-:, or...

>
> I suffer from the same lack of imagination, as I sure many others here 
> do.  To make up for that, we've learned to follow a philosophy of 
> empowering the users (and not imposing arbitrary limits just because we 
> couldn't think of good reasons to go beyond them).
>

I'm all for empowering the users who want power, and the changeset is, 
intentionally, fully backward compatible.  In fact, strictly speaking, 
this changeset does exactly that: it empowers users, they can do things 
they couldn't do earlier.  But I'm also for protecting regular users who 
want a powerful editor that "just works".  In the same vein, we are 
marking a number of local variables as risky, to protect regular users, 
without preventing users from shooting themselves in the foot by adding

(defalias 'risky-local-variable-p 'ignore)

in their init file if they see fit.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> You're a mathematician, aren't you?  I'm not speaking of mathematical
> (i.e. absolute) necessity here, but of a practical necessity.

I agree with the practical necessity to narrow.
I strongly disagree with the necessity to make re-widening
technically impossible, I find it fundamentally incompatible with
Emacs's philosophy and can't see any practical justification here.
Just narrow and make sure jit-lock.el and font-lock.el don't
accidentally widen it.  Any other accidental widening should be
considered as a bug anyway (and we could even easily cook up some ad-hoc
advice to try and detect those cases for people like me who lke to run
their Emacs with lots of extra runtime debugging checks).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I strongly disagree with the necessity to make re-widening technically 
> impossible, I find it fundamentally incompatible with Emacs's philosophy 
> and can't see any practical justification here.
>

As I said earlier, offering a way to escape the locked narrowing right 
away simply means that the current solution isn't one anymore.  Elisp 
programmers would take the habit of using (widen-unlock) instead of 
(widen) in their programs, and in a couple of years we'll see again bug 
reports by users who cannot edit buffers with long lines.

>
> Just narrow and make sure jit-lock.el and font-lock.el don't 
> accidentally widen it.
>

Now I'm lost.  Isn't this what is happening right now: "narrow and make 
sure jit-lock and font-lock don't accidentally widen it"?  What am I 
missing?

>
> Any other accidental widening should be considered as a bug anyway (and 
> we could even easily cook up some ad-hoc advice to try and detect those 
> cases for people like me who like to run their Emacs with lots of extra 
> runtime debugging checks).
>

There are I fear too many bugs related to that problem (you just said that 
half of the modes in core do use widen), and it does not seem reasonable 
to hope that they will all be fixed anytime soon.  If at some point they 
are, the current solution will not be necessary anymore, and it will be 
very easy to reset the default value of long-line-threshold to nil.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 02.08.2022 19:19, Gregory Heytings wrote:
> 
>>> Try to open the dictionary.json with Emacs on master a month ago.
>>
>> I believe it can also be done with the current master, just after 
>> setting long-line-threshold to the nil value.  Right?
>>
> 
> Indeed.  With master from one month ago it's even more crystal-clear 
> that you see the statu quo ante.

If I set long-line-threshold to nil, does that also disable the 
redisplay optimizations related to long lines?

Ones that caused scrolling delays even after the buffer has been fully 
fontified.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 03.08.2022 03:00, Dmitry Gutov wrote:
> On 02.08.2022 19:19, Gregory Heytings wrote:
>>
>>>> Try to open the dictionary.json with Emacs on master a month ago.
>>>
>>> I believe it can also be done with the current master, just after 
>>> setting long-line-threshold to the nil value.  Right?
>>>
>>
>> Indeed.  With master from one month ago it's even more crystal-clear 
>> that you see the statu quo ante.
> 
> If I set long-line-threshold to nil, does that also disable the 
> redisplay optimizations related to long lines?
> 
> Ones that caused scrolling delays even after the buffer has been fully 
> fontified.

I mean those that *fixed* the said scrolling delays, of course.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  56682@debbugs.gnu.org,  Eli Zaretskii
>  <eliz@gnu.org>
> Date: Tue, 02 Aug 2022 17:19:53 -0400
> 
> > Indeed, that's my conclusion too.  So until syntax-ppss (at least) is made
> > an order of magnitude faster, the right thing to do is to use the forced
> > narrowing method.
> 
> By tying this to "long lines" is wrong, since it has nothing to do with
> long lines, only with large buffers.

I thought you told me once that syntax-propertize needs to consider
complete lines in some (frequent) situations?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  56682@debbugs.gnu.org,
>   gregory@heytings.org
> Date: Tue, 02 Aug 2022 17:46:49 -0400
> 
> >> So ideally font-lock is either called with undo-able narrowing, or is 
> >> simply passed a range of positions, and shouldn't fontify too far from them.
> > Many major-modes do widen the buffer, though.
> 
> Actually, IIUC this should be considered a bug (it breaks the use of
> that major in mmm-mode and friends).
> 
> [ And `grep` suggests that less than half of progmodes/*.el use `widen`.  ]

That fits my definition of "many".

And we also need to consider specifically those modes which are likely
to happen in files with long lines.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1350 bytes --]


>>>>> Try to open the dictionary.json with Emacs on master a month ago.
>>>> 
>>>> I believe it can also be done with the current master, just after 
>>>> setting long-line-threshold to the nil value.  Right?
>>> 
>>> Indeed.  With master from one month ago it's even more crystal-clear 
>>> that you see the statu quo ante.
>> 
>> If I set long-line-threshold to nil, does that also disable the 
>> redisplay optimizations related to long lines?
>> 
>> Ones that caused scrolling delays even after the buffer has been fully 
>> fontified.
>
> I mean those that *fixed* the said scrolling delays, of course.
>

To clarify, you need to (setq long-line-threshold nil syntax-wholeline-max 
most-positive-fixnum) to "recover" (more or less) how master was behaving 
a month ago.  Otherwise you'll see the effect of syntax-wholeline-max, 
which can be either positive or negative.

Three recipes (with today's master):

emacs -Q
M-: (setq long-line-threshold nil syntax-wholeline-max most-positive-fixnum) RET
C-x C-f dictionary.json RET y ;; takes 160 seconds
C-e ;; takes 200 seconds

emacs -Q
M-: (setq long-line-threshold nil) RET
C-x C-f dictionary.json RET y ;; immediate
C-e ;; not finished after 1200 seconds (20 minutes), I killed Emacs

emacs -Q
C-x C-f dictionary.json RET y ;; immediate
C-e ;; immediate

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> As I said earlier, offering a way to escape the locked narrowing right away
> simply means that the current solution isn't one anymore.
> Elisp programmers would take the habit of using (widen-unlock) instead of
> (widen) in their programs, and in a couple of years we'll see again bug
> reports by users who cannot edit buffers with long lines.

I don't think we need such a paternalistic view of ELisp programmers.
ELisp programmers aren't out there looking for ways to mess things up.
If we give them good tools that make it easy to solve the usual problems
with needing `widen-unlock`, they won't start using it recklessly everywhere.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> > Indeed, that's my conclusion too.  So until syntax-ppss (at least) is made
>> > an order of magnitude faster, the right thing to do is to use the forced
>> > narrowing method.
>> 
>> By tying this to "long lines" is wrong, since it has nothing to do with
>> long lines, only with large buffers.
>
> I thought you told me once that syntax-propertize needs to consider
> complete lines in some (frequent) situations?

Yes, but we're talking about `syntax-ppss` here.  Admittedly,
`syntax-ppss` uses `syntax-propertize` internally, but I think the two
need to be considered separately (and `syntax-propertize` already tries
to bound its work via `syntax-wholeline-max`).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> But once you step into "impossible to circumvent", you start going
>> directly against the goals of Free Software (and Emacs) to empower
>> the user.
> But it is *not* impossible to circumvent.  Just add (setq
> long-line-threshold nil) in your init file, or evaluate it with M-:, or...

That's like saying you can fix the ugly color on strings by turning off font-lock.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> With a locked narrowing that can't be circumvented, this itch can't be
>> scratched any more :-(
> Once again, it can!  Just M-: (setq long-line-threshold nil) RET.

The itch is to make long-lines work better than with your code.
Not to make them completely unusable.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I don't think we need such a paternalistic view of ELisp programmers. 
> ELisp programmers aren't out there looking for ways to mess things up. 
> If we give them good tools that make it easy to solve the usual problems 
> with needing `widen-unlock`, they won't start using it recklessly 
> everywhere.
>

I'd say it is realistic, based on the observation that half of the 
programming language modes in core do use widen, which they weren't 
supposed to do.

But this discussion is leading nowhere.  If you could point out to an 
actual (or even potential) problem caused by this locked narrowing, apart 
from an occasional mis-fontification, that would perhaps help it to 
advance.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  56682@debbugs.gnu.org,  Eli Zaretskii
>  <eliz@gnu.org>
> Date: Tue, 02 Aug 2022 18:26:51 -0400
> 
> I strongly disagree with the necessity to make re-widening
> technically impossible, I find it fundamentally incompatible with
> Emacs's philosophy and can't see any practical justification here.
> Just narrow and make sure jit-lock.el and font-lock.el don't
> accidentally widen it.

What do you mean by "accidentally" in this context?  A program rarely
does anything "accidentally", it almost always does what the
programmer wanted it to do.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Wed, 3 Aug 2022 03:26:02 +0300
> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> On 03.08.2022 03:00, Dmitry Gutov wrote:
> > On 02.08.2022 19:19, Gregory Heytings wrote:
> >>
> >>>> Try to open the dictionary.json with Emacs on master a month ago.
> >>>
> >>> I believe it can also be done with the current master, just after 
> >>> setting long-line-threshold to the nil value.  Right?
> >>>
> >>
> >> Indeed.  With master from one month ago it's even more crystal-clear 
> >> that you see the statu quo ante.
> > 
> > If I set long-line-threshold to nil, does that also disable the 
> > redisplay optimizations related to long lines?
> > 
> > Ones that caused scrolling delays even after the buffer has been fully 
> > fontified.
> 
> I mean those that *fixed* the said scrolling delays, of course.

I don't think I understand which changes you had in mind (could you
give a less vague pointer?), but I think the answer is NO: that
variable disables only the recent changes related to long lines.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gregory@heytings.org,  dgutov@yandex.ru,  56682@debbugs.gnu.org
> Date: Wed, 03 Aug 2022 04:37:55 -0400
> 
> >> > Indeed, that's my conclusion too.  So until syntax-ppss (at least) is made
> >> > an order of magnitude faster, the right thing to do is to use the forced
> >> > narrowing method.
> >> 
> >> By tying this to "long lines" is wrong, since it has nothing to do with
> >> long lines, only with large buffers.
> >
> > I thought you told me once that syntax-propertize needs to consider
> > complete lines in some (frequent) situations?
> 
> Yes, but we're talking about `syntax-ppss` here.  Admittedly,
> `syntax-ppss` uses `syntax-propertize` internally, but I think the two
> need to be considered separately (and `syntax-propertize` already tries
> to bound its work via `syntax-wholeline-max`).

What about parse-partial-sexp, which calls scan_sexps_forward?  It
looks like I've misremembered, and that was the culprit in the
scenario we discussed, see
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45898#92.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I'd say it is realistic, based on the observation that half of the
> programming language modes in core do use widen, which they weren't supposed
> to do.

AFAICT, most uses of widen in lisp/progmodes is *not* within code
related to font-lock (unsurprinsingly, since font-lock.el has been
widening "for ever", so widening in the major mode has always been
redundant).

> But this discussion is leading nowhere.  If you could point out to an
> actual (or even potential) problem caused by this locked narrowing,
> apart from an occasional mis-fontification, that would perhaps help it
> to advance.

If I were maintainer I'd just refuse such a change without corresponding
escape hatch, based on the experience gained over the years about how
Emacs is and should be designed.

As a package contributor I just find it offensive that the C code would
go to extra trouble in order to cut me out under the premise that "Elisp
programmers would take the habit of using (widen-unlock) instead of
(widen) in their programs", which I read as "we have to protect ELisp
contributors from themselves".

Also, I'm trying to imagine  scenario that leads to such an abuse:
- under normal circumstances, there are no long lines, so they'll never
  hit a "locked" narrowing and it will thus never occur to them to use
  a `widen-unlock`.
- when they get a bug report with a locked narrowing because of long
  lines, using `widen-unlock` naively is likely to lead to an immediate
  performance problem, so it's unlikely they'll use it.
I just don't buy it.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> What do you mean by "accidentally" in this context?  A program rarely
> does anything "accidentally", it almost always does what the
> programmer wanted it to do.

I mean that currently font-lock just blindly widens all the time, so we
should fix it so it only widens when it's expected to (i.e. not when
the widening is installed by the LLT code).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> What about parse-partial-sexp, which calls scan_sexps_forward?  It
> looks like I've misremembered, and that was the culprit in the
> scenario we discussed, see
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45898#92.

The wholelines problem did not kick in because of PPS nor `syntax-ppss`
but because of font-lock (which then called `syntax-ppss` which then
called PPS).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-08-02 21:43:35] wrote:
>> Hmmm was that using the GNU ELPA `json-mode` package?
> No, it's with emacs -Q.

Sorry, I'm not sure how I missed that you said you used `js-mode`.

> Yes, that's correct.  (But did you test each mode with the same 5 MB buffer?

Yup, a 5MB binary file.

> If so, that's perhaps not representative of what happens in reality.)

Indeed, tho it gives a rough idea of how it can change.

> The general idea is that syntax-ppss is currently an order of
> magnitude too slow for "too large" buffers.

And that it can be almost another order of magnitude worse in some major
modes because of `syntax-propertize`.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Also, I'm trying to imagine scenario that leads to such an abuse:
>
> - under normal circumstances, there are no long lines, so they'll never 
> hit a "locked" narrowing and it will thus never occur to them to use a 
> `widen-unlock`.
>
> - when they get a bug report with a locked narrowing because of long 
> lines, using `widen-unlock` naively is likely to lead to an immediate 
> performance problem, so it's unlikely they'll use it.
>

When I read this, I thought you had a point, but there's a fallacy in your 
reasoning: using widen-unlock is in fact not likely to lead to an 
immediate performance problem.  The long-line-threshold limit is 
sufficiently high to never be reached in "normal" files, but nothing would 
happen if you cross that limit by a small amount, and nothing would even 
happen at twice or even thrice that limit.

If a mode author gets a bug report that is caused by locked narrowing, 
there is something wrong in the way the mode fontifies the buffer.  There 
is no reason to require access the whole buffer to fontify a small chunk 
of that buffer.  IOW, using widen-unlock there is nearly always wrong (I 
add "nearly" to leave open the possibility that there might be an 
exception).

This is becoming so litigious (you're now telling me that you're offended) 
that I start to believe that the right thing might in fact be to 
completely disable font locking in such buffers.  Would "no highlighting" 
be better than "occasional mis-highlighting" from your point of view?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> The general idea is that syntax-ppss is currently an order of magnitude 
>> too slow for "too large" buffers.
>
> And that it can be almost another order of magnitude worse in some major 
> modes because of `syntax-propertize`.
>

That's possible, I don't know.  I might take a look at that later.  But 
note that your reactions are not exactly encouraging me to do so.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

By the way, I found at least one other case in which Emacs silently 
refuses to do what the programmer asked it to do.

A recipe:

emacs -Q
M-: (setq max-specpdl-size 100) RET
M-: max-specpdl-size RET ;; this returns 100, as expected
C-x C-f lisp/simple.el RET
M-: max-specpdl-size RET ;; this returns 400, why?
M-: (setq max-specpdl-size 100) RET
M-: max-specpdl-size RET ;; this returns 100 again, as expected
C-v ;; lean on the key until the end of the buffer
M-: max-specpdl-size RET ;; this returns 265, why?

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> - when they get a bug report with a locked narrowing because of long
>> lines, using `widen-unlock` naively is likely to lead to an immediate
>> performance problem, so it's unlikely they'll use it.
> When I read this, I thought you had a point, but there's a fallacy in your
> reasoning: using widen-unlock is in fact not likely to lead to an immediate
> performance problem.  The long-line-threshold limit is sufficiently high to
> never be reached in "normal" files, but nothing would happen if you cross
> that limit by a small amount, and nothing would even happen at twice or even
> thrice that limit.

That's a valid point.  A bit like Alan's bug report, where he gets
a regression for 10K-long lines where the performance would be tolerable.

> If a mode author gets a bug report that is caused by locked narrowing, there
> is something wrong in the way the mode fontifies the buffer.  There is no
> reason to require access the whole buffer to fontify a small chunk of that
> buffer.  IOW, using widen-unlock there is nearly always wrong (I add
> "nearly" to leave open the possibility that there might be an exception).

As I explained already, it's basically always wrong for a major mode's
font-lock rules to widen, regardless if the narrowing is due to
something like LLT or MMM-mode.

> This is becoming so litigious (you're now telling me that you're offended)
> that I start to believe that the right thing might in fact be to completely
> disable font locking in such buffers.  Would "no highlighting" be better
> than "occasional mis-highlighting" from your point of view?

I don't care about the mishighlighting and find the current behavior
perfectly acceptable from an end-user point of view.  I only care about
the extra enforcement done in C code without providing any mechanism to
circumvent it.  Especially since this discussion seems to suggest that
if I were to propose a patch that makes this locking a bit more "soft",
it might be rejected on the grounds that it opens the door to abuse, so
not only I strongly dislike this design but I can't even try and
improve it.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> The general idea is that syntax-ppss is currently an order of magnitude
>>> too slow for "too large" buffers.
>> And that it can be almost another order of magnitude worse in some major
>> modes because of `syntax-propertize`.
> That's possible, I don't know.  I might take a look at that later.  But note
> that your reactions are not exactly encouraging me to do so.

I don't see any need to look into it, so don't waste your time there,
because whether what I said above is true or not is mostly irrelevant
(e.g. what would you do with that info anyway?).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2900 bytes --]

On 28.07.2022 10:49, Gregory Heytings wrote:
> 
>>> Yes, occasional mis-fontification is expected.  It's a compromise 
>>> between "no fontification" and "slow fontification".
>>
>> I wonder now if the majority of the slowdown was caused by the 
>> redisplay, whereas font-lock (which only has to run once per 
>> screenful) was actually "fast enough".
>>
> 
> Those two statements are not mutually exclusive.  The majority of the 
> slowdown was indeed caused by redisplay, but font-lock was not fast 
> enough.  Try to open a sufficiently large file (e.g. the dictionary.json 
> one) with the code on master, and type M->.

Where does one obtain dictionary.json from?

> You'll see that Emacs needs 
> about five seconds (on my laptop) to display the end of the buffer.

Does that come from the long lines, or solely from the size of the buffer?

> Now 
> compare that with the feature branch, with which the end of the buffer 
> is displayed instantaneously.  That five seconds delay is caused by 
> fontification-functions.

At some point we should accept that visiting a huge file might take some 
time (5 seconds doesn't sound terrible, depending on the context). 
Because the alternative is mis-fontifications and broken display.

>> Could you clarify what you mean by "access ... to the place where ... 
>> is defined"? "new Downloadify.Container" is highlighted by a regular 
>> regexp matcher, not some custom elisp code which has to visit the 
>> position where the identifier is defined.
>>
> 
> Sorry, I cannot be more precise, I don't have the "downloadify.js" file 
> here.  It was just a guess, based on what I saw on the screenshot, that 
> one function called by fontification-functions collects all class 
> definitions and highlights their identifiers elsewhere in the buffer 
> with a specific face.  When the buffer is narrowed, that function may 
> not see the Downloadify.Container definition (which is, I guess, placed 
> near the beginning of the file) anymore.

Here I'm attaching a version of downloadify.js we can use for comparison 
(please rename the extension from .sj to .js locally; Gmail was not 
letting it through otherwise). It's not a huge file, just about 88K.

As long as I keep my Emacs window/frame width half of the desktop, I can 
reliably reproduce the problem with the lack of highlighting for 
"Downloadify.Container" while other tokens are still highlighted.

I'm also attaching a screenshot of another problem: suddenly the bottom 
several screens of the buffer are mis-highlighted as if starting inside 
a string. That very much look like a result of breaking syntax-ppss's 
visibility of the buffer.

So the buffer scrolls quickly but looks bad.

Branch feature/long-lines-and-font-locking, revision cd41ce8c6c1079 from 
July 25. That branch is not there anymore, so let me know if I should 
re-test this with some later version of your work.

[-- Attachment #2: downloadify-min-example.sj --]
[-- Type: application/x-javascript, Size: 90077 bytes --]

[-- Attachment #3: Screenshot from 2022-08-04 03-32-22.png --]
[-- Type: image/png, Size: 1050723 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 03.08.2022 14:56, Eli Zaretskii wrote:
>> Date: Wed, 3 Aug 2022 03:26:02 +0300
>> From: Dmitry Gutov<dgutov@yandex.ru>
>> Cc:56682@debbugs.gnu.org,monnier@iro.umontreal.ca
>>
>> On 03.08.2022 03:00, Dmitry Gutov wrote:
>>> On 02.08.2022 19:19, Gregory Heytings wrote:
>>>>>> Try to open the dictionary.json with Emacs on master a month ago.
>>>>> I believe it can also be done with the current master, just after
>>>>> setting long-line-threshold to the nil value.  Right?
>>>>>
>>>> Indeed.  With master from one month ago it's even more crystal-clear
>>>> that you see the statu quo ante.
>>> If I set long-line-threshold to nil, does that also disable the
>>> redisplay optimizations related to long lines?
>>>
>>> Ones that caused scrolling delays even after the buffer has been fully
>>> fontified.
>> I mean those that*fixed*  the said scrolling delays, of course.
> I don't think I understand which changes you had in mind (could you
> give a less vague pointer?), but I think the answer is NO: that
> variable disables only the recent changes related to long lines.

I'm not sure if there are separate/specific changes to speak of (sorry, 
I'm flying blind), but previously I remarked in this discussion that on 
master pushing C-p can still result in sluggish response (at the end of 
a long line), even after the current window has been fontified (meaning 
font-lock has finished its work), and Gregory remarked that I should try 
the branch feature/long-lines-and-font-locking where this was fixed.

I did try the branch and indeed did not experience that sluggishness 
anymore. But that probably is a separate issue from slow font-lock.

So to try to debug any remaining speed issues with font-lock, it would 
be great to eliminate other sources of slow display first. In particular 
ones coming from a low-level subsystem I cannot as easily 
benchmark/debug/etc as Lisp code.

The setting

(setq long-line-threshold nil syntax-wholeline-max most-positive-fixnum)

indeed makes all the speed issues come back, including the 
aforementioned sluggishness.

You can try it yourself with downloadify.js I attached to the previous 
email. It's not a huge file either: the line is 90077 characters long.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 02.08.2022 18:46, Eli Zaretskii wrote:
>> Date: Tue, 2 Aug 2022 17:10:53 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 02.08.2022 05:27, Eli Zaretskii wrote:
>>>> syntax-ppss cache is a list of checkpoints spread along the buffer.
>>>>
>>>> After a modification, only the checkpoints below it are invalidated (to
>>>> be recomputed on-demand later).
>>> So a suitably-concocted replace command will still invalidate a lot of
>>> that cache, right?
>>
>> For any cache, one can invent an operation that would result in
>> thrashing it repeatedly.
> 
> Yes, but when thrashing causes delays of dozens of seconds, the result
> is not just a rare delay, the result is simply unacceptable.

What is unacceptable is the behavior I see from the narrowing solution. 
See the screenshots I attached in this thread.

>> A regular search-replace should work well enough, though. Because when
>> the buffer is long, the user is likely, on average, to spend a lot more
>> time examining the occurrences and deciding whether to replace each one.
>> And since the operation goes from top to bottom, this will likely
>> invalidate the list of caches once, and then rebuild it from the
>> beginning (or from wherever the first replacement was).
> 
> We want every basic operation in such buffers to perform reasonably
> well.  That's the goal of this activity.  Because partial solutions
> that sometimes work we already have: there's so-long-mode, there's
> longlines.el, and a couple of other trick up our sleeve.

We cannot perform every basic operation in fixed time for any 
arbitrarily sized file. There are limits of what we can possibly do.

If we narrow willy-nilly, we step on the toes of syntax parsing and get 
other weird behaviors as a result.

Which means we got another partial solution.

so-long-mode, longlines, etc, were all targeted as buffers with long 
lines. I'd really like it if we could scope this discussion to solving 
that particular problem. Not the speed of operations in large files in 
general.

The long lines problem is caused by pathologic complexity of some 
operations (like O(N^2) of line length, I guess). syntax-ppss's 
performance is nothing like that: it's O(N) for initial full scan, and 
O(1) for most operations afterward.

You can't really get better than that. Maye get a better multiplier with 
tree-sitter or a more optimized version of parse-partial-sexp, but take 
a 10x bigger file (or 100x bigger, or 1000x bigger) - and voila, the 
delay can be observed again.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 2346 bytes --]


>
> Where does one obtain dictionary.json from?
>

From here, for example: https://www.heytings.org/data/dictionary.json .

>> You'll see that Emacs needs about five seconds (on my laptop) to 
>> display the end of the buffer.
>
> Does that come from the long lines, or solely from the size of the 
> buffer?
>

You're replying to a week old email.  IIRC that delay comes solely from 
font locking.

>> Now compare that with the feature branch, with which the end of the 
>> buffer is displayed instantaneously.  That five seconds delay is caused 
>> by fontification-functions.
>
> At some point we should accept that visiting a huge file might take some 
> time (5 seconds doesn't sound terrible, depending on the context). 
> Because the alternative is mis-fontifications and broken display.
>

If you dislike mis-fontification, turn font-lock mode off.

>
> Here I'm attaching a version of downloadify.js we can use for comparison 
> (please rename the extension from .sj to .js locally; Gmail was not 
> letting it through otherwise). It's not a huge file, just about 88K.
>

It's a tiny file, not in any way representative of the ones we're dealing 
with.  But amusingly, even with that tiny file, you can see the problem at 
hand.  Do M-: (setq long-line-threshold nil) RET, and open it in a large 
enough window (e.g. 160 characters).  Type M->, and try to move point 
there with C-p or C-n.  You'll see that Emacs is already sluggish.

>
> I'm also attaching a screenshot of another problem: suddenly the bottom 
> several screens of the buffer are mis-highlighted as if starting inside 
> a string. That very much look like a result of breaking syntax-ppss's 
> visibility of the buffer.
>
> So the buffer scrolls quickly but looks bad.
>

If you dislike mis-fontification, turn font-lock mode off.  It's as easy 
as that.  Mis-fontification is expected in such cases.  The docstring of 
syntax-wholeline-max also mentions that "misfontification may then occur". 
Why did you not protest at that time?

>
> Branch feature/long-lines-and-font-locking, revision cd41ce8c6c1079 from 
> July 25. That branch is not there anymore, so let me know if I should 
> re-test this with some later version of your work.
>

That branch doesn't exist anymore, it has been merged in master.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I only care about the extra enforcement done in C code without providing 
> any mechanism to circumvent it.  Especially since this discussion seems 
> to suggest that if I were to propose a patch that makes this locking a 
> bit more "soft", it might be rejected on the grounds that it opens the 
> door to abuse, so not only I strongly dislike this design but I can't 
> even try and improve it.
>

I did not say that.  I do consider that it's not the right thing to do, 
for all the reasons I already gave, but if you have something concrete in 
mind, please show it.  I'm not the one who decides anyway.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>>> The general idea is that syntax-ppss is currently an order of 
>>>> magnitude too slow for "too large" buffers.
>>>
>>> And that it can be almost another order of magnitude worse in some 
>>> major modes because of `syntax-propertize`.
>>
>> That's possible, I don't know.  I might take a look at that later. 
>> But note that your reactions are not exactly encouraging me to do so.
>
> I don't see any need to look into it, so don't waste your time there,
>

To make syntax-ppss faster, if possible?  Is that not a sensible thing to 
do?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> So to try to debug any remaining speed issues with font-lock, it would 
> be great to eliminate other sources of slow display first. In particular 
> ones coming from a low-level subsystem I cannot as easily 
> benchmark/debug/etc as Lisp code.
>
> The setting
>
> (setq long-line-threshold nil syntax-wholeline-max most-positive-fixnum)
>
> indeed makes all the speed issues come back, including the 
> aforementioned sluggishness.
>

If you want to see only the slowdowns cause by font locking, just comment 
out the "if (current_buffer->long_line_optimizations_p)" in 
xdisp.c:handle_fontified_prop.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> We cannot perform every basic operation in fixed time for any 
> arbitrarily sized file. There are limits of what we can possibly do.
>

Apparently the limits are lower than what you think.  Provided that we 
accept some compromises, such as mis-fontification, which is also what 
syntax-wholeline-max does, and against which you didn't protest.

>
> I'd really like it if we could scope this discussion to solving that 
> particular problem. Not the speed of operations in large files in 
> general.
>

I don't understand what you mean.  Which "particular problem"?  The point 
of this discussion is of course the speed of operations in large files in 
general.  If you take that out of the picture, everything is of course 
possible.  I'm not even sure what remains in fact, Emacs is an editor, not 
a displayer.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gregory@heytings.org,  dgutov@yandex.ru,  56682@debbugs.gnu.org
> Date: Wed, 03 Aug 2022 16:36:05 -0400
> 
> > What do you mean by "accidentally" in this context?  A program rarely
> > does anything "accidentally", it almost always does what the
> > programmer wanted it to do.
> 
> I mean that currently font-lock just blindly widens all the time, so we
> should fix it so it only widens when it's expected to (i.e. not when
> the widening is installed by the LLT code).

So let's please fix that, and take it from there.  We have plenty of
time to make followup decisions after we see what that does.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gregory@heytings.org,  dgutov@yandex.ru,  56682@debbugs.gnu.org
> Date: Wed, 03 Aug 2022 16:38:57 -0400
> 
> > What about parse-partial-sexp, which calls scan_sexps_forward?  It
> > looks like I've misremembered, and that was the culprit in the
> > scenario we discussed, see
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45898#92.
> 
> The wholelines problem did not kick in because of PPS nor `syntax-ppss`
> but because of font-lock (which then called `syntax-ppss` which then
> called PPS).

If it's font-lock that forces syntax-ppss to examine the whole huge
line, then what is your proposal for avoiding that which doesn't
involve some more-or-less arbitrary restrictions on the part of the
buffer that can be examined by syntax-ppss?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 56682@debbugs.gnu.org,  Eli Zaretskii <eliz@gnu.org>,  Dmitry Gutov
>  <dgutov@yandex.ru>
> Date: Wed, 03 Aug 2022 18:42:03 -0400
> 
> I don't care about the mishighlighting and find the current behavior
> perfectly acceptable from an end-user point of view.  I only care about
> the extra enforcement done in C code without providing any mechanism to
> circumvent it.  Especially since this discussion seems to suggest that
> if I were to propose a patch that makes this locking a bit more "soft",
> it might be rejected on the grounds that it opens the door to abuse

"Show me the code", instead of imagining what would be the reaction to
that.  (To say nothing of the fact that what I already wrote several
times explicitly contradicts your impression.)

bug#56682: Fix the long lines font locking related slowdowns

[Lars Ingebrigtsen]

By the way, playing with Alan's example here a bit...  To recap, this is
the test case (in a .cc file):

---
char long_line[] = R"foo(

)foo"
---

If I insert a 1M long line there (with `C-y'), Emacs will hang
indefinitely.  Wasn't the long-line stuff supposed to trigger in these
situations?  Or is it hanging in some cc-mode stuff before we get that
far?

`C-g' a number of times will eventually get out of the hang, but then
Emacs hangs again.  `C-g' again a few times breaks out of that, and then
finally Emacs becomes responsive.  If I load the file with the 1M line
from the start, then Emacs is responsive all the time.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 4 Aug 2022 04:08:20 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 03.08.2022 14:56, Eli Zaretskii wrote:
> >> Date: Wed, 3 Aug 2022 03:26:02 +0300
> >> From: Dmitry Gutov<dgutov@yandex.ru>
> >> Cc:56682@debbugs.gnu.org,monnier@iro.umontreal.ca
> >>
> >> On 03.08.2022 03:00, Dmitry Gutov wrote:
> >>> On 02.08.2022 19:19, Gregory Heytings wrote:
> >>>>>> Try to open the dictionary.json with Emacs on master a month ago.
> >>>>> I believe it can also be done with the current master, just after
> >>>>> setting long-line-threshold to the nil value.  Right?
> >>>>>
> >>>> Indeed.  With master from one month ago it's even more crystal-clear
> >>>> that you see the statu quo ante.
> >>> If I set long-line-threshold to nil, does that also disable the
> >>> redisplay optimizations related to long lines?
> >>>
> >>> Ones that caused scrolling delays even after the buffer has been fully
> >>> fontified.
> >> I mean those that*fixed*  the said scrolling delays, of course.
> > I don't think I understand which changes you had in mind (could you
> > give a less vague pointer?), but I think the answer is NO: that
> > variable disables only the recent changes related to long lines.
> 
> I'm not sure if there are separate/specific changes to speak of (sorry, 
> I'm flying blind), but previously I remarked in this discussion that on 
> master pushing C-p can still result in sluggish response (at the end of 
> a long line), even after the current window has been fontified (meaning 
> font-lock has finished its work), and Gregory remarked that I should try 
> the branch feature/long-lines-and-font-locking where this was fixed.
> 
> I did try the branch and indeed did not experience that sluggishness 
> anymore. But that probably is a separate issue from slow font-lock.
> 
> So to try to debug any remaining speed issues with font-lock, it would 
> be great to eliminate other sources of slow display first. In particular 
> ones coming from a low-level subsystem I cannot as easily 
> benchmark/debug/etc as Lisp code.
> 
> The setting
> 
> (setq long-line-threshold nil syntax-wholeline-max most-positive-fixnum)
> 
> indeed makes all the speed issues come back, including the 
> aforementioned sluggishness.

OK, so I guess my answer to your question was accurate, and you now
have a way to reproduce those issues by disabling the recent speedups.

The changes in what was the long-lines-and-font-locking branch are now
on master, AFAIK.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 4 Aug 2022 03:49:43 +0300
> Cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Now 
> > compare that with the feature branch, with which the end of the buffer 
> > is displayed instantaneously.  That five seconds delay is caused by 
> > fontification-functions.
> 
> At some point we should accept that visiting a huge file might take some 
> time (5 seconds doesn't sound terrible, depending on the context). 
> Because the alternative is mis-fontifications and broken display.
> 
> >> Could you clarify what you mean by "access ... to the place where ... 
> >> is defined"? "new Downloadify.Container" is highlighted by a regular 
> >> regexp matcher, not some custom elisp code which has to visit the 
> >> position where the identifier is defined.
> >>
> > 
> > Sorry, I cannot be more precise, I don't have the "downloadify.js" file 
> > here.  It was just a guess, based on what I saw on the screenshot, that 
> > one function called by fontification-functions collects all class 
> > definitions and highlights their identifiers elsewhere in the buffer 
> > with a specific face.  When the buffer is narrowed, that function may 
> > not see the Downloadify.Container definition (which is, I guess, placed 
> > near the beginning of the file) anymore.
> 
> Here I'm attaching a version of downloadify.js we can use for comparison 
> (please rename the extension from .sj to .js locally; Gmail was not 
> letting it through otherwise). It's not a huge file, just about 88K.
> 
> As long as I keep my Emacs window/frame width half of the desktop, I can 
> reliably reproduce the problem with the lack of highlighting for 
> "Downloadify.Container" while other tokens are still highlighted.
> 
> I'm also attaching a screenshot of another problem: suddenly the bottom 
> several screens of the buffer are mis-highlighted as if starting inside 
> a string. That very much look like a result of breaking syntax-ppss's 
> visibility of the buffer.
> 
> So the buffer scrolls quickly but looks bad.

I don't understand the point you are trying to make.

On the one hand, you say "At some point we should accept that visiting
a huge file might take some time", which seems to imply that you agree
in general with some (hopefully graceful) degradation when editing
files with such long lines.  But OTOH you object to have that
degradation in the fontification?  IOW, you prefer Emacs to become
much slower, but still fontify correctly?  If so, just enlarge the
value of long-line-threshold, with the effect that Emacs will become
more sluggish before the long-line optimizations kick in.  If this is
your point, then maybe lobby for enlarging the default value of that
variable.

But if you are saying that Emacs should behave as it does with that
variable being nil, then I don't understand your position.  With that
variable nil, Emacs becomes _unusable_, not just slow, with files that
are not even too large by any modern measure, just because the lines
are very long.  And in those cases, why is it wrong to decide that
occasional glitches in fontifications are a lesser evil than a
complete lockup of the Emacs UI, which usually results in users
killing the session?  We decided that some imperfection in
fontifications are the "graceful degradation" we are willing to endure
in order to make Emacs reasonably performant in those cases.  What is
wrong with that tradeoff?  And what alternative tradeoff would you
suggest instead?

(You mention "broken display" in addition to inaccurate
fontifications, but I don't understand what does that allude to.
Which instances of broken display did you see, and how to reproduce
them?)

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 4 Aug 2022 04:08:20 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Yes, but when thrashing causes delays of dozens of seconds, the result
> > is not just a rare delay, the result is simply unacceptable.
> 
> What is unacceptable is the behavior I see from the narrowing solution. 
> See the screenshots I attached in this thread.

So that behavior is unacceptable, but declaring we are unable to allow
editing of such files is acceptable?  If that is your position, we
will have to agree to disagree.  We have decided that we want a more
graceful degradation in these cases, and in particular to sacrifice
some accuracy of font-lock to allow reasonable editing of such files.
If you consider that unacceptable, you can still have the lockups you
prefer by setting long-line-threshold to nil.  But we disagree with
having nil be the default value of that variable.

> > We want every basic operation in such buffers to perform reasonably
> > well.  That's the goal of this activity.  Because partial solutions
> > that sometimes work we already have: there's so-long-mode, there's
> > longlines.el, and a couple of other trick up our sleeve.
> 
> We cannot perform every basic operation in fixed time for any 
> arbitrarily sized file. There are limits of what we can possibly do.

I said "reasonably well", not "in fixed time".  Some slowdown is
acceptable.  But when a simple editing operation takes several
minutes, we cannot in good faith claim that Emacs is still usable.  If
you prefer to have that to some degradation in font-lock, then we
disagree.

> If we narrow willy-nilly, we step on the toes of syntax parsing and get 
> other weird behaviors as a result.
> 
> Which means we got another partial solution.

The experience till now seems to indicate that the degradation caused
by this solution is much milder and rarer than any other solution, and
allows users to edit files with much longer lines.  It certainly
sounds like a better solution than completely disabling font-lock in
such buffers (but that is still an option, if someone prefers it).
Reporting specific problems with that solution will allow us to solve
at least some of them, with the goal of making the degradation even
more graceful -- this is what this bug report is for.

But if you want to claim that it is better to have Emacs lock up for
minutes in these cases, you will have to do that in your local
customization, because we disagree with having Emacs behave like that
when it can be avoided.

> so-long-mode, longlines, etc, were all targeted as buffers with long 
> lines. I'd really like it if we could scope this discussion to solving 
> that particular problem. Not the speed of operations in large files in 
> general.

The "long lines problem" is directly related to the speed of
operations.  We didn't yet make any changes that affect large files in
general, only files with long lines.  In those cases, the speed of
operations becomes unacceptably slow beyond some threshold, and we
want Emacs to remain usable beyond that threshold, even if that makes
fontifications sometimes inaccurate.

> The long lines problem is caused by pathologic complexity of some 
> operations (like O(N^2) of line length, I guess).

That assumption is incorrect, at least according to my analysis of the
relevant bottlenecks.  The slowdown is mostly linear in the number of
buffer positions redisplay and its subroutines need to traverse.

> syntax-ppss's performance is nothing like that: it's O(N) for
> initial full scan, and O(1) for most operations afterward.
> 
> You can't really get better than that. Maye get a better multiplier with 
> tree-sitter or a more optimized version of parse-partial-sexp, but take 
> a 10x bigger file (or 100x bigger, or 1000x bigger) - and voila, the 
> delay can be observed again.

No one denies that beyond some threshold the performance will be too
slow again.  We just want to make that threshold much farther.

In addition, the idea of using narrowing is a good one precisely
because it is unaffected by the buffer size.  So its effect doesn't
deteriorate when the buffer or the line length becomes larger.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 01:26:04 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, 
>     Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca
> 
> > Branch feature/long-lines-and-font-locking, revision cd41ce8c6c1079 from 
> > July 25. That branch is not there anymore, so let me know if I should 
> > re-test this with some later version of your work.
> 
> That branch doesn't exist anymore, it has been merged in master.

Right.

Gregory, is there a reason why the long-lines-improvements branch is
not yet merged?  I think it already includes important improvements
that should be exposed to a larger population for testing and
feedback.

If you for some reason prefer to keep that branch active, can you
please merge the current master to it, so that the changes in
narrow-to-region will be on the branch?  As things are now, the
problems you solved on master in that part are not yet solved on the
branch, and so working on the branch runs the risk of hitting problems
unrelated to changes in display code.

Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Gregory, is there a reason why the long-lines-improvements branch is not 
> yet merged?  I think it already includes important improvements that 
> should be exposed to a larger population for testing and feedback.
>

I was planning to add more improvements, but indeed merging that one and 
creating yet another one seems better at this point.

I just pushed an improvement for Bidi.  I did not do so earlier because I 
was trying to improve it further, but somehow I'm hitting a brick wall 
here.  Could you please check, and tell me if what I already did is okay?

In this case I did not find a good test case in the wild (or more 
precisely, I had no idea how to find one), so I concoted one myself. 
You'll find it here: https://www.heytings.org/data/locales.json .  Emacs 
is still a bit sluggish with that file, but behaves much better than 
before.

>
> If you for some reason prefer to keep that branch active, can you please 
> merge the current master to it, so that the changes in narrow-to-region 
> will be on the branch?
>

Is that feasible?  Non fast forwards are not allowed in the Emacs 
repository, so rebasing a feature branch is not possible (without using 
the workaround of deleting and recreating the branch).

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 09:24:00 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: dgutov@yandex.ru, gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, 
>     monnier@iro.umontreal.ca
> 
> I just pushed an improvement for Bidi.  I did not do so earlier because I 
> was trying to improve it further, but somehow I'm hitting a brick wall 
> here.  Could you please check, and tell me if what I already did is okay?

Will do, thanks.

> > If you for some reason prefer to keep that branch active, can you please 
> > merge the current master to it, so that the changes in narrow-to-region 
> > will be on the branch?
> 
> Is that feasible?  Non fast forwards are not allowed in the Emacs 
> repository, so rebasing a feature branch is not possible (without using 
> the workaround of deleting and recreating the branch).

I didn't mean a rebase, I meant a merge.  A merge from master to a
feature branch will work exactly as a merge from emacs-28 to master
or, indeed, a merge from a feature branch to master.  There are no
problems here.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 09:24:00 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: dgutov@yandex.ru, gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, 
>     monnier@iro.umontreal.ca
> 
> I just pushed an improvement for Bidi.  I did not do so earlier because I 
> was trying to improve it further, but somehow I'm hitting a brick wall 
> here.  Could you please check, and tell me if what I already did is okay?

Hmm... which part(s) of the recent commit(s) on the branch are related
to bidi?  I only see changes related to composed characters.  What did
I miss?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> If you for some reason prefer to keep that branch active, can you 
>>> please merge the current master to it, so that the changes in 
>>> narrow-to-region will be on the branch?
>>
>> Is that feasible?  Non fast forwards are not allowed in the Emacs 
>> repository, so rebasing a feature branch is not possible (without using 
>> the workaround of deleting and recreating the branch).
>
> I didn't mean a rebase, I meant a merge.
>

Indeed, I did not read what you wrote with enough attention.  Sorry for 
that.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> I just pushed an improvement for Bidi.  I did not do so earlier because 
>> I was trying to improve it further, but somehow I'm hitting a brick 
>> wall here.  Could you please check, and tell me if what I already did 
>> is okay?
>
> Hmm... which part(s) of the recent commit(s) on the branch are related 
> to bidi?  I only see changes related to composed characters.  What did I 
> miss?
>

82b602dc2f improves bidi in long lines, without indeed touching bidi.c.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Kangas]

Gregory Heytings <gregory@heytings.org> writes:

>> If you for some reason prefer to keep that branch active, can you please
>> merge the current master to it, so that the changes in narrow-to-region
>> will be on the branch?
>
> Is that feasible?  Non fast forwards are not allowed in the Emacs
> repository, so rebasing a feature branch is not possible (without using
> the workaround of deleting and recreating the branch).

Rebasing is not possible, but merging is.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 09:46:17 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: dgutov@yandex.ru, gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, 
>     monnier@iro.umontreal.ca
> 
> 
> >> I just pushed an improvement for Bidi.  I did not do so earlier because 
> >> I was trying to improve it further, but somehow I'm hitting a brick 
> >> wall here.  Could you please check, and tell me if what I already did 
> >> is okay?
> >
> > Hmm... which part(s) of the recent commit(s) on the branch are related 
> > to bidi?  I only see changes related to composed characters.  What did I 
> > miss?
> >
> 
> 82b602dc2f improves bidi in long lines, without indeed touching bidi.c.

The changes are related to compositions, not to bidi.  Displaying
Arabic (and maybe also other characters in that file) requires
character composition, but it has nothing in particular to do with
bidi per se.

Are you saying that if you replace the Arabic text there with some
other script that also requires composition processing (like one of
the Indic scripts, see lisp/language/indian.el), editing this file is
significantly faster?

Btw, I'm unable to edit that file on the branch, because
show-paren--default causes an assertion violation.  I'm pretty sure
that's due to the issues in narrow-to-region that were already fixed
on master.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> The changes are related to compositions, not to bidi.  Displaying Arabic 
> (and maybe also other characters in that file) requires character 
> composition, but it has nothing in particular to do with bidi per se.
>

I know next to nothing about bidi, so it's very well possible indeed that 
I confused "bidi" and "composition" (or "bidi composition"?).  Anyway, 
navigating through the locales.json file was slow (at some positions) 
before the change and is now reasonably fast (but alas not instantaneous).

>
> Are you saying that if you replace the Arabic text there with some other 
> script that also requires composition processing (like one of the Indic 
> scripts, see lisp/language/indian.el), editing this file is 
> significantly faster?
>

I think the locales.json file contains samples of pretty much all 
available scripts.  Devanagari for example is around position 3260000. 
As far as I can tell, navigating in that part of the file is not 
significantly faster with the change.  It is only in the parts of the file 
that contain e.g. Arabic text that the speedup is visible, around position 
70000 for example.

>
> Btw, I'm unable to edit that file on the branch, because 
> show-paren--default causes an assertion violation.  I'm pretty sure 
> that's due to the issues in narrow-to-region that were already fixed on 
> master.
>

I just merged master into the feature branch.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 04.08.2022 04:26, Gregory Heytings wrote:

>> Here I'm attaching a version of downloadify.js we can use for 
>> comparison (please rename the extension from .sj to .js locally; Gmail 
>> was not letting it through otherwise). It's not a huge file, just 
>> about 88K.
>>
> 
> It's a tiny file, not in any way representative of the ones we're 
> dealing with.  But amusingly, even with that tiny file, you can see the 
> problem at hand.  Do M-: (setq long-line-threshold nil) RET, and open it 
> in a large enough window (e.g. 160 characters).  Type M->, and try to 
> move point there with C-p or C-n.  You'll see that Emacs is already 
> sluggish.

That's the scenario I described, and that's my point: this file's 
display is sluggish. Even though font-lock has already finished its 
work. And it didn't have to spend any significant time in syntax-ppss.

So there is a particular performance problem with the display of 
fontified buffers which I'd really like your help in fixing.

Fixing in a way that doesn't add narrowing around 
fontification-functions, because as we can see it's not necessary in 
examples like this.

Then it would be much easier to evaluate font-lock's effect on 
performance in larger files.

>> I'm also attaching a screenshot of another problem: suddenly the 
>> bottom several screens of the buffer are mis-highlighted as if 
>> starting inside a string. That very much look like a result of 
>> breaking syntax-ppss's visibility of the buffer.
>>
>> So the buffer scrolls quickly but looks bad.
>>
> 
> If you dislike mis-fontification, turn font-lock mode off.  It's as easy 
> as that.  Mis-fontification is expected in such cases.  The docstring of 
> syntax-wholeline-max also mentions that "misfontification may then 
> occur". Why did you not protest at that time?

I think we could have both speed and correctness, at least for files of 
this size.

>> Branch feature/long-lines-and-font-locking, revision cd41ce8c6c1079 
>> from July 25. That branch is not there anymore, so let me know if I 
>> should re-test this with some later version of your work.
>>
> 
> That branch doesn't exist anymore, it has been merged in master.

Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> By the way, playing with Alan's example here a bit...  To recap, this is 
> the test case (in a .cc file):
>
> ---
> char long_line[] = R"foo(
>
> )foo"
> ---
>
> If I insert a 1M long line there (with `C-y'), Emacs will hang 
> indefinitely.  Wasn't the long-line stuff supposed to trigger in these 
> situations?  Or is it hanging in some cc-mode stuff before we get that 
> far?
>

No wonder.  CC Mode is a slow mode, and one of the worst offenders here. 
In this cas, IIUC, what you see is because CC Mode adds c-after-change to 
after-change-functions, which has the effect that

(put-text-property 27 1000028 'face 'font-lock-string-face)

is called no less than 1335 times after that C-y.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> So there is a particular performance problem with the display of 
> fontified buffers which I'd really like your help in fixing.
>

Thank you, but that problem has, as you can see, already been fixed.

>
> Fixing in a way that doesn't add narrowing around 
> fontification-functions, because as we can see it's not necessary in 
> examples like this.
>

As I said, your 88K example file is not representative.  So the fact that 
it's not necessary in that tiny example file says nothing about much 
larger files.

>
> I think we could have both speed and correctness, at least for files of 
> this size.
>

Feel free to enlarge long-line-threshold in your init file.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Kangas]

Gregory Heytings <gregory@heytings.org> writes:

>> I think we could have both speed and correctness, at least for files of
>> this size.
>
> Feel free to enlarge long-line-threshold in your init file.

Just a curious question: How did we arrive at the current value of
`long-line-threshold'?

I've been trying to follow the discussion but I don't think this has
been mentioned anywhere.  My apologies if I missed it.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 568 bytes --]


>>> I think we could have both speed and correctness, at least for files 
>>> of this size.
>>
>> Feel free to enlarge long-line-threshold in your init file.
>
> Just a curious question: How did we arrive at the current value of 
> `long-line-threshold'?
>

I took the same value that Stefan had chosen for syntax-wholeline-max.

>
> I've been trying to follow the discussion but I don't think this has 
> been mentioned anywhere.  My apologies if I missed it.
>

No need to apologize, you're a careful reader 😃, it hasn't been mentioned 
indeed.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 12:05:14 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Dmitry Gutov <dgutov@yandex.ru>, gerd.moellmann@gmail.com, 
>     56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>, 
>     monnier@iro.umontreal.ca
> 
> >> Feel free to enlarge long-line-threshold in your init file.
> >
> > Just a curious question: How did we arrive at the current value of 
> > `long-line-threshold'?
> >
> 
> I took the same value that Stefan had chosen for syntax-wholeline-max.

And it isn't sacred in any way.  If we decide a different value
strikes a better balance, we will change it.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 4 Aug 2022 13:35:39 +0300
> Cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org,
>  Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > It's a tiny file, not in any way representative of the ones we're 
> > dealing with.  But amusingly, even with that tiny file, you can see the 
> > problem at hand.  Do M-: (setq long-line-threshold nil) RET, and open it 
> > in a large enough window (e.g. 160 characters).  Type M->, and try to 
> > move point there with C-p or C-n.  You'll see that Emacs is already 
> > sluggish.
> 
> That's the scenario I described, and that's my point: this file's 
> display is sluggish. Even though font-lock has already finished its 
> work. And it didn't have to spend any significant time in syntax-ppss.
> 
> So there is a particular performance problem with the display of 
> fontified buffers which I'd really like your help in fixing.

Maybe it is in display, and maybe it isn't.  Do you have any evidence
that the sluggish response is due to redisplay?  C-n, for example, is
mostly not redisplay, but Lisp code in simple.el and occasional calls
to vertical-motion.

But even if the slow response is due to redisplay, we just have
another cause that we need to investigate and try fixing.  It says
nothing about the measures we've already taken on master.  They
definitely make even this case faster, and with an unoptimized build I
can now reasonably edit this file, something I couldn't do before.

> Fixing in a way that doesn't add narrowing around 
> fontification-functions, because as we can see it's not necessary in 
> examples like this.

If that is possible, sure.  No one said that from now on every problem
in Emacs that causes slow responses will be handled by narrowing.  But
if, for example, it turns out that the slow responses is due to time
it takes some code to traverse a long stretch of fontified buffer,
what other solution would you suggest except making the portion to be
traversed shorter?

> > If you dislike mis-fontification, turn font-lock mode off.  It's as easy 
> > as that.  Mis-fontification is expected in such cases.  The docstring of 
> > syntax-wholeline-max also mentions that "misfontification may then 
> > occur". Why did you not protest at that time?
> 
> I think we could have both speed and correctness, at least for files of 
> this size.

That is not a given, and the experience till now suggests otherwise.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 10:33:48 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> I know next to nothing about bidi, so it's very well possible indeed that 
> I confused "bidi" and "composition" (or "bidi composition"?).  Anyway, 
> navigating through the locales.json file was slow (at some positions) 
> before the change and is now reasonably fast (but alas not instantaneous).
> 
> >
> > Are you saying that if you replace the Arabic text there with some other 
> > script that also requires composition processing (like one of the Indic 
> > scripts, see lisp/language/indian.el), editing this file is 
> > significantly faster?
> >
> 
> I think the locales.json file contains samples of pretty much all 
> available scripts.  Devanagari for example is around position 3260000. 
> As far as I can tell, navigating in that part of the file is not 
> significantly faster with the change.  It is only in the parts of the file 
> that contain e.g. Arabic text that the speedup is visible, around position 
> 70000 for example.

OK, I will take a look.

> > Btw, I'm unable to edit that file on the branch, because 
> > show-paren--default causes an assertion violation.  I'm pretty sure 
> > that's due to the issues in narrow-to-region that were already fixed on 
> > master.
> 
> I just merged master into the feature branch.

Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>>> Feel free to enlarge long-line-threshold in your init file.
>>>
>>> Just a curious question: How did we arrive at the current value of 
>>> `long-line-threshold'?
>>
>> I took the same value that Stefan had chosen for syntax-wholeline-max.
>
> And it isn't sacred in any way.  If we decide a different value strikes 
> a better balance, we will change it.
>

Indeed (although I fear a bikeshedding here).

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 10:33:48 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> navigating through the locales.json file was slow (at some positions) 
> before the change and is now reasonably fast (but alas not instantaneous).

Which navigation commands were slow, as compared to the same commands
in other portions of this file?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca,
>  dgutov@yandex.ru
> Date: Thu, 04 Aug 2022 17:14:58 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Date: Thu, 04 Aug 2022 10:33:48 +0000
> > From: Gregory Heytings <gregory@heytings.org>
> > cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
> >     dgutov@yandex.ru
> > 
> > navigating through the locales.json file was slow (at some positions) 
> > before the change and is now reasonably fast (but alas not instantaneous).
> 
> Which navigation commands were slow, as compared to the same commands
> in other portions of this file?

Just before the Devanagari portion of the file, there's the Hebrew
portion, starting around buffer position 3243400.  If you go there and
try the same navigation commands that were slow with Arabic, are they
as slow with Hebrew (which is also a right-to-left script, but doesn't
use character compositions nearly as heavily as Arabic)?  Here it
looks like Hebrew is noticeably faster, as fast as Devanagari.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> navigating through the locales.json file was slow (at some positions) 
>> before the change and is now reasonably fast (but alas not 
>> instantaneous).
>
> Which navigation commands were slow, as compared to the same commands in 
> other portions of this file?
>

For example C-p, C-n, C-v, M-v.  C-f and C-b also, but much less so.

For example, open the file and do M-g c 70000 RET.  This took about 5 
seconds.  Now do C-n, this again took about 5 seconds.  With the 
optimizations, M-g c 70000 RET is almost immediate, and C-n there takes 
less than a second.

But you're right, this slowdown has little to do with bidi.  A file with a 
sufficiently long single line of Arabic text has the same problem.  (But 
not one with a line of Hebrew text.)

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Just before the Devanagari portion of the file, there's the Hebrew 
> portion, starting around buffer position 3243400.  If you go there and 
> try the same navigation commands that were slow with Arabic, are they as 
> slow with Hebrew (which is also a right-to-left script, but doesn't use 
> character compositions nearly as heavily as Arabic)?  Here it looks like 
> Hebrew is noticeably faster, as fast as Devanagari.
>

Sorry, I hadn't seen this post before replying to the previous one. 
Indeed, as I said in my previous reply navigation commands in Hebrew texts 
are much faster than in Arabic texts.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 15:08:07 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> For example C-p, C-n, C-v, M-v.  C-f and C-b also, but much less so.
> 
> For example, open the file and do M-g c 70000 RET.  This took about 5 
> seconds.  Now do C-n, this again took about 5 seconds.  With the 
> optimizations, M-g c 70000 RET is almost immediate, and C-n there takes 
> less than a second.

I'm on the branch, so I am _after_ the optimizations.  I thought you
said even after that the navigation is sluggish.  I see somewhat
slower response than in "normal" files, but my build is unoptimized,
so where it takes 3 or 4 seconds, and optimized build should be almost
instantaneous.  And that looks good enough to me, since being a bit
slower in such files is IMO fine.

> But you're right, this slowdown has little to do with bidi.  A file with a 
> sufficiently long single line of Arabic text has the same problem.  (But 
> not one with a line of Hebrew text.)

OK.  Text that goes through character compositions is expected to be
slower in redisplay, because character composition in Emacs works by
calling into Lisp.  So I think we are good here, do you agree?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> For example C-p, C-n, C-v, M-v.  C-f and C-b also, but much less so.
>>
>> For example, open the file and do M-g c 70000 RET.  This took about 5 
>> seconds.  Now do C-n, this again took about 5 seconds.  With the 
>> optimizations, M-g c 70000 RET is almost immediate, and C-n there takes 
>> less than a second.
>
> I'm on the branch, so I am _after_ the optimizations.  I thought you 
> said even after that the navigation is sluggish.
>

Yes, that's what I said indeed.

>
> I see somewhat slower response than in "normal" files, but my build is 
> unoptimized, so where it takes 3 or 4 seconds, and optimized build 
> should be almost instantaneous. And that looks good enough to me, since 
> being a bit slower in such files is IMO fine.
>

It's not really "almost instantaneous", moving point can take (depending 
on factors I do not understand at the moment) something between 0.2 
seconds and 2 seconds.

>
> OK.  Text that goes through character compositions is expected to be 
> slower in redisplay, because character composition in Emacs works by 
> calling into Lisp.
>

So Arabic text goes through character compositions and Hebrew text 
doesn't, is that correct?

>
> So I think we are good here, do you agree?
>

Hmmm...  I still think it would be possible to do better.  With the above 
recipe (M-g g 70000 RET C-n), composition_compute_stop_pos is called 
627663 times and uses about 2 seconds of CPU time.  What surprises me (and 
makes me believe it's perhaps possible to do better) is that it is called 
repeatedly with the same arguments.  For example, when doing C-n it is 
called 26 times with charpos = 69980.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 16:25:28 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> > I'm on the branch, so I am _after_ the optimizations.  I thought you 
> > said even after that the navigation is sluggish.
> 
> Yes, that's what I said indeed.
> 
> > I see somewhat slower response than in "normal" files, but my build is 
> > unoptimized, so where it takes 3 or 4 seconds, and optimized build 
> > should be almost instantaneous. And that looks good enough to me, since 
> > being a bit slower in such files is IMO fine.
> 
> It's not really "almost instantaneous", moving point can take (depending 
> on factors I do not understand at the moment) something between 0.2 
> seconds and 2 seconds.

I saw slower response when point was at a parentesis or a brace -- due
to show-paren-mode.  Try disabling it and see if that affects
performance.  Another problem could be the BPA algorithm in bidi.c,
but in my testing setting bidi-inhibit-bpa non-nil didn't affect
performance in any tangible way, with this file.

> > OK.  Text that goes through character compositions is expected to be 
> > slower in redisplay, because character composition in Emacs works by 
> > calling into Lisp.
> 
> So Arabic text goes through character compositions and Hebrew text 
> doesn't, is that correct?

More accurately, Arabic text needs to call the shaping engine
(HarfBuzz) for all the characters, whereas Hebrew does that only
rarely (and not at all in locales.json).  You can see from the
composition rules in, respectively, lisp/language/misc-lang.el and
lisp/language/hebrew.el that for Arabic, the entire range of Arabic
characters is populated with composition rules in
composition-function-table, whereas for Hebrew, only some relatively
rare characters have non-nil rules.

> > So I think we are good here, do you agree?
> 
> Hmmm...  I still think it would be possible to do better.  With the above 
> recipe (M-g g 70000 RET C-n), composition_compute_stop_pos is called 
> 627663 times and uses about 2 seconds of CPU time.  What surprises me (and 
> makes me believe it's perhaps possible to do better) is that it is called 
> repeatedly with the same arguments.  For example, when doing C-n it is 
> called 26 times with charpos = 69980.

I'll see where these come from and whether some of them could be
avoided.

Are you capable of running under perf and producing the profile of
these commands?  Because I wonder whether we correctly identify the
main bottlenecks in these scenarios.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> It's not really "almost instantaneous", moving point can take 
>> (depending on factors I do not understand at the moment) something 
>> between 0.2 seconds and 2 seconds.
>
> I saw slower response when point was at a parentesis or a brace -- due 
> to show-paren-mode.  Try disabling it and see if that affects 
> performance.
>

I forgot to mention that this is with show-paren-mode disabled.

>
> Another problem could be the BPA algorithm in bidi.c, but in my testing 
> setting bidi-inhibit-bpa non-nil didn't affect performance in any 
> tangible way, with this file.
>

Indeed.

>
> More accurately, Arabic text needs to call the shaping engine (HarfBuzz) 
> for all the characters, whereas Hebrew does that only rarely (and not at 
> all in locales.json).  You can see from the composition rules in, 
> respectively, lisp/language/misc-lang.el and lisp/language/hebrew.el 
> that for Arabic, the entire range of Arabic characters is populated with 
> composition rules in composition-function-table, whereas for Hebrew, 
> only some relatively rare characters have non-nil rules.
>

Okay, thanks!

>> Hmmm...  I still think it would be possible to do better.  With the 
>> above recipe (M-g g 70000 RET C-n), composition_compute_stop_pos is 
>> called 627663 times and uses about 2 seconds of CPU time.  What 
>> surprises me (and makes me believe it's perhaps possible to do better) 
>> is that it is called repeatedly with the same arguments.  For example, 
>> when doing C-n it is called 26 times with charpos = 69980.
>
> I'll see where these come from and whether some of them could be 
> avoided.
>
> Are you capable of running under perf and producing the profile of these 
> commands?  Because I wonder whether we correctly identify the main 
> bottlenecks in these scenarios.
>

I can't speak in general, but in this particular scenario, the bottleneck 
is clearly composition_compute_stop_pos.

You didn't tell me whether it's okay to merge the branch with the latest 
changes?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 18:16:02 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> >> Hmmm...  I still think it would be possible to do better.  With the 
> >> above recipe (M-g g 70000 RET C-n), composition_compute_stop_pos is 
> >> called 627663 times and uses about 2 seconds of CPU time.  What 
> >> surprises me (and makes me believe it's perhaps possible to do better) 
> >> is that it is called repeatedly with the same arguments.  For example, 
> >> when doing C-n it is called 26 times with charpos = 69980.
> >
> > I'll see where these come from and whether some of them could be 
> > avoided.
> >
> > Are you capable of running under perf and producing the profile of these 
> > commands?  Because I wonder whether we correctly identify the main 
> > bottlenecks in these scenarios.
> >
> 
> I can't speak in general, but in this particular scenario, the bottleneck 
> is clearly composition_compute_stop_pos.

I looked into this, and I don't see how this could be avoided,
unfortunately, not without disabling auto-composition-mode (which I'm
told produces display of Arabic that makes readers of that script turn
away in disgust).  Disabling auto-composition-mode makes navigation
there about twice faster, as fast as in the other parts of the file.

The problem here is that bidi iteration through buffer text is
non-linear (it follows the visual order, not the order of buffer
positions), so the iterator frequently finds itself out of sync with
the next known "stop position", and needs to resync, to be able to
find which faces, overlays, and invisible and display properties are
in effect.  That's what handle_stop_backwards does, and that involves
going back and rescanning text.  With Arabic, this is exacerbated by
the fact that every Arabic character is "composable" (in the sense of
the CHAR_COMPOSED_P macro), and triggers a call to
composition_compute_stop_pos to find the next (or previous) one.

And on top of that, C-n calls pos-visible-in-window-p two or 3 times,
posn-at-point 2 times, and then vertical-motion.  Each one of these
needs to scan text around point starting from the beginning of the
previous visible line.  The narrowing limits that to some reasonable
distance, but it is still several thousand characters back.

So if composition_compute_stop_pos is the bottleneck, perhaps some
simple caching could help?  But note that when this function is called
twice with the same character position, it is called to search in
different directions -- once forward and another time back.  So even
such low-hanging fruit is not simple to reap, as these two calls will
return two different results.

For now, I don't see how to speed this up, without producing woefully
incorrect display.  I will keep thinking, but I'm not too worried
about this case, since the current performance is tolerable enough,
even if somewhat sluggish.

> You didn't tell me whether it's okay to merge the branch with the latest 
> changes?

I think you can merge.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> So if composition_compute_stop_pos is the bottleneck, perhaps some 
> simple caching could help?
>

This is exactly what I tried...

>
> But note that when this function is called twice with the same character 
> position, it is called to search in different directions -- once forward 
> and another time back.
>

... but I missed that important piece of the puzzle, thanks!

>
> For now, I don't see how to speed this up, without producing woefully 
> incorrect display.  I will keep thinking, but I'm not too worried about 
> this case, since the current performance is tolerable enough, even if 
> somewhat sluggish.
>

Alas, it becomes much more sluggish if with a larger frame (160 columns 
instead of 80), and/or with only Arabic characters.  Emacs takes ~10 
seconds to open a file with only 6000 characters on a single line, and 
motion commands are slow.

>> You didn't tell me whether it's okay to merge the branch with the 
>> latest changes?
>
> I think you can merge.
>

Done.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> To make syntax-ppss faster, if possible?
> Is that not a sensible thing to do?

It's surely possible and useful, regardless if what I said is true
or not :-)


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Just a curious question: How did we arrive at the current value of
>> `long-line-threshold'?
> I took the same value that Stefan had chosen for syntax-wholeline-max.

For the record, I chose 10K based on ... a wild guess that a 10K is
"definitely" in the camp of "very long lines" (it's not just a normal
line that happens to be a bit long) yet at the same time it is hopefully
short enough that it shouldn't take too much time to process.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> The wholelines problem did not kick in because of PPS nor `syntax-ppss`
>> but because of font-lock (which then called `syntax-ppss` which then
>> called PPS).
> If it's font-lock that forces syntax-ppss to examine the whole huge
> line, then what is your proposal for avoiding that which doesn't
> involve some more-or-less arbitrary restrictions on the part of the
> buffer that can be examined by syntax-ppss?

The use of `syntax-wholeline-max` in
`font-lock-extend-region-wholelines` supposedly fixed this problem since
it changed `font-lock` so it doesn't ask `syntax-ppss` to compute the
whole line/buffer.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 04.08.2022 16:09, Eli Zaretskii wrote:

>> That's the scenario I described, and that's my point: this file's
>> display is sluggish. Even though font-lock has already finished its
>> work. And it didn't have to spend any significant time in syntax-ppss.
>>
>> So there is a particular performance problem with the display of
>> fontified buffers which I'd really like your help in fixing.
> 
> Maybe it is in display, and maybe it isn't.  Do you have any evidence
> that the sluggish response is due to redisplay?  C-n, for example, is
> mostly not redisplay, but Lisp code in simple.el and occasional calls
> to vertical-motion.

I come to that conclusion by observing said sluggish movement in a 
buffer that is fully fontified. And yet the delays after pressing 'C-n' 
or 'C-p' or 'C-n' seemed noticeable and very similar to delays on 
PgUp/PgDown.

Are these delays in fontification-functions? That seems unlikely given 
the buffer is fontified already in full, and none of the commands are 
causing modifications (which would force syntax-ppss cache and 
fontifications to be recalculated).

If the delays are in font-lock anyway, that might be in the code which 
checks that the area between window-start and window-end is fontified. I 
don't see why that code has to be slow (long lines or not), so it should 
be fixable easily enough. Unless 'get-text-property' or 
'next-single-property-change' can exhibit pathologic performance in the 
presence of long lines, of course. But that doesn't show up in my testing.

> But even if the slow response is due to redisplay, we just have
> another cause that we need to investigate and try fixing.

It seems to me that that cause actually has larger impact than 
font-lock, because it does show itself in a moderately-sized (88K) 
buffer, where font-lock doesn't feel like a problem. It stands to reason 
that the same "cause" might have a proportionally bigger impact in large 
buffers as well, and only after we remove it (alone), then we can 
evaluate how font-lock itself affects user experience, and how much of 
its correctness (and for buffers of which size) we want to sacrifice.

> It says
> nothing about the measures we've already taken on master.  They
> definitely make even this case faster, and with an unoptimized build I
> can now reasonably edit this file, something I couldn't do before.

If my guess is right, the fix on master whammied all over the redisplay 
with narrowing, both fixing the "cause" and restricting font-lock to the 
same narrowed region. The latter part might be unnecessary in the usual 
case (we might still decide to do that later for much larger buffers, 
but that should be decided by a separate threshold variable).

>> Fixing in a way that doesn't add narrowing around
>> fontification-functions, because as we can see it's not necessary in
>> examples like this.
> 
> If that is possible, sure.  No one said that from now on every problem
> in Emacs that causes slow responses will be handled by narrowing.  But
> if, for example, it turns out that the slow responses is due to time
> it takes some code to traverse a long stretch of fontified buffer,
> what other solution would you suggest except making the portion to be
> traversed shorter?

For all I know, the most optimal fix might still be implemented through 
narrowing, but it would be temporarily widened while 
fontificiation-functions are run.

>>> If you dislike mis-fontification, turn font-lock mode off.  It's as easy
>>> as that.  Mis-fontification is expected in such cases.  The docstring of
>>> syntax-wholeline-max also mentions that "misfontification may then
>>> occur". Why did you not protest at that time?
>>
>> I think we could have both speed and correctness, at least for files of
>> this size.
> 
> That is not a given, and the experience till now suggests otherwise.

I have commented out the code which applies the narrowing in 
'handle_fontified_prop' and recompiled.

The result:

- My 88K file is fontified correctly now. The redisplay and scrolling 
performance seem unaffected (meaning still fast).
- dictionary.json (18M) seems to be fontified correctly as well now 
(it's a mess by default on master), its scrolling performance is 
unaffected too. The difference: I have to wait ~2 seconds the first time 
I press 'M->'.

BTW, 'M-> M-<' triggers some puzzling long wait (~3 seconds) both on 
master and with my change, every time I issue this sequence of commands.

diff --git a/src/xdisp.c b/src/xdisp.c
index 099efed2db..02d7f6c562 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4391,19 +4391,19 @@ handle_fontified_prop (struct it *it)

        eassert (it->end_charpos == ZV);

-      if (current_buffer->long_line_optimizations_p)
-	{
-	  ptrdiff_t begv = it->narrowed_begv;
-	  ptrdiff_t zv = it->narrowed_zv;
-	  ptrdiff_t charpos = IT_CHARPOS (*it);
-	  if (charpos < begv || charpos > zv)
-	    {
-	      begv = get_narrowed_begv (it->w, charpos);
-	      zv = get_narrowed_zv (it->w, charpos);
-	    }
-	  narrow_to_region_internal (make_fixnum (begv), make_fixnum (zv), true);
-	  specbind (Qrestrictions_locked, Qt);
-	}
+      /* if (current_buffer->long_line_optimizations_p) */
+      /* 	{ */
+      /* 	  ptrdiff_t begv = it->narrowed_begv; */
+      /* 	  ptrdiff_t zv = it->narrowed_zv; */
+      /* 	  ptrdiff_t charpos = IT_CHARPOS (*it); */
+      /* 	  if (charpos < begv || charpos > zv) */
+      /* 	    { */
+      /* 	      begv = get_narrowed_begv (it->w, charpos); */
+      /* 	      zv = get_narrowed_zv (it->w, charpos); */
+      /* 	    } */
+      /* 	  narrow_to_region_internal (make_fixnum (begv), make_fixnum 
(zv), true); */
+      /* 	  specbind (Qrestrictions_locked, Qt); */
+      /* 	} */

        /* Don't allow Lisp that runs from 'fontification-functions'
  	 clear our face and image caches behind our back.  */

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 02.08.2022 17:57, Gregory Heytings wrote:
> 
>>
>> Regarding the long-standing bug reports, we did solve a bunch of 
>> issues already. One major one, IIUC, was redisplay of already 
>> fontified text on long lines.
>>
> 
> Try to open the dictionary.json with Emacs on master a month ago.  It's 
> a small file (only 18 MB).  On my computer just opening the file with 
> emacs -Q takes 220 seconds.  220 seconds during which Emacs is 
> completely locked, because of font-lock mode.  If you're not convinced, 
> turn font-lock mode off, open the file, and turn font-lock mode on.

I downloaded that file, and I commented out the code in 
'handle_fontified_prop' which performs the narrowing on master.

And recompiled, leaving all the other settings the same.

Visiting dictionary.json takes about 1 second.

M-> takes ~2 seconds, but only the first time (and until the next 
modification near the beginning of the buffer, I guess).

Scrolling is as fast as without my change. All fontification seems 
correct (which is not the case on master).

>> Another piece of the puzzle was added by Stefan in 15b2138719b340.
>>
> 
> That looked promising, but sadly it had only a very limited effect.
> 
>>
>> So perhaps we should re-evaluate the testing scenario to see where the 
>> current bottlenecks are. If we current main issue is the 55s spent in 
>> syntax-ppss, a more constructive approach would be to look into 
>> optimizing parse-partial-sexp. Or even give up on certain scenarios, 
>> admitting that waiting 55s once to visit the end of a 1 GB buffer is 
>> not so bad (and that could part could also be sped up by setting 
>> syntax-propertize-function to nil and using a very simple syntax 
>> table, for instance).
>>
> 
> It is bad, especially now that it became clear that in fact it's not 
> "waiting 55s once" but "waiting 55s each time the buffer is modified and 
> you move to another position in the buffer".

That was about a 1 GB buffer, right?

Let's take care of buffers with more reasonable sizes first, and then we 
can consider extremes. A separate threshold for syntax-ppss to avoid 
parsing the whole buffer might fit the bill.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 19:26:03 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> Alas, it becomes much more sluggish if with a larger frame (160 columns 
> instead of 80), and/or with only Arabic characters.  Emacs takes ~10 
> seconds to open a file with only 6000 characters on a single line, and 
> motion commands are slow.

It doesn't surprise me.  If you disable font-lock and show-paren-mode,
does it become significantly faster?  And how does disabling font-lock
that measure vs disabling auto-composition-mode with that file?

Can you post that file?  I'd like to try some ideas I might have with
it.

> >> You didn't tell me whether it's okay to merge the branch with the 
> >> latest changes?
> >
> > I think you can merge.
> 
> Done.

Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gregory@heytings.org,  dgutov@yandex.ru,  56682@debbugs.gnu.org
> Date: Thu, 04 Aug 2022 18:35:36 -0400
> 
> >> The wholelines problem did not kick in because of PPS nor `syntax-ppss`
> >> but because of font-lock (which then called `syntax-ppss` which then
> >> called PPS).
> > If it's font-lock that forces syntax-ppss to examine the whole huge
> > line, then what is your proposal for avoiding that which doesn't
> > involve some more-or-less arbitrary restrictions on the part of the
> > buffer that can be examined by syntax-ppss?
> 
> The use of `syntax-wholeline-max` in
> `font-lock-extend-region-wholelines` supposedly fixed this problem since
> it changed `font-lock` so it doesn't ask `syntax-ppss` to compute the
> whole line/buffer.

It did?

And if it did, how is that better or different from a locked
narrowing?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 04:39:46 +0300
> Cc: gregory@heytings.org, gerd.moellmann@gmail.com, 56682@debbugs.gnu.org,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> So there is a particular performance problem with the display of
> >> fontified buffers which I'd really like your help in fixing.
> > 
> > Maybe it is in display, and maybe it isn't.  Do you have any evidence
> > that the sluggish response is due to redisplay?  C-n, for example, is
> > mostly not redisplay, but Lisp code in simple.el and occasional calls
> > to vertical-motion.
> 
> I come to that conclusion by observing said sluggish movement in a 
> buffer that is fully fontified. And yet the delays after pressing 'C-n' 
> or 'C-p' or 'C-n' seemed noticeable and very similar to delays on 
> PgUp/PgDown.

There's much more to C-n/C-p/C-v/M-v than just fontifications and
redisplay.  If fontification-functions are out of the picture, it
doesn't yet follow that the only factor that is left is redisplay.

In general, redisplay _is_ slower when there are non-default faces in
the buffer, but (a) that is inevitable due to additional processing,
and (b) I hope you are not arguing against having font-lock faces, are
you?  The question, therefore, is whether the additional processing
due to font-lock faces, by itself, is or isn't a significant factor in
the slowdown you observe?  I don't see how that question could be
answered except by profiling.

> Are these delays in fontification-functions? That seems unlikely given 
> the buffer is fontified already in full

When the buffer is fully fontified, Emacs will not call
fontification-functions at all, unless the buffer becomes modified.

> If the delays are in font-lock anyway, that might be in the code which 
> checks that the area between window-start and window-end is fontified.

Unlikely.  That test uses the interval tree, and is reasonably fast.
But again, only profiling will tell the truth.

> I 
> don't see why that code has to be slow (long lines or not), so it should 
> be fixable easily enough. Unless 'get-text-property' or 
> 'next-single-property-change' can exhibit pathologic performance in the 
> presence of long lines, of course. But that doesn't show up in my testing.

get-text-property and its ilk don't treat newlines as special
characters, and don't even access buffer text at all.  So no, that is
most probably not a significant factor here.

> > But even if the slow response is due to redisplay, we just have
> > another cause that we need to investigate and try fixing.
> 
> It seems to me that that cause actually has larger impact than 
> font-lock, because it does show itself in a moderately-sized (88K) 
> buffer, where font-lock doesn't feel like a problem.

"Doesn't feel" like a problem?  On what is that feeling based?

We need profiling and other hard data.  It is impossible to argue
based on feelings.

> It stands to reason 
> that the same "cause" might have a proportionally bigger impact in large 
> buffers as well, and only after we remove it (alone), then we can 
> evaluate how font-lock itself affects user experience, and how much of 
> its correctness (and for buffers of which size) we want to sacrifice.

The buffer size shouldn't matter if lines are not too long, because
redisplay always examines a small portion of the buffer that fits in
the window, and sometimes also a couple of lines above and below.  If
lines _are_ long, then yes, their length (not the buffer size) is a
significant factor, because the redisplay algorithms many times need
to start from a previous line's beginning, to anchor their layout
calculations (because only there the X coordinate is known in
advance).

That is the root cause which these changes we are discussing are
trying to eliminate or at least alleviate.

There are no other causes we know about.  If you claim that there are
such causes, you need to show that, not just by reasoning, but by
measurements, after you eliminate the already-known effects, like the
length of a physical line, character composition (when the script of
the characters requires it), etc.

And please keep in mind that without long lines, any slowdown factors
that we could have are unlikely to cause unreasonably slow responses,
because otherwise we'd have complaints about that long ago.  (In fact
we did have such complaints, and the problems they reported were fixed
in past releases.)  The only known situation with large buffers that
is yet unsolved in core is when the buffer is larger than the
available memory, so it causes paging when buffer text is accessed for
display or navigation.

The mental model you are building and on whose basis you are trying to
reason about the ways to solve this problem should take all of the
above into account.  Only after that you may have a chance of
identifying some hidden factor that eluded us, whose elimination could
then allow to solve these problems without the narrowing we are now
using.

> > It says
> > nothing about the measures we've already taken on master.  They
> > definitely make even this case faster, and with an unoptimized build I
> > can now reasonably edit this file, something I couldn't do before.
> 
> If my guess is right, the fix on master whammied all over the redisplay 
> with narrowing, both fixing the "cause" and restricting font-lock to the 
> same narrowed region. The latter part might be unnecessary in the usual 
> case (we might still decide to do that later for much larger buffers, 
> but that should be decided by a separate threshold variable).

We need hard data, not guesses.  It is impractical to argue about
guesses, especially when they are based on incomplete or inaccurate
understanding of how the relevant code really behaves.  If you produce
measurements or other facts that contradict our understanding, then we
will be forced to reconsider and adapt to those facts.  For now, you
didn't yet say or show anything that amounts to such a contradiction.

> >> Fixing in a way that doesn't add narrowing around
> >> fontification-functions, because as we can see it's not necessary in
> >> examples like this.
> > 
> > If that is possible, sure.  No one said that from now on every problem
> > in Emacs that causes slow responses will be handled by narrowing.  But
> > if, for example, it turns out that the slow responses is due to time
> > it takes some code to traverse a long stretch of fontified buffer,
> > what other solution would you suggest except making the portion to be
> > traversed shorter?
> 
> For all I know, the most optimal fix might still be implemented through 
> narrowing, but it would be temporarily widened while 
> fontificiation-functions are run.

The first step of these changes didn't narrow when
fontification-functions were run.  It was still insufficient, because
font-lock still made Emacs extremely slow with long lines, whereas
disabling font-lock removed that slowness.  The next step then applied
narrowing to fontification-functions as well, and that solved the slow
cases.  This is how this activity proceeded, and this is why we are
reasonably sure at this point that fontification-functions _are_
indeed a significant slowdown factor when very long lines are
involved.

> >> I think we could have both speed and correctness, at least for files of
> >> this size.
> > 
> > That is not a given, and the experience till now suggests otherwise.
> 
> I have commented out the code which applies the narrowing in 
> 'handle_fontified_prop' and recompiled.
> 
> The result:
> 
> - My 88K file is fontified correctly now. The redisplay and scrolling 
> performance seem unaffected (meaning still fast).
> - dictionary.json (18M) seems to be fontified correctly as well now 
> (it's a mess by default on master), its scrolling performance is 
> unaffected too. The difference: I have to wait ~2 seconds the first time 
> I press 'M->'.

We consider 2 seconds of wait in this case to be "too slow".

But if all you are saying is that the value of long-line-threshold
should be changed, or that perhaps the portion of the buffer around
the window used for narrowing should be enlarged and/or exposed to
control of Lisp programs, we can discuss that.  My impression, though,
was that your arguments are much more basic: that they argue against
the very methods of solving this problem that we currently have on
master.  And that is an entirely different discussion than the one
about the default values of these thresholds.

> BTW, 'M-> M-<' triggers some puzzling long wait (~3 seconds) both on 
> master and with my change, every time I issue this sequence of commands.

It would be useful to look into the reasons of that, thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 05:03:39 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> That was about a 1 GB buffer, right?
> 
> Let's take care of buffers with more reasonable sizes first, and then we 
> can consider extremes.

We want to solve all of them.

> A separate threshold for syntax-ppss to avoid parsing the whole
> buffer might fit the bill.

Don't we already have such a threshold?

But again, if you are just saying that the current balance between
response time and font-lock correctness is sub-optimal, and should be
made better by changing the values of the thresholds, that's a
different discussion.  For that discussion, we'd need a representative
enough sample of real-life files with long lines and in as many major
modes as possible, to make our balance really close to optimal.  I,
for one, will welcome more examples of such files, especially if they
use major modes we didn't consider until now.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> If my guess is right, the fix on master whammied all over the redisplay 
> with narrowing, both fixing the "cause" and restricting font-lock to the 
> same narrowed region.
>

No, as Eli told you the restriction around font-lock was added after the 
other fixes, when it became clear that fontification-functions were the 
main remaining cause of the remaining slowdowns.

>
> - dictionary.json (18M) seems to be fontified correctly as well now 
> (it's a mess by default on master), its scrolling performance is 
> unaffected too. The difference: I have to wait ~2 seconds the first time 
> I press 'M->'.
>

You have to wait 2 seconds.  I have to wait 4, and Eli has to wait 8. 
And for someone else, with older hardware, it might be 20.  And you'll see 
similar slowdowns when you modify the buffer at one place and move 
somewhere else.  This is hardly acceptable for a still relatively small 
file, if the only reason of that additional wait it to put colors on the 
buffer characters.

>
> BTW, 'M-> M-<' triggers some puzzling long wait (~3 seconds) both on 
> master and with my change, every time I issue this sequence of commands.
>

This is caused by show-paren-mode.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> "waiting 55s once" but "waiting 55s each time the buffer is modified 
>> and you move to another position in the buffer".
>
> That was about a 1 GB buffer, right?
>
> Let's take care of buffers with more reasonable sizes first, and then we 
> can consider extremes.
>

No, because the point of considering extreme cases is that they reveal on 
your computer what happens on other people's less powerful computers with 
much smaller files.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> The use of `syntax-wholeline-max` in
>> `font-lock-extend-region-wholelines` supposedly fixed this problem since
>> it changed `font-lock` so it doesn't ask `syntax-ppss` to compute the
>> whole line/buffer.
>
> It did?
> And if it did, how is that better or different from a locked
> narrowing?

In terms of end-user behavior, it's very similar: it can break the
`font-lock-keywords` part of font-lock but it still lets `syntax-ppss`
look at the whole buffer and will thus still provide correct recognition
of strings and comments, except when the major mode relies on
`syntax-propertize-function` since that one also obeys
`syntax-wholeline-max` and can thus misbehave in a similar way to the
narrowing.

The more important difference is that it can be
tweaked/changed/broken/improved by any ELisp package without
necessitating a recompilation of Emacs's C code, or ugly workarounds to
escape the narrowing, like postponing the actual font-lock to a timer or
some such.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> It doesn't surprise me.  If you disable font-lock and show-paren-mode, 
> does it become significantly faster?  And how does disabling font-lock 
> that measure vs disabling auto-composition-mode with that file?
>

Disabling show-paren-mode and font-lock has no effect, no.  What is 
surprising is that the speed seems to depend both on the mode and on the 
presence of bidirectionality (so it seems that after all bidi is involved 
in one way or another).

>
> Can you post that file?  I'd like to try some ideas I might have with 
> it.
>

There are now eight files:

1. https://www.heytings.org/data/arabic-large.json
2. https://www.heytings.org/data/arabic-large.json.txt
3. https://www.heytings.org/data/arabic-large.txt
4. https://www.heytings.org/data/arabic-large.txt.json
5. https://www.heytings.org/data/arabic-small.json
6. https://www.heytings.org/data/arabic-small.json.txt
7. https://www.heytings.org/data/arabic-small.txt
8. https://www.heytings.org/data/arabic-small.txt.json

1 and 2, 3 and 4, 5 and 6, 7 and 8 are the same file, the only difference 
is the added extension.  1, 2, 3 and 4 on the one hand, and 5, 6, 7 and 8 
on the other hand, are almost the same file, the only difference is that 
in 1 and 2 and 5 and 6 the arabic text is enclosed into '{"ar":"<arabic 
text>"}'.

Now what you'll see is that 5 is slow, 6 is also slow (so it's not only 
js-mode which is doing something wrong), 7 is fast, and 8 is again slow 
(so js-mode is perhaps doing something wrong).  Also, the motion commands 
C-n and C-p do not work as expected in 5, 6 and 8.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> If it's font-lock that forces syntax-ppss to examine the whole huge 
>> line, then what is your proposal for avoiding that which doesn't 
>> involve some more-or-less arbitrary restrictions on the part of the 
>> buffer that can be examined by syntax-ppss?
>
> The use of `syntax-wholeline-max` in 
> `font-lock-extend-region-wholelines` supposedly fixed this problem since 
> it changed `font-lock` so it doesn't ask `syntax-ppss` to compute the 
> whole line/buffer.
>

Actually, it didn't, it made things better in some cases, but (much) worse 
in other cases.  You may have seen the recipes I sent to Dmitry a few days 
ago:

emacs -Q
M-: (setq long-line-threshold nil syntax-wholeline-max most-positive-fixnum) RET
C-x C-f dictionary.json RET y ;; takes 160 seconds
C-e ;; takes 200 seconds

emacs -Q
M-: (setq long-line-threshold nil) RET
C-x C-f dictionary.json RET y ;; immediate
C-e ;; not finished after 1200 seconds (20 minutes), I killed Emacs

emacs -Q
C-x C-f dictionary.json RET y ;; immediate
C-e ;; immediate

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 05 Aug 2022 08:21:30 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, gerd.moellmann@gmail.com, 
>     56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> 
> > BTW, 'M-> M-<' triggers some puzzling long wait (~3 seconds) both on 
> > master and with my change, every time I issue this sequence of commands.
> >
> 
> This is caused by show-paren-mode.

Interesting.  I guess the problematic part is the code in
show-paren-mode which searches for the matching parenthesis, not the
part that puts the overlay on the buffer?  IOW, does making
blink-matching-paren-distance smaller shortens the delay?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gregory@heytings.org,  dgutov@yandex.ru,  56682@debbugs.gnu.org
> Date: Fri, 05 Aug 2022 05:03:25 -0400
> 
> >> The use of `syntax-wholeline-max` in
> >> `font-lock-extend-region-wholelines` supposedly fixed this problem since
> >> it changed `font-lock` so it doesn't ask `syntax-ppss` to compute the
> >> whole line/buffer.
> >
> > It did?
> > And if it did, how is that better or different from a locked
> > narrowing?
> 
> In terms of end-user behavior, it's very similar: it can break the
> `font-lock-keywords` part of font-lock but it still lets `syntax-ppss`
> look at the whole buffer and will thus still provide correct recognition
> of strings and comments, except when the major mode relies on
> `syntax-propertize-function` since that one also obeys
> `syntax-wholeline-max` and can thus misbehave in a similar way to the
> narrowing.
> 
> The more important difference is that it can be
> tweaked/changed/broken/improved by any ELisp package without
> necessitating a recompilation of Emacs's C code, or ugly workarounds to
> escape the narrowing, like postponing the actual font-lock to a timer or
> some such.

AFAIK, the problem is not entirely solved by syntax-wholeline-max.  If
and when it is solved, we could revisit this issue.  However, since
syntactic fontifications are invoked by a major-mode's font-lock
setup, there's still a problem of how to prevent the rest of font-lock
from causing significant slowdown.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 10:43, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 05:03:39 +0300
>> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>>   monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> That was about a 1 GB buffer, right?
>>
>> Let's take care of buffers with more reasonable sizes first, and then we
>> can consider extremes.
> 
> We want to solve all of them.

I didn't say we don't. But different issues call for different solutions.

>> A separate threshold for syntax-ppss to avoid parsing the whole
>> buffer might fit the bill.
> 
> Don't we already have such a threshold?

Not exactly: the buffer is still fully parsed by parse-partial-sexp 
(once). AFAICT, the variable makes the application of 
syntax-propertize-rules more lax, but at least it keeps counting the 
simple parens/quotes from the beginning of the buffer. That's why the 
fontification remained correct in both examples that I posted.

The time it takes to (parse-partial-sexp 1 (point-max)) accounts for the 
whole fontification delay at the end of dictionary.json.

Now, if nobody manages to speed up parse-partial-sexp itself further, we 
can add an additional tweak/size threshold, after which syntax-ppss 
won't parse the whole buffer anymore. But if we do that in Lisp, we can 
later improve that bit of logic so that the result is not entirely 
arbitrary, like it is now on master with dictionary.json.

> But again, if you are just saying that the current balance between
> response time and font-lock correctness is sub-optimal, and should be
> made better by changing the values of the thresholds, that's a
> different discussion.  For that discussion, we'd need a representative
> enough sample of real-life files with long lines and in as many major
> modes as possible, to make our balance really close to optimal.  I,
> for one, will welcome more examples of such files, especially if they
> use major modes we didn't consider until now.

We really have different problems and thus need different solutions for 
them. Not just one blunt instrument.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 05 Aug 2022 09:37:35 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> > It doesn't surprise me.  If you disable font-lock and show-paren-mode, 
> > does it become significantly faster?  And how does disabling font-lock 
> > that measure vs disabling auto-composition-mode with that file?
> 
> Disabling show-paren-mode and font-lock has no effect, no.  What is 
> surprising is that the speed seems to depend both on the mode and on the 
> presence of bidirectionality (so it seems that after all bidi is involved 
> in one way or another).

That the presence of R2L characters makes this slower shouldn't
surprise.  The bidi iteration through buffer text _can_ be non-linear,
but it's only _actually_ non-linear when R2L characters are present.
So the complications with handle_stop_backwards and
compute_stop_backwards actually happen only when there are R2L
characters in the buffer; otherwise the iterator behaves exactly like
the unidirectional display in Emacs 23 and before did: it examines
characters one by one it strict increasing order of buffer positions,
and skips or shortcuts some processing that is only needed for truly
bidirectional text.

IOW, the display engine is specially optimized for the very frequent
case of buffers that don't include R2L characters, and is expected to
be slower otherwise.

If font-lock doesn't produce any tangible difference, it probably
means that the amount of stop-positions added due to faces is
negligible when compared to the stop-positions due to character
composition.  Which I guess is also reasonable with a script like
Arabic.

> 1. https://www.heytings.org/data/arabic-large.json
> 2. https://www.heytings.org/data/arabic-large.json.txt
> 3. https://www.heytings.org/data/arabic-large.txt
> 4. https://www.heytings.org/data/arabic-large.txt.json
> 5. https://www.heytings.org/data/arabic-small.json
> 6. https://www.heytings.org/data/arabic-small.json.txt
> 7. https://www.heytings.org/data/arabic-small.txt
> 8. https://www.heytings.org/data/arabic-small.txt.json
> 
> 1 and 2, 3 and 4, 5 and 6, 7 and 8 are the same file, the only difference 
> is the added extension.  1, 2, 3 and 4 on the one hand, and 5, 6, 7 and 8 
> on the other hand, are almost the same file, the only difference is that 
> in 1 and 2 and 5 and 6 the arabic text is enclosed into '{"ar":"<arabic 
> text>"}'.
> 
> Now what you'll see is that 5 is slow, 6 is also slow (so it's not only 
> js-mode which is doing something wrong), 7 is fast, and 8 is again slow 
> (so js-mode is perhaps doing something wrong).  Also, the motion commands 
> C-n and C-p do not work as expected in 5, 6 and 8.

Thanks, I will use these.

There's (at least) one more aspect of this, as long as Text mode is
being used: Text mode doesn't force bidi-paragraph-direction to be
left-to-right, whereas all descendants of prog-mode, including
js-mode, do.  Leaving bidi-paragraph-direction at nil means Emacs
needs to determine the base paragraph direction each time it's about
to redisplay a window, and that might be expensive, especially in a
large buffer without any paragraph breaks (by default, an empty line),
because that is determined by the first strong directional character
of the paragraph.  So for a more fair comparison with Text mode, you
should set bidi-paragraph-direction to the value left-to-right in
text-mode buffers.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 14:34:12 +0300
> Cc: gregory@heytings.org, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> We really have different problems and thus need different solutions for 
> them. Not just one blunt instrument.

The current opinion of both the head maintainers and of Gregory is
that these are all parts of the same problem, and a single class of
solutions can solve most of them.  The problem being that many
portions of Emacs code involved in navigation and redisplay don't
expect lines to be too long, and therefore employ algorithms that
don't scale well with line length.  Preventing such code from going
far back to the beginning of the previous line, and then coming back
through all that text, is therefore an idea that should appear very
reasonable.  It also works surprisingly well in practice, at least
according to what we know at this point.

I get it that you disagree, but I haven't seen any real data behind
your dissenting opinions, and thus I don't yet see any reason to
reconsider changing the direction of development in this regard.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> There's (at least) one more aspect of this, as long as Text mode is 
> being used: Text mode doesn't force bidi-paragraph-direction to be 
> left-to-right, whereas all descendants of prog-mode, including js-mode, 
> do.  Leaving bidi-paragraph-direction at nil means Emacs needs to 
> determine the base paragraph direction each time it's about to redisplay 
> a window, and that might be expensive, especially in a large buffer 
> without any paragraph breaks (by default, an empty line), because that 
> is determined by the first strong directional character of the 
> paragraph.  So for a more fair comparison with Text mode, you should set 
> bidi-paragraph-direction to the value left-to-right in text-mode 
> buffers.
>

Indeed, that seems to be the culprit here, I didn't know that text-mode 
was an exception here.  If I set bidi-paragraph-direction to 
'left-to-right after visiting the arabic-small.txt file, Emacs (mis) 
behaves like it does for the other Arabic files: it becomes slow, and C-n 
C-p do not work correctly anymore.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> AFAIK, the problem is not entirely solved by syntax-wholeline-max.

No, indeed.

> If and when it is solved, we could revisit this issue.

The locked narrowing currently in effect makes it impossible to
investigate this problem or improve the behavior :-(


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 14:48, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 14:34:12 +0300
>> Cc: gregory@heytings.org, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> We really have different problems and thus need different solutions for
>> them. Not just one blunt instrument.
> 
> The current opinion of both the head maintainers and of Gregory is
> that these are all parts of the same problem, and a single class of
> solutions can solve most of them.

This kind of approach fails to optimize for the behavior in medium-sized 
files, like the downloadify.js I showed previously.

Simply customizing long-line-threshold to a much higher value will bring 
back redisplay stutters on C-n/C-p/etc, which *were* a real problem that 
some of Gregory's changes solved.

> The problem being that many
> portions of Emacs code involved in navigation and redisplay don't
> expect lines to be too long, and therefore employ algorithms that
> don't scale well with line length.

As I demonstrated, font-lock itself doesn't have that issue.

Furthermore, the performance problem with syntax-ppss which we are 
talking about now doesn't have anything to do long lines.

Go ahead and pretty-print dictionary.json (you can use 'M-x 
json-pretty-print', write the buffer to a new file, then re-visit it). 
There won't be any long lines in the resulting file, but 'M->' will 
still make you wait a few seconds the first time.

> Preventing such code from going
> far back to the beginning of the previous line, and then coming back
> through all that text, is therefore an idea that should appear very
> reasonable.  It also works surprisingly well in practice, at least
> according to what we know at this point.
> 
> I get it that you disagree, but I haven't seen any real data behind
> your dissenting opinions, and thus I don't yet see any reason to
> reconsider changing the direction of development in this regard.

I don't understand why you dismiss the more subtle approach which still 
seems to reach the stated goals.

Gregory's changes, along with my suggested tweak, indeed bring work 
"surprisingly well" already. All without breaking font-lock in the 
common case.

Like, we're going from a 255 (?) second delay to 2 second delay already 
without breaking fontification. And yet you're eager to go from 2 
seconds down to ~0 and sacrifice highlighting correctness?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> If and when it is solved, we could revisit this issue.
>
> The locked narrowing currently in effect makes it impossible to 
> investigate this problem or improve the behavior :-(
>

How so?  It is fully backward compatible, just (setq long-line-threshold 
nil).  If what you want is only to remove the locked narrowing, comment 
out the relevant line in xdisp.c.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 11:23, Gregory Heytings wrote:
> No, because the point of considering extreme cases is that they reveal 
> on your computer what happens on other people's less powerful computers 
> with much smaller files.

The problem with considering extreme cases, however, is that one can 
also provide an even more extreme case no matter how we optimize our 
implementations, which, with the chosen approach, will force us to 
cripple or drop most features outright.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Like, we're going from a 255 (?) second delay to 2 second delay already 
> without breaking fontification. And yet you're eager to go from 2 
> seconds down to ~0 and sacrifice highlighting correctness?
>

Yes.  Because as I told you your 2 seconds are 4 for me and 8 for Eli and 
20 for someone else.  And that's in a relatively small file.

Note that if it were 2/4/8/20 seconds once, and then no further slowdowns 
while editing the file, that would perhaps be okay.  But that's not the 
case, you will regularly see a similar 2/4/8/20 seconds delay.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 15:16, Gregory Heytings wrote:
> 
>>> If and when it is solved, we could revisit this issue.
>>
>> The locked narrowing currently in effect makes it impossible to 
>> investigate this problem or improve the behavior :-(
>>
> 
> How so?  It is fully backward compatible, just (setq long-line-threshold 
> nil).  If what you want is only to remove the locked narrowing, comment 
> out the relevant line in xdisp.c.

That brings back the performance problems in redisplay, which have a 
more pronounced effect, overshadowing both issues and potential 
improvements in font-lock.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 04.08.2022 04:41, Gregory Heytings wrote:
> 
>>
>> We cannot perform every basic operation in fixed time for any 
>> arbitrarily sized file. There are limits of what we can possibly do.
>>
> 
> Apparently the limits are lower than what you think.  Provided that we 
> accept some compromises, such as mis-fontification, which is also what 
> syntax-wholeline-max does, and against which you didn't protest.

syntax-wholelines-max indeed can potentially cause problems too, but in 
a much narrow range of situations (and only with major modes which have 
non-trivial syntax-propertize-function).

In any case, I would be happy to investigate further improvements in 
Lisp which would be implemented in the same area of code that 
syntax-wholelines-max lives in.

>> I'd really like it if we could scope this discussion to solving that 
>> particular problem. Not the speed of operations in large files in 
>> general.
>>
> 
> I don't understand what you mean.  Which "particular problem"?  The 
> point of this discussion is of course the speed of operations in large 
> files in general.

The particular problem is long lines. It not the same as "large files in 
general".

> If you take that out of the picture, everything is of 
> course possible. I'm not even sure what remains in fact, Emacs is an 
> editor, not a displayer.

The well-known problem we have had for a while is that Emacs screeches 
to a halt even on medium-sized buffers as soon as it encounters a long line.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]


>>> The locked narrowing currently in effect makes it impossible to 
>>> investigate this problem or improve the behavior :-(
>> 
>> How so?  It is fully backward compatible, just (setq 
>> long-line-threshold nil).  If what you want is only to remove the 
>> locked narrowing, comment out the relevant line in xdisp.c.
>
> That brings back the performance problems in redisplay, which have a 
> more pronounced effect, overshadowing both issues and potential 
> improvements in font-lock.
>

The first option does, the second does not.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 15:20, Gregory Heytings wrote:
> 
>>
>> Like, we're going from a 255 (?) second delay to 2 second delay 
>> already without breaking fontification. And yet you're eager to go 
>> from 2 seconds down to ~0 and sacrifice highlighting correctness?
>>
> 
> Yes.  Because as I told you your 2 seconds are 4 for me and 8 for Eli 
> and 20 for someone else.  And that's in a relatively small file.

Is it a "relatively small file" if we've effectively been unable to edit 
such files for all of 40 years of Emacs's existence?

> Note that if it were 2/4/8/20 seconds once, and then no further 
> slowdowns while editing the file, that would perhaps be okay.  But 
> that's not the case, you will regularly see a similar 2/4/8/20 seconds 
> delay.

Do our users regularly edit 30MB files? And do a lot of changes in them? 
In different areas?

Note that, again, to see the same delay you would have to edit that file 
near the beginning, and then visit its end again.

If I did that, though, I'm not sure whether I would be more 
inconvenienced by performance, or by broken syntax highlighting and sexp 
navigation. font-lock is not just eye candy: it also assists you when 
editing code.

For instance, if I were to edit dictionary.json, I might have needed to 
look for a certain key and change it somewhere. But if the said key is 
highlighted as a part of a string value in some places, and only as a 
key in some others, that can look and feel very puzzling, and slow down 
my work just the same.

Similarly, if I'm editing a large JSON file, I might want to write a 
small Lisp program which searches for a word, checks that it's inside a 
string (or, conversely, outside and thus looks like a key), makes all 
the necessary changes in an automated fashion, and saves the buffer. A 
broken syntax-ppss wouldn't let me do that.

Finally, yes, for some buffer size the initial wait is going to be too 
much. But that can have a separate solution with a separate threshold.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 01.08.2022 14:58, Eli Zaretskii wrote:

> As I wrote elsewhere, I'm okay with extending 'widen' so that it could
> "unlock" the locked narrowing, which could then be used in major modes
> that convince us their performance is adequate (or clearly announce in
> their docs that they don't care about files with long lines ;-).

And to address the idea of "unlocking" the narrowing: I think I have 
demonstrated that the remaining slowdown can be caused purely by the 
length of the buffer and how long 'parse-partial-sexp' takes to parse 
it. That part doesn't have much to do with individual modes.

And of course the more, let's say, *complex* modes like CC Mode will opt 
for unlocking narrowing right away because its font-lock logic has to 
jump around to previously-saved positions in its syntax cache, which 
will inevitably spam errors here and there when those positions are not 
accessible.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]


>> Yes.  Because as I told you your 2 seconds are 4 for me and 8 for Eli 
>> and 20 for someone else.  And that's in a relatively small file.
>
> Is it a "relatively small file" if we've effectively been unable to edit 
> such files for all of 40 years of Emacs's existence?
>

There are many other editors out there, isn't it?

>> Note that if it were 2/4/8/20 seconds once, and then no further 
>> slowdowns while editing the file, that would perhaps be okay.  But 
>> that's not the case, you will regularly see a similar 2/4/8/20 seconds 
>> delay.
>
> Do our users regularly edit 30MB files? And do a lot of changes in them? 
> In different areas?
>

You seem to think they do not.  Then why is it a problem if such files are 
mis-fontified?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: gregory@heytings.org,  dgutov@yandex.ru,  56682@debbugs.gnu.org
> Date: Fri, 05 Aug 2022 08:06:32 -0400
> 
> > If and when it is solved, we could revisit this issue.
> 
> The locked narrowing currently in effect makes it impossible to
> investigate this problem or improve the behavior :-(

??? Just set long-line-threshold to nil, and you can investigate all
you want.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 16:00, Gregory Heytings wrote:
>>> Note that if it were 2/4/8/20 seconds once, and then no further 
>>> slowdowns while editing the file, that would perhaps be okay.  But 
>>> that's not the case, you will regularly see a similar 2/4/8/20 
>>> seconds delay.
>>
>> Do our users regularly edit 30MB files? And do a lot of changes in 
>> them? In different areas?
>>
> 
> You seem to think they do not.  Then why is it a problem if such files 
> are mis-fontified?

 From experience, I might visit a medium-to-large sized file, and I 
might search for a particular identifier inside it. Much more rarely, I 
would apply some edit inside it, in one place, and then save the buffer.

On balance, there is more likely more reading than editing involved. 
That's why I think font-lock should be assigned more importance.

In all likelihood, the file I would visit would be even smaller than 
30MB, but big enough that the majority of your improvements will be 
noticeable and welcome there. Just not the part that breaks font-lock.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Do our users regularly edit 30MB files?

Thanks to Gregory's changes, it may start becoming more common.

> And do a lot of changes in them?

In my experience, not very much, no.  Usually large files are
machine-generated and rarely edited by hand.  But there are still
relevant use-cases like opening a large JSON/XML/younameit file and
applying a search&replace or a keyboard macro to it.

I think my machines are slow enough (and my Emacs has enough extra
sluggishness thanks to the many assertion checking compiled into it)
that I'd be better served with an approach like that of so-long where
large files use a dumbed down major mode to avoid most source of extra
slow down like font-lock.

What I'm not sure of is how useful is a "font-lock with arbitrary
narrowing", where portions will be highlighted as strings rather than
code (and vice-versa).  I don't have enough experience with it yet to
be sure.  Taking a step back, I suspect that the only "real" solution is
something like `jit-lock-defer` coupled with a way to perform the
font-lock (and syntax-ppss/propertize) in the background.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 16:17, Stefan Monnier wrote:
> What I'm not sure of is how useful is a "font-lock with arbitrary
> narrowing", where portions will be highlighted as strings rather than
> code (and vice-versa).  I don't have enough experience with it yet to
> be sure.  Taking a step back, I suspect that the only "real" solution is
> something like `jit-lock-defer` coupled with a way to perform the
> font-lock (and syntax-ppss/propertize) in the background.

IIRC I heard that "some other editors" take the approach of restricting 
syntax-highlighting to just the beginning of a large file.

10000 seems too low, but if the 2 seconds in the dictionary.json example 
feels too much to people (and the file is 18 MB), maybe restrict syntax 
highlighting to the first 1 MB of each file? At least until someone 
optimizes parse-partial-sexp to work much faster.

Or 10MB. Not too important as long as the value is separately customizable.

Anyway, I think I'd prefer no highlighting at the end of those large 
files, rather than arbitrarily incorrect one.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> What I'm not sure of is how useful is a "font-lock with arbitrary 
>> narrowing", where portions will be highlighted as strings rather than 
>> code (and vice-versa).  I don't have enough experience with it yet to 
>> be sure.  Taking a step back, I suspect that the only "real" solution 
>> is something like `jit-lock-defer` coupled with a way to perform the 
>> font-lock (and syntax-ppss/propertize) in the background.
>
> IIRC I heard that "some other editors" take the approach of restricting 
> syntax-highlighting to just the beginning of a large file.
>

Other editors just give up syntax highlighting altogether even with the 18 
MB file (and you cannot edit really large files with them).

But if you're so annoyed by mis-fontification, why don't you just turn 
font-lock mode off?

Also, why did you not protest vehemently when Stefan added 
syntax-wholeline-max, which also causes occasional mis-fontification?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 05 Aug 2022 11:50:56 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> > There's (at least) one more aspect of this, as long as Text mode is 
> > being used: Text mode doesn't force bidi-paragraph-direction to be 
> > left-to-right, whereas all descendants of prog-mode, including js-mode, 
> > do.  Leaving bidi-paragraph-direction at nil means Emacs needs to 
> > determine the base paragraph direction each time it's about to redisplay 
> > a window, and that might be expensive, especially in a large buffer 
> > without any paragraph breaks (by default, an empty line), because that 
> > is determined by the first strong directional character of the 
> > paragraph.  So for a more fair comparison with Text mode, you should set 
> > bidi-paragraph-direction to the value left-to-right in text-mode 
> > buffers.
> 
> Indeed, that seems to be the culprit here, I didn't know that text-mode 
> was an exception here.

Actually, it's the other way around: prog-mode and its descendants are
the exception.  The default value of bidi-paragraph-direction is nil,
but in prog-mode's we _know_ that program source code is written
left-to-right, even if some of strings and comments could include R2L
text, so we force bidi-paragraph-direction in these modes.

> If I set bidi-paragraph-direction to 'left-to-right after visiting
> the arabic-small.txt file, Emacs (mis) behaves like it does for the
> other Arabic files: it becomes slow, and C-n C-p do not work
> correctly anymore.

OK, so there's one more subtle issue about bidi, and it definitely
affects some of these files: whether the file actually includes both
L2R and R2L characters.  If all of the characters are L2R or all of
them are R2L (the latter is the case with arabic-small.txt), and
bidi-paragraph-direction is either nil or left-to-right for L2R text
and right-to-left for R2L text, then from the display engine's POV the
text is not truly bidirectional, it actually has a single direction.
Text that has a single direction doesn't require (expensive)
reordering for display, and we have optimizations in place for such
situations, to make redisplay faster in those cases, because that is
the natural situation with human-readable text: text that is
predominantly R2L is relatively rarely mixed with L2R text and is
normally displayed in right-to-left paragraphs.

By adding the "ar" prefix to the files, you made what was a
unidirectional text be bidirectional.  The effect on display is
dramatic: the first Arabic letter will now be displayed _last_,
because the paragraph now has left-to-right direction (the first
strong character in it is 'a', a L2R character), and the entire Arabic
string that follows needs to be reversed on display.

Compare arabic-small.txt with arabic-small.json after setting
bidi-paragraph-direction to be right-to-left in the latter, and you
should see a similar performance in both.  (And watch what happens on
display when you change the value of bidi-paragraph-direction in
arabic-small.json.)

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 16:41, Gregory Heytings wrote:
> 
>>> What I'm not sure of is how useful is a "font-lock with arbitrary 
>>> narrowing", where portions will be highlighted as strings rather than 
>>> code (and vice-versa).  I don't have enough experience with it yet to 
>>> be sure.  Taking a step back, I suspect that the only "real" solution 
>>> is something like `jit-lock-defer` coupled with a way to perform the 
>>> font-lock (and syntax-ppss/propertize) in the background.
>>
>> IIRC I heard that "some other editors" take the approach of 
>> restricting syntax-highlighting to just the beginning of a large file.
>>
> 
> Other editors just give up syntax highlighting altogether even with the 
> 18 MB file (and you cannot edit really large files with them).

In my testing, with my patch, the 18 MB file works reasonably well and 
has good syntax highlighting.

> But if you're so annoyed by mis-fontification, why don't you just turn 
> font-lock mode off?

"If you're annoyed by Emacs's performance with large files, why don't 
you just never open them?"

I like font-lock and the visual cues that come with it. Only 
font-locking the first 1 MB of a large file seems like a good 
compromise: show correct highlighting where we can with reasonable 
performance, and omit it in the rest of the file.

> Also, why did you not protest vehemently when Stefan added 
> syntax-wholeline-max, which also causes occasional mis-fontification?

I have replied to this exact question in an earlier email. We can 
continue this line of inquiry in that subthread.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> But if you're so annoyed by mis-fontification, why don't you just turn 
>> font-lock mode off?
>
> "If you're annoyed by Emacs's performance with large files, why don't 
> you just never open them?"
>

That's just wrong: in one case we're talking about your personal 
feelings/preferences, in the other one about Emacs' capabilities for all 
its users.

>
> I like font-lock and the visual cues that come with it. Only 
> font-locking the first 1 MB of a large file seems like a good 
> compromise: show correct highlighting where we can with reasonable 
> performance, and omit it in the rest of the file.
>

So what you prefer IIUC would be to call fontification-functions with a 
locked narrowing to 1 MB if point is before that threshold, and to not 
call fontification-functions at all after that threshold?  That might be 
another doable approach.

>> Also, why did you not protest vehemently when Stefan added 
>> syntax-wholeline-max, which also causes occasional mis-fontification?
>
> I have replied to this exact question in an earlier email. We can 
> continue this line of inquiry in that subthread.
>

Sorry, I missed that part of your earlier post:

>
> syntax-wholelines-max indeed can potentially cause problems too, but in 
> a much narrow range of situations (and only with major modes which have 
> non-trivial syntax-propertize-function).
>

You forget to metion that syntax-wholelines-max can in fact make things 
much worse, see the recipes I sent you a few days ago.  So it doesn't seem 
like it's the right approach.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 15:08:22 +0300
> Cc: gregory@heytings.org, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > The current opinion of both the head maintainers and of Gregory is
> > that these are all parts of the same problem, and a single class of
> > solutions can solve most of them.
> 
> This kind of approach fails to optimize for the behavior in medium-sized 
> files, like the downloadify.js I showed previously.
> 
> Simply customizing long-line-threshold to a much higher value will bring 
> back redisplay stutters on C-n/C-p/etc, which *were* a real problem that 
> some of Gregory's changes solved.

You want a separate threshold for calling font-lock from redisplay?
We could discuss that.

> > The problem being that many
> > portions of Emacs code involved in navigation and redisplay don't
> > expect lines to be too long, and therefore employ algorithms that
> > don't scale well with line length.
> 
> As I demonstrated, font-lock itself doesn't have that issue.

What do you mean by "font-lock itself"?

> Furthermore, the performance problem with syntax-ppss which we are 
> talking about now doesn't have anything to do long lines.

The facts are that it has problems in large files, but it also has
problems in not-so-large files with long lines, due to whole-line
approach.

> Go ahead and pretty-print dictionary.json (you can use 'M-x 
> json-pretty-print', write the buffer to a new file, then re-visit it). 
> There won't be any long lines in the resulting file, but 'M->' will 
> still make you wait a few seconds the first time.

No one said that the changes being discussed solve all the problems in
Emacs.

> > I get it that you disagree, but I haven't seen any real data behind
> > your dissenting opinions, and thus I don't yet see any reason to
> > reconsider changing the direction of development in this regard.
> 
> I don't understand why you dismiss the more subtle approach which still 
> seems to reach the stated goals.
> 
> Gregory's changes, along with my suggested tweak, indeed bring work 
> "surprisingly well" already. All without breaking font-lock in the 
> common case.

I don't think I understand what "tweak" you are suggesting.  can you
show a patch relative to the current master?

> Like, we're going from a 255 (?) second delay to 2 second delay already 
> without breaking fontification. And yet you're eager to go from 2 
> seconds down to ~0 and sacrifice highlighting correctness?

Yes, because 2 sec (in an optimized build) is a very long time.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 15:19:45 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 05.08.2022 11:23, Gregory Heytings wrote:
> > No, because the point of considering extreme cases is that they reveal 
> > on your computer what happens on other people's less powerful computers 
> > with much smaller files.
> 
> The problem with considering extreme cases, however, is that one can 
> also provide an even more extreme case no matter how we optimize our 
> implementations, which, with the chosen approach, will force us to 
> cripple or drop most features outright.

The chosen approach does the same in a 10KB file and in 100GB file, so
I don't understand what you have in mind here.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 15:59:57 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 01.08.2022 14:58, Eli Zaretskii wrote:
> 
> > As I wrote elsewhere, I'm okay with extending 'widen' so that it could
> > "unlock" the locked narrowing, which could then be used in major modes
> > that convince us their performance is adequate (or clearly announce in
> > their docs that they don't care about files with long lines ;-).
> 
> And to address the idea of "unlocking" the narrowing: I think I have 
> demonstrated that the remaining slowdown can be caused purely by the 
> length of the buffer and how long 'parse-partial-sexp' takes to parse 
> it.

No, you haven't demonstrated that.

> And of course the more, let's say, *complex* modes like CC Mode will opt 
> for unlocking narrowing right away because its font-lock logic has to 
> jump around to previously-saved positions in its syntax cache, which 
> will inevitably spam errors here and there when those positions are not 
> accessible.

CC Mode is extremely unlikely to happen in files with such long lines,
so what it does is largely irrelevant to this discussion.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 17:20, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 15:59:57 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 01.08.2022 14:58, Eli Zaretskii wrote:
>>
>>> As I wrote elsewhere, I'm okay with extending 'widen' so that it could
>>> "unlock" the locked narrowing, which could then be used in major modes
>>> that convince us their performance is adequate (or clearly announce in
>>> their docs that they don't care about files with long lines ;-).
>>
>> And to address the idea of "unlocking" the narrowing: I think I have
>> demonstrated that the remaining slowdown can be caused purely by the
>> length of the buffer and how long 'parse-partial-sexp' takes to parse
>> it.
> 
> No, you haven't demonstrated that.

Apply the patch for xdisp.c that I have sent previously (it will be at 
the end of this email too) and recompile Emacs.

Now try two different scenarios. 1,2a and 1,2b.

1. Visit dictionary.json. It will ask you whether to open such big file 
literally, but after you answer 'y', it will display the beginning of 
the file quickly.
2a) Evaluate (benchmark 1 '(save-excursion (parse-partial-sexp 1 
(point-max)))), note the reported delay.

Kill and re-visit the file.

1. (same as before)
2b) Press M->, note the delay you see.

The delays in scenarios 1,2a and 1,2b should be ~the same. They are so 
in my testing.

Or try this scenario: 1,2a,2b. Step 2b should work instantly here.

The patch:

diff --git a/src/xdisp.c b/src/xdisp.c
index 099efed2db..02d7f6c562 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4391,19 +4391,19 @@ handle_fontified_prop (struct it *it)

        eassert (it->end_charpos == ZV);

-      if (current_buffer->long_line_optimizations_p)
-	{
-	  ptrdiff_t begv = it->narrowed_begv;
-	  ptrdiff_t zv = it->narrowed_zv;
-	  ptrdiff_t charpos = IT_CHARPOS (*it);
-	  if (charpos < begv || charpos > zv)
-	    {
-	      begv = get_narrowed_begv (it->w, charpos);
-	      zv = get_narrowed_zv (it->w, charpos);
-	    }
-	  narrow_to_region_internal (make_fixnum (begv), make_fixnum (zv), true);
-	  specbind (Qrestrictions_locked, Qt);
-	}
+      /* if (current_buffer->long_line_optimizations_p) */
+      /* 	{ */
+      /* 	  ptrdiff_t begv = it->narrowed_begv; */
+      /* 	  ptrdiff_t zv = it->narrowed_zv; */
+      /* 	  ptrdiff_t charpos = IT_CHARPOS (*it); */
+      /* 	  if (charpos < begv || charpos > zv) */
+      /* 	    { */
+      /* 	      begv = get_narrowed_begv (it->w, charpos); */
+      /* 	      zv = get_narrowed_zv (it->w, charpos); */
+      /* 	    } */
+      /* 	  narrow_to_region_internal (make_fixnum (begv), make_fixnum 
(zv), true); */
+      /* 	  specbind (Qrestrictions_locked, Qt); */
+      /* 	} */

        /* Don't allow Lisp that runs from 'fontification-functions'
  	 clear our face and image caches behind our back.  */

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 17:41:51 +0300
> Cc: monnier@iro.umontreal.ca, 56682@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> And to address the idea of "unlocking" the narrowing: I think I have
> >> demonstrated that the remaining slowdown can be caused purely by the
> >> length of the buffer and how long 'parse-partial-sexp' takes to parse
> >> it.
> > 
> > No, you haven't demonstrated that.
> 
> Apply the patch for xdisp.c that I have sent previously (it will be at 
> the end of this email too) and recompile Emacs.
> 
> Now try two different scenarios. 1,2a and 1,2b.
> 
> 1. Visit dictionary.json. It will ask you whether to open such big file 
> literally, but after you answer 'y', it will display the beginning of 
> the file quickly.
> 2a) Evaluate (benchmark 1 '(save-excursion (parse-partial-sexp 1 
> (point-max)))), note the reported delay.
> 
> Kill and re-visit the file.
> 
> 1. (same as before)
> 2b) Press M->, note the delay you see.
> 
> The delays in scenarios 1,2a and 1,2b should be ~the same. They are so 
> in my testing.
> 
> Or try this scenario: 1,2a,2b. Step 2b should work instantly here.

How is (parse-partial-sexp 1 (point-max)) related to the issue at
hand?

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 18:33, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 17:41:51 +0300
>> Cc: monnier@iro.umontreal.ca, 56682@debbugs.gnu.org, gregory@heytings.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
I think I have
>>>> demonstrated that the remaining slowdown can be caused purely by the
>>>> length of the buffer and how long 'parse-partial-sexp' takes to parse
>>>> it.
>>>
>>> No, you haven't demonstrated that.
>>
>> Apply the patch for xdisp.c that I have sent previously (it will be at
>> the end of this email too) and recompile Emacs.
>>
>> Now try two different scenarios. 1,2a and 1,2b.
>>
>> 1. Visit dictionary.json. It will ask you whether to open such big file
>> literally, but after you answer 'y', it will display the beginning of
>> the file quickly.
>> 2a) Evaluate (benchmark 1 '(save-excursion (parse-partial-sexp 1
>> (point-max)))), note the reported delay.
>>
>> Kill and re-visit the file.
>>
>> 1. (same as before)
>> 2b) Press M->, note the delay you see.
>>
>> The delays in scenarios 1,2a and 1,2b should be ~the same. They are so
>> in my testing.
>>
>> Or try this scenario: 1,2a,2b. Step 2b should work instantly here.
> 
> How is (parse-partial-sexp 1 (point-max)) related to the issue at
> hand?

I said:

 >>>> I think I have
 >>>> demonstrated that the remaining slowdown can be caused purely by the
 >>>> length of the buffer and how long 'parse-partial-sexp' takes to parse
 >>>> it.

You said:

 >>> No, you haven't demonstrated that.

...and now you are asking why we are talking about parse-partial-sexp?

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 18:33, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 17:41:51 +0300
>> Cc:monnier@iro.umontreal.ca,56682@debbugs.gnu.org,gregory@heytings.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>>>> And to address the idea of "unlocking" the narrowing: I think I have
>>>> demonstrated that the remaining slowdown can be caused purely by the
>>>> length of the buffer and how long 'parse-partial-sexp' takes to parse
>>>> it.
>>> No, you haven't demonstrated that.
>> Apply the patch for xdisp.c that I have sent previously (it will be at
>> the end of this email too) and recompile Emacs.
>>
>> Now try two different scenarios. 1,2a and 1,2b.
>>
>> 1. Visit dictionary.json. It will ask you whether to open such big file
>> literally, but after you answer 'y', it will display the beginning of
>> the file quickly.
>> 2a) Evaluate (benchmark 1 '(save-excursion (parse-partial-sexp 1
>> (point-max)))), note the reported delay.
>>
>> Kill and re-visit the file.
>>
>> 1. (same as before)
>> 2b) Press M->, note the delay you see.
>>
>> The delays in scenarios 1,2a and 1,2b should be ~the same. They are so
>> in my testing.
>>
>> Or try this scenario: 1,2a,2b. Step 2b should work instantly here.
> How is (parse-partial-sexp 1 (point-max)) related to the issue at
> hand?

Or perhaps I should answer this way:

We move to near EOB.
fontification-functions are called.

jit-lock
calls
(font-lock-fontify-region point-near-buffer-end (point-max))
which calls
font-lock-fontify-syntactically-region
which calls both
   (syntax-propertize (point-max))
   and
   (syntax-ppss point-near-buffer-end) -> and it calls parse-partial-sexp

syntax-propertize will also likely call syntax-ppss itself, probably 
through the major mode's syntax-propertize-function. But if 
syntax-propertize-function is nil, parse-partial-sexp gets called 
anyway, over the whole buffer, which makes it the main workload in 
fontifying near EOB.

Now, if syntax-propertize-function is non-nil, parse-partial-sexp will 
also call it, and it adds its overhead (sometimes a multiple of p-p-s), 
which also scales linearly with the length of the buffer.

So if one can demonstrate that (parse-partial-sexp (point-min) 
(point-max)) takes about the same time as it takes to fontify the last 
screen-ful of a buffer, then that says that everything else that 
jit-lock does to fontify, is negligible, time-wise.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 20:32:32 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 05.08.2022 18:33, Eli Zaretskii wrote:
> >> Date: Fri, 5 Aug 2022 17:41:51 +0300
> >> Cc: monnier@iro.umontreal.ca, 56682@debbugs.gnu.org, gregory@heytings.org
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >>
> I think I have
> >>>> demonstrated that the remaining slowdown can be caused purely by the
> >>>> length of the buffer and how long 'parse-partial-sexp' takes to parse
> >>>> it.
> >>>
> >>> No, you haven't demonstrated that.
> >>
> >> Apply the patch for xdisp.c that I have sent previously (it will be at
> >> the end of this email too) and recompile Emacs.
> >>
> >> Now try two different scenarios. 1,2a and 1,2b.
> >>
> >> 1. Visit dictionary.json. It will ask you whether to open such big file
> >> literally, but after you answer 'y', it will display the beginning of
> >> the file quickly.
> >> 2a) Evaluate (benchmark 1 '(save-excursion (parse-partial-sexp 1
> >> (point-max)))), note the reported delay.
> >>
> >> Kill and re-visit the file.
> >>
> >> 1. (same as before)
> >> 2b) Press M->, note the delay you see.
> >>
> >> The delays in scenarios 1,2a and 1,2b should be ~the same. They are so
> >> in my testing.
> >>
> >> Or try this scenario: 1,2a,2b. Step 2b should work instantly here.
> > 
> > How is (parse-partial-sexp 1 (point-max)) related to the issue at
> > hand?
> 
> I said:
> 
>  >>>> I think I have
>  >>>> demonstrated that the remaining slowdown can be caused purely by the
>  >>>> length of the buffer and how long 'parse-partial-sexp' takes to parse
>  >>>> it.
> 
> You said:
> 
>  >>> No, you haven't demonstrated that.
> 
> ...and now you are asking why we are talking about parse-partial-sexp?

Yes, because a user who visits a file doesn't invoke
parse-partial-sexp.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 21:02:35 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > How is (parse-partial-sexp 1 (point-max)) related to the issue at
> > hand?
> 
> Or perhaps I should answer this way:
> 
> We move to near EOB.
> fontification-functions are called.
> 
> jit-lock
> calls
> (font-lock-fontify-region point-near-buffer-end (point-max))
> which calls
> font-lock-fontify-syntactically-region
> which calls both
>    (syntax-propertize (point-max))
>    and
>    (syntax-ppss point-near-buffer-end) -> and it calls parse-partial-sexp
> 
> syntax-propertize will also likely call syntax-ppss itself, probably 
> through the major mode's syntax-propertize-function. But if 
> syntax-propertize-function is nil, parse-partial-sexp gets called 
> anyway, over the whole buffer, which makes it the main workload in 
> fontifying near EOB.
> 
> Now, if syntax-propertize-function is non-nil, parse-partial-sexp will 
> also call it, and it adds its overhead (sometimes a multiple of p-p-s), 
> which also scales linearly with the length of the buffer.
> 
> So if one can demonstrate that (parse-partial-sexp (point-min) 
> (point-max)) takes about the same time as it takes to fontify the last 
> screen-ful of a buffer, then that says that everything else that 
> jit-lock does to fontify, is negligible, time-wise.

So you have demonstrated that, if visiting a file and moving inside it
calls parse-partial-sexp to scan the entire buffer, then this could be
some, perhaps a large, part of the slowdown.

First, we need to establish that indeed parse-partial-sexp is called
in that manner in the relevant major modes (not just one of them), or
by font-lock itself regardless of the mode.

Second, we need to establish that indeed this takes a large portion of
the time in the slow operations.  Not just one particular operation,
but most or all of them.

And after that, we may have some food for thought.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 21:14, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 21:02:35 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> How is (parse-partial-sexp 1 (point-max)) related to the issue at
>>> hand?
>>
>> Or perhaps I should answer this way:
>>
>> We move to near EOB.
>> fontification-functions are called.
>>
>> jit-lock
>> calls
>> (font-lock-fontify-region point-near-buffer-end (point-max))
>> which calls
>> font-lock-fontify-syntactically-region
>> which calls both
>>     (syntax-propertize (point-max))
>>     and
>>     (syntax-ppss point-near-buffer-end) -> and it calls parse-partial-sexp
>>
>> syntax-propertize will also likely call syntax-ppss itself, probably
>> through the major mode's syntax-propertize-function. But if
>> syntax-propertize-function is nil, parse-partial-sexp gets called
>> anyway, over the whole buffer, which makes it the main workload in
>> fontifying near EOB.
>>
>> Now, if syntax-propertize-function is non-nil, parse-partial-sexp will
>> also call it, and it adds its overhead (sometimes a multiple of p-p-s),
>> which also scales linearly with the length of the buffer.
>>
>> So if one can demonstrate that (parse-partial-sexp (point-min)
>> (point-max)) takes about the same time as it takes to fontify the last
>> screen-ful of a buffer, then that says that everything else that
>> jit-lock does to fontify, is negligible, time-wise.
> 
> So you have demonstrated that, if visiting a file and moving inside it
> calls parse-partial-sexp to scan the entire buffer, then this could be
> some, perhaps a large, part of the slowdown.

Yes.

> First, we need to establish that indeed parse-partial-sexp is called
> in that manner in the relevant major modes (not just one of them), or
> by font-lock itself regardless of the mode.

It is called by font-lock itself, which ends up calling syntax-ppss, 
which does its job with parse-partial-sexp. I have outlined the chain of 
calls in the previous message, you can verify it by looking at the sources.

> Second, we need to establish that indeed this takes a large portion of
> the time in the slow operations.  Not just one particular operation,
> but most or all of them.

To establish that, I have described the experiment in the grandparent 
email (with scenarios 1,2a;1,2b;1,2a,2b), and performed it myself as well.

But I'm talking about the slowdown observed when doing 'M->'. Not about 
any operations one might try to perform. Having said that, after the 
initial 'M->' most of navigation operations look snappy to me. So that's 
the slowdown I decided to investigate.

> And after that, we may have some food for thought.

Here's some more:

All major modes we can currently use for JSON (the built-in js-mode and 
the two json-mode's in ELPA) inherit the value of 
syntax-propertize-function from js-mode. But there's no need for it: 
JSON doesn't have division, or regexps, or preprocessor directives, or 
embedded JSX structures.

Setting syntax-propertize-function to nil speeds up parse-partial-sexp 
significantly. Here's a patch you can try to evaluate the effect on 
dictionary.json of that change combined with the previous tweak I 
suggested. Now it takes about 5x faster to fontify the last screenful, 
on my machine. Meaning, 'M->' feels almost (but not quite) instant. And 
the fontification is still correct.

A "proper" change would involve creating a new major mode, probably, 
rather than regexp-matching against buffer-file-name. But I'm not sure 
what name to pick: 'json-mode' would step on the toes of two existing 
packages now. 'js-json-mode', maybe? Or we bring in json-mode from GNU 
ELPA (with a similar change).

Anyway, try this please:

diff --git a/lisp/progmodes/js.el b/lisp/progmodes/js.el
index eb2a1e4fcc..ae8e980125 100644
--- a/lisp/progmodes/js.el
+++ b/lisp/progmodes/js.el
@@ -3418,7 +3418,8 @@ js-mode
                (list js--font-lock-keywords nil nil nil nil
                      '(font-lock-syntactic-face-function
                        . js-font-lock-syntactic-face-function)))
-  (setq-local syntax-propertize-function #'js-syntax-propertize)
+  (unless (and buffer-file-name (string-match-p "\\.json\\'" 
buffer-file-name))
+    (setq-local syntax-propertize-function #'js-syntax-propertize))
    (add-hook 'syntax-propertize-extend-region-functions
              #'syntax-propertize-multiline 'append 'local)
    (add-hook 'syntax-propertize-extend-region-functions
diff --git a/src/xdisp.c b/src/xdisp.c
index 099efed2db..fcb2be8768 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4391,20 +4391,6 @@ handle_fontified_prop (struct it *it)

        eassert (it->end_charpos == ZV);

-      if (current_buffer->long_line_optimizations_p)
-	{
-	  ptrdiff_t begv = it->narrowed_begv;
-	  ptrdiff_t zv = it->narrowed_zv;
-	  ptrdiff_t charpos = IT_CHARPOS (*it);
-	  if (charpos < begv || charpos > zv)
-	    {
-	      begv = get_narrowed_begv (it->w, charpos);
-	      zv = get_narrowed_zv (it->w, charpos);
-	    }
-	  narrow_to_region_internal (make_fixnum (begv), make_fixnum (zv), true);
-	  specbind (Qrestrictions_locked, Qt);
-	}
-
        /* Don't allow Lisp that runs from 'fontification-functions'
  	 clear our face and image caches behind our back.  */
        it->f->inhibit_clear_image_cache = true;

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 22:01:24 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > First, we need to establish that indeed parse-partial-sexp is called
> > in that manner in the relevant major modes (not just one of them), or
> > by font-lock itself regardless of the mode.
> 
> It is called by font-lock itself, which ends up calling syntax-ppss, 
> which does its job with parse-partial-sexp.

In all modes, regardless of what the mode wants to highlight?

> > Second, we need to establish that indeed this takes a large portion of
> > the time in the slow operations.  Not just one particular operation,
> > but most or all of them.
> 
> To establish that, I have described the experiment in the grandparent 
> email (with scenarios 1,2a;1,2b;1,2a,2b), and performed it myself as well.
> 
> But I'm talking about the slowdown observed when doing 'M->'. Not about 
> any operations one might try to perform. Having said that, after the 
> initial 'M->' most of navigation operations look snappy to me. So that's 
> the slowdown I decided to investigate.

We need to look at more than just M->.  C-n/C-p, C-v/M-v, C-l are also
important, as are the time it takes from typing M-x or M-: until you
see the prompt in the minibuffer, and the time to update the display
after inserting or deleting a single character.

> All major modes we can currently use for JSON (the built-in js-mode and 
> the two json-mode's in ELPA) inherit the value of 
> syntax-propertize-function from js-mode. But there's no need for it: 
> JSON doesn't have division, or regexps, or preprocessor directives, or 
> embedded JSX structures.
> 
> Setting syntax-propertize-function to nil speeds up parse-partial-sexp 
> significantly.

Thanks, so I guess we may have a solution for JSON files, if disabling
syntax-propertize-function doesn't have any downsides.  What about
other modes that we see in files with long lines, like XML?

And how scalable is the solution you propose, i.e. how it behaves in
JSON files with a much longer lines?

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 22:14, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 22:01:24 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> First, we need to establish that indeed parse-partial-sexp is called
>>> in that manner in the relevant major modes (not just one of them), or
>>> by font-lock itself regardless of the mode.
>>
>> It is called by font-lock itself, which ends up calling syntax-ppss,
>> which does its job with parse-partial-sexp.
> 
> In all modes, regardless of what the mode wants to highlight?

Yes. As part of highlighting strings, for instance. It needs to know 
which intervals of characters are strings. And you can't really know 
that without scanning the buffer from the beginning.

>>> Second, we need to establish that indeed this takes a large portion of
>>> the time in the slow operations.  Not just one particular operation,
>>> but most or all of them.
>>
>> To establish that, I have described the experiment in the grandparent
>> email (with scenarios 1,2a;1,2b;1,2a,2b), and performed it myself as well.
>>
>> But I'm talking about the slowdown observed when doing 'M->'. Not about
>> any operations one might try to perform. Having said that, after the
>> initial 'M->' most of navigation operations look snappy to me. So that's
>> the slowdown I decided to investigate.
> 
> We need to look at more than just M->.  C-n/C-p, C-v/M-v, C-l are also
> important, as are the time it takes from typing M-x or M-: until you
> see the prompt in the minibuffer, and the time to update the display
> after inserting or deleting a single character.

I'm not seeing any particular sluggishness in these operations when 
visiting dictionary.json.

>> All major modes we can currently use for JSON (the built-in js-mode and
>> the two json-mode's in ELPA) inherit the value of
>> syntax-propertize-function from js-mode. But there's no need for it:
>> JSON doesn't have division, or regexps, or preprocessor directives, or
>> embedded JSX structures.
>>
>> Setting syntax-propertize-function to nil speeds up parse-partial-sexp
>> significantly.
> 
> Thanks, so I guess we may have a solution for JSON files, if disabling
> syntax-propertize-function doesn't have any downsides.  What about
> other modes that we see in files with long lines, like XML?

Someone will need to test it with some typical large file. xml-mode 
(alias to nxml-mode) does have a syntax-propertize-function, but it's 
probably faster than js-syntax-propertize.

> And how scalable is the solution you propose, i.e. how it behaves in
> JSON files with a much longer lines?

parse-partial-sexp is O(length of text span)

Meaning, it scales linearly. You'll see a 10x delay in a JSON file that 
is 10x as large.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 05.08.2022 17:09, Gregory Heytings wrote:
> 
>>> But if you're so annoyed by mis-fontification, why don't you just 
>>> turn font-lock mode off?
>>
>> "If you're annoyed by Emacs's performance with large files, why don't 
>> you just never open them?"
>>
> 
> That's just wrong: in one case we're talking about your personal 
> feelings/preferences, in the other one about Emacs' capabilities for all 
> its users.

Being able to edit a medium-size file with correct syntax highlighting 
without redisplay-related stuttering is also a capability.

>> I like font-lock and the visual cues that come with it. Only 
>> font-locking the first 1 MB of a large file seems like a good 
>> compromise: show correct highlighting where we can with reasonable 
>> performance, and omit it in the rest of the file.
>>
> 
> So what you prefer IIUC would be to call fontification-functions with a 
> locked narrowing to 1 MB if point is before that threshold, and to not 
> call fontification-functions at all after that threshold?  That might be 
> another doable approach.

If we have to support huge files with max responsiveness, then that 
would be my preference, yes.

I don't see the point of using a "locked" narrowing for this, though. 
Maybe not even a narrowing at all: just avoid calling 
fontification-functions with START > value_of(large_file_fontification_max).

Or even implement that limitation in font-lock itself (in Lisp). Not 
sure which place is better. Depends on how we want 
'font-lock-fontify-region' to behave: would it still fontify a region 
when asked, or would it abort when BEG is too high. I'd prefer the 
former approach since it gives more power to programmers, if we don't 
find any significant pitfalls with it.

>>> Also, why did you not protest vehemently when Stefan added 
>>> syntax-wholeline-max, which also causes occasional mis-fontification?
>>
>> I have replied to this exact question in an earlier email. We can 
>> continue this line of inquiry in that subthread.
>>
> 
> Sorry, I missed that part of your earlier post:
> 
>>
>> syntax-wholelines-max indeed can potentially cause problems too, but 
>> in a much narrow range of situations (and only with major modes which 
>> have non-trivial syntax-propertize-function).
>>
> 
> You forget to metion that syntax-wholelines-max can in fact make things 
> much worse, see the recipes I sent you a few days ago.  So it doesn't 
> seem like it's the right approach.

These recipes, I guess:

 > emacs -Q
 > M-: (setq long-line-threshold nil syntax-wholeline-max most-positive-
 > fixnum) RET
 > C-x C-f dictionary.json RET y ;; takes 160 seconds
 > C-e ;; takes 200 seconds
 >
 > emacs -Q
 > M-: (setq long-line-threshold nil) RET
 > C-x C-f dictionary.json RET y ;; immediate
 > C-e ;; not finished after 1200 seconds (20 minutes), I killed Emacs

I get what you're saying, but the approach does seem "right" to me 
because it works, letting me view and edit dictionary.json with good 
performance and behavior. And because it fits the overall 
font-lock/syntax-propertize/syntax-ppss scheme.

I'm not considering it without the improvements that are enabled with 
non-nil long-line-threshold, though. As you show it above, Emacs is 
simply not usable with those numbers. 200 seconds might as well be 
"never finished". It took like 10-20 seconds on my machine (and with the 
latest patch I sent to this thread), but that's still a lot. Hadn't 
managed to see the second scenario finish either.

Why the default value of syntax-wholeline-max makes things worse in this 
scenario, is a good question, but is it a useful one? Perhaps if you 
could show a similar comparison with a non-nil value of 
long-line-threshold (so that Emacs actually works rather than grind to a 
halt), we could debug it and produce the answer.

Or, if the scenario requires long-line-threshold to be nil, I guess the 
example file needs to be smaller so that the second scenario can finish 
in a bearable amount of time, and Emacs is still usable.

Either way, I believe the change is at the right level of abstraction, 
and if it has bugs, they should be solvable without major redesign.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 5 Aug 2022 23:23:25 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > We need to look at more than just M->.  C-n/C-p, C-v/M-v, C-l are also
> > important, as are the time it takes from typing M-x or M-: until you
> > see the prompt in the minibuffer, and the time to update the display
> > after inserting or deleting a single character.
> 
> I'm not seeing any particular sluggishness in these operations when 
> visiting dictionary.json.

Numbers, please.  You have a very fast machine, so what doesn't look
sluggish on your system could very well be so on others.

> > Thanks, so I guess we may have a solution for JSON files, if disabling
> > syntax-propertize-function doesn't have any downsides.  What about
> > other modes that we see in files with long lines, like XML?
> 
> Someone will need to test it with some typical large file. xml-mode 
> (alias to nxml-mode) does have a syntax-propertize-function, but it's 
> probably faster than js-syntax-propertize.
> 
> > And how scalable is the solution you propose, i.e. how it behaves in
> > JSON files with a much longer lines?
> 
> parse-partial-sexp is O(length of text span)

Isn't that the part you proposed to remove?  I was asking about the
remaining parts.

> Meaning, it scales linearly. You'll see a 10x delay in a JSON file that 
> is 10x as large.

Linear scaling is less optimal than O(0), which is what the current
solution produces.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 6 Aug 2022 01:38:05 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > So what you prefer IIUC would be to call fontification-functions with a 
> > locked narrowing to 1 MB if point is before that threshold, and to not 
> > call fontification-functions at all after that threshold?  That might be 
> > another doable approach.
> 
> If we have to support huge files with max responsiveness, then that 
> would be my preference, yes.
> 
> I don't see the point of using a "locked" narrowing for this, though. 
> Maybe not even a narrowing at all: just avoid calling 
> fontification-functions with START > value_of(large_file_fontification_max).

I don't understand why this would be a good idea.  First, if we are
able to fontify the first 1MB of a file, why won't we be able to
fontify the 2nd, 3rd, ... Nth 1MB in a similarly fast fashion?  Do you
assume that fontification must go to BOB, but will never go forward
towards EOB?  Why would we make such an assumption, especially with
long lines, where EOB could also be EOL?

Next, why assume that your personal preferences in this case will be
shared by everyone else?  I, for one, will find it very strange to
have font-lock suddenly disappear at an arbitrary point.  If we don't
want to make the assumption it will be good for everyone, your
proposal should be one optional behavior, but not the only one.

And finally, there's no way to restrict fontification-functions from
accessing the buffer beyond the first 1MB, even if we want to, without
something like the locked narrowing.

>  > emacs -Q
>  > M-: (setq long-line-threshold nil syntax-wholeline-max most-positive-
>  > fixnum) RET
>  > C-x C-f dictionary.json RET y ;; takes 160 seconds
>  > C-e ;; takes 200 seconds
>  >
>  > emacs -Q
>  > M-: (setq long-line-threshold nil) RET
>  > C-x C-f dictionary.json RET y ;; immediate
>  > C-e ;; not finished after 1200 seconds (20 minutes), I killed Emacs
> 
> I get what you're saying, but the approach does seem "right" to me 
> because it works, letting me view and edit dictionary.json with good 
> performance and behavior.

On a Core i9 machine and with an optimized build, I presume?  And with
"good" being your subjective judgment that places more importance on
fontification than on responsiveness?  That's not necessarily a good
data point on which to base our decisions regarding the default
behavior.

> Either way, I believe the change is at the right level of abstraction, 
> and if it has bugs, they should be solvable without major redesign.

Alas, Mr. ShouldBe is not available for this project, and probably
won't be any time soon.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 06.08.2022 09:07, Eli Zaretskii wrote:
>> Date: Fri, 5 Aug 2022 23:23:25 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> We need to look at more than just M->.  C-n/C-p, C-v/M-v, C-l are also
>>> important, as are the time it takes from typing M-x or M-: until you
>>> see the prompt in the minibuffer, and the time to update the display
>>> after inserting or deleting a single character.
>>
>> I'm not seeing any particular sluggishness in these operations when
>> visiting dictionary.json.
> 
> Numbers, please.  You have a very fast machine, so what doesn't look
> sluggish on your system could very well be so on others.

How do you measure these operations including the redisplay lag?

Anyway, all of these look instant.

>>> Thanks, so I guess we may have a solution for JSON files, if disabling
>>> syntax-propertize-function doesn't have any downsides.  What about
>>> other modes that we see in files with long lines, like XML?
>>
>> Someone will need to test it with some typical large file. xml-mode
>> (alias to nxml-mode) does have a syntax-propertize-function, but it's
>> probably faster than js-syntax-propertize.
>>
>>> And how scalable is the solution you propose, i.e. how it behaves in
>>> JSON files with a much longer lines?
>>
>> parse-partial-sexp is O(length of text span)
> 
> Isn't that the part you proposed to remove?  I was asking about the
> remaining parts.

No, removing syntax-propertize-function just made parse-partial-sexp 5x 
faster here, resulting in much faster fontification near EOB.

But in a large enough buffer, it's still going to be the slowest part.

>> Meaning, it scales linearly. You'll see a 10x delay in a JSON file that
>> is 10x as large.
> 
> Linear scaling is less optimal than O(0), which is what the current
> solution produces.

Correct syntax highlighting requires parsing the buffer from the 
beginning. Otherwise we get random results, essentially.

Just like depicted on my latest screenshot for downloadify.js: the 
beginning of the narrowed region ended up inside a string, and as a 
result several screenfuls were entirely mis-fontified, with strings and 
non-strings inverted.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 6 Aug 2022 13:50:41 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> I'm not seeing any particular sluggishness in these operations when
> >> visiting dictionary.json.
> > 
> > Numbers, please.  You have a very fast machine, so what doesn't look
> > sluggish on your system could very well be so on others.
> 
> How do you measure these operations including the redisplay lag?

By timing them.  With benchmark-run or similar, if it isn't slow
enough to be measured "by hand".

> > Linear scaling is less optimal than O(0), which is what the current
> > solution produces.
> 
> Correct syntax highlighting requires parsing the buffer from the 
> beginning. Otherwise we get random results, essentially.
> 
> Just like depicted on my latest screenshot for downloadify.js: the 
> beginning of the narrowed region ended up inside a string, and as a 
> result several screenfuls were entirely mis-fontified, with strings and 
> non-strings inverted.

Yes, and IMNSHO responsiveness is more important than correctness in
these cases.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Just like depicted on my latest screenshot for downloadify.js: the 
>> beginning of the narrowed region ended up inside a string, and as a 
>> result several screenfuls were entirely mis-fontified, with strings and 
>> non-strings inverted.
>
> Yes, and IMNSHO responsiveness is more important than correctness in 
> these cases.
>

Especially given that the said correctness has always been relative.  So 
it's not like we're leaving the aven of 100% correctness and falling into 
the hell of 10% correctness.  A more reasonable view of the situation is 
that we had 90% correctness and now have, in files with "loo long lines" 
and only in those files, 60% correctness.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> not like we're leaving the aven
>

the heaven

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 05 Aug 2022 11:50:56 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> > There's (at least) one more aspect of this, as long as Text mode is 
> > being used: Text mode doesn't force bidi-paragraph-direction to be 
> > left-to-right, whereas all descendants of prog-mode, including js-mode, 
> > do.  Leaving bidi-paragraph-direction at nil means Emacs needs to 
> > determine the base paragraph direction each time it's about to redisplay 
> > a window, and that might be expensive, especially in a large buffer 
> > without any paragraph breaks (by default, an empty line), because that 
> > is determined by the first strong directional character of the 
> > paragraph.  So for a more fair comparison with Text mode, you should set 
> > bidi-paragraph-direction to the value left-to-right in text-mode 
> > buffers.
> 
> Indeed, that seems to be the culprit here, I didn't know that text-mode 
> was an exception here.  If I set bidi-paragraph-direction to 
> 'left-to-right after visiting the arabic-small.txt file, Emacs (mis) 
> behaves like it does for the other Arabic files: it becomes slow, and C-n 
> C-p do not work correctly anymore.

The problems with C-n/C-p were unrelated, and seem to be a very old
bug.  I've now mostly fixed that on master (and hopefully didn't
introduce any regressions while at that).

In general, layout calculations when we have very long stretches of
R2L text in a left-to-right paragraph are very tricky, because many
assumptions become false.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 05 Aug 2022 11:50:56 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> Indeed, that seems to be the culprit here, I didn't know that text-mode 
> was an exception here.  If I set bidi-paragraph-direction to 
> 'left-to-right after visiting the arabic-small.txt file, Emacs (mis) 
> behaves like it does for the other Arabic files: it becomes slow, and C-n 
> C-p do not work correctly anymore.

I've installed a change that makes Arabic rendering roughly twice
faster.  It is still quite slow, but less so.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

On 06.08.2022 14:49, Gregory Heytings wrote:
> 
>>> Just like depicted on my latest screenshot for downloadify.js: the 
>>> beginning of the narrowed region ended up inside a string, and as a 
>>> result several screenfuls were entirely mis-fontified, with strings 
>>> and non-strings inverted.
>>
>> Yes, and IMNSHO responsiveness is more important than correctness in 
>> these cases.
>>
> 
> Especially given that the said correctness has always been relative.  So 
> it's not like we're leaving the aven of 100% correctness and falling 
> into the hell of 10% correctness.  A more reasonable view of the 
> situation is that we had 90% correctness and now have,

As someone who worked on different major mode and syntax-ppss itself a 
little, that feels moderately insulting.

No: we strive to close to 100% correctness in supporting language syntax 
and can often reach it with moderate effort (programming-wise).

> in files with 
> "loo long lines" and only in those files, 60% correctness.

Does the attached screenshot look like 60% correctness to you?

To me, it's more like -60%. Or at least, that's what the utility of such 
highlighting will be (negative).

And I didn't have to hunt for this screenshot, tweaking parameters, or 
files, or window sizes here and there.

This is just how half of dictionary.json looks with Emacs from master on 
default settings.

[-- Attachment #2: Screenshot from 2022-08-06 22-55-58.png --]
[-- Type: image/png, Size: 823680 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 06.08.2022 14:17, Eli Zaretskii wrote:
>> Date: Sat, 6 Aug 2022 13:50:41 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> I'm not seeing any particular sluggishness in these operations when
>>>> visiting dictionary.json.
>>>
>>> Numbers, please.  You have a very fast machine, so what doesn't look
>>> sluggish on your system could very well be so on others.
>>
>> How do you measure these operations including the redisplay lag?
> 
> By timing them.  With benchmark-run or similar, if it isn't slow
> enough to be measured "by hand".

How do I time 'M-x'?

Anyway, here are some numbers:

(benchmark 1 '(next-line 1)) =>
Elapsed time: 0.035137s
Elapsed time: 0.061863s
Elapsed time: 0.027261s
(benchmark 1 '(cua-scroll-down 1)) =>
Elapsed time: 0.049362s
Elapsed time: 0.050060s
(benchmark 1 '(cua-scroll-up 1)) =>
Elapsed time: 0.012383s
Elapsed time: 0.039984s
(benchmark 1 '(recenter-top-bottom 1)) =>
Elapsed time: 0.042014s
Elapsed time: 0.013934s

I get very similar numbers on these operations with unmodified master in 
this file.

>>> Linear scaling is less optimal than O(0), which is what the current
>>> solution produces.
>>
>> Correct syntax highlighting requires parsing the buffer from the
>> beginning. Otherwise we get random results, essentially.
>>
>> Just like depicted on my latest screenshot for downloadify.js: the
>> beginning of the narrowed region ended up inside a string, and as a
>> result several screenfuls were entirely mis-fontified, with strings and
>> non-strings inverted.
> 
> Yes, and IMNSHO responsiveness is more important than correctness in
> these cases.

That's why I suggested that, possibly after some further optimization 
work on parse-partial-sexp and syntax-propertize, we decide which 
initial length of the file we can fontify correctly with decent speed. 1 
MB, or 10 MB, something like that.

And of course make that customizable.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 06.08.2022 10:28, Eli Zaretskii wrote:
>> Date: Sat, 6 Aug 2022 01:38:05 +0300
>> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>>   Stefan Monnier <monnier@iro.umontreal.ca>
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> So what you prefer IIUC would be to call fontification-functions with a
>>> locked narrowing to 1 MB if point is before that threshold, and to not
>>> call fontification-functions at all after that threshold?  That might be
>>> another doable approach.
>>
>> If we have to support huge files with max responsiveness, then that
>> would be my preference, yes.
>>
>> I don't see the point of using a "locked" narrowing for this, though.
>> Maybe not even a narrowing at all: just avoid calling
>> fontification-functions with START > value_of(large_file_fontification_max).
> 
> I don't understand why this would be a good idea.  First, if we are
> able to fontify the first 1MB of a file, why won't we be able to
> fontify the 2nd, 3rd, ... Nth 1MB in a similarly fast fashion?

Fontifying a window-ful of buffer near the end of 2N will take ~2x as 
long as doing that near the end of 1N, 3N - ~3x as long, etc.

> Do you
> assume that fontification must go to BOB, but will never go forward
> towards EOB?

That's my experience, yes. Language parsers either don't need to look 
forward at all, or only need to look forward a little. And especially 
since we regularly have to deal with unfinished code (and code is most 
often unfinished from the tail end, not from the front), our 
syntax-propertize and font-lock matchers have to account for that.

> Why would we make such an assumption, especially with
> long lines, where EOB could also be EOL?

That's where syntax-wholeline-max comes into play: it won't extend the 
current region past additional 10000 chars.

Even if that fails to work somehow, we could code up font-lock to apply 
narrowing past a certain point, to definitely limit the possible 
overhead. But I'd rather we not do that without a concrete problem case.

> Next, why assume that your personal preferences in this case will be
> shared by everyone else?  I, for one, will find it very strange to
> have font-lock suddenly disappear at an arbitrary point.

Is it more strange than syntax highlighting starting to go crazy around 
the same arbitrary point?

I'm assuming that people will quickly understand that very big files are 
treated in a special way, and I seem to recall that certain other 
editors use a similar approach (switch off some features past a certain 
file length).

> If we don't
> want to make the assumption it will be good for everyone, your
> proposal should be one optional behavior, but not the only one.

I'm happy for this to be customizable, and for everyone to be able to 
choose the tradeoffs they want. For that to be possible, this choice 
need to happen in Lisp. But I made my arguments for what seems like a 
good default behavior, both reasonable-looking and fast in (hopefully) 
all significant cases.

The important part is for this to be customizable separately from 
long-line-threshold because these are different issues, and the 
slowdowns happen around different amounts of text.

> And finally, there's no way to restrict fontification-functions from
> accessing the buffer beyond the first 1MB, even if we want to, without
> something like the locked narrowing.

font-lock rules really shouldn't call 'widen', we've talked about this 
before (in mmm-mode threads), then went through the built-in major modes 
which did so, and fixed that.

The sole (big) exception is CC Mode, but its internals aren't compatible 
with the current forced narrowing either, and Gregory said it's not 
important in the context of long lines or large files anyway.

So as long as major modes behave, we won't need the locked narrowing in 
this area. And hopefully won't need narrowing at all.

>>   > emacs -Q
>>   > M-: (setq long-line-threshold nil syntax-wholeline-max most-positive-
>>   > fixnum) RET
>>   > C-x C-f dictionary.json RET y ;; takes 160 seconds
>>   > C-e ;; takes 200 seconds
>>   >
>>   > emacs -Q
>>   > M-: (setq long-line-threshold nil) RET
>>   > C-x C-f dictionary.json RET y ;; immediate
>>   > C-e ;; not finished after 1200 seconds (20 minutes), I killed Emacs
>>
>> I get what you're saying, but the approach does seem "right" to me
>> because it works, letting me view and edit dictionary.json with good
>> performance and behavior.
> 
> On a Core i9 machine and with an optimized build, I presume?  And with
> "good" being your subjective judgment that places more importance on
> fontification than on responsiveness?  That's not necessarily a good
> data point on which to base our decisions regarding the default
> behavior.

On a slower machine, you'll just need to find a correspondingly smaller 
file to see the same combination: the line is long enough for redisplay 
to stutter with (setq long-line-threshold nil), but the file is not big 
enough for font-lock to really become the bottleneck.

Even with my "big and beefy" i9, I've been seeing redisplay slowdowns 
with lines containing 1000s of chars. Very noticeable around 40K.

Whereas font-lock can remain fast with files up to 100x that size. And 
even then remain bearable so long as the user doesn't jump around too 
much, editing the file at BOB then going to EOB, time and time again.

>> Either way, I believe the change is at the right level of abstraction,
>> and if it has bugs, they should be solvable without major redesign.
> 
> Alas, Mr. ShouldBe is not available for this project, and probably
> won't be any time soon.

No need to mock my request for a proper reproduction scenario.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]


>> Especially given that the said correctness has always been relative.  
>> So it's not like we're leaving the heaven of 100% correctness and 
>> falling into the hell of 10% correctness.  A more reasonable view of 
>> the situation is that we had 90% correctness and now have,
>
> As someone who worked on different major mode and syntax-ppss itself a 
> little, that feels moderately insulting.
>
> No: we strive to close to 100% correctness in supporting language syntax 
> and can often reach it with moderate effort (programming-wise).
>

It isn't meant to be insulting, and I think you understood that.  My 
experience is simply that highlighting in Emacs is definitely not "close 
to 100% correct", whatever be the mode.  The most correct one is perhaps 
emacs-lisp-mode (unsurprisingy).

>> in files with "loo long lines" and only in those files, 60% 
>> correctness.
>
> Does the attached screenshot look like 60% correctness to you?
>

A small sample of a big file is definitely not representative, and I think 
you understood that.

>
> To me, it's more like -60%. Or at least, that's what the utility of such 
> highlighting will be (negative).
>

In which case you should turn highlighting off in such files.  That's what 
all other editors do anyway.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> I'm not seeing any particular sluggishness in these operations when 
>>> visiting dictionary.json.
>> 
>> Numbers, please.  You have a very fast machine, so what doesn't look 
>> sluggish on your system could very well be so on others.
>
> How do you measure these operations including the redisplay lag?
>
> Anyway, all of these look instant.
>

They aren't, no.  Of course it all depends on where you are in the file, 
on your CPU, and so forth.  For me (with locked narrowing disabled, in 
dictionary.json, with emacs -Q) M-> takes about 3 seconds.  Likewise, if I 
do for example C-s aan, and then repeat C-s, whenever the next match is 
far enough in the buffer I see a delay of about 2 seconds.  Another test 
you can do is to lean on C-v after opening the buffer, you'll see that 
Emacs becomes very sluggish, sometimes I have to wait more than 5 seconds 
to avance from one screenfull.

>
> Correct syntax highlighting requires parsing the buffer from the 
> beginning. Otherwise we get random results, essentially.
>

It all depends how you define "correct".  You may remind that I suggested 
to introduce some heuristics in the algorithm.  With appropriate 
heuristics, you could probably get reasonably good results even if you 
only have access to a small portion of the buffer.  For example, instead 
of counting the number of '"' characters from BOB to know for sure if you 
are in a string, you mark those that are likely at the beginning of a 
string and those that are likely at the end of a string, and you select 
the most likely possibility among all combinations to highlight the buffer 
portion as well as possible.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]


>
> The problems with C-n/C-p were unrelated, and seem to be a very old bug. 
> I've now mostly fixed that on master (and hopefully didn't introduce any 
> regressions while at that).
>

Thanks!  Indeed these problem seem fixed now, it's much better.  I did not 
see any regressions, if I see anything I'll tell you.  One case that still 
isn't quite right apparently is the arabic-small.txt.json file.  If I open 
that file with emacs -Q and type C-f, or C-n, or C-p, point moves to EOB 
(but strangly, not in 100% cases), and there C-a, or M-<, or C-p, or C-n 
do not seem to work anymore.

>
> In general, layout calculations when we have very long stretches of R2L 
> text in a left-to-right paragraph are very tricky, because many 
> assumptions become false.
>

Frankly, would programming be as fun as it is if assumptions were true? 😉

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Indeed, that seems to be the culprit here, I didn't know that text-mode 
>> was an exception here.  If I set bidi-paragraph-direction to 
>> 'left-to-right after visiting the arabic-small.txt file, Emacs (mis) 
>> behaves like it does for the other Arabic files: it becomes slow, and 
>> C-n C-p do not work correctly anymore.
>
> I've installed a change that makes Arabic rendering roughly twice 
> faster.  It is still quite slow, but less so.
>

Yes, I've seen that change, too.  It's better now, I wonder if I'll manage 
to make it even better.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> The sole (big) exception is CC Mode, but its internals aren't compatible 
> with the current forced narrowing either, and Gregory said it's not 
> important in the context of long lines or large files anyway.
>

I did not say that, no.  What I can say is that the way CC Mode (ab)uses 
font locking is very unfortunate, and that it is indeed, as far as I can 
tell, incompatible with locked narrowing.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 7 Aug 2022 01:58:06 +0300
> Cc: gregory@heytings.org, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> Either way, I believe the change is at the right level of abstraction,
> >> and if it has bugs, they should be solvable without major redesign.
> > 
> > Alas, Mr. ShouldBe is not available for this project, and probably
> > won't be any time soon.
> 
> No need to mock my request for a proper reproduction scenario.

It isn't mockery.  My point is simple: we would like to solve this
issue in the best possible manner in Emacs 29.  Keeping talking about
what should and shouldn't be done doesn't advance us on that path.  If
you have practical ideas to solve this in a way different from what is
now on master, feel free to push a feature branch with your proposed
changes, or even install them as optional features on master, and
let's compare the merits and demerits of each idea in practical use
cases and on a variety of systems.  Then we can eventually make an
intelligent decision about the optimal mix of features we want to
have.  But as long as there's only one practical idea that works
reasonably well, it is going to win if only for lack of "competition",
because leaving the situation as it is in Emacs 28 is not an option
anymore.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 07.08.2022 02:29, Gregory Heytings wrote:
> 
>>> Especially given that the said correctness has always been relative. 
>>> So it's not like we're leaving the heaven of 100% correctness and 
>>> falling into the hell of 10% correctness.  A more reasonable view of 
>>> the situation is that we had 90% correctness and now have,
>>
>> As someone who worked on different major mode and syntax-ppss itself a 
>> little, that feels moderately insulting.
>>
>> No: we strive to close to 100% correctness in supporting language 
>> syntax and can often reach it with moderate effort (programming-wise).
>>
> 
> It isn't meant to be insulting, and I think you understood that.  My 
> experience is simply that highlighting in Emacs is definitely not "close 
> to 100% correct", whatever be the mode.  The most correct one is perhaps 
> emacs-lisp-mode (unsurprisingy).

Have you ever managed to trigger incorrect highlighting in a JSON file?

Without the recent changes, that is.

>>> in files with "loo long lines" and only in those files, 60% correctness.
>>
>> Does the attached screenshot look like 60% correctness to you?
>>
> 
> A small sample of a big file is definitely not representative, and I 
> think you understood that.

Did I need to send the screenshots of every screen? Like I said, more 
than half of the file was like that.

And then I restarted with 'emacs -Q', and the result looked even worse.

>> To me, it's more like -60%. Or at least, that's what the utility of 
>> such highlighting will be (negative).
>>
> 
> In which case you should turn highlighting off in such files.  That's 
> what all other editors do anyway.

That's the same as saying the user should turn off font-lock if they 
experience performance problems.

Except in this case they would still have choice to suffer a little in 
the performance department (and the amount of said suffering will highly 
depend on their use case) but keep good syntax highlighting.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 07.08.2022 03:11, Gregory Heytings wrote:
> 
>>>> I'm not seeing any particular sluggishness in these operations when 
>>>> visiting dictionary.json.
>>>
>>> Numbers, please.  You have a very fast machine, so what doesn't look 
>>> sluggish on your system could very well be so on others.
>>
>> How do you measure these operations including the redisplay lag?
>>
>> Anyway, all of these look instant.
>>
> 
> They aren't, no.  Of course it all depends on where you are in the file, 
> on your CPU, and so forth.  For me (with locked narrowing disabled, in 
> dictionary.json, with emacs -Q) M-> takes about 3 seconds.

And it's about 1.5s here. Once.

> Likewise, if 
> I do for example C-s aan, and then repeat C-s, whenever the next match 
> is far enough in the buffer I see a delay of about 2 seconds.

Sounds like a performance problem inside isearch. I might take a look.

It shouldn't force font-lock over a big region, and if syntax-ppss cache 
is already filled, there is no other obvious place for that delay to 
come from.

> Another 
> test you can do is to lean on C-v after opening the buffer, you'll see 
> that Emacs becomes very sluggish, sometimes I have to wait more than 5 
> seconds to avance from one screenfull.

First of all, these aren't the tests Eli asked for or I performed. Why 
don't you compare apples to oranges?

Leaning on C-v isn't a scientific test anyway: it compares the speed of 
two processes internal to Emacs (event handling and rendering), rather 
than something objective. And waiting for X seconds means you wait for 
however many screens you managed to page through, for all of them to get 
syntax-highlighted. That is indeed a way to break the "jit" in 
"jit-lock", but how is that a useful test?

Anyway, could you please try the combined patch I posted recently 
together with dictionary.json?

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56682#1084

>> Correct syntax highlighting requires parsing the buffer from the 
>> beginning. Otherwise we get random results, essentially.
>>
> 
> It all depends how you define "correct".  You may remind that I 
> suggested to introduce some heuristics in the algorithm.

Stefan mentioned such possibility as well.

> With 
> appropriate heuristics, you could probably get reasonably good results 
> even if you only have access to a small portion of the buffer.  For 
> example, instead of counting the number of '"' characters from BOB to 
> know for sure if you are in a string, you mark those that are likely at 
> the beginning of a string and those that are likely at the end of a 
> string, and you select the most likely possibility among all 
> combinations to highlight the buffer portion as well as possible.

Any such heuristics are possible, but major modes are the best place to 
make that decision. And until the major mode has installed something 
like this, syntax-ppss should default to the "correct" behavior. Which 
means operating outside of narrowing.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]


>> They aren't, no.  Of course it all depends on where you are in the 
>> file, on your CPU, and so forth.  For me (with locked narrowing 
>> disabled, in dictionary.json, with emacs -Q) M-> takes about 3 seconds.
>
> And it's about 1.5s here.
>

So you're the lucky owner of a fast computer.  And using an optimized 
build.  Not everyone is in that fortunate situation.

>
> Once.
>

No, not once.  It happens every time you modify the buffer and move far 
enough in that buffer.

>> Likewise, if I do for example C-s aan, and then repeat C-s, whenever 
>> the next match is far enough in the buffer I see a delay of about 2 
>> seconds.
>
> Sounds like a performance problem inside isearch. I might take a look.
>

No, isearch is entirely agnostic about font locking.

>> Another test you can do is to lean on C-v after opening the buffer, 
>> you'll see that Emacs becomes very sluggish, sometimes I have to wait 
>> more than 5 seconds to avance from one screenfull.
>
> First of all, these aren't the tests Eli asked for or I performed. Why 
> don't you compare apples to oranges?
>
> Leaning on C-v isn't a scientific test anyway: it compares the speed of 
> two processes internal to Emacs (event handling and rendering), rather 
> than something objective.
>

It's yet another test meant to test Emacs' responsiveness, and it is as 
"objective" as possible: does Emacs choke or does it not?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 07 Aug 2022 00:29:05 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: gerd.moellmann@gmail.com, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> Thanks!  Indeed these problem seem fixed now, it's much better.  I did not 
> see any regressions, if I see anything I'll tell you.  One case that still 
> isn't quite right apparently is the arabic-small.txt.json file.  If I open 
> that file with emacs -Q and type C-f, or C-n, or C-p, point moves to EOB 
> (but strangly, not in 100% cases), and there C-a, or M-<, or C-p, or C-n 
> do not seem to work anymore.

Indeed.  But what you perhaps don't realize is that in this case what
is shown on the last screen-full is not EOB, but BOB!  That's because
everything is reordered for display, so the first character appears
last.

I've fixed some more issues with such (relatively rare) situations, so
now these commands should work more reasonably (but very slowly) after
C-f in that file.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 07.08.2022 15:46, Gregory Heytings wrote:
> 
>>> They aren't, no.  Of course it all depends on where you are in the 
>>> file, on your CPU, and so forth.  For me (with locked narrowing 
>>> disabled, in dictionary.json, with emacs -Q) M-> takes about 3 seconds.
>>
>> And it's about 1.5s here.
>>
> 
> So you're the lucky owner of a fast computer.  And using an optimized 
> build.  Not everyone is in that fortunate situation.

If your computer is 2x slower, that should mean it can handle twice 
smaller files with approximately the same speed.

So, qualitatively, you should be in the same boat.

>>
>> Once.
>>
> 
> No, not once.  It happens every time you modify the buffer and move far 
> enough in that buffer.

Yes. That's the price of re-parsing the buffer the way that we know how 
(with parse-partial-sexp). The only feasible comparable alternative I 
know of is tree-sitter which has a more advanced parser that knows how 
to amortize certain operations modifications faster than O(N).

So, with the tree-sitter integration that's hopefully coming, we should 
be able to get a speed improvement in handling of huge files, too. If it 
doesn't blow up the memory requirements, that is.

>>> Likewise, if I do for example C-s aan, and then repeat C-s, whenever 
>>> the next match is far enough in the buffer I see a delay of about 2 
>>> seconds.
>>
>> Sounds like a performance problem inside isearch. I might take a look.
>>
> 
> No, isearch is entirely agnostic about font locking.

I wonder where it spends those 2 seconds, then.

>>> Another test you can do is to lean on C-v after opening the buffer, 
>>> you'll see that Emacs becomes very sluggish, sometimes I have to wait 
>>> more than 5 seconds to avance from one screenfull.
>>
>> First of all, these aren't the tests Eli asked for or I performed. Why 
>> don't you compare apples to oranges?
>>
>> Leaning on C-v isn't a scientific test anyway: it compares the speed 
>> of two processes internal to Emacs (event handling and rendering), 
>> rather than something objective.
>>
> 
> It's yet another test meant to test Emacs' responsiveness, and it is as 
> "objective" as possible: does Emacs choke or does it not?

It's not a test of font-lock's performance, however. Because it compares 
that to a process that's internal to Emacs as well (moving across the 
buffer with C-v). Like, the faster we're able to make the latter 
command, the faster font-lock has to be to keep up. As an objective 
test, it's not meaningful.

Now, if we measured how many screens one could page through with 
fontification in 10 seconds, that's something more objective (putting 
aside one's screen dimensions, of course). But anyway, it's a nice race, 
but not as essential as quickly fontifying any particular screen. That's 
what we're measured responsiveness with in this discussion so far.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 07.08.2022 14:28, Dmitry Gutov wrote:
> Anyway, could you please try the combined patch I posted recently 
> together with dictionary.json?
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56682#1084

Gregory, could you do this?

And report on how your own responsiveness metrics hold up after that.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 07.08.2022 09:11, Eli Zaretskii wrote:
>> Date: Sun, 7 Aug 2022 01:58:06 +0300
>> Cc: gregory@heytings.org, 56682@debbugs.gnu.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> Either way, I believe the change is at the right level of abstraction,
>>>> and if it has bugs, they should be solvable without major redesign.
>>>
>>> Alas, Mr. ShouldBe is not available for this project, and probably
>>> won't be any time soon.
>>
>> No need to mock my request for a proper reproduction scenario.
> 
> It isn't mockery.

How else would I call that response to me saying that I need an 
actionable complaint. I.e. a scenario that reflects an realistic usage 
scenario and is possible to debug.

> My point is simple: we would like to solve this
> issue in the best possible manner in Emacs 29.  Keeping talking about
> what should and shouldn't be done doesn't advance us on that path.  If
> you have practical ideas to solve this in a way different from what is
> now on master,

That "if" doesn't sound very friendly either. I've been explaining 
"practical ideas" for the last couple of days.

And I've sent one patch to try already. One I'm assuming nobody aside 
from me even applied, as of now.

> feel free to push a feature branch with your proposed
> changes, or even install them as optional features on master, and
> let's compare the merits and demerits of each idea in practical use
> cases and on a variety of systems.
I will.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 7 Aug 2022 21:27:21 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > feel free to push a feature branch with your proposed
> > changes, or even install them as optional features on master, and
> > let's compare the merits and demerits of each idea in practical use
> > cases and on a variety of systems.
> I will.

Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> No, isearch is entirely agnostic about font locking.
>
> I wonder where it spends those 2 seconds, then.
>

The answer is simple: it is not isearch that is slow, it's redisplay that 
is slowed down by font locking.

>> It's yet another test meant to test Emacs' responsiveness, and it is as 
>> "objective" as possible: does Emacs choke or does it not?
>
> It's not a test of font-lock's performance, however. Because it compares 
> that to a process that's internal to Emacs as well (moving across the 
> buffer with C-v). Like, the faster we're able to make the latter 
> command, the faster font-lock has to be to keep up. As an objective 
> test, it's not meaningful.
>

It is not by itself a test of font locking performance.  It becomes one 
when you compare it with what you see (a) when font locking is completely 
disabled, and (b) when font locking is enabled by constrained by a locked 
narrowing.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Anyway, could you please try the combined patch I posted recently 
>> together with dictionary.json?
>> 
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=56682#1084
>
> Gregory, could you do this?
>
> And report on how your own responsiveness metrics hold up after that.
>

Now done, see my reply on emacs-devel.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 07.08.2022 23:21, Gregory Heytings wrote:
> 
>>> No, isearch is entirely agnostic about font locking.
>>
>> I wonder where it spends those 2 seconds, then.
>>
> 
> The answer is simple: it is not isearch that is slow, it's redisplay 
> that is slowed down by font locking.

Turns out, that was true. Unexpectedly, there was an area near the 
beginning of that file where a particular matcher (that I have now 
removed in fcd2d14db1) slowed things down considerably. The effect was 
such that fontifying a particular screen was about as slow as fontifying 
the end of the buffer.

That might or might not be a general design bug in font-lock with its 
MATCH-ANCHORED syntax (when a line is long, the "anchored" thing might 
lose its meaning).

isearch looks fast again, you're welcome to try.

>>> It's yet another test meant to test Emacs' responsiveness, and it is 
>>> as "objective" as possible: does Emacs choke or does it not?
>>
>> It's not a test of font-lock's performance, however. Because it 
>> compares that to a process that's internal to Emacs as well (moving 
>> across the buffer with C-v). Like, the faster we're able to make the 
>> latter command, the faster font-lock has to be to keep up. As an 
>> objective test, it's not meaningful.
>>
> 
> It is not by itself a test of font locking performance.  It becomes one 
> when you compare it with what you see (a) when font locking is 
> completely disabled, and (b) when font locking is enabled by constrained 
> by a locked narrowing.

What do you get from that comparison? That one thing is faster than the 
other? That accurate font-lock might take a little more time?

In any case, when I floor 'C-v' in dictionary.json now, whether 
narrowing is applied in 'handle_fontified_prop' or not, I see the buffer 
stutter around 2%, then go on and freeze around the 9% mark. When I 
release 'C-v' after waiting a while, it ends up at 12% or 14%.

Or if I go to EOB and floor 'M-v', both versions freeze almost right 
away. So it doesn't look like one is an order of a magnitude faster than 
the other, or at least not anymore.

C-n and C-p are also pretty sluggish with both versions.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> In any case, when I floor 'C-v' in dictionary.json now, whether 
> narrowing is applied in 'handle_fontified_prop' or not, I see the buffer 
> stutter around 2%, then go on and freeze around the 9% mark. When I 
> release 'C-v' after waiting a while, it ends up at 12% or 14%.
>
> Or if I go to EOB and floor 'M-v', both versions freeze almost right 
> away. So it doesn't look like one is an order of a magnitude faster than 
> the other, or at least not anymore.
>
> C-n and C-p are also pretty sluggish with both versions.
>

You must be doing something wrong in your tests.  You have a faster 
computer than mine, and on my computer, with emacs -Q, neither C-v nor M-v 
nor C-n nor C-p are sluggish.

bug#56682: Fix the long lines font locking related slowdowns

[Stephen Berman]

On Wed, 10 Aug 2022 07:47:44 +0000 Gregory Heytings <gregory@heytings.org> wrote:

>>
>> In any case, when I floor 'C-v' in dictionary.json now, whether narrowing is
>> applied in 'handle_fontified_prop' or not, I see the buffer stutter around
>> 2%, then go on and freeze around the 9% mark. When I release 'C-v' after
>> waiting a while, it ends up at 12% or 14%.
>>
>> Or if I go to EOB and floor 'M-v', both versions freeze almost right
>> away. So it doesn't look like one is an order of a magnitude faster than the
>> other, or at least not anymore.
>>
>> C-n and C-p are also pretty sluggish with both versions.
>>
>
> You must be doing something wrong in your tests.  You have a faster computer
> than mine, and on my computer, with emacs -Q, neither C-v nor M-v nor C-n nor
> C-p are sluggish.

When I visit addons.json (from my Firefox directory), which is a single
line of (only) 19540 characters, and type `C-n' and hold down both keys,
the cursor stops moving almost immediately (on the third visual line)
and after letting go of the keys and waiting several seconds, point
jumps down the buffer to its actual position; likewise with `C-p'.  This
is on master with -Q, and is no different from Emacs 28 (the only
difference I notice is that in master the buffer is wrongly fontified
from position 8235 to the end, while in 28 the entire buffer is
correctly fontified).  Is this expected?  (When I enable so-long-mode in
the buffer, holding down `C-n' or `C-p' produces no delay, both in
master and 28.)

Steve Berman

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> When I visit addons.json (from my Firefox directory), which is a single 
> line of (only) 19540 characters, and type `C-n' and hold down both keys, 
> the cursor stops moving almost immediately (on the third visual line) 
> and after letting go of the keys and waiting several seconds, point 
> jumps down the buffer to its actual position; likewise with `C-p'. 
> This is on master with -Q, and is no different from Emacs 28 (the only 
> difference I notice is that in master the buffer is wrongly fontified 
> from position 8235 to the end, while in 28 the entire buffer is 
> correctly fontified).  Is this expected?  (When I enable so-long-mode in 
> the buffer, holding down `C-n' or `C-p' produces no delay, both in 
> master and 28.)
>

That's not expected, no, and rather surprising because it would contradict 
what we've seen so far.  Can you post that file somewhere?  The only 
possible reason I can see ATM is that the file contains Arabic characters, 
which are (and will remain) slow.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> When I visit addons.json (from my Firefox directory), which is a single 
>> line of (only) 19540 characters, and type `C-n' and hold down both 
>> keys, the cursor stops moving almost immediately (on the third visual 
>> line) and after letting go of the keys and waiting several seconds, 
>> point jumps down the buffer to its actual position; likewise with 
>> `C-p'. This is on master with -Q, and is no different from Emacs 28 
>> (the only difference I notice is that in master the buffer is wrongly 
>> fontified from position 8235 to the end, while in 28 the entire buffer 
>> is correctly fontified).  Is this expected?  (When I enable 
>> so-long-mode in the buffer, holding down `C-n' or `C-p' produces no 
>> delay, both in master and 28.)
>
> That's not expected, no, and rather surprising because it would 
> contradict what we've seen so far.  Can you post that file somewhere? 
> The only possible reason I can see ATM is that the file contains Arabic 
> characters, which are (and will remain) slow.
>

I forgot to mention: did you try after disabling show-paren-mode?  It is 
known to slow down redisplay at the beginning and end of the buffer, and 
will eventually be automatically turned off in these buffers.

bug#56682: Fix the long lines font locking related slowdowns

[Stephen Berman]

[-- Attachment #1: Type: text/plain, Size: 1644 bytes --]

On Wed, 10 Aug 2022 09:04:06 +0000 Gregory Heytings <gregory@heytings.org> wrote:

>>> When I visit addons.json (from my Firefox directory), which is a single
>>> line of (only) 19540 characters, and type `C-n' and hold down both keys,
>>> the cursor stops moving almost immediately (on the third visual line) and
>>> after letting go of the keys and waiting several seconds, point jumps down
>>> the buffer to its actual position; likewise with `C-p'. This is on master
>>> with -Q, and is no different from Emacs 28 (the only difference I notice is
>>> that in master the buffer is wrongly fontified from position 8235 to the
>>> end, while in 28 the entire buffer is correctly fontified).  Is this
>>> expected?  (When I enable so-long-mode in the buffer, holding down `C-n' or
>>> `C-p' produces no delay, both in master and 28.)
>>
>> That's not expected, no, and rather surprising because it would contradict
>> what we've seen so far.  Can you post that file somewhere? 

It contains nothing particularly personal, so I've attached it.

>>                                                            The only possible
>> reason I can see ATM is that the file contains Arabic characters, which are
>> (and will remain) slow.

No Arabic and only four non-ASCII characters (three occurrences of `’'
and one of `É').

> I forgot to mention: did you try after disabling show-paren-mode?  It is known
> to slow down redisplay at the beginning and end of the buffer, and will
> eventually be automatically turned off in these buffers.

Disabling show-paren-mode makes no noticeable difference.

Steve Berman


[-- Attachment #2: addons.json --]
[-- Type: application/octet-stream, Size: 19547 bytes --]

{"schema":6,"addons":[{"id":"uBlock0@raymondhill.net","icons":{"32":"https://addons.mozilla.org/user-media/addon_icons/607/607454-32.png?modified=mcrushed","64":"https://addons.mozilla.org/user-media/addon_icons/607/607454-64.png?modified=mcrushed","128":"https://addons.mozilla.org/user-media/addon_icons/607/607454-128.png?modified=mcrushed"},"name":"uBlock Origin","version":"1.43.0","sourceURI":"https://addons.mozilla.org/firefox/downloads/file/3961087/ublock_origin-1.43.0.xpi","homepageURL":"https://github.com/gorhill/uBlock#ublock-origin","supportURL":"https://old.reddit.com/r/uBlockOrigin/","description":"Finally, an efficient wide-spectrum content blocker. Easy on CPU and memory.","fullDescription":"uBlock Origin is not an \"ad blocker\", it's a wide-spectrum content blocker with CPU and memory efficiency as a primary feature.\n\n***\n\nOut of the box, uBO blocks ads, trackers, coin miners, popups, etc. through the following lists of filters, enabled by default:\n\n- EasyList (ads)\n- EasyPrivacy (tracking)\n- Peter Lowe’s Ad server list (ads and tracking)\n- Online Malicious URL Blocklist\n- uBO's own lists\n\nMore lists are available for you to select if you wish:\n\n- EasyList Cookie\n- Fanboy Annoyances\n- AdGuard Annoyances\n- Dan Pollock’s hosts file\n- And many others\n\nAdditionally, you can point-and-click to block JavaScript locally or globally, create your own global or local rules to override entries from filter lists, and many more advanced features.\n\n***\n\nFree.\nOpen source with public license (GPLv3)\nFor users by users.\n\nIf ever you really do want to contribute something, think about the people working hard to maintain the filter lists you are using, which were made available to use by all for free.\n\n***\n\n    Documentation\n    Release notes\n    Community support @ Reddit\n    Contributors @ GitHub\n    Contributors @ Crowdin","weeklyDownloads":122583,"type":"extension","creator":{"name":"Raymond Hill","url":"https://addons.mozilla.org/en-US/firefox/user/11423598/"},"developers":[],"screenshots":[{"url":"https://addons.mozilla.org/user-media/previews/full/238/238546.png?modified=1622132421","width":1011,"height":758,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/238/238546.jpg?modified=1622132421","thumbnailWidth":533,"thumbnailHeight":400,"caption":"The popup panel: default mode"},{"url":"https://addons.mozilla.org/user-media/previews/full/238/238548.png?modified=1622132423","width":1011,"height":758,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/238/238548.jpg?modified=1622132423","thumbnailWidth":533,"thumbnailHeight":400,"caption":"The dashboard: stock filter lists"},{"url":"https://addons.mozilla.org/user-media/previews/full/238/238547.png?modified=1622132425","width":1011,"height":758,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/238/238547.jpg?modified=1622132425","thumbnailWidth":533,"thumbnailHeight":400,"caption":"The popup panel: default-deny mode"},{"url":"https://addons.mozilla.org/user-media/previews/full/238/238549.png?modified=1622132426","width":1011,"height":758,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/238/238549.jpg?modified=1622132426","thumbnailWidth":533,"thumbnailHeight":400,"caption":"The dashboard: settings"},{"url":"https://addons.mozilla.org/user-media/previews/full/238/238552.png?modified=1622132430","width":970,"height":1800,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/238/238552.jpg?modified=1622132430","thumbnailWidth":216,"thumbnailHeight":400,"caption":"The popup panel in Firefox Preview: default mode with more blocking options revealed"},{"url":"https://addons.mozilla.org/user-media/previews/full/230/230370.png?modified=1622132432","width":800,"height":600,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/230/230370.jpg?modified=1622132432","thumbnailWidth":533,"thumbnailHeight":400,"caption":"The unified logger tells you all that uBO is seeing and doing"}],"contributionURL":"","averageRating":4.7731,"reviewCount":3675,"reviewURL":"https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/reviews/","updateDate":1659988217000},{"id":"addon@darkreader.org","icons":{"32":"https://addons.mozilla.org/user-media/addon_icons/855/855413-32.png?modified=3f34e276","64":"https://addons.mozilla.org/user-media/addon_icons/855/855413-64.png?modified=3f34e276","128":"https://addons.mozilla.org/user-media/addon_icons/855/855413-128.png?modified=3f34e276"},"name":"Dark Reader","version":"4.9.52","sourceURI":"https://addons.mozilla.org/firefox/downloads/file/3968561/darkreader-4.9.52.xpi","homepageURL":"https://darkreader.org/","supportURL":"https://github.com/darkreader/darkreader","description":"Dark mode for every website. Take care of your eyes, use dark theme for night and daily browsing.","fullDescription":"This eye-care extension enables night mode creating dark themes for websites on the fly. Dark Reader inverts bright colors making them high contrast and easy to read at night.\n\nYou can adjust brightness, contrast, sepia filter, dark mode, font settings and ignore-list.\n\nDark Reader doesn't show ads and doesn't send user's data anywhere. It is fully open-source https://github.com/darkreader/darkreader\n\nBefore you install disable similar extensions. Enjoy watching!","weeklyDownloads":20979,"type":"extension","creator":{"name":"Alexander Shutau","url":"https://addons.mozilla.org/en-US/firefox/user/13299734/"},"developers":[],"screenshots":[{"url":"https://addons.mozilla.org/user-media/previews/full/201/201069.png?modified=1638883247","width":2400,"height":1500,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/201/201069.jpg?modified=1638883247","thumbnailWidth":533,"thumbnailHeight":333,"caption":"Dark Reader night theme mode"},{"url":"https://addons.mozilla.org/user-media/previews/full/201/201070.png?modified=1638883247","width":2400,"height":1500,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/201/201070.jpg?modified=1638883247","thumbnailWidth":533,"thumbnailHeight":333,"caption":"Dark Reader filter settings"}],"contributionURL":"https://opencollective.com/darkreader?utm_content=product-page-contribute&utm_medium=referral&utm_source=addons.mozilla.org","averageRating":4.5528,"reviewCount":1454,"reviewURL":"https://addons.mozilla.org/en-US/firefox/addon/darkreader/reviews/","updateDate":1656418435000},{"id":"{5384767E-00D9-40E9-B72F-9CC39D655D6F}","icons":{"32":"https://addons.mozilla.org/user-media/addon_icons/45/45281-32.png?modified=mcrushed","64":"https://addons.mozilla.org/user-media/addon_icons/45/45281-64.png?modified=mcrushed","128":"https://addons.mozilla.org/user-media/addon_icons/45/45281-128.png?modified=mcrushed"},"name":"EPUBReader","version":"2.0.13","sourceURI":"https://addons.mozilla.org/firefox/downloads/file/3594370/epubreader-2.0.13.xpi","homepageURL":"https://www.epubread.com/","supportURL":"https://www.epubread.com/","description":"Read ePub files right in Firefox. No additional software needed!","fullDescription":"Avoid annoying download prompts and easily read ePub files directly in Firefox. Conveniently access previously saved ePub files right from the toolbar button.\n\nListen to your eBooks by installing the Read Aloud extension!\n\nThe requested permissions are needed so EPUBReader can download and open epub files directly from websites and save them.\n\nIf you experience any problems or issues, please let me know through my support site. I'll be very happy to help you!","weeklyDownloads":1543,"type":"extension","creator":{"name":"epubreader","url":"https://addons.mozilla.org/en-US/firefox/user/4960765/"},"developers":[],"screenshots":[{"url":"https://addons.mozilla.org/user-media/previews/full/184/184970.png?modified=1622132570","width":700,"height":525,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/184/184970.jpg?modified=1622132570","thumbnailWidth":533,"thumbnailHeight":400,"caption":"Adventures of Captain Hatteras opened in EPUBReader"},{"url":"https://addons.mozilla.org/user-media/previews/full/184/184972.png?modified=1622132570","width":700,"height":525,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/184/184972.jpg?modified=1622132570","thumbnailWidth":533,"thumbnailHeight":400,"caption":"Table of contents"},{"url":"https://addons.mozilla.org/user-media/previews/full/184/184973.png?modified=1622132570","width":700,"height":525,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/184/184973.jpg?modified=1622132570","thumbnailWidth":533,"thumbnailHeight":400,"caption":"Preferences"},{"url":"https://addons.mozilla.org/user-media/previews/full/184/184971.png?modified=1622132570","width":700,"height":525,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/184/184971.jpg?modified=1622132570","thumbnailWidth":533,"thumbnailHeight":400,"caption":"Toolbar"}],"contributionURL":"","averageRating":3.7266,"reviewCount":110,"reviewURL":"https://addons.mozilla.org/en-US/firefox/addon/epubreader/reviews/","updateDate":1592832627000},{"id":"jid1-KtlZuoiikVfFew@jetpack","icons":{"32":"https://addons.mozilla.org/user-media/addon_icons/350/350590-32.png?modified=1637253033","64":"https://addons.mozilla.org/user-media/addon_icons/350/350590-64.png?modified=1637253033","128":"https://addons.mozilla.org/user-media/addon_icons/350/350590-128.png?modified=1637253033"},"name":"GNU LibreJS","version":"7.21.0","sourceURI":"https://addons.mozilla.org/firefox/downloads/file/3978554/librejs-7.21.0.xpi","homepageURL":"https://www.gnu.org/software/librejs/","supportURL":"https://gnu.org/software/librejs","description":"GNU LibreJS aims to address the JavaScript problem described in Richard Stallman's article The JavaScript Trap. It is a free add-on for Mozilla browsers. It blocks nonfree nontrivial JavaScript while allowing JavaScript that is free and/or trivial.","fullDescription":"GNU LibreJS --an add-on for GNU IceCat and Mozilla Firefox-- detects and blocks nonfree nontrivial JavaScript while allowing its execution on pages containing code that is either trivial and/or free.\n\nMany websites run nontrivial JavaScript on your computer. Some use it for complex tasks; many use it gratuitously for minor jobs that could be done easily with plain HTML. Sometimes this JavaScript code is malicious. Either way, the JavaScript code is often nonfree. For explanation of the issue, see The JavaScript Trap.\n\nIf you care about freedom in your computing, and don't wish to let all and sundry make you run nonfree programs, now you can prevent it by using LibreJS.","weeklyDownloads":25,"type":"extension","creator":{"name":"Loic J. Duros","url":"https://addons.mozilla.org/en-US/firefox/user/5942745/"},"developers":[{"name":"nnyby","url":"https://addons.mozilla.org/en-US/firefox/user/11079870/"},{"name":"Ruben Rodriguez","url":"https://addons.mozilla.org/en-US/firefox/user/11729378/"},{"name":"NateN1222","url":"https://addons.mozilla.org/en-US/firefox/user/12985142/"},{"name":"Firefox user 13499009","url":"https://addons.mozilla.org/en-US/firefox/user/13499009/"},{"name":"Yuchen Pei","url":"https://addons.mozilla.org/en-US/firefox/user/17132119/"}],"screenshots":[{"url":"https://addons.mozilla.org/user-media/previews/full/202/202329.png?modified=1622132158","width":1152,"height":557,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/202/202329.jpg?modified=1622132158","thumbnailWidth":533,"thumbnailHeight":258,"caption":"LibreJS showing the panel on a page with free JavaScript."}],"contributionURL":"","averageRating":4.25,"reviewCount":15,"reviewURL":"https://addons.mozilla.org/en-US/firefox/addon/librejs/reviews/","updateDate":1658394623000},{"id":"kkb-ext@fairlyterrible.name","icons":{"32":"https://addons.mozilla.org/user-media/addon_icons/980/980765-32.png?modified=2a69a5c9","64":"https://addons.mozilla.org/user-media/addon_icons/980/980765-64.png?modified=2a69a5c9","128":"https://addons.mozilla.org/user-media/addon_icons/980/980765-128.png?modified=2a69a5c9"},"name":"Keyboard Browsing","version":"1.2.0","sourceURI":"https://addons.mozilla.org/firefox/downloads/file/982967/keyboard_browsing-1.2.0.xpi","homepageURL":"https://github.com/potockyt/kbb-ext","supportURL":"https://github.com/potockyt/kbb-ext/issues","description":"Adds numbered navigation markers to a page.","fullDescription":null,"weeklyDownloads":1,"type":"extension","creator":{"name":"Tomas Potocky","url":"https://addons.mozilla.org/en-US/firefox/user/13580917/"},"developers":[],"screenshots":[{"url":"https://addons.mozilla.org/user-media/previews/full/203/203729.png?modified=1622133059","width":1270,"height":1035,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/203/203729.jpg?modified=1622133059","thumbnailWidth":491,"thumbnailHeight":400,"caption":"Numbered navigation markers in action"}],"contributionURL":"","averageRating":5,"reviewCount":1,"reviewURL":"https://addons.mozilla.org/en-US/firefox/addon/keyboard-browsing/reviews/","updateDate":1528547706000},{"id":"@keyboardsurfing","icons":{"32":"https://addons.mozilla.org/user-media/addon_icons/701/701020-32.png?modified=1511914780","64":"https://addons.mozilla.org/user-media/addon_icons/701/701020-64.png?modified=1511914780","128":"https://addons.mozilla.org/user-media/addon_icons/701/701020-128.png?modified=1511914780"},"name":"Keyboard Surfing","version":"1.3.20","sourceURI":"https://addons.mozilla.org/firefox/downloads/file/555072/keyboard_surfing-1.3.20.xpi","homepageURL":null,"supportURL":null,"description":"Easy keyboard web browsing with computer assisted cursor home-in. Mouseless browsing. Smart cursor-based link selection, simple keyboard navigation, single-hand control, laptop friendly.","fullDescription":"Provides a keyboard-controlled cursor for navigating and browsing web pages. See the demonstration video.\n\nThe cursor can be moved to nearby links by pressing the up, down, left and right keys. The addon will select nearby links on the page intelligently.\n\nThree sets of keys are provided for controlling the cursor and the addon: the numpad set, the left-hand set and the right-hand set.\n\nAdditional functions include: smooth scrolling, open links in new tab and in new window, close windows and tabs, go to homepage.","weeklyDownloads":0,"type":"extension","creator":{"name":"Somnia","url":"https://addons.mozilla.org/en-US/firefox/user/12273604/"},"developers":[],"screenshots":[{"url":"https://addons.mozilla.org/user-media/previews/full/173/173840.png?modified=1622132423","width":700,"height":457,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/173/173840.jpg?modified=1622132423","thumbnailWidth":533,"thumbnailHeight":348,"caption":"Help Page"},{"url":"https://addons.mozilla.org/user-media/previews/full/173/173841.png?modified=1622132423","width":660,"height":525,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/173/173841.jpg?modified=1622132423","thumbnailWidth":503,"thumbnailHeight":400,"caption":"Preferences Page"}],"contributionURL":"","averageRating":4.6364,"reviewCount":8,"reviewURL":"https://addons.mozilla.org/en-US/firefox/addon/keyboard-surfing/reviews/","updateDate":1481927177000},{"id":"simple_hinting@umaneti.net","icons":{"32":"https://addons.mozilla.org/user-media/addon_icons/695/695684-32.png?modified=1630517684","64":"https://addons.mozilla.org/user-media/addon_icons/695/695684-64.png?modified=1630517684","128":"https://addons.mozilla.org/user-media/addon_icons/695/695684-128.png?modified=1630517684"},"name":"Simple Hinting","version":"0.5.4","sourceURI":"https://addons.mozilla.org/firefox/downloads/file/3812461/simple_hinting-0.5.4.xpi","homepageURL":"https://git.umaneti.net/simple_hinting/about/","supportURL":"https://git.umaneti.net/simple_hinting/about/","description":"Allow keyboard browsing through links hinting.","fullDescription":"This addon displays small green numbered labels next to the links of the current page when you press Alt + Shift + L.\n\nIf you enter one of the displayed numbers, followed by a special keycode, you'll be redirected to the link target, either directly or by opening a new window or tab.\n\nThe keycodes are the following:\n\n    f (follow), browse in current tab;\n    t (tab), open in a new tab;\n    w (window), open in a new window;\n    i (incognito) or p (private), open in a new private window (only if you’ve authorized this addons to run on private windows);\n    v (view), don't do anything, just display the target URL.\n\n\nAt any time, pressing c or Esc will cancel the link selection and remove all green labels.\n\nPrivacy\n\nThis addon try to protect your privacy by removing unwanted tracking parameters from the URL you are trying to go to. Another widespread privacy invading behavior consists by hiding real links behind a short link, which is responsible to track who click on which link, and when. In this later case, this addon call an external web service, which will expand a tiny URL, before removing its tracking parameters, if any. That is why this addon ask for the permission to contact unshorten.umaneti.net host.\n\nTo avoid asking too frequently to this same remote web service what to do about a link, this addon will download from it two lists:\n\n    a list of known tracking parameters, which will help it to remove them from already expanded URLs without having to contact the remote service;\n    a list of known shorten URL services, for which this addons must contact the remote web service to resolve and clean them.\n\n\nThis two lists are then stored in your web browser addons storage area (hence the permission to use it).\n\nThe source code of the previously mentioned external web service is available too.","weeklyDownloads":0,"type":"extension","creator":{"name":"Étienne Deparis","url":"https://addons.mozilla.org/en-US/firefox/user/162211/"},"developers":[],"screenshots":[{"url":"https://addons.mozilla.org/user-media/previews/full/172/172680.png?modified=1622132531","width":700,"height":351,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/172/172680.jpg?modified=1622132531","thumbnailWidth":533,"thumbnailHeight":267,"caption":"Hinting is activated and the user just press \"5\" to select \"firefox\" link"},{"url":"https://addons.mozilla.org/user-media/previews/full/196/196467.png?modified=1622132532","width":502,"height":525,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/196/196467.jpg?modified=1622132532","thumbnailWidth":382,"thumbnailHeight":400,"caption":"Context menu item"},{"url":"https://addons.mozilla.org/user-media/previews/full/196/196468.png?modified=1622132532","width":389,"height":392,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/196/196468.jpg?modified=1622132532","thumbnailWidth":389,"thumbnailHeight":392},{"url":"https://addons.mozilla.org/user-media/previews/full/196/196469.png?modified=1622132532","width":380,"height":358,"thumbnailURL":"https://addons.mozilla.org/user-media/previews/thumbs/196/196469.jpg?modified=1622132532","thumbnailWidth":380,"thumbnailHeight":358,"caption":"The link is expanded and clean"}],"contributionURL":"https://paypal.me/milouse?utm_content=product-page-contribute&utm_medium=referral&utm_source=addons.mozilla.org","averageRating":2.5,"reviewCount":1,"reviewURL":"https://addons.mozilla.org/en-US/firefox/addon/simple-hinting/reviews/","updateDate":1626715250000}]}

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> That's not expected, no, and rather surprising because it would 
>> contradict what we've seen so far.  Can you post that file somewhere?
>
> It contains nothing particularly personal, so I've attached it.
>

Thanks.  A last question: what CPU does your computer use?  I do see a 
(very slight) stuttering here, certainly not "several seconds".

Anyway, this file is too small for you to see the effect/benefits of long 
line optimizations.  The cause of the delays you see is the "Bidirectional 
Parentheses Algorithm" which is used by Emacs, and which you can disable 
by setting bidi-inhibit-bpa to t.  It makes C-n and C-p an order of 
magnitude slower in that file.  On my computer and in that file, they take 
50 ms with bidi-inhibit-bpa nil and 5 ms with bidi-inhibit-bpa t.  So far 
I don't think these delays are annoying enough to disable that algorithm 
by default in files with long lines.  Another option might be to add some 
limits to the algorithm in buffers with long lines, without disabling it 
completely.

bug#56682: Fix the long lines font locking related slowdowns

[Stephen Berman]

On Wed, 10 Aug 2022 10:34:32 +0000 Gregory Heytings <gregory@heytings.org> wrote:

>>> That's not expected, no, and rather surprising because it would contradict
>>> what we've seen so far.  Can you post that file somewhere?
>>
>> It contains nothing particularly personal, so I've attached it.
>>
>
> Thanks.  A last question: what CPU does your computer use?  I do see a (very
> slight) stuttering here, certainly not "several seconds".

i7-8700 CPU @ 3.20GHz with 6 cores (12 processors).  (It's somewhat
throttled due to Meltdown and Spectre mitigations.)

> Anyway, this file is too small for you to see the effect/benefits of long line
> optimizations.  The cause of the delays you see is the "Bidirectional
> Parentheses Algorithm" which is used by Emacs, and which you can disable by
> setting bidi-inhibit-bpa to t.  It makes C-n and C-p an order of magnitude
> slower in that file.  On my computer and in that file, they take 50 ms with
> bidi-inhibit-bpa nil and 5 ms with bidi-inhibit-bpa t.  So far I don't think
> these delays are annoying enough to disable that algorithm by default in files
> with long lines.  Another option might be to add some limits to the algorithm
> in buffers with long lines, without disabling it completely.

Indeed, setting bidi-inhibit-bpa to t eliminates the delay.  Thanks.

Steve Berman

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Wed, 10 Aug 2022 10:34:32 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>, 
>     monnier@iro.umontreal.ca, Dmitry Gutov <dgutov@yandex.ru>
> 
> Another option might be to add some limits to the algorithm in
> buffers with long lines, without disabling it completely.

It already has such limits and other optimizations.  Inhibiting the
BPA is an option if it is known that the buffer doesn't include ant
R2L characters anywhere; otherwise some braces/parens could be
displayed incorrectly mirrored.

If you remove the 3 closing braces/brackets at EOB and the
corresponding opening braces/brackets, does navigation become
significantly faster?  These are the only brackets I could spot in the
file that enclose very large portions of text.  If removing them
doesn't help much, the only possible reason is that there are a lot of
bracketed stuff in this file.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 10.08.2022 13:55, Stephen Berman wrote:
> Indeed, setting bidi-inhibit-bpa to t eliminates the delay.  Thanks.

Same here.

And the "faster computer" didn't help, apparently.

After flooring 'C-n', I even had to 'C-g' out of it for Emacs to regain 
responsiveness, because waiting for a while didn't help.

bug#56682: Fix the long lines font locking related slowdowns

[Stephen Berman]

On Wed, 10 Aug 2022 14:44:59 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> Date: Wed, 10 Aug 2022 10:34:32 +0000
>> From: Gregory Heytings <gregory@heytings.org>
>> cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>>     monnier@iro.umontreal.ca, Dmitry Gutov <dgutov@yandex.ru>
>>
>> Another option might be to add some limits to the algorithm in
>> buffers with long lines, without disabling it completely.
>
> It already has such limits and other optimizations.  Inhibiting the
> BPA is an option if it is known that the buffer doesn't include ant
> R2L characters anywhere; otherwise some braces/parens could be
> displayed incorrectly mirrored.
>
> If you remove the 3 closing braces/brackets at EOB and the
> corresponding opening braces/brackets, does navigation become
> significantly faster?

After removing those brackets, holding down `C-n' or `C-p' does seem a
bit faster with bidi-inhibit-bpa nil, but not anywhere near as fast as
with bidi-inhibit-bpa non-nil.

>                        These are the only brackets I could spot in the
> file that enclose very large portions of text.  If removing them
> doesn't help much, the only possible reason is that there are a lot of
> bracketed stuff in this file.

I suppose so, though I guess this is a pretty typical one-line JSON
file.

Steve Berman

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Wed, 10 Aug 2022 14:54:51 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 10.08.2022 13:55, Stephen Berman wrote:
> > Indeed, setting bidi-inhibit-bpa to t eliminates the delay.  Thanks.
> 
> Same here.
> 
> And the "faster computer" didn't help, apparently.
> 
> After flooring 'C-n', I even had to 'C-g' out of it for Emacs to regain 
> responsiveness, because waiting for a while didn't help.

That's strange, because on my "slower" computer I didn't see anything
like that.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: Gregory Heytings <gregory@heytings.org>,  56682@debbugs.gnu.org,
>   monnier@iro.umontreal.ca,  dgutov@yandex.ru
> Date: Wed, 10 Aug 2022 13:59:45 +0200
> 
> >                        These are the only brackets I could spot in the
> > file that enclose very large portions of text.  If removing them
> > doesn't help much, the only possible reason is that there are a lot of
> > bracketed stuff in this file.
> 
> I suppose so, though I guess this is a pretty typical one-line JSON
> file.

I think it's quite unique in that its bracketed segments are very
long.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 10.08.2022 15:04, Eli Zaretskii wrote:
>> Date: Wed, 10 Aug 2022 14:54:51 +0300
>> Cc:56682@debbugs.gnu.org, Eli Zaretskii<eliz@gnu.org>,
>>   monnier@iro.umontreal.ca
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> On 10.08.2022 13:55, Stephen Berman wrote:
>>> Indeed, setting bidi-inhibit-bpa to t eliminates the delay.  Thanks.
>> Same here.
>>
>> And the "faster computer" didn't help, apparently.
>>
>> After flooring 'C-n', I even had to 'C-g' out of it for Emacs to regain
>> responsiveness, because waiting for a while didn't help.
> That's strange, because on my "slower" computer I didn't see anything
> like that.

I think that just confirms what I said: the test "floor the button and 
see of UI freezes" is not a good one because it doesn't work against any 
objective numbers. It compares the speed of command execution with the 
speed of redisplay, and there is no backpressure mechanism.

If the "faster computer" executes the commands faster as well, it can 
likewise enter the unresponsive state.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Another option might be to add some limits to the algorithm in buffers 
>> with long lines, without disabling it completely.
>
> It already has such limits and other optimizations.
>

Yes, I meant making them a bit lower in such cases.

>
> Inhibiting the BPA is an option if it is known that the buffer doesn't 
> include ant R2L characters anywhere; otherwise some braces/parens could 
> be displayed incorrectly mirrored.
>

But is there an efficient way to determine if that the buffer doesn't 
include any R2L characters?

>
> If you remove the 3 closing braces/brackets at EOB and the corresponding 
> opening braces/brackets, does navigation become significantly faster?
>

Yes, removing the 3 brackets at positions 1, 22, 23 and 19538, 19539 and 
19540 solves that problem.  In fact, just removing the brackets at 1, 22 
and 23 solves that problem.  You can see the same problem in 
dictionary.json, just removing the first character makes C-n and C-p 
faster at BOB.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Wed, 10 Aug 2022 12:31:24 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, stephen.berman@gmx.net, monnier@iro.umontreal.ca, 
>     dgutov@yandex.ru
> 
> > Inhibiting the BPA is an option if it is known that the buffer doesn't 
> > include ant R2L characters anywhere; otherwise some braces/parens could 
> > be displayed incorrectly mirrored.
> 
> But is there an efficient way to determine if that the buffer doesn't 
> include any R2L characters?

Not without searching the buffer, not that I know of.  If the buffer
includes only ASCII characters (which this one doesn't), i.e. its
sizes in characters and bytes are identical, we could deduce that
there are no R2L characters without any heavy lifting.

> > If you remove the 3 closing braces/brackets at EOB and the corresponding 
> > opening braces/brackets, does navigation become significantly faster?
> 
> Yes, removing the 3 brackets at positions 1, 22, 23 and 19538, 19539 and 
> 19540 solves that problem.  In fact, just removing the brackets at 1, 22 
> and 23 solves that problem.  You can see the same problem in 
> dictionary.json, just removing the first character makes C-n and C-p 
> faster at BOB.

Stephen said it had a much smaller effect than inhibiting the BPA.
But yes, very long bracketed text segments could be a problem.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> But is there an efficient way to determine if that the buffer doesn't 
>> include any R2L characters?
>
> Not without searching the buffer, not that I know of.
>

Thanks, that's what I thought.

>>> If you remove the 3 closing braces/brackets at EOB and the 
>>> corresponding opening braces/brackets, does navigation become 
>>> significantly faster?
>>
>> Yes, removing the 3 brackets at positions 1, 22, 23 and 19538, 19539 
>> and 19540 solves that problem.  In fact, just removing the brackets at 
>> 1, 22 and 23 solves that problem.  You can see the same problem in 
>> dictionary.json, just removing the first character makes C-n and C-p 
>> faster at BOB.
>
> Stephen said it had a much smaller effect than inhibiting the BPA. But 
> yes, very long bracketed text segments could be a problem.
>

I'm not completely sure he did indeed remove those three brackets. 
Stephen, can you remove the three characters at positions 1, 22 and 23, 
and try again?  These are the first two '{' and the first '[' characters 
in the buffer.

bug#56682: Fix the long lines font locking related slowdowns

[Stephen Berman]

On Wed, 10 Aug 2022 12:48:49 +0000 Gregory Heytings <gregory@heytings.org> wrote:

>>>> If you remove the 3 closing braces/brackets at EOB and the corresponding
>>>> opening braces/brackets, does navigation become significantly faster?
>>>
>>> Yes, removing the 3 brackets at positions 1, 22, 23 and 19538, 19539 and
>>> 19540 solves that problem.  In fact, just removing the brackets at 1, 22
>>> and 23 solves that problem.  You can see the same problem in
>>> dictionary.json, just removing the first character makes C-n and C-p faster
>>> at BOB.
>>
>> Stephen said it had a much smaller effect than inhibiting the BPA. But yes,
>> very long bracketed text segments could be a problem.
>>
>
> I'm not completely sure he did indeed remove those three brackets. Stephen,
> can you remove the three characters at positions 1, 22 and 23, and try again?
> These are the first two '{' and the first '[' characters in the buffer.

I did remove those brackets previously; the improvement I see is that,
without them (and even without removing the corresponding closing
brackets at the end of the file, and of course with bidi-inhibit-bpa
nil), holding down `C-n' moves the cursor almost a whole screenful (~60
visual lines) before coming to a halt, instead of just three visual
lines with the initial brackets, and after letting go of `C-n', the
cursor catches up to the actual position of point noticeably quicker
than with the unaltered file.  Still, the movement is not at all smooth,
unlike with bidi-inhibit-bpa non-nil (and that is not as smooth as
moving through the same file with so-long-mode enabled).

Steve Berman

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> I'm not completely sure he did indeed remove those three brackets. 
>> Stephen, can you remove the three characters at positions 1, 22 and 23, 
>> and try again? These are the first two '{' and the first '[' characters 
>> in the buffer.
>
> I did remove those brackets previously; the improvement I see is that, 
> without them (and even without removing the corresponding closing 
> brackets at the end of the file, and of course with bidi-inhibit-bpa 
> nil), holding down `C-n' moves the cursor almost a whole screenful (~60 
> visual lines) before coming to a halt, instead of just three visual 
> lines with the initial brackets, and after letting go of `C-n', the 
> cursor catches up to the actual position of point noticeably quicker 
> than with the unaltered file.  Still, the movement is not at all smooth, 
> unlike with bidi-inhibit-bpa non-nil (and that is not as smooth as 
> moving through the same file with so-long-mode enabled).
>

Is that with emacs -Q? And with show-paren-mode disabled?

bug#56682: Fix the long lines font locking related slowdowns

[Stephen Berman]

On Wed, 10 Aug 2022 13:11:52 +0000 Gregory Heytings <gregory@heytings.org> wrote:

>>> I'm not completely sure he did indeed remove those three brackets. Stephen,
>>> can you remove the three characters at positions 1, 22 and 23, and try
>>> again? These are the first two '{' and the first '[' characters in the
>>> buffer.
>>
>> I did remove those brackets previously; the improvement I see is that,
>> without them (and even without removing the corresponding closing brackets
>> at the end of the file, and of course with bidi-inhibit-bpa nil), holding
>> down `C-n' moves the cursor almost a whole screenful (~60 visual lines)
>> before coming to a halt, instead of just three visual lines with the initial
>> brackets, and after letting go of `C-n', the cursor catches up to the actual
>> position of point noticeably quicker than with the unaltered file.  Still,
>> the movement is not at all smooth, unlike with bidi-inhibit-bpa non-nil (and
>> that is not as smooth as moving through the same file with so-long-mode
>> enabled).
>>
>
> Is that with emacs -Q? And with show-paren-mode disabled?

I had forgotten to disable show-paren-mode; doing that (with -Q) seems
to increase the length of the initial scrolling but it still comes to a
halt and there's a delay (though no longer several seconds) before the
cursor catches up.  Further scrolling in the file seems not to show the
same improvement (i.e., scrolling halts again after ~3 visual lines),
though the file is too short to do that more than once or twice.

Steve Berman

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I had forgotten to disable show-paren-mode; doing that (with -Q) seems 
> to increase the length of the initial scrolling but it still comes to a 
> halt and there's a delay (though no longer several seconds) before the 
> cursor catches up.  Further scrolling in the file seems not to show the 
> same improvement (i.e., scrolling halts again after ~3 visual lines), 
> though the file is too short to do that more than once or twice.
>

I can now reproduce what you describe.  The short halt you see at some 
point is AFAIU due to fontification, which (for some reason) takes a bit 
longer at some point in the file.  Note that the goal of long line 
optimizations is to get the best possible performance while disabling as 
least functionalities as possible.  It is not to become as fast as what 
you would see with M-x find-find-literally in all circumstances, which 
wouldn't be possible.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 10.08.2022 15:43, Eli Zaretskii wrote:
>> Date: Wed, 10 Aug 2022 12:31:24 +0000
>> From: Gregory Heytings<gregory@heytings.org>
>> cc:56682@debbugs.gnu.org,stephen.berman@gmx.net,monnier@iro.umontreal.ca,
>>      dgutov@yandex.ru
>>
>>> Inhibiting the BPA is an option if it is known that the buffer doesn't
>>> include ant R2L characters anywhere; otherwise some braces/parens could
>>> be displayed incorrectly mirrored.
>> But is there an efficient way to determine if that the buffer doesn't
>> include any R2L characters?
> Not without searching the buffer, not that I know of.  If the buffer
> includes only ASCII characters (which this one doesn't), i.e. its
> sizes in characters and bytes are identical, we could deduce that
> there are no R2L characters without any heavy lifting.

Shouldn't the scanner be limited by the same narrowing that the 
long-line optimizations create? Given its existing effect on font-lock, 
limiting bidi searches seems like a no-brainer.

Alternatively, I would perhaps propose to perform said searching at two 
junctions: when the buffer is visited (and the text is inserted and, 
thus, unavoidably read), and when the buffer is saved (perhaps at that 
point Emacs might have info about the added ranges, to avoid searching 
through the whole buffer). The downside would be that the first added 
R2L character might be displayed incorrectly until the buffer is saved.

But the upside would be pretty huge.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Wed, 10 Aug 2022 18:44:53 +0300
> Cc: 56682@debbugs.gnu.org, stephen.berman@gmx.net, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> Shouldn't the scanner be limited by the same narrowing that the 
> long-line optimizations create? Given its existing effect on font-lock, 
> limiting bidi searches seems like a no-brainer.

Like I said: it _is_ limited.  But the limitation is smarter than just
some arbitrary number of character positions, because it is tuned to
this particular processing.

Note that the effect of it in this case is not as dramatic as
font-lock: it does slow down C-n/C-p, but not catastrophically so, not
like we've seen with the files we used for the long-lines speedups.
And if someone does think it's a catastrophe, they can always disable
the BPA, perhaps even globally, if they know they will never need to
look closely at bidirectional text: the effects are hardly visible
unless you actually read the text.

> Alternatively, I would perhaps propose to perform said searching at two 
> junctions: when the buffer is visited (and the text is inserted and, 
> thus, unavoidably read), and when the buffer is saved (perhaps at that 
> point Emacs might have info about the added ranges, to avoid searching 
> through the whole buffer). The downside would be that the first added 
> R2L character might be displayed incorrectly until the buffer is saved.

This is impossible without a complete redesign of how the bidi display
works: it saves no information in the buffer object, none whatsoever.
The information is recomputed every time the display code is called,
and only for the portion of the buffer that needs to be traversed or
displayed; then it is discarded.  There are no caches of any kind that
keep the information after redisplay has done its job.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 10.08.2022 19:08, Eli Zaretskii wrote:
>> Date: Wed, 10 Aug 2022 18:44:53 +0300
>> Cc: 56682@debbugs.gnu.org, stephen.berman@gmx.net, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> Shouldn't the scanner be limited by the same narrowing that the
>> long-line optimizations create? Given its existing effect on font-lock,
>> limiting bidi searches seems like a no-brainer.
> 
> Like I said: it _is_ limited.  But the limitation is smarter than just
> some arbitrary number of character positions, because it is tuned to
> this particular processing.
> 
> Note that the effect of it in this case is not as dramatic as
> font-lock: it does slow down C-n/C-p, but not catastrophically so,

It's much worse. In the aforementioned dictionary.json, with 
fully-featured font-lock (narrowing disabled in handle_fontified_props) 
the buffer is perfectly responsive to edit as long as I (setq 
bidi-inhibit-bpa nil). Even with show-paren-mode enabled.

Without that, however, simple navigation commands take a significant 
amount of time. These measurements are with show-paren-mode off:

Near BOB:

(benchmark 1 '(next-line 1)) =>
Elapsed time: 0.246551s
Elapsed time: 0.247237s
Elapsed time: 0.247392s

Near EOB:

(benchmark 1 '(next-line 1)) =>
Elapsed time: 0.059583s
Elapsed time: 0.040372s
Elapsed time: 0.059508s

> not
> like we've seen with the files we used for the long-lines speedups.
> And if someone does think it's a catastrophe, they can always disable
> the BPA, perhaps even globally,

That reminds me of "the user can always disable font-lock" and your 
dismissal of Alan's advice to change font-lock-maximum-decoration to 2, 
saying that we should have good editing performance OOTB.

> if they know they will never need to
> look closely at bidirectional text: the effects are hardly visible
> unless you actually read the text.

We could add (setq bidi-inhibit-bpa nil) to prog-mode, I suppose.

It already has a bidi-paragraph-direction setting anyway.

>> Alternatively, I would perhaps propose to perform said searching at two
>> junctions: when the buffer is visited (and the text is inserted and,
>> thus, unavoidably read), and when the buffer is saved (perhaps at that
>> point Emacs might have info about the added ranges, to avoid searching
>> through the whole buffer). The downside would be that the first added
>> R2L character might be displayed incorrectly until the buffer is saved.
> 
> This is impossible without a complete redesign of how the bidi display
> works: it saves no information in the buffer object, none whatsoever.
> The information is recomputed every time the display code is called,
> and only for the portion of the buffer that needs to be traversed or
> displayed; then it is discarded.  There are no caches of any kind that
> keep the information after redisplay has done its job.

Perhaps a cache similar to syntax-ppss one could do the job? Not sure 
what would be the trafeoffs, though. Like, how that would change the 
performance in the simple case.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Wed, 10 Aug 2022 20:02:20 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, stephen.berman@gmx.net,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> It's much worse. In the aforementioned dictionary.json, with 
> fully-featured font-lock (narrowing disabled in handle_fontified_props) 
> the buffer is perfectly responsive to edit as long as I (setq 
> bidi-inhibit-bpa nil). Even with show-paren-mode enabled.
> 
> Without that, however, simple navigation commands take a significant 
> amount of time. These measurements are with show-paren-mode off:
> 
> Near BOB:
> 
> (benchmark 1 '(next-line 1)) =>
> Elapsed time: 0.246551s
> Elapsed time: 0.247237s
> Elapsed time: 0.247392s
> 
> Near EOB:
> 
> (benchmark 1 '(next-line 1)) =>
> Elapsed time: 0.059583s
> Elapsed time: 0.040372s
> Elapsed time: 0.059508s

That's what I meant: it isn't a catastrophe.  1/4th of a second for
C-n is barely perceptible.

"Catastrophe" in my book is when it takes more than a couple of
seconds, like 10 sec or more.

> > not
> > like we've seen with the files we used for the long-lines speedups.
> > And if someone does think it's a catastrophe, they can always disable
> > the BPA, perhaps even globally,
> 
> That reminds me of "the user can always disable font-lock" and your 
> dismissal of Alan's advice to change font-lock-maximum-decoration to 2, 
> saying that we should have good editing performance OOTB.

No, because font-lock is much more important than the effect of the
BPA.  (Do you even understand what the BPA does, and did you ever see
it in action?  Without that, this part of the discussion is just waste
of time.)

> > if they know they will never need to
> > look closely at bidirectional text: the effects are hardly visible
> > unless you actually read the text.
> 
> We could add (setq bidi-inhibit-bpa nil) to prog-mode, I suppose.

Not to prog-mode, that I won't agree.  But maybe for JSON files, if we
think they are unlikely to suffer.

> It already has a bidi-paragraph-direction setting anyway.

That's not even similar.  That's just saving Emacs to figure that out
itself, wasting CPU cycles on what is known in advance.  Because a PL
buffer _always_ has its paragraphs left-to-right.

> > This is impossible without a complete redesign of how the bidi display
> > works: it saves no information in the buffer object, none whatsoever.
> > The information is recomputed every time the display code is called,
> > and only for the portion of the buffer that needs to be traversed or
> > displayed; then it is discarded.  There are no caches of any kind that
> > keep the information after redisplay has done its job.
> 
> Perhaps a cache similar to syntax-ppss one could do the job? Not sure 
> what would be the trafeoffs, though. Like, how that would change the 
> performance in the simple case.

There are no caches for a good reason: the Unicode Bidirectional
Algorithm that we implement results in very far-reaching non-local
effects of small changes.  Even inserting or deleting a single
character can dramatically change how the buffer looks on display very
far from the locus of the change.  Figuring out which changes
invalidate which caches is extremely non-trivial, and in many cases
requires the same amount of work as would take to just recompute
everything from scratch.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 10.08.2022 20:19, Eli Zaretskii wrote:
>> Date: Wed, 10 Aug 2022 20:02:20 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, stephen.berman@gmx.net,
>>   monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> It's much worse. In the aforementioned dictionary.json, with
>> fully-featured font-lock (narrowing disabled in handle_fontified_props)
>> the buffer is perfectly responsive to edit as long as I (setq
>> bidi-inhibit-bpa nil). Even with show-paren-mode enabled.
>>
>> Without that, however, simple navigation commands take a significant
>> amount of time. These measurements are with show-paren-mode off:
>>
>> Near BOB:
>>
>> (benchmark 1 '(next-line 1)) =>
>> Elapsed time: 0.246551s
>> Elapsed time: 0.247237s
>> Elapsed time: 0.247392s
>>
>> Near EOB:
>>
>> (benchmark 1 '(next-line 1)) =>
>> Elapsed time: 0.059583s
>> Elapsed time: 0.040372s
>> Elapsed time: 0.059508s
> 
> That's what I meant: it isn't a catastrophe.  1/4th of a second for
> C-n is barely perceptible.

It's 2.5 the upper limit on what's perceived as "instant". And it's 
higher than the "uncomfortable" font-lock delays I have seen.

> "Catastrophe" in my book is when it takes more than a couple of
> seconds, like 10 sec or more.

For such a basic operation as next-line, it seems excessive to me. But 
if you're sure about this, all right.

This delay doesn't seem to extend to other commands, such as C-f/C-b or 
self-insert-command. It does seem to affect isearch, though.

>>> not
>>> like we've seen with the files we used for the long-lines speedups.
>>> And if someone does think it's a catastrophe, they can always disable
>>> the BPA, perhaps even globally,
>>
>> That reminds me of "the user can always disable font-lock" and your
>> dismissal of Alan's advice to change font-lock-maximum-decoration to 2,
>> saying that we should have good editing performance OOTB.
> 
> No, because font-lock is much more important than the effect of the
> BPA.  (Do you even understand what the BPA does, and did you ever see
> it in action?  Without that, this part of the discussion is just waste
> of time.)

I have looked at examples now. Seems important for text in pertinent 
human languages.

>>> if they know they will never need to
>>> look closely at bidirectional text: the effects are hardly visible
>>> unless you actually read the text.
>>
>> We could add (setq bidi-inhibit-bpa nil) to prog-mode, I suppose.
> 
> Not to prog-mode, that I won't agree.  But maybe for JSON files, if we
> think they are unlikely to suffer.

Sure, maybe.

Where do we draw the line, though? Not in prog-mode because a .js file, 
say, can have comments in Persian or Hebrew?

By that measurement, JSON is fine (no comments supported). But JSON 
contains a bunch of strings, could store localizations, etc.
I suppose the contents of said strings can store i18n text, with the 
expectation of a user being able to edit it.

>> It already has a bidi-paragraph-direction setting anyway.
> 
> That's not even similar.  That's just saving Emacs to figure that out
> itself, wasting CPU cycles on what is known in advance.  Because a PL
> buffer _always_ has its paragraphs left-to-right.
> 
>>> This is impossible without a complete redesign of how the bidi display
>>> works: it saves no information in the buffer object, none whatsoever.
>>> The information is recomputed every time the display code is called,
>>> and only for the portion of the buffer that needs to be traversed or
>>> displayed; then it is discarded.  There are no caches of any kind that
>>> keep the information after redisplay has done its job.
>>
>> Perhaps a cache similar to syntax-ppss one could do the job? Not sure
>> what would be the trafeoffs, though. Like, how that would change the
>> performance in the simple case.
> 
> There are no caches for a good reason: the Unicode Bidirectional
> Algorithm that we implement results in very far-reaching non-local
> effects of small changes.  Even inserting or deleting a single
> character can dramatically change how the buffer looks on display very
> far from the locus of the change.

In both directions?

> Figuring out which changes
> invalidate which caches is extremely non-trivial, and in many cases
> requires the same amount of work as would take to just recompute
> everything from scratch.

Sounds a little similar to syntax-ppss cache: changing a character near 
BOB invalidates the whole cache for all positions > POS.

This one sounds more complex, though.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Wed, 10 Aug 2022 21:54:04 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, stephen.berman@gmx.net,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > No, because font-lock is much more important than the effect of the
> > BPA.  (Do you even understand what the BPA does, and did you ever see
> > it in action?  Without that, this part of the discussion is just waste
> > of time.)
> 
> I have looked at examples now. Seems important for text in pertinent 
> human languages.

Yes.  Which JSON isn't.  But most PL modes have comments and strings,
and that's where this is important.

> >> We could add (setq bidi-inhibit-bpa nil) to prog-mode, I suppose.
> > 
> > Not to prog-mode, that I won't agree.  But maybe for JSON files, if we
> > think they are unlikely to suffer.
> 
> Sure, maybe.
> 
> Where do we draw the line, though? Not in prog-mode because a .js file, 
> say, can have comments in Persian or Hebrew?
>
> By that measurement, JSON is fine (no comments supported). But JSON 
> contains a bunch of strings, could store localizations, etc.

I presume that JSON files are rarely for human consumption, so the
strings are not very important.  And they will rarely include R2L
characters to begin with.

> I suppose the contents of said strings can store i18n text, with the 
> expectation of a user being able to edit it.

If and when that happens, users can turn BPA back on.

> > There are no caches for a good reason: the Unicode Bidirectional
> > Algorithm that we implement results in very far-reaching non-local
> > effects of small changes.  Even inserting or deleting a single
> > character can dramatically change how the buffer looks on display very
> > far from the locus of the change.
> 
> In both directions?

Yes (IIUC what you mean by that).  The effect back happens rarer than
forward, but it does happen.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 07.08.2022 21:38, Eli Zaretskii wrote:
>> Date: Sun, 7 Aug 2022 21:27:21 +0300
>> Cc:56682@debbugs.gnu.org,gregory@heytings.org,monnier@iro.umontreal.ca
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>>> feel free to push a feature branch with your proposed
>>> changes, or even install them as optional features on master, and
>>> let's compare the merits and demerits of each idea in practical use
>>> cases and on a variety of systems.
>> I will.
> Thanks.

I've pushed a branch scratch/font_lock_large_files.

Please try out different values of the new variable.

Preferably test together with (setq bidi-inhibit-bpa t), to eliminate 
its contribution.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I've pushed a branch scratch/font_lock_large_files.
>

Thank you.  But as I told you earlier, what that branch contains is a too 
local fix.  You created a new minimal major mode to edit JSON files, in 
which you removed everything that made font locking slow in js-mode.  As a 
consequence, now editing the dictionary.json file is indeed reasonably 
fast without disabling font locking or using locked narrowing (although it 
is still too slow to my taste for larger files, e.g. I have to wait five 
seconds after M-> in a ten times larger JSON file).  And the original 
problem is still present if you edit, for example, a 20 MB minified 
JavaScript file...

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

[-- Attachment #1: Type: text/html, Size: 2180 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: Eli Zaretskii <eliz@gnu.org>,
> 	"56682@debbugs.gnu.org" <56682@debbugs.gnu.org>,
> 	"monnier@iro.umontreal.ca" <monnier@iro.umontreal.ca>
> Date: Fri, 12 Aug 2022 19:35:57 +0300
> 
> I have eliminated two performance problems frim js-json-mode that are uncharacteristic for font-lock in
> general. One used a pathological matcher (simply removed), and another uses the "frameworks" facility
> which simply always rescans the full buffer from the beginning. That one can/should be fixed later, but I have
> disabled it for JSON (where it doesn't make sense anyway).
> 
> You should know that the rest of js-mode's font-lock keywords are still there. I experimented with leaving just
> the bare minimum needed for JSON, but the effect was very minimal (like 20% improvement).

I didn't yet have time to try the branch, but what you say begs a
question: does the branch attempt to solve the problems of JSON files
only, or does it attempt to solve a more general problem?

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

[-- Attachment #1: Type: text/html, Size: 1717 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

> 
> I have eliminated two performance problems frim js-json-mode that are 
> uncharacteristic for font-lock in general.
>

This is where we disagree: they _are_ characteristic of font locking in 
general, in all modes (in core and in the wild) that use widen in their 
font locking routines and/or do complex calculations far away from the 
region that must be fontified.

>> although it is still too slow to my taste for larger files, e.g. I have 
>> to wait five seconds after M-> in a ten times larger JSON file). And 
>> the original problem is still present if you edit, for example, a 20 MB 
>> minified JavaScript file...
>
> Hence the new user option.
>

The goal of this effort is not, and never was, to make editing the 
dictionary.json file reasonably fast with an optimized build on a Core i9 
computer.  That file is, again, just an example.  The goal of this effort 
is to make editing any large file as smooth as possible, out of the box, 
regardless of the major (and minor) modes that are are enabled, even with 
a less efficient CPU, e.g. a Raspberry Pi computer, or a lower end laptop 
with its CPU running at a lower frequency.

Which is why fixing js-mode, and adding a json-mode, as you did, is a "too 
local" fix.  Of course, that doesn't mean what you did is useless; it only 
means that it cannot be considered as a general solution to the problem at 
hand.

I can only repeat that the only ways to improve font locking in a 
non-local way (or at least, the only ways that were identified in the 
discussions so far) are either to improve the speed of syntax-ppss by an 
order of magnitude, in which case it would perhaps become feasible to 
allow syntax-ppss to have access to the whole buffer, or to equip 
syntax-ppss with heuristics to make it work "as well as possible" even 
when it only has access to a portion of the buffer.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> I didn't yet have time to try the branch, but what you say begs a 
>> question: does the branch attempt to solve the problems of JSON files 
>> only, or does it attempt to solve a more general problem?
> 
> There is nothing json-specific in the branch. All the js/json stuff is 
> on master already.
>

That doesn't answer Eli's question, AFAIU.  The branch demonstrates that, 
with the Javascript/JSON improvements you installed recently on master, 
editing the 20 MB dictionary.json file without using locked narrowing 
around fontification-functions is reasonably efficient.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I can only repeat that the only ways to improve font locking in a non-local
> way (or at least, the only ways that were identified in the discussions so
> far) are either to improve the speed of syntax-ppss by an order of
> magnitude, in which case it would perhaps become feasible to allow
> syntax-ppss to have access to the whole buffer, or to equip syntax-ppss with
> heuristics to make it work "as well as possible" even when it only has
> access to a portion of the buffer.

But the locked narrowing is applied based on the presence of long-lines
rather than based on the total size of the buffer.

I think we'll want to decouple the two because they aren't quite the
same (e.g. in many cases the time taken by `syntax-ppss` doesn't depend
on the length of lines, whereas the time taken by `font-lock-keywords`
rues does not depend on the total buffer size).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Fri, 12 Aug 2022 21:41:36 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, 56682@debbugs.gnu.org, 
>     monnier@iro.umontreal.ca
> 
> The branch demonstrates that, with the Javascript/JSON improvements
> you installed recently on master, editing the 20 MB dictionary.json
> file without using locked narrowing around fontification-functions
> is reasonably efficient.

Is 20MB single-line JSON file fairly representative of such files out
there?  Do we know of JSON file with much longer lines, for example
50MB or 100MB?

If 20MB is reasonably representative of the long-line JSON, then
perhaps at least json-mode could consider resetting
long-line-threshold to nil.  But if there are significantly larger
files out there, we should first assess the performance in those
cases.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  Eli Zaretskii <eliz@gnu.org>,
>   56682@debbugs.gnu.org
> Date: Fri, 12 Aug 2022 22:01:54 -0400
> 
> > I can only repeat that the only ways to improve font locking in a non-local
> > way (or at least, the only ways that were identified in the discussions so
> > far) are either to improve the speed of syntax-ppss by an order of
> > magnitude, in which case it would perhaps become feasible to allow
> > syntax-ppss to have access to the whole buffer, or to equip syntax-ppss with
> > heuristics to make it work "as well as possible" even when it only has
> > access to a portion of the buffer.
> 
> But the locked narrowing is applied based on the presence of long-lines
> rather than based on the total size of the buffer.

That's true, but the two are closely related: a buffer with very long
lines tends also to be a large buffer.

> I think we'll want to decouple the two because they aren't quite the
> same (e.g. in many cases the time taken by `syntax-ppss` doesn't depend
> on the length of lines, whereas the time taken by `font-lock-keywords`
> rues does not depend on the total buffer size).

The solutions for stuff that depends on buffer size will indeed be
different from solutions for stuff that depends on line size.
However, font-lock currently depends on both, and narrowing solves
both, albeit in non-specific ways.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

[-- Attachment #1: Type: text/html, Size: 1269 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[dgutov]

[-- Attachment #1: Type: text/html, Size: 1749 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

[-- Attachment #1: Type: text/html, Size: 1079 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

[-- Attachment #1: Type: text/html, Size: 3524 bytes --]

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Is 20MB single-line JSON file fairly representative of such files out
> there?  Do we know of JSON file with much longer lines, for example
> 50MB or 100MB?

IIUC it's very common for JSON files to be made of a single line, so the
line length is the file's length and since those files are
machine-generated I don't expect 100MB to be much less common than 20MB.

> If 20MB is reasonably representative of the long-line JSON, then
> perhaps at least json-mode could consider resetting
> long-line-threshold to nil.

IIUC a nil value for `long-line-threshold` makes Emacs completely
unusable with such long lines even if you disable all the bells
and whistles.  Dmitry's suggestion is to skip the locked-narrowing in
some cases, but not to set `long-line-threshold` to nil.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I think we'll want to decouple the two because they aren't quite the 
> same (e.g. in many cases the time taken by `syntax-ppss` doesn't depend 
> on the length of lines, whereas the time taken by `font-lock-keywords` 
> rues does not depend on the total buffer size).
>

That's what I initially thought, too.  Now I tend to think that it would 
be better to not decouple these two cases, because it would be too 
expensive to distinguish the four possible cases, namely:

1. "normal buffers"

2. non-large buffers with long lines

3. large buffers without long lines

4. large buffers with long lines

It is probably better to group the three last cases in a single group, and 
to use the same optimization techniques for these three cases.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> But the locked narrowing is applied based on the presence of long-lines
>> rather than based on the total size of the buffer.
> That's true, but the two are closely related: a buffer with very long
> lines tends also to be a large buffer.

They are related but I don't think it "closely"-enough to conflate the
two: e.g. many large buffers have no long lines (e.g. log files) and as
Dmitry mentions the thresholds are quite different (back-of-the-envelope
suggests a factor 100 or so).

> The solutions for stuff that depends on buffer size will indeed be
> different from solutions for stuff that depends on line size.
> However, font-lock currently depends on both,

Various parts of it depends on only one of the two, tho.

> and narrowing solves both, albeit in non-specific ways.

And I'm all for it as a blunt-but-effective "fallback", as long as it
can be selectively overridden from ELisp.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-08-13 13:42:25] wrote:
> 1. "normal buffers"
> 2. non-large buffers with long lines
> 3. large buffers without long lines
> 4. large buffers with long lines
>
> It is probably better to group the three last cases in a single group, and
> to use the same optimization techniques for these three cases.

(with-half-joking-mode
  Hmm... so should be increase `syntax-wholeline-max` from 10kB to 10MB or
  should we decrease `large-file-warning-threshold` from 10MB down
  to 10kB?)


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

> 
> But they're not. Of course, modes might have problematic rules 
> (font-lock-keywords are turing-complete, after all), but it might be 
> just as easy to fix all the main major modes rather than hobble user 
> experience.
>

You've made it quite clear that in your opinion incorrect font locking 
hobbles user experience more than multi-second delays between motion or 
search commands.  Let's agree to disagree.

>
> Especially if we limit ourselves to modes that are likely to be used 
> with large files.
>

That's not the intention, no.

>
> So please go ahead and experiment, and report on your findings later.
>

I already did, and told you that these multi-second delays were still 
present in larger JSON files (and therefore also in smaller files with 
slower CPUs).

>
> Then the goal failed because we don't disable bidi hpa? It has a much 
> more noticeable effect on editing that font-lock.
>

That's not correct, no.  The BPA algorithm makes C-n and C-p slower at the 
beginning of large files only in a very specific case: when the beginning 
of the buffer contains opening brackets whose matching closing bracket is 
at the end of the buffer.  And the BPA algorithm does not cause 
multi-second delays.

>
> And of course if we wanted to make everything as smooth as possible on 
> every possible machine down to TI calculators,
>

That's not what I aim at, and I think you understood that.

>> Which is why fixing js-mode, and adding a json-mode, as you did, is a 
>> "too local" fix. Of course, that doesn't mean what you did is useless; 
>> it only means that it cannot be considered as a general solution to the 
>> problem at hand.
> 
> That's not what the branch contains.
>

You're playing with words.  The fixes to js-mode and the added json-mode 
are not on the branch because you installed them on master, but what you 
claim the branch demonstrates depends on what you installed on master. 
Or do you claim that what the branch contains makes editing any other file 
(that is, not a .js or a .json file) almost as fast as what is on master?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Gregory Heytings <gregory@heytings.org>,  dgutov@yandex.ru,
>   56682@debbugs.gnu.org
> Date: Sat, 13 Aug 2022 09:37:06 -0400
> 
> > Is 20MB single-line JSON file fairly representative of such files out
> > there?  Do we know of JSON file with much longer lines, for example
> > 50MB or 100MB?
> 
> IIUC it's very common for JSON files to be made of a single line, so the
> line length is the file's length and since those files are
> machine-generated I don't expect 100MB to be much less common than 20MB.

I think I'd prefer some actual data points.  It's clear that larger
JSON files can happen, the question is how frequently they do happen.

> > If 20MB is reasonably representative of the long-line JSON, then
> > perhaps at least json-mode could consider resetting
> > long-line-threshold to nil.
> 
> IIUC a nil value for `long-line-threshold` makes Emacs completely
> unusable with such long lines even if you disable all the bells
> and whistles.  Dmitry's suggestion is to skip the locked-narrowing in
> some cases, but not to set `long-line-threshold` to nil.

That's what I actually meant.

However, if much larger JSON files exist, we should first see what
that proposal produces with such larger files.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> However, if much larger JSON files exist, we should first see what that 
> proposal produces with such larger files.
>

As I just said to Dmitry: the same multi-second delays we saw in smaller 
JSON files.  And those who have a slower CPU will see these multi-second 
delays with smaller files.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> (with-half-joking-mode Hmm... so should be increase 
> `syntax-wholeline-max` from 10kB to 10MB or should we decrease 
> `large-file-warning-threshold` from 10MB down to 10kB?)
>

C-u C-x C-e => 42

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 14:32:08 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, dgutov@yandex.ru, 
>     56682@debbugs.gnu.org
> 
> > However, if much larger JSON files exist, we should first see what that 
> > proposal produces with such larger files.
> 
> As I just said to Dmitry: the same multi-second delays we saw in smaller 
> JSON files.  And those who have a slower CPU will see these multi-second 
> delays with smaller files.

Where can I download such files?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> However, if much larger JSON files exist, we should first see what 
>>> that proposal produces with such larger files.
>>
>> As I just said to Dmitry: the same multi-second delays we saw in 
>> smaller JSON files.  And those who have a slower CPU will see these 
>> multi-second delays with smaller files.
>
> Where can I download such files?
>

As Stefan said, JSON files are machine generated, so you can just 
concatenate the dictionary.json file as many times as you want to obtain a 
larger file.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 13.08.2022 17:32, Gregory Heytings wrote:
> 
>>
>> However, if much larger JSON files exist, we should first see what 
>> that proposal produces with such larger files.
>>
> 
> As I just said to Dmitry: the same multi-second delays we saw in smaller 
> JSON files.  And those who have a slower CPU will see these multi-second 
> delays with smaller files.

No exactly the same ones, though, since the previous tests we did were 
done with the older version of JSON support (not the current 
js-json-mode), with somewhat different performance characteristics.

So re-testing would be a good idea.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 13.08.2022 09:13, Eli Zaretskii wrote:
>>> I can only repeat that the only ways to improve font locking in a non-local
>>> way (or at least, the only ways that were identified in the discussions so
>>> far) are either to improve the speed of syntax-ppss by an order of
>>> magnitude, in which case it would perhaps become feasible to allow
>>> syntax-ppss to have access to the whole buffer, or to equip syntax-ppss with
>>> heuristics to make it work "as well as possible" even when it only has
>>> access to a portion of the buffer.
>> But the locked narrowing is applied based on the presence of long-lines
>> rather than based on the total size of the buffer.
> That's true, but the two are closely related: a buffer with very long
> lines tends also to be a large buffer.
> 

The long-line related problems start to become noticeable much earlier 
than the buffer becomes "large".

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 13.08.2022 16:42, Gregory Heytings wrote:
> 
>>
>> I think we'll want to decouple the two because they aren't quite the 
>> same (e.g. in many cases the time taken by `syntax-ppss` doesn't 
>> depend on the length of lines, whereas the time taken by 
>> `font-lock-keywords` rues does not depend on the total buffer size).
>>
> 
> That's what I initially thought, too.  Now I tend to think that it would 
> be better to not decouple these two cases, because it would be too 
> expensive to distinguish the four possible cases, namely:

I'm fairly sure that my branch demonstrates that there's nothing 
expensive about it.

It's just one commit, 20 additions, 10 deletions. It should take 2 
minutes max to read and understand it.

Yes, it adds a new custom var (for readability's sake we could split it 
in two later). But that's only logical since it handles an orthogonal 
problem to the one that long-line-threshold is about.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]


>> As I just said to Dmitry: the same multi-second delays we saw in 
>> smaller JSON files.  And those who have a slower CPU will see these 
>> multi-second delays with smaller files.
>
> No exactly the same ones, though, since the previous tests we did were 
> done with the older version of JSON support (not the current 
> js-json-mode), with somewhat different performance characteristics.
>

Okay, not "the same ones", but delays of the same order of magnitude 
nonetheless.  We had that discussion a month ago, and concluded that local 
fixes in every major (and minor) modes were not the way to go.  Having 
that discussion again does not seem fruitful.  You improved js-mode and 
created json-mode, that's fine, but it doesn't change anything to the way 
all other major (and minor) modes behave.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 13.08.2022 17:24, Gregory Heytings wrote:
> 
>>
>> But they're not. Of course, modes might have problematic rules 
>> (font-lock-keywords are turing-complete, after all), but it might be 
>> just as easy to fix all the main major modes rather than hobble user 
>> experience.
>>
> 
> You've made it quite clear that in your opinion incorrect font locking 
> hobbles user experience more than multi-second delays between motion or 
> search commands.

I did not.

As soon as the file becomes very large, such that font-lock actually 
becomes costly even in properly written major modes, we can still do one 
of the two things: apply narrowing during font-lock, or fontify only the 
beginning of the file.

That's what two alternative values of font-lock-large-files (added on 
the branch) aim to do. But what is the good default value, and if it's 
not t, what thresholds to use, those questions require more testing and 
user feedback.

>> Especially if we limit ourselves to modes that are likely to be used 
>> with large files.
>>
> 
> That's not the intention, no.
> 
>>
>> So please go ahead and experiment, and report on your findings later.
>>
> 
> I already did, and told you that these multi-second delays were still 
> present in larger JSON files (and therefore also in smaller files with 
> slower CPUs).

Delays or "a delay"? Just the first time you scroll to EOB? Or did you 
do the whole dance with editing near BOB, then goto EOB, and repeat?

Did you try that with other file formats? What is the size threshold 
where the performance becomes uncomfortable?

>> Then the goal failed because we don't disable bidi hpa? It has a much 
>> more noticeable effect on editing that font-lock.
>>
> 
> That's not correct, no.  The BPA algorithm makes C-n and C-p slower at 
> the beginning of large files only in a very specific case: when the 
> beginning of the buffer contains opening brackets whose matching closing 
> bracket is at the end of the buffer.

To be fair, that's pretty much all JSON files. But I can't quickly 
recall any other format with that properly, OK.

> And the BPA algorithm does not 
> cause multi-second delays.

The ones I saw seemed pretty bad. And they happened on every command 
invocation, not only after you edit the buffer on some distant position.

And bidi-inhibit-hpa is the main cause of C-v/M-v stuttering over here.

>>> Which is why fixing js-mode, and adding a json-mode, as you did, is a 
>>> "too local" fix. Of course, that doesn't mean what you did is 
>>> useless; it only means that it cannot be considered as a general 
>>> solution to the problem at hand.
>>
>> That's not what the branch contains.
>>
> 
> You're playing with words.  The fixes to js-mode and the added json-mode 
> are not on the branch because you installed them on master, but what you 
> claim the branch demonstrates depends on what you installed on master. 
> Or do you claim that what the branch contains makes editing any other 
> file (that is, not a .js or a .json file) almost as fast as what is on 
> master?

The branch:

1) Allows everyone interested to evaluate the performance of 
unrestricted font-lock even in large files (single-line or not) and see 
how big on a problem the delays caused by syntax-ppss actually are in 
their experience. It's an important question.
2) Figure out the file sizes where syntax-ppss's performance really does 
become a problem. That can give us data for better defaults later.
3) Play around with two easy solutions that we discussed previously: 
narrowing during font-lock (one of the values for font-lock-large-files 
pretty much matches the current behavior on master), or fontifying only 
the first X characters (e.g. 1000000) of the buffer, and skipping on the 
rest.
4) It should be plain to see that implementing additional approaches 
should be easy enough. For instance, a hybrid like "fontify the first 1 
MB correctly, and the rest - on best-effort basis". Although the value 
'(narrow . 1000000) should provide behavior a very similar behavior 
already. Maybe ever a better one: the boundaries are stable. Maybe sure 
to try it together with (setq bidi-inhibit-hpa t): the result looks very 
fast to me.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]


>> That's what I initially thought, too.  Now I tend to think that it 
>> would be better to not decouple these two cases, because it would be 
>> too expensive to distinguish the four possible cases, namely:
>
> I'm fairly sure that my branch demonstrates that there's nothing 
> expensive about it.
>

If you think this, it means you do not fully understand the problem, 
sorry.  Detecting the presence of long lines in a buffer takes time, and 
it takes more time as the buffer becomes larger.  Again all this has been 
discussed, tested and fine-tuned a month ago.

>
> It's just one commit, 20 additions, 10 deletions. It should take 2 
> minutes max to read and understand it.
>

As I said, I tested that commit.  And I understood it, too.  What you do 
there does not even depend on the presence of long lines in the buffer (or 
on the length of the buffer).  Moreover it does not offer any protection 
against modes which use widen in their font locking routines.  Which was 
the main reason to add the locked narrowing feature.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 2557 bytes --]


>> I already did, and told you that these multi-second delays were still 
>> present in larger JSON files (and therefore also in smaller files with 
>> slower CPUs).
>
> Delays or "a delay"? Just the first time you scroll to EOB? Or did you 
> do the whole dance with editing near BOB, then goto EOB, and repeat?
>

Delays.  When scrolling to EOB and when using isearch, for example.

>>> Then the goal failed because we don't disable bidi hpa? It has a much 
>>> more noticeable effect on editing that font-lock.
>> 
>> That's not correct, no.  The BPA algorithm makes C-n and C-p slower at 
>> the beginning of large files only in a very specific case: when the 
>> beginning of the buffer contains opening brackets whose matching 
>> closing bracket is at the end of the buffer.
>
> To be fair, that's pretty much all JSON files.
>

Again, the aim of this effort has _nothing_ to do with JSON files.

>
> The branch:
>
> 1) Allows everyone interested to evaluate the performance of 
> unrestricted font-lock even in large files (single-line or not) and see 
> how big on a problem the delays caused by syntax-ppss actually are in 
> their experience. It's an important question.
>
> 2) Figure out the file sizes where syntax-ppss's performance really does 
> become a problem. That can give us data for better defaults later.
>

Buffers with long lines are rare enough.  It wouldn't make sense (and 
would be a nightmare in terms of long term maintenance) to add 25 
defcustoms to allow anyone to fine-tune what Emacs does in such buffers.

>
> 3) Play around with two easy solutions that we discussed previously: 
> narrowing during font-lock (one of the values for font-lock-large-files 
> pretty much matches the current behavior on master), or fontifying only 
> the first X characters (e.g. 1000000) of the buffer, and skipping on the 
> rest.
>
> 4) It should be plain to see that implementing additional approaches 
> should be easy enough. For instance, a hybrid like "fontify the first 1 
> MB correctly, and the rest - on best-effort basis". Although the value 
> '(narrow . 1000000) should provide behavior a very similar behavior 
> already. Maybe ever a better one: the boundaries are stable. Maybe sure 
> to try it together with (setq bidi-inhibit-hpa t): the result looks very 
> fast to me.
>

These 3) and 4) assume that the major (and minor) modes are well-behaving, 
and will obey the restrictions.  Which isn't true, and was the reason why 
locked narrowing was introduced.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 20:20:04 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> As soon as the file becomes very large, such that font-lock actually 
> becomes costly even in properly written major modes, we can still do one 
> of the two things: apply narrowing during font-lock, or fontify only the 
> beginning of the file.

Fontifying only the beginning of the file doesn't help when the file
is first shown at another point, like when Emacs or emacsclient is
invoked with the +NN[:nn] argument, or the user uses saveplace or
similar package.  That's admittedly rarer than starting at the
beginning, but it's a valid use case, and I wouldn't like us to
dismiss it.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 13.08.2022 20:54, Eli Zaretskii wrote:
>> Date: Sat, 13 Aug 2022 20:20:04 +0300
>> Cc:56682@debbugs.gnu.org, Eli Zaretskii<eliz@gnu.org>,
>>   monnier@iro.umontreal.ca
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> As soon as the file becomes very large, such that font-lock actually
>> becomes costly even in properly written major modes, we can still do one
>> of the two things: apply narrowing during font-lock, or fontify only the
>> beginning of the file.
> Fontifying only the beginning of the file doesn't help when the file
> is first shown at another point, like when Emacs or emacsclient is
> invoked with the +NN[:nn] argument, or the user uses saveplace or
> similar package.  That's admittedly rarer than starting at the
> beginning, but it's a valid use case, and I wouldn't like us to
> dismiss it.

The beginning of the file is the part of it which we can fontify quickly 
enough while still doing it correctly.

Starting with some position P, jumping to it will show a larger initial 
delay than you might be comfortable with. Maybe it's 1 MB, maybe 10 MB, 
maybe 100 MB. Or more. But such file size exists because syntax-ppss's 
performance depends linearly on the buffer size. So after that position 
we'll not fontify it anymore. That's really the simplest solution.

Because the main alternative on offer is to use narrowing and thus risk 
getting invalid syntax information. The effect is very noticeable in 
dictionary.json: you can just as easily get strings and non-strings 
inverted (the odds are pretty much 50/50). So the "don't fontify past X" 
strategy is simply based on the idea that no fontification is probably 
better than unreliable and obviously incorrect one.

Now, we could develop more complex approaches from there. But this can 
be a starting point, and the user option allows people to choose the 
strategy they're most comfortable with.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 13.08.2022 20:16, Gregory Heytings wrote:
> 
>>> As I just said to Dmitry: the same multi-second delays we saw in 
>>> smaller JSON files.  And those who have a slower CPU will see these 
>>> multi-second delays with smaller files.
>>
>> No exactly the same ones, though, since the previous tests we did were 
>> done with the older version of JSON support (not the current 
>> js-json-mode), with somewhat different performance characteristics.
>>
> 
> Okay, not "the same ones", but delays of the same order of magnitude 
> nonetheless.  We had that discussion a month ago, and concluded that 
> local fixes in every major (and minor) modes were not the way to go.  
> Having that discussion again does not seem fruitful.

Did you have that discussion with other people who have not authored a 
single major mode? That's ridiculous.

> You improved 
> js-mode and created json-mode, that's fine, but it doesn't change 
                       ^
                        js-json-mode

> anything to the way all other major (and minor) modes behave.

It showed specific pitfalls we can take care of in other major modes as 
well.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 13.08.2022 20:26, Gregory Heytings wrote:
> 
>>> That's what I initially thought, too.  Now I tend to think that it 
>>> would be better to not decouple these two cases, because it would be 
>>> too expensive to distinguish the four possible cases, namely:
>>
>> I'm fairly sure that my branch demonstrates that there's nothing 
>> expensive about it.
>>
> 
> If you think this, it means you do not fully understand the problem, 
> sorry.  Detecting the presence of long lines in a buffer takes time, and 
> it takes more time as the buffer becomes larger.

How is that relevant? font-lock's performance does not depend on the 
presence of long lines.

>> It's just one commit, 20 additions, 10 deletions. It should take 2 
>> minutes max to read and understand it.
>>
> 
> As I said, I tested that commit.  And I understood it, too.  What you do 
> there does not even depend on the presence of long lines in the buffer 
> (or on the length of the buffer).

Exactly.

> Moreover it does not offer any 
> protection against modes which use widen in their font locking 
> routines.  Which was the main reason to add the locked narrowing feature.

The only modes we know that do that (CC Mode) are also incompatible with 
your code. Your "protection" breaks it.

And when CC Mode is fixed to work inside a narrowing, it will 
automatically become compatible with my proposal.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 13.08.2022 20:43, Gregory Heytings wrote:
> 
>>> I already did, and told you that these multi-second delays were still 
>>> present in larger JSON files (and therefore also in smaller files 
>>> with slower CPUs).
>>
>> Delays or "a delay"? Just the first time you scroll to EOB? Or did you 
>> do the whole dance with editing near BOB, then goto EOB, and repeat?
>>
> 
> Delays.  When scrolling to EOB and when using isearch, for example.
> 
>>>> Then the goal failed because we don't disable bidi hpa? It has a 
>>>> much more noticeable effect on editing that font-lock.
>>>
>>> That's not correct, no.  The BPA algorithm makes C-n and C-p slower 
>>> at the beginning of large files only in a very specific case: when 
>>> the beginning of the buffer contains opening brackets whose matching 
>>> closing bracket is at the end of the buffer.
>>
>> To be fair, that's pretty much all JSON files.
>>
> 
> Again, the aim of this effort has _nothing_ to do with JSON files.

What kind of files are your main target?

>> The branch:
>>
>> 1) Allows everyone interested to evaluate the performance of 
>> unrestricted font-lock even in large files (single-line or not) and 
>> see how big on a problem the delays caused by syntax-ppss actually are 
>> in their experience. It's an important question.
>>
>> 2) Figure out the file sizes where syntax-ppss's performance really 
>> does become a problem. That can give us data for better defaults later.
>>
> 
> Buffers with long lines are rare enough.  It wouldn't make sense (and 
> would be a nightmare in terms of long term maintenance) to add 25 
> defcustoms to allow anyone to fine-tune what Emacs does in such buffers.

I have regularly encountered redisplay slowdown caused by long lines in 
my work. One doesn't need to have a 18 MB files to see them.

Having a 5000-10000 character line is enough to see redisplay starting 
to stutter. And, of course, there's nothing font-lock related in those 
stutters.

>> 3) Play around with two easy solutions that we discussed previously: 
>> narrowing during font-lock (one of the values for 
>> font-lock-large-files pretty much matches the current behavior on 
>> master), or fontifying only the first X characters (e.g. 1000000) of 
>> the buffer, and skipping on the rest.
>>
>> 4) It should be plain to see that implementing additional approaches 
>> should be easy enough. For instance, a hybrid like "fontify the first 
>> 1 MB correctly, and the rest - on best-effort basis". Although the 
>> value '(narrow . 1000000) should provide behavior a very similar 
>> behavior already. Maybe ever a better one: the boundaries are stable. 
>> Maybe sure to try it together with (setq bidi-inhibit-hpa t): the 
>> result looks very fast to me.
>>
> 
> These 3) and 4) assume that the major (and minor) modes are 
> well-behaving, and will obey the restrictions.  Which isn't true, and 
> was the reason why locked narrowing was introduced.

What are your examples of misbehaving major/minor modes?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> We had that discussion a month ago, and concluded that local fixes in 
>> every major (and minor) modes were not the way to go.  Having that 
>> discussion again does not seem fruitful.
>
> Did you have that discussion with other people who have not authored a 
> single major mode? That's ridiculous.
>

You mean, Eli, Lars and me?  I agree with you, we're indeed ridiculous.

>
> It showed specific pitfalls we can take care of in other major modes as 
> well.
>

It doesn't.  Please do (on you Core i9 computer):

for i in $(seq 1 60); do cat dictionary.json; done > hugedictionary.json

Now open that file with the code on master, and with your branch, and type 
M->.  With the code on master the effect is instantaneous (about 50 ms on 
my computer), with your branch it takes ~20 seconds (21 seconds on my 
computer).  If you cannot see the difference, I don't know what else I 
could say.  And please don't tell me "but this is not the kind of file 
size we should care about".  That would be wrong, both because we do care 
about that file size, and because if that basic operation takes 20 seconds 
in a 1 GB file on your Core i9 computer, that same basic operation will 
take 20 seconds on a 20 MB or 50 MB file for someone else who has a slower 
CPU.

Again, the right thing to do here is to improve the efficiency of 
syntax-ppss in one way or another.  Of course, doing that is several 
orders of magnitude more difficult than just undoing what others have 
already done and claiming that the result is better.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 13.08.2022 22:54, Gregory Heytings wrote:
>> It showed specific pitfalls we can take care of in other major modes 
>> as well.
>>
> 
> It doesn't.  Please do (on you Core i9 computer):
> 
> for i in $(seq 1 60); do cat dictionary.json; done > hugedictionary.json
> 
> Now open that file with the code on master, and with your branch, and 
> type M->.  With the code on master the effect is instantaneous (about 50 
> ms on my computer), with your branch it takes ~20 seconds (21 seconds on 
> my computer).  If you cannot see the difference, I don't know what else 
> I could say.  And please don't tell me "but this is not the kind of file 
> size we should care about".

Why would I tell you that after I already spent the effort (yesterday 
morning) to make sure the branch contains whole two alternative working 
strategies for handling such very large files?

Should I explain them again?

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> fixes in every major (and minor) modes were not the way to go.

I think we do want to encourage such fixes since they have other
benefits.  So while we can't rely on those happening everywhere, indeed,
we do want to be able to take advantage of them, otherwise there won't
be any incentive to fix those problems.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 22:08:25 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Fontifying only the beginning of the file doesn't help when the file
> > is first shown at another point, like when Emacs or emacsclient is
> > invoked with the +NN[:nn] argument, or the user uses saveplace or
> > similar package.  That's admittedly rarer than starting at the
> > beginning, but it's a valid use case, and I wouldn't like us to
> > dismiss it.
> 
> The beginning of the file is the part of it which we can fontify quickly 
> enough while still doing it correctly.

I know, but we are not doing only what is easy to do, do we?  We do
(or should do) what the users expect.  In this case, if we want to
fontify some relatively small portion of the document, it should be
the portion around where the file is first displayed.

> Because the main alternative on offer is to use narrowing and thus risk 
> getting invalid syntax information.

That alternative is a direct and somewhat simplistic way of
restricting misbehaving fontification, so as to prevent them from
making Emacs unresponsive.  Smarter alternatives -- and you seem to be
arguing for such -- should be smarter, not less so.  They should be
like what you installed for JSON on master.

> So the "don't fontify past X" strategy is simply based on the idea
> that no fontification is probably better than unreliable and
> obviously incorrect one.

I disagree with that idea, but if someone agrees with you, they can
simply turn off font-lock.  As was already mentioned many times in
this endless discussion.

> Now, we could develop more complex approaches from there. But this can 
> be a starting point, and the user option allows people to choose the 
> strategy they're most comfortable with.

Sorry, I don't consider this (fontifying the beginning of a file) a
good starting point for making any progress towards smarter, faster
fontifications.  The new json-mode is such a starting point, but we
should do something similar for other major modes as well.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 22:12:42 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Moreover it does not offer any 
> > protection against modes which use widen in their font locking 
> > routines.  Which was the main reason to add the locked narrowing feature.
> 
> The only modes we know that do that (CC Mode)

AFAIU, currently _all_ modes widen, because font-lock does.  That part
was not yet fixed in font-0lock, was it?

> are also incompatible with your code. Your "protection" breaks it.

For some value of "breaks".

And that is a deliberate "breakage": we consider responsive Emacs more
important than correct fontification or "accurate" operation of modes
such as CC Mode.  As was already explained numerous times here, and
arguing about that should by now be understood as being futile.
People who disagree with that default can customize
long-line-threshold to a larger value or to a nil value.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 22:19:26 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Buffers with long lines are rare enough.  It wouldn't make sense (and 
> > would be a nightmare in terms of long term maintenance) to add 25 
> > defcustoms to allow anyone to fine-tune what Emacs does in such buffers.
> 
> I have regularly encountered redisplay slowdown caused by long lines in 
> my work. One doesn't need to have a 18 MB files to see them.
> 
> Having a 5000-10000 character line is enough to see redisplay starting 
> to stutter. And, of course, there's nothing font-lock related in those 
> stutters.

Stuttering in redisplay (and I think you actually mean stuttering in
cursor motion commands, but let's ignore that difference for now) is a
known issue, but it is NOT the goal of this effort.  The goal of this
effort is to avoid making Emacs _completely_unusable_ due to the need
to wait for many seconds for commands to complete and their results to
be shown on display.  Stuttering, while somewhat annoying, doesn't
make Emacs unusable.

So if you are talking about the cases of stuttering, you are focused
on cases that are of no particular interest to us here.  Of course,
any improvement in those other cases will also be welcome, but it is
not directly related to what we are trying to do here, and may or may
not solve the cases that are of interest to us.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 08:36, Eli Zaretskii wrote:
>> Date: Sat, 13 Aug 2022 22:19:26 +0300
>> Cc:56682@debbugs.gnu.org, Eli Zaretskii<eliz@gnu.org>,
>>   monnier@iro.umontreal.ca
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>>> Buffers with long lines are rare enough.  It wouldn't make sense (and
>>> would be a nightmare in terms of long term maintenance) to add 25
>>> defcustoms to allow anyone to fine-tune what Emacs does in such buffers.
>> I have regularly encountered redisplay slowdown caused by long lines in
>> my work. One doesn't need to have a 18 MB files to see them.
>>
>> Having a 5000-10000 character line is enough to see redisplay starting
>> to stutter. And, of course, there's nothing font-lock related in those
>> stutters.
> Stuttering in redisplay (and I think you actually mean stuttering in
> cursor motion commands, but let's ignore that difference for now) is a
> known issue, but it is NOT the goal of this effort.  The goal of this
> effort is to avoid making Emacs_completely_unusable_  due to the need
> to wait for many seconds for commands to complete and their results to
> be shown on display.  Stuttering, while somewhat annoying, doesn't
> make Emacs unusable.
> 
> So if you are talking about the cases of stuttering, you are focused
> on cases that are of no particular interest to us here.  Of course,
> any improvement in those other cases will also be welcome, but it is
> not directly related to what we are trying to do here, and may or may
> not solve the cases that are of interest to us.

They *are* solved by Gregory's work.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 08:29, Eli Zaretskii wrote:
>> Date: Sat, 13 Aug 2022 22:12:42 +0300
>> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>>   Stefan Monnier <monnier@iro.umontreal.ca>
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> Moreover it does not offer any
>>> protection against modes which use widen in their font locking
>>> routines.  Which was the main reason to add the locked narrowing feature.
>>
>> The only modes we know that do that (CC Mode)
> 
> AFAIU, currently _all_ modes widen, because font-lock does.  That part
> was not yet fixed in font-0lock, was it?

The changes on the branch are right in that area:

-    (unless font-lock-dont-widen (widen))
+    (pcase-exhaustive font-lock-large-files
+      (`t nil)
+      (`(head . ,length)
+       (setq beg (min length beg)
+             end (min length end)))
+      (`(narrow . ,width)
+       (narrow-to-region (max (point-min) (* (1- (/ beg width)) width))
+                         (min (point-max) (* (1+ (/ beg width)) width)))
+       (setq end (min end (point-max)))))
+    (unless (or font-lock-dont-widen
+                (eq (car-safe font-lock-large-files) 'narrow))
+      (widen))

>> are also incompatible with your code. Your "protection" breaks it.
> 
> For some value of "breaks".
> 
> And that is a deliberate "breakage": we consider responsive Emacs more
> important than correct fontification or "accurate" operation of modes
> such as CC Mode.  As was already explained numerous times here, and
> arguing about that should by now be understood as being futile.
> People who disagree with that default can customize
> long-line-threshold to a larger value or to a nil value.

Due to its current way of operation, it might spam errors to the echo 
area, break some navigation commands, or even freeze. I imagine it'll be 
not so easy to rework that.

Though I'd of course welcome that: if you recall the older talks about 
switching it to the syntax-propertize approach, it's one way to do that.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 08:23, Eli Zaretskii wrote:
>> Date: Sat, 13 Aug 2022 22:08:25 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> Fontifying only the beginning of the file doesn't help when the file
>>> is first shown at another point, like when Emacs or emacsclient is
>>> invoked with the +NN[:nn] argument, or the user uses saveplace or
>>> similar package.  That's admittedly rarer than starting at the
>>> beginning, but it's a valid use case, and I wouldn't like us to
>>> dismiss it.
>>
>> The beginning of the file is the part of it which we can fontify quickly
>> enough while still doing it correctly.
> 
> I know, but we are not doing only what is easy to do, do we?  We do
> (or should do) what the users expect.  In this case, if we want to
> fontify some relatively small portion of the document, it should be
> the portion around where the file is first displayed.

There's no point in doing that. Either we narrow to some area around 
point (might even using a larger radius like 1 MB), or we we only 
fontify up to some position. The former easily creates bad fontification.

The alternative, of course, is to pay the price of syntax-ppss on larger 
spans and wait the corresponding amount of time the first time the user 
scrolls to EOB. That's what the current default on the branch does.

>> Because the main alternative on offer is to use narrowing and thus risk
>> getting invalid syntax information.
> 
> That alternative is a direct and somewhat simplistic way of
> restricting misbehaving fontification, so as to prevent them from
> making Emacs unresponsive.  Smarter alternatives -- and you seem to be
> arguing for such -- should be smarter, not less so.  They should be
> like what you installed for JSON on master.

js-json-mode is not an "alternative", it's just doing what font-lock is 
supposed to do (I fixed the outstanding problems with the rules).

But as Gregory shows, when you get to _really_ large files (like 1 GB 
JSON file in his example), pressing M-> will still make you wait (I have 
to wait around 20 seconds).

It's still a great improvement from what we had in Emacs 28, though, so 
I'm fine with this default too.

>> So the "don't fontify past X" strategy is simply based on the idea
>> that no fontification is probably better than unreliable and
>> obviously incorrect one.
> 
> I disagree with that idea, but if someone agrees with you, they can
> simply turn off font-lock.  As was already mentioned many times in
> this endless discussion.

If someone agrees with me, they will simply be able to customize 
font-lock-large-files to choose this strategy.

I do not really insist on it being the default. But being able to choose 
this approach (in the absence of better upcoming alternatives) is a good 
thing.

>> Now, we could develop more complex approaches from there. But this can
>> be a starting point, and the user option allows people to choose the
>> strategy they're most comfortable with.
> 
> Sorry, I don't consider this (fontifying the beginning of a file) a
> good starting point for making any progress towards smarter, faster
> fontifications.  The new json-mode is such a starting point, but we
> should do something similar for other major modes as well.

I'm still waiting for people to come forward with other major modes 
which have the same kind of problems. Preferably ones that are likely to 
be used with large files.

But as mentioned, that does not preclude us from having to choose what 
to do with _really_ large files.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 14 Aug 2022 12:41:33 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Stuttering in redisplay (and I think you actually mean stuttering in
> > cursor motion commands, but let's ignore that difference for now) is a
> > known issue, but it is NOT the goal of this effort.  The goal of this
> > effort is to avoid making Emacs_completely_unusable_  due to the need
> > to wait for many seconds for commands to complete and their results to
> > be shown on display.  Stuttering, while somewhat annoying, doesn't
> > make Emacs unusable.
> > 
> > So if you are talking about the cases of stuttering, you are focused
> > on cases that are of no particular interest to us here.  Of course,
> > any improvement in those other cases will also be welcome, but it is
> > not directly related to what we are trying to do here, and may or may
> > not solve the cases that are of interest to us.
> 
> They *are* solved by Gregory's work.

Not in all cases and not on all systems.  I still see stuttering in
some cases, at least with my unoptimized builds, and I don't consider
that a problem.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 14 Aug 2022 13:29:42 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > I know, but we are not doing only what is easy to do, do we?  We do
> > (or should do) what the users expect.  In this case, if we want to
> > fontify some relatively small portion of the document, it should be
> > the portion around where the file is first displayed.
> 
> There's no point in doing that. Either we narrow to some area around 
> point (might even using a larger radius like 1 MB), or we we only 
> fontify up to some position. The former easily creates bad fontification.
> 
> The alternative, of course, is to pay the price of syntax-ppss on larger 
> spans and wait the corresponding amount of time the first time the user 
> scrolls to EOB. That's what the current default on the branch does.

You are still thinking in terms of the original design of syntactical
analysis which strives to produce 100% accurate results.  That design
principle doesn't work with very long lines, so sticking to it would
indeed lead us to give up on solving the problem.

The better way is to acknowledge that some inaccuracies are acceptable
in those cases.  With that in mind, one can design a syntax analyzer
that looks back only a short ways, until it finds some place that
could reasonably serve as an anchor point for heuristic decisions
about whether we are inside or outside a string or comment, and then
verifying that guess with some telltale syntactic elements that follow
(like semi-colons or comment-end delimiters in C).  While this kind of
heuristics can sometimes fail, if they only fail rarely, the result is
a huge win.

> But as Gregory shows, when you get to _really_ large files (like 1 GB 
> JSON file in his example), pressing M-> will still make you wait (I have 
> to wait around 20 seconds).

Try with the latest master, it might have improved (fingers crossed).

In any case, the way to speed up these cases is to look at the profile
and identify the code that is slowing us down; then attempt to make it
faster.  (20 sec is actually long enough for us to interrupt Emacs
under a debugger and look at the backtrace to find the culprit.)

> >> So the "don't fontify past X" strategy is simply based on the idea
> >> that no fontification is probably better than unreliable and
> >> obviously incorrect one.
> > 
> > I disagree with that idea, but if someone agrees with you, they can
> > simply turn off font-lock.  As was already mentioned many times in
> > this endless discussion.
> 
> If someone agrees with me, they will simply be able to customize 
> font-lock-large-files to choose this strategy.

If that solves the problems in a reasonable way for very long lines,
maybe we will eventually have such an option.

> I'm still waiting for people to come forward with other major modes 
> which have the same kind of problems. Preferably ones that are likely to 
> be used with large files.

One such major mode and one such file was presented long ago : a
single-line XML file.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 16:02, Eli Zaretskii wrote:
>> Date: Sun, 14 Aug 2022 12:41:33 +0300
>> Cc:56682@debbugs.gnu.org,gregory@heytings.org,monnier@iro.umontreal.ca
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>>> Stuttering in redisplay (and I think you actually mean stuttering in
>>> cursor motion commands, but let's ignore that difference for now) is a
>>> known issue, but it is NOT the goal of this effort.  The goal of this
>>> effort is to avoid making Emacs_completely_unusable_  due to the need
>>> to wait for many seconds for commands to complete and their results to
>>> be shown on display.  Stuttering, while somewhat annoying, doesn't
>>> make Emacs unusable.
>>>
>>> So if you are talking about the cases of stuttering, you are focused
>>> on cases that are of no particular interest to us here.  Of course,
>>> any improvement in those other cases will also be welcome, but it is
>>> not directly related to what we are trying to do here, and may or may
>>> not solve the cases that are of interest to us.
>> They*are*  solved by Gregory's work.
> Not in all cases and not on all systems.  I still see stuttering in
> some cases, at least with my unoptimized builds, and I don't consider
> that a problem.

Point is, Gregory's work improves performance in many real-world 
scenarios, not just in rare special files.

So being able to fine-tune the behavior makes sense. Certainly not a 
"maintenance nightmare".

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> The alternative, of course, is to pay the price of syntax-ppss on larger 
> spans and wait the corresponding amount of time the first time the user 
> scrolls to EOB.
>

Or to speed up syntax-ppss in a way or another.

>
> But as Gregory shows, when you get to _really_ large files (like 1 GB 
> JSON file in his example), pressing M-> will still make you wait (I have 
> to wait around 20 seconds).
>

Or when you have a slower CPU, with moderately large files.

>
> But being able to choose this approach (in the absence of better 
> upcoming alternatives) is a good thing.
>

The problem is (again) that what you would like does not provide any 
protection against modes which use widen inside their fontification 
routines.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> So being able to fine-tune the behavior makes sense. Certainly not a 
> "maintenance nightmare".
>

If the conclusion is, after some reasonable effort, that there is no way 
to make syntax-ppss significantly faster in one way or another in such 
cases, and that there is no way to make font locking reasonably accurate 
even when it doesn't have access to the whole buffer, it might make sense 
to provide user options to fine-tune the behavior.  But we are not there 
yet.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 16:15, Eli Zaretskii wrote:

>> There's no point in doing that. Either we narrow to some area around
>> point (might even using a larger radius like 1 MB), or we we only
>> fontify up to some position. The former easily creates bad fontification.
>>
>> The alternative, of course, is to pay the price of syntax-ppss on larger
>> spans and wait the corresponding amount of time the first time the user
>> scrolls to EOB. That's what the current default on the branch does.
> 
> You are still thinking in terms of the original design of syntactical
> analysis which strives to produce 100% accurate results.  That design
> principle doesn't work with very long lines, so sticking to it would
> indeed lead us to give up on solving the problem.

s/very long lines/very large files

In any case, the "original design" is not going anywhere (as the only 
way to achieve correctness), and I'm talking in terms of balance between 
accuracy and performance. To use Gregory's narrowing approach in 
font-lock, checkout the branch under discussion 
(scratch/font_lock_large_files) and evaluate

   (setq font-lock-large-files '(narrow . 5000))

You'll see the same behavior as on master now (except narrowing isn't 
"hard"), with the same performance characteristics.

> The better way is to acknowledge that some inaccuracies are acceptable
> in those cases.  With that in mind, one can design a syntax analyzer
> that looks back only a short ways, until it finds some place that
> could reasonably serve as an anchor point for heuristic decisions
> about whether we are inside or outside a string or comment, and then
> verifying that guess with some telltale syntactic elements that follow
> (like semi-colons or comment-end delimiters in C).  While this kind of
> heuristics can sometimes fail, if they only fail rarely, the result is
> a huge win.

You cannot design a language-agnostic syntax analyzer like that. It's 
something every major mode would have to consider how to implement.

It's relatively easy to design for JSON (again) because the syntax is so 
simple, but for others -- not so much.

So we need to settle on the basic design first. The code on the branch 
includes the narrowing approach which is trivially extended to use the 
"find safe place" hook when it's available. But it won't be always 
available.

>> But as Gregory shows, when you get to _really_ large files (like 1 GB
>> JSON file in his example), pressing M-> will still make you wait (I have
>> to wait around 20 seconds).
> 
> Try with the latest master, it might have improved (fingers crossed).

All improvements are welcome, but that's unlikely:

> In any case, the way to speed up these cases is to look at the profile
> and identify the code that is slowing us down; then attempt to make it
> faster.  (20 sec is actually long enough for us to interrupt Emacs
> under a debugger and look at the backtrace to find the culprit.)

I've profiled and benchmarked this scenario already: all of the delay 
(17 seconds, to be precise) come from parse-partial-sexp. 1 GB is a lot.

>>>> So the "don't fontify past X" strategy is simply based on the idea
>>>> that no fontification is probably better than unreliable and
>>>> obviously incorrect one.
>>>
>>> I disagree with that idea, but if someone agrees with you, they can
>>> simply turn off font-lock.  As was already mentioned many times in
>>> this endless discussion.
>>
>> If someone agrees with me, they will simply be able to customize
>> font-lock-large-files to choose this strategy.
> 
> If that solves the problems in a reasonable way for very long lines,
> maybe we will eventually have such an option.

Can I merge the branch, then?

I was hoping for a stylistic review, perhaps. Like, whether you like the 
name of the variable, and should it be split in two.

A change of the default value(s) is on the table too.

>> I'm still waiting for people to come forward with other major modes
>> which have the same kind of problems. Preferably ones that are likely to
>> be used with large files.
> 
> One such major mode and one such file was presented long ago : a
> single-line XML file.

XMl is indeed slower. It takes almost 3 seconds for me to scroll to the 
end of a 20 MB XML file.

Most of it comes from sgml--syntax-propertize-ppss, which is probably 
justified: XML is a more complex language.

But other than the initial delay, scrolling, and isearch, and local 
editing, all work fast, unlike the original situation with JSON.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 19:24, Gregory Heytings wrote:
> 
>>
>> So being able to fine-tune the behavior makes sense. Certainly not a 
>> "maintenance nightmare".
>>
> 
> If the conclusion is, after some reasonable effort, that there is no way 
> to make syntax-ppss significantly faster in one way or another in such 
> cases, and that there is no way to make font locking reasonably accurate 
> even when it doesn't have access to the whole buffer, it might make 
> sense to provide user options to fine-tune the behavior.  But we are not 
> there yet.

Both conclusions lead to removing the applications of narrowing from 
handle_fontified_prop. So how about we either do that (defaulting to 
accurate font-lock), or merge the branch I proposed, and then continue 
on to the more complex developments?

Implementing the "font locking reasonably accurate even when it doesn't 
have access to the whole buffer" would also have to be implemented in 
Lisp, so narrowing outside of font-lock doesn't make sense.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]


>> If the conclusion is, after some reasonable effort, that there is no 
>> way to make syntax-ppss significantly faster in one way or another in 
>> such cases, and that there is no way to make font locking reasonably 
>> accurate even when it doesn't have access to the whole buffer, it might 
>> make sense to provide user options to fine-tune the behavior.  But we 
>> are not there yet.
>
> Both conclusions lead to removing the applications of narrowing from 
> handle_fontified_prop.
>

I have seen no effort whatsoever to make syntax-ppss significantly faster 
in one way or another.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 19:16, Gregory Heytings wrote:
> 
>>
>> The alternative, of course, is to pay the price of syntax-ppss on 
>> larger spans and wait the corresponding amount of time the first time 
>> the user scrolls to EOB.
>>
> 
> Or to speed up syntax-ppss in a way or another.

I suggest you try tweaking parse-partial-sexp's implementation.

>> But as Gregory shows, when you get to _really_ large files (like 1 GB 
>> JSON file in his example), pressing M-> will still make you wait (I 
>> have to wait around 20 seconds).
>>
> 
> Or when you have a slower CPU, with moderately large files.

I doubt there are any CPUs in wide use that are even 5x slower than mine.

And even 200 MB is pretty large, and not something we've been reasonable 
able to edit before.

>> But being able to choose this approach (in the absence of better 
>> upcoming alternatives) is a good thing.
>>
> 
> The problem is (again) that what you would like does not provide any 
> protection against modes which use widen inside their fontification 
> routines.

Do you have examples of such modes?

Calling 'widen' inside font-lock-keywords is a bug, and the bugs should 
be fixed at the source.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 14 Aug 2022 20:47:40 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > The better way is to acknowledge that some inaccuracies are acceptable
> > in those cases.  With that in mind, one can design a syntax analyzer
> > that looks back only a short ways, until it finds some place that
> > could reasonably serve as an anchor point for heuristic decisions
> > about whether we are inside or outside a string or comment, and then
> > verifying that guess with some telltale syntactic elements that follow
> > (like semi-colons or comment-end delimiters in C).  While this kind of
> > heuristics can sometimes fail, if they only fail rarely, the result is
> > a huge win.
> 
> You cannot design a language-agnostic syntax analyzer like that.

_I_ cannot, but hopefully someone else will.

> > In any case, the way to speed up these cases is to look at the profile
> > and identify the code that is slowing us down; then attempt to make it
> > faster.  (20 sec is actually long enough for us to interrupt Emacs
> > under a debugger and look at the backtrace to find the culprit.)
> 
> I've profiled and benchmarked this scenario already: all of the delay 
> (17 seconds, to be precise) come from parse-partial-sexp. 1 GB is a lot.

Before we get to 1GB files, there are 20MB files and 250MB files.  I
found quite a few low-hanging fruit in those that are worth plucking,
while we wait for parse-partial-sexp to get its act together.

> > If that solves the problems in a reasonable way for very long lines,
> > maybe we will eventually have such an option.
> 
> Can I merge the branch, then?

Please wait until I have time to review it.

> I was hoping for a stylistic review, perhaps. Like, whether you like the 
> name of the variable, and should it be split in two.
> 
> A change of the default value(s) is on the table too.

Will definitely do, I'm just busy with "other things" right now, most
of them related to other aspects of long lines.

> > One such major mode and one such file was presented long ago : a
> > single-line XML file.
> 
> XMl is indeed slower. It takes almost 3 seconds for me to scroll to the 
> end of a 20 MB XML file.
> 
> Most of it comes from sgml--syntax-propertize-ppss, which is probably 
> justified: XML is a more complex language.

Did you wait till nxml-mode did its initial scan and displayed "Valid"
in the mode line?  The performance is quite different before and after
that.

> But other than the initial delay, scrolling, and isearch, and local 
> editing, all work fast, unlike the original situation with JSON.

With which branch?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 14 Aug 2022 20:51:14 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > If the conclusion is, after some reasonable effort, that there is no way 
> > to make syntax-ppss significantly faster in one way or another in such 
> > cases, and that there is no way to make font locking reasonably accurate 
> > even when it doesn't have access to the whole buffer, it might make 
> > sense to provide user options to fine-tune the behavior.  But we are not 
> > there yet.
> 
> Both conclusions lead to removing the applications of narrowing from 
> handle_fontified_prop.

We will not remove that, no.

> So how about we either do that (defaulting to accurate font-lock),
> or merge the branch I proposed, and then continue on to the more
> complex developments?

Please wait with requests to merge until I had time to review the
branch.

> Implementing the "font locking reasonably accurate even when it doesn't 
> have access to the whole buffer" would also have to be implemented in 
> Lisp, so narrowing outside of font-lock doesn't make sense.

Cannot parse this, sorry.  I guess some typo?

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 21:01, Eli Zaretskii wrote:
>> Date: Sun, 14 Aug 2022 20:51:14 +0300
>> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>>   monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> If the conclusion is, after some reasonable effort, that there is no way
>>> to make syntax-ppss significantly faster in one way or another in such
>>> cases, and that there is no way to make font locking reasonably accurate
>>> even when it doesn't have access to the whole buffer, it might make
>>> sense to provide user options to fine-tune the behavior.  But we are not
>>> there yet.
>>
>> Both conclusions lead to removing the applications of narrowing from
>> handle_fontified_prop.
> 
> We will not remove that, no.

Please look at the branch. It moves the (potential) narrowing from C to 
Lisp.

>> So how about we either do that (defaulting to accurate font-lock),
>> or merge the branch I proposed, and then continue on to the more
>> complex developments?
> 
> Please wait with requests to merge until I had time to review the
> branch.

Waiting, and thanks.

>> Implementing the "font locking reasonably accurate even when it doesn't
>> have access to the whole buffer" would also have to be implemented in
>> Lisp, so narrowing outside of font-lock doesn't make sense.
> 
> Cannot parse this, sorry.  I guess some typo?

The implementation of the idea that Gregory mentioned (font locking 
reasonably accurate even when it doesn't have access to the whole 
buffer) will have to be done in Lisp anyway. So that's where the 
narrowing should be applied too.

Does that parse? Not sure how to phrase it better.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 20:54, Gregory Heytings wrote:
> 
>>> If the conclusion is, after some reasonable effort, that there is no 
>>> way to make syntax-ppss significantly faster in one way or another in 
>>> such cases, and that there is no way to make font locking reasonably 
>>> accurate even when it doesn't have access to the whole buffer, it 
>>> might make sense to provide user options to fine-tune the behavior.  
>>> But we are not there yet.
>>
>> Both conclusions lead to removing the applications of narrowing from 
>> handle_fontified_prop.
>>
> 
> I have seen no effort whatsoever to make syntax-ppss significantly 
> faster in one way or another.

syntax-ppss is limited in its performance by parse-partial-sexp.

Could you take a look at it? Seems like more of your area of expertise.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 14 Aug 2022 21:14:00 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> The implementation of the idea that Gregory mentioned (font locking 
> reasonably accurate even when it doesn't have access to the whole 
> buffer) will have to be done in Lisp anyway. So that's where the 
> narrowing should be applied too.

I don't see how it follows.  If we decide (and I don't say we did, but
if we do) that fontification-functions must run narrowed, then they
will run narrowed, and the best place to do that is in the caller.

> Does that parse?

Yes, it's clear now, thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Or when you have a slower CPU, with moderately large files.
>
> I doubt there are any CPUs in wide use that are even 5x slower than 
> mine.
>

Have a look at e.g. https://www.cpubenchmark.net/singleThread.html , 
you'll see that many CPUs in wide use are 5x slower, or more than 5x 
slower, than your CPU.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> Or when you have a slower CPU, with moderately large files.
>> I doubt there are any CPUs in wide use that are even 5x slower than mine.
> Have a look at e.g. https://www.cpubenchmark.net/singleThread.html , you'll
> see that many CPUs in wide use are 5x slower, or more than 5x slower, than
> your CPU.

Which one of those CPUs that are 5x slower than the top performer do you
consider as being "in wide use"?

AFAICT the highest score there is 4400.  So 5x slower would mean a score
below 880.  My 2006-vintage Thinkpad T60 is just below at 848.  But my
T61 (released in 2007, tho I bought it refurbished in 2012) is already
rated at 1K.

Admittedly, these were top-of-the-line machines when they were released,
and are hence faster than many machine released later (e.g. netbooks),
but still: I have seen *very* few people recently using machines slower
than my T61.

I don't know what machine Dmitry is using, but based on that
"cpubenchmark" data I'd indeed be very surprised if "there are any CPUs
in wide use that are even 5x slower than" his.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 20:59, Eli Zaretskii wrote:
>> Date: Sun, 14 Aug 2022 20:47:40 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> The better way is to acknowledge that some inaccuracies are acceptable
>>> in those cases.  With that in mind, one can design a syntax analyzer
>>> that looks back only a short ways, until it finds some place that
>>> could reasonably serve as an anchor point for heuristic decisions
>>> about whether we are inside or outside a string or comment, and then
>>> verifying that guess with some telltale syntactic elements that follow
>>> (like semi-colons or comment-end delimiters in C).  While this kind of
>>> heuristics can sometimes fail, if they only fail rarely, the result is
>>> a huge win.
>>
>> You cannot design a language-agnostic syntax analyzer like that.
> 
> _I_ cannot, but hopefully someone else will.

That seems unlikely. Nothing's impossible, of course, but I wouldn't 
want to wait for such an invention to come up before we make the 
decision on how to proceed now.

What _can_ be done is make syntax-ppss's cache invalidations more local 
by introducing a "repair" step. That would only speed up certain 
operations, at most, and the initial wait near EOB can't be avoided this 
way.

>>> In any case, the way to speed up these cases is to look at the profile
>>> and identify the code that is slowing us down; then attempt to make it
>>> faster.  (20 sec is actually long enough for us to interrupt Emacs
>>> under a debugger and look at the backtrace to find the culprit.)
>>
>> I've profiled and benchmarked this scenario already: all of the delay
>> (17 seconds, to be precise) come from parse-partial-sexp. 1 GB is a lot.
> 
> Before we get to 1GB files, there are 20MB files and 250MB files.  I
> found quite a few low-hanging fruit in those that are worth plucking,
> while we wait for parse-partial-sexp to get its act together.

Definitely.

But when the profiler output in a 1 GB file comes down to syntax-ppss 
only, that means the low-handing fruit has been picked.

>>> If that solves the problems in a reasonable way for very long lines,
>>> maybe we will eventually have such an option.
>>
>> Can I merge the branch, then?
> 
> Please wait until I have time to review it.
> 
>> I was hoping for a stylistic review, perhaps. Like, whether you like the
>> name of the variable, and should it be split in two.
>>
>> A change of the default value(s) is on the table too.
> 
> Will definitely do, I'm just busy with "other things" right now, most
> of them related to other aspects of long lines.

Roger that.

>>> One such major mode and one such file was presented long ago : a
>>> single-line XML file.
>>
>> XMl is indeed slower. It takes almost 3 seconds for me to scroll to the
>> end of a 20 MB XML file.
>>
>> Most of it comes from sgml--syntax-propertize-ppss, which is probably
>> justified: XML is a more complex language.
> 
> Did you wait till nxml-mode did its initial scan and displayed "Valid"
> in the mode line?  The performance is quite different before and after
> that.

It takes a while to switch from "Validated: 0" to "Valid", but the 
performance seems about the same in both states.

Maybe some other example file would show different behavior, IDK.

>> But other than the initial delay, scrolling, and isearch, and local
>> editing, all work fast, unlike the original situation with JSON.
> 
> With which branch?

scratch/font_lock_large_files, with 'emacs -Q'

I've also run this test on master now, and M-> is not instant there 
either. Apparently, a fair amount time is also spent in 
nxml-extend-region (which calls sgml-syntax-propertize and syntax-ppss).

Not sure why it would spend any significant time in either, though, if 
they're called inside a narrowing.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 22:15, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
> I don't know what machine Dmitry is using, but based on that
> "cpubenchmark" data I'd indeed be very surprised if "there are any CPUs
> in wide use that are even 5x slower than" his.

Right. I have a beefy CPU, but not the latest-and-greatest one anyway:

Intel Core i9-8950HK @ 2.90GHz: single-threaded score 2,492

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 14.08.2022 21:27, Eli Zaretskii wrote:
>> Date: Sun, 14 Aug 2022 21:14:00 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> The implementation of the idea that Gregory mentioned (font locking
>> reasonably accurate even when it doesn't have access to the whole
>> buffer) will have to be done in Lisp anyway. So that's where the
>> narrowing should be applied too.
> 
> I don't see how it follows.  If we decide (and I don't say we did, but
> if we do) that fontification-functions must run narrowed, then they
> will run narrowed, and the best place to do that is in the caller.

Only if we somehow decided that it makes sense to always use the same 
narrowing bounds. But as experiment shows, font-lock can use 100x larger 
narrowing, and still perform well.

Further, the thing which is going to call the language-specific 
safe-position-finding logic is likely to want to update the bounds of 
the narrowing (to then call syntax-ppss within those bounds, where START 
is a "safe" position).

So yeah, the font-lock related widen/narrowing logic should indeed live 
in one place. And until now it has resided in font-lock-fontify-region.

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> The problem is (again) that what you would like does not provide any 
>> protection against modes which use widen inside their fontification 
>> routines.
>
> Do you have examples of such modes?
>
> Calling 'widen' inside font-lock-keywords is a bug, and the bugs should 
> be fixed at the source.

widen inside save-restriction is likely used by Org mode (or will be).
This is needed by Org parser to determine the syntax element at point.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ihor Radchenko [2022-08-15 09:23:21] wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
>> Do you have examples of such modes?
>> Calling 'widen' inside font-lock-keywords is a bug, and the bugs should 
>> be fixed at the source.
> widen inside save-restriction is likely used by Org mode (or will be).

Why?

> This is needed by Org parser to determine the syntax element at point.

But font-lock widens before using Org-mode's `font-lock-keywords`, so
there should be no need for Org-mode to widen in that case.


        Stefan


PS: Not sure if Org-mode qualifies as a mode that is likely to suffer
from the kinds of long-lines or large-file problems that Gregory has
tried to tackle, tho.  I've never seen an Org file with lines >10kB nor
an Org file larger than 10MB.

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Ihor Radchenko [2022-08-15 09:23:21] wrote:
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>> Do you have examples of such modes?
>>> Calling 'widen' inside font-lock-keywords is a bug, and the bugs should 
>>> be fixed at the source.
>> widen inside save-restriction is likely used by Org mode (or will be).
>
> Why?

Because Org syntax is context-dependent, and we may need to look
backwards to determine the "container" element before knowing what the
element at point is.

>> This is needed by Org parser to determine the syntax element at point.
>
> But font-lock widens before using Org-mode's `font-lock-keywords`, so
> there should be no need for Org-mode to widen in that case.

Yes, but Org parser is not written explicitly for font-locking purposes.
org-element-at-point knows nothing about buffer restriction upon
calling and thus has to widen to ensure correctness.

> PS: Not sure if Org-mode qualifies as a mode that is likely to suffer
> from the kinds of long-lines or large-file problems that Gregory has
> tried to tackle, tho.  I've never seen an Org file with lines >10kB nor
> an Org file larger than 10MB.

I have seen very long lines in Org. Specifically, in source block
output. (http query)

My notes.org file is over 20Mb.
Even larger files can be auto-generated by https://gitlab.com/publicvoit/Memacs

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 15 Aug 2022 01:07:08 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 14.08.2022 21:27, Eli Zaretskii wrote:
> >> Date: Sun, 14 Aug 2022 21:14:00 +0300
> >> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >>
> >> The implementation of the idea that Gregory mentioned (font locking
> >> reasonably accurate even when it doesn't have access to the whole
> >> buffer) will have to be done in Lisp anyway. So that's where the
> >> narrowing should be applied too.
> > 
> > I don't see how it follows.  If we decide (and I don't say we did, but
> > if we do) that fontification-functions must run narrowed, then they
> > will run narrowed, and the best place to do that is in the caller.
> 
> Only if we somehow decided that it makes sense to always use the same 
> narrowing bounds. But as experiment shows, font-lock can use 100x larger 
> narrowing, and still perform well.

So what?  The proponents of the widening claim that they need to go to
BOB, and no "arbitrary" narrowing smaller than that will suffice.

And btw, I have very different impression of what happens with 100x
larger narrowing on my machine and with unoptimized builds.

> Further, the thing which is going to call the language-specific 
> safe-position-finding logic is likely to want to update the bounds of 
> the narrowing (to then call syntax-ppss within those bounds, where START 
> is a "safe" position).
> 
> So yeah, the font-lock related widen/narrowing logic should indeed live 
> in one place. And until now it has resided in font-lock-fontify-region.

Since when is font-lock-fontify-region specific to a language?  It is
as general as xdisp.c.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> This is needed by Org parser to determine the syntax element at point.
>> But font-lock widens before using Org-mode's `font-lock-keywords`, so
>> there should be no need for Org-mode to widen in that case.
> Yes, but Org parser is not written explicitly for font-locking purposes.
> org-element-at-point knows nothing about buffer restriction upon
> calling and thus has to widen to ensure correctness.

That will break uses of MMM-mode where one of the chunks is using
Org-mode.  Admittedly, this is unlikely, but still: It would be much
better to arrange to do `widen` *outside* of Org-mode's font-lock (and
indentation) code.

>> PS: Not sure if Org-mode qualifies as a mode that is likely to suffer
>> from the kinds of long-lines or large-file problems that Gregory has
>> tried to tackle, tho.  I've never seen an Org file with lines >10kB nor
>> an Org file larger than 10MB.
>
> I have seen very long lines in Org. Specifically, in source block
> output. (http query)
>
> My notes.org file is over 20Mb.
> Even larger files can be auto-generated by https://gitlab.com/publicvoit/Memacs

I stand corrected,


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> This is needed by Org parser to determine the syntax element at point.
>>> But font-lock widens before using Org-mode's `font-lock-keywords`, so
>>> there should be no need for Org-mode to widen in that case.
>> Yes, but Org parser is not written explicitly for font-locking purposes.
>> org-element-at-point knows nothing about buffer restriction upon
>> calling and thus has to widen to ensure correctness.
>
> That will break uses of MMM-mode where one of the chunks is using
> Org-mode.  Admittedly, this is unlikely, but still: It would be much
> better to arrange to do `widen` *outside* of Org-mode's font-lock (and
> indentation) code.

The fact that MMM-mode is relying on narrowing is a bug in MMM-mode.
Polymode (an alternative to MMM-mode) is using a different approach.

Correct me if I am wrong, but I assume that having a specific major mode
in buffer implies that the whole buffer is considered to contribute to
the major mode.

If the Org parser does not ensure that buffer is widened then the parser
output may depend on the narrowing state. For font-lock, it also means
that different fontification must be used with/without narrowing -
something that font-lock will fail to achieve unless buffer is
re-fontified every time the narrowing changes.

Further, Org parser is caching parse results and reuse them to not
re-parse on every invocation. The need to maintain separate parser
caches for every possible narrowing state will be a nightmare.
Especially considering the user narrowing may also be an option and
having a different parser/command behaviour will be unexpected to the
user.

More generally, prohibiting the innocent-looking
(save-restriction (widen) ...) means that font-lock-keywords cannot
safely call any kind of general-purpose API function without checking
(widen) usage inside that function and inside all the deeper nesting
levels. This will be fragile, especially for user-defined keywords.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Which one of those CPUs that are 5x slower than the top performer do you 
> consider as being "in wide use"?
>
> AFAICT the highest score there is 4400.  So 5x slower would mean a score 
> below 880.  My 2006-vintage Thinkpad T60 is just below at 848.  But my 
> T61 (released in 2007, tho I bought it refurbished in 2012) is already 
> rated at 1K.
>
> Admittedly, these were top-of-the-line machines when they were released, 
> and are hence faster than many machine released later (e.g. netbooks), 
> but still: I have seen *very* few people recently using machines slower 
> than my T61.
>

This is more and more tangent, but anyway: I think you should look around 
more carefully.  The Raspberry Pi 4B has a quad-core ARM CPU (Cortex A72) 
whose single core benchmark on that website is 456.  And nowadays "thin 
client" computers, in which slow CPUs are used, are very common both in 
industry and in schools/universities.  One random example, which I've seen 
a few days ago, is the Dell Wyse 3040, which is equipped with an Intel 
Atom X5 Z8350 CPU, whose single core benchmark is 480.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> I have seen no effort whatsoever to make syntax-ppss significantly 
>> faster in one way or another.
>
> syntax-ppss is limited in its performance by parse-partial-sexp.
>
> Could you take a look at it? Seems like more of your area of expertise.
>

I already said (at least twice) that I would do that.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> The implementation of the idea that Gregory mentioned (font locking 
> reasonably accurate even when it doesn't have access to the whole 
> buffer) will have to be done in Lisp anyway. So that's where the 
> narrowing should be applied too.
>

It would be helpful if you did not reiterate your arguments again and 
again.  I think everybody here has understood what your preference would 
be, and your reasons for that preference.  Please give us some time to 
explore the current direction, before asking us to go back and take 
another direction.  The current situation is already, in many aspects, way 
better than it was two months ago.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> Further, Org parser is caching parse results and reuse them to not 
> re-parse on every invocation. The need to maintain separate parser 
> caches for every possible narrowing state will be a nightmare. 
> Especially considering the user narrowing may also be an option and 
> having a different parser/command behaviour will be unexpected to the 
> user.
>

I'm not sure I correctly understand what you mean, but it seems to me that 
the fact that the Org parser caches its results implies that Org doesn't 
need to access the whole buffer to make "local" decisions.  It can use the 
data in cache for everything that is outside of the current narrowing, and 
update the portion of the cache corresponding to the current narrowing. 
IOW, there is no need to maintain separate parser caches for each possible 
narrowing state.  Am I misunderstanding something?

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 06:10, Ihor Radchenko wrote:
> The fact that MMM-mode is relying on narrowing is a bug in MMM-mode.
> Polymode (an alternative to MMM-mode) is using a different approach.

Does it really?

 From what I see, Polymode does use narrowing: 
https://github.com/polymode/polymode/search?q=narrow&type=code

But it also contains provisions for modes whose syntax highlighting 
cannot function with it. With different tradeoffs.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 05:32, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2022 01:07:08 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 14.08.2022 21:27, Eli Zaretskii wrote:
>>>> Date: Sun, 14 Aug 2022 21:14:00 +0300
>>>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>>>> From: Dmitry Gutov <dgutov@yandex.ru>
>>>>
>>>> The implementation of the idea that Gregory mentioned (font locking
>>>> reasonably accurate even when it doesn't have access to the whole
>>>> buffer) will have to be done in Lisp anyway. So that's where the
>>>> narrowing should be applied too.
>>>
>>> I don't see how it follows.  If we decide (and I don't say we did, but
>>> if we do) that fontification-functions must run narrowed, then they
>>> will run narrowed, and the best place to do that is in the caller.
>>
>> Only if we somehow decided that it makes sense to always use the same
>> narrowing bounds. But as experiment shows, font-lock can use 100x larger
>> narrowing, and still perform well.
> 
> So what?  The proponents of the widening claim that they need to go to
> BOB, and no "arbitrary" narrowing smaller than that will suffice.

I'm trying to support several approaches, including narrowing.

> And btw, I have very different impression of what happens with 100x
> larger narrowing on my machine and with unoptimized builds.

Given what we've seen about your parse-partial-sexp's performance (10x 
slower than mine), I would hope for someone to figure out why it's so 
slow in your unoptimized build. But if it doesn't happen, you would 
probably use a different threshold/narrowing radius (through customization).

It would make sense for our defaults not to be tailored to this very 
particular development rig.

>> Further, the thing which is going to call the language-specific
>> safe-position-finding logic is likely to want to update the bounds of
>> the narrowing (to then call syntax-ppss within those bounds, where START
>> is a "safe" position).
>>
>> So yeah, the font-lock related widen/narrowing logic should indeed live
>> in one place. And until now it has resided in font-lock-fontify-region.
> 
> Since when is font-lock-fontify-region specific to a language?  It is
> as general as xdisp.c.

syntax-propertize is general, and yet it invokes language-specific rules 
(through syntax-propertize-function).

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  Gregory Heytings
>  <gregory@heytings.org>,  56682@debbugs.gnu.org,  Eli Zaretskii
>  <eliz@gnu.org>
> Date: Mon, 15 Aug 2022 10:10:36 +0800
> 
> > PS: Not sure if Org-mode qualifies as a mode that is likely to suffer
> > from the kinds of long-lines or large-file problems that Gregory has
> > tried to tackle, tho.  I've never seen an Org file with lines >10kB nor
> > an Org file larger than 10MB.
> 
> I have seen very long lines in Org. Specifically, in source block
> output. (http query)

How long is "very long" in those cases?

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

> I'm not sure I correctly understand what you mean, but it seems to me that 
> the fact that the Org parser caches its results implies that Org doesn't 
> need to access the whole buffer to make "local" decisions.  It can use the 
> data in cache for everything that is outside of the current narrowing, and 
> update the portion of the cache corresponding to the current narrowing. 
> IOW, there is no need to maintain separate parser caches for each possible 
> narrowing state.  Am I misunderstanding something?

Cache existence is not guaranteed.
When cache does not exist, Org must widen. If not, the cache will be
valid only for the current narrowing.
When cache do exist, Org may not need to widen. But that also mean that
widening has happened in the past when the cache was generated.

At the end, Org does need to be able to widen safely. At least in some
cases.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 15 Aug 2022 13:06:14 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > And btw, I have very different impression of what happens with 100x
> > larger narrowing on my machine and with unoptimized builds.
> 
> Given what we've seen about your parse-partial-sexp's performance (10x 
> slower than mine), I would hope for someone to figure out why it's so 
> slow in your unoptimized build. But if it doesn't happen, you would 
> probably use a different threshold/narrowing radius (through customization).
> 
> It would make sense for our defaults not to be tailored to this very 
> particular development rig.

If you are willing to lose my ability to debug complex redisplay
problems in many cases, sure, go ahead.

> >> So yeah, the font-lock related widen/narrowing logic should indeed live
> >> in one place. And until now it has resided in font-lock-fontify-region.
> > 
> > Since when is font-lock-fontify-region specific to a language?  It is
> > as general as xdisp.c.
> 
> syntax-propertize is general, and yet it invokes language-specific rules 
> (through syntax-propertize-function).

Nothing prevents xdisp.c from invoking those same rules if needed.

bug#56682: Fix the long lines font locking related slowdowns

[dick]

EZ> If you are willing to lose my ability to debug complex redisplay
EZ> problems in many cases, sure, go ahead.

Willing!  I'll consider "sure, go ahead" as ratification.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 14:51, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2022 13:06:14 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>> And btw, I have very different impression of what happens with 100x
>>> larger narrowing on my machine and with unoptimized builds.
>>
>> Given what we've seen about your parse-partial-sexp's performance (10x
>> slower than mine), I would hope for someone to figure out why it's so
>> slow in your unoptimized build. But if it doesn't happen, you would
>> probably use a different threshold/narrowing radius (through customization).
>>
>> It would make sense for our defaults not to be tailored to this very
>> particular development rig.
> 
> If you are willing to lose my ability to debug complex redisplay
> problems in many cases, sure, go ahead.

Why? You would customize an option to a lower value, and go on to debugging.

Though I guess reproducing the exact conditions on the user's machine 
might get harder sometimes.

>>>> So yeah, the font-lock related widen/narrowing logic should indeed live
>>>> in one place. And until now it has resided in font-lock-fontify-region.
>>>
>>> Since when is font-lock-fontify-region specific to a language?  It is
>>> as general as xdisp.c.
>>
>> syntax-propertize is general, and yet it invokes language-specific rules
>> (through syntax-propertize-function).
> 
> Nothing prevents xdisp.c from invoking those same rules if needed.

True. We could basically rewrite any Lisp function we have in C 
(font-lock-fontify-region, for example).

And by doing that bar a lot of power users/potential developers from 
being able to read that code, debug and contribute improvements.

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 15.08.2022 06:10, Ihor Radchenko wrote:
>> The fact that MMM-mode is relying on narrowing is a bug in MMM-mode.
>> Polymode (an alternative to MMM-mode) is using a different approach.
>
> Does it really?
>
>  From what I see, Polymode does use narrowing: 
> https://github.com/polymode/polymode/search?q=narrow&type=code

> But it also contains provisions for modes whose syntax highlighting 
> cannot function with it. With different tradeoffs.

AFAIK, it allows running commands across multiple chunks of the same
foreign mode.

In any case, do not take my word on this. I have been studying MMM-mode
and polymode years back.

However, I do insist on my statement that expecting major-mode to not
widen is a bug. There are valid cases when a major mode needs to examine
the whole buffer.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 15 Aug 2022 15:06:35 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> It would make sense for our defaults not to be tailored to this very
> >> particular development rig.
> > 
> > If you are willing to lose my ability to debug complex redisplay
> > problems in many cases, sure, go ahead.
> 
> Why? You would customize an option to a lower value, and go on to debugging.

I will forget to do it and waste time.

> >> syntax-propertize is general, and yet it invokes language-specific rules
> >> (through syntax-propertize-function).
> > 
> > Nothing prevents xdisp.c from invoking those same rules if needed.
> 
> True. We could basically rewrite any Lisp function we have in C 
> (font-lock-fontify-region, for example).

No need to go ad absurdum.  I was responding to your assertion that
this must be done from font-lock.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  56682@debbugs.gnu.org,
>   Gregory Heytings <gregory@heytings.org>,  Eli Zaretskii <eliz@gnu.org>
> Date: Mon, 15 Aug 2022 20:08:12 +0800
> 
> However, I do insist on my statement that expecting major-mode to not
> widen is a bug.

We only don't expect that when the mode is invoked from redisplay.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 20:20:04 +0300
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> The branch:
> 
> 1) Allows everyone interested to evaluate the performance of 
> unrestricted font-lock even in large files (single-line or not) and see 
> how big on a problem the delays caused by syntax-ppss actually are in 
> their experience. It's an important question.
> 2) Figure out the file sizes where syntax-ppss's performance really does 
> become a problem. That can give us data for better defaults later.
> 3) Play around with two easy solutions that we discussed previously: 
> narrowing during font-lock (one of the values for font-lock-large-files 
> pretty much matches the current behavior on master), or fontifying only 
> the first X characters (e.g. 1000000) of the buffer, and skipping on the 
> rest.
> 4) It should be plain to see that implementing additional approaches 
> should be easy enough. For instance, a hybrid like "fontify the first 1 
> MB correctly, and the rest - on best-effort basis". Although the value 
> '(narrow . 1000000) should provide behavior a very similar behavior 
> already. Maybe ever a better one: the boundaries are stable. Maybe sure 
> to try it together with (setq bidi-inhibit-hpa t): the result looks very 
> fast to me.

Thanks.

I tested this branch with an unoptimized build and a 416MB file made
of 23 copies of dictionary.json, after disabling show-paren-mode.

Observations:

  . With the default value of font-lock-large-files, which allows
    font-lock to widen as it pleases, the initial M-> in takes 5 min
    here.  That's way too long.  On master, this is instantaneous.
  . Thereafter jumping into various random places inside the file is
    much faster, but still takes between 1 min and 2.25 min -- again
    too long, even if I divide by 10 to get an approximation to your
    faster machine.  On master this takes maybe 1 sec, maybe less.
  . Horizontal and vertical motion commands are as fast (or as slow)
    as on master, which is not surprising.

Bottom line: with the default value of font-lock-large-files, this
scales much worse than the master branch -- which again is no
surprise.  My conclusion is that we do need to prevent uncontrolled
widening in fontification-functions, if we want a scalable solution.

How small or large should the narrowing be is a separate issue.  If we
want to allow a more flexible control of that, we could introduce yet
another variable exposed to Lisp, so that users and perhaps Lisp
programs could tune that, instead of hard-coding the value as we do
now on master.

I honestly don't see any reason to explore more sophisticated
alternatives for where and how to narrow, until and unless major modes
will on their side develop capabilities which will make such
complications (both in code, but mainly on the user part) justified.

If you agree, I'll add on master a variable to control the narrowing
when we call fontification-functions.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Sun, 14 Aug 2022 23:46:13 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> I've also run this test on master now, and M-> is not instant there 
> either.

It's instantaneous here, so I'm not sure why you see something
different.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> If you are willing to lose my ability to debug complex redisplay
>> problems in many cases, sure, go ahead.
> Why? You would customize an option to a lower value, and go on to debugging.

I suspect that the "large-file-threshold" and the "long-line-threshold"
can both be kept small enough that they accommodate the case of Eli (or
myself) running a slow build of Emacs on a slowish machine without
significantly affecting those who could live with larger thresholds.

IOW, I think this part of your discussion should wait to see what actual
thresholds seem to work well in practice because it may prove to be
a non-problem.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> However, I do insist on my statement that expecting major-mode to not
> widen is a bug.

No, it's a design decision, which makes it possible to use specific
implementation techniques.  Org-mode's support for multiple major modes
(org-babel) uses an implementation technique that does not require
narrowing.  Both approaches have their virtues and downsides, but if the
submode widens then the supermode has less freedom in terms of which
implementation technique to use.

> There are valid cases when a major mode needs to examine
> the whole buffer.

Not if the whole buffer contains text which the major mode can't
understand.  There are valid cases where a major mode needs to widen,
indeed, but not when called from font-lock or from
`indent-line-function`.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 17:06, Eli Zaretskii wrote:
>> Date: Sun, 14 Aug 2022 23:46:13 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> I've also run this test on master now, and M-> is not instant there
>> either.
> 
> It's instantaneous here, so I'm not sure why you see something
> different.

No idea. Size of file, or even the contents might the reason.

Here's the profiler output anyway:

         1067  85% - command-execute
         1067  85%  - call-interactively
         1023  82%   - funcall-interactively
         1012  81%    - end-of-buffer
         1008  81%     - recenter
         1008  81%      - jit-lock-function
         1008  81%       - jit-lock-fontify-now
         1008  81%        - jit-lock--run-functions
         1008  81%         - run-hook-wrapped
         1008  81%          - #<compiled -0x149e7c2f08a063c3>
         1008  81%           - font-lock-fontify-region
         1008  81%            - font-lock-default-fontify-region
         1008  81%             - nxml-extend-region
          720  57%              - skip-syntax-forward
          720  57%               - internal--syntax-propertize
          720  57%                - syntax-propertize
          720  57%                 - nxml-syntax-propertize
          720  57%                  - sgml-syntax-propertize
          378  30%                     #<compiled -0x12505696cac83570>
          294  23%                     sgml--syntax-propertize-ppss
          288  23%              - nxml-move-outside-backwards
          288  23%               - nxml-inside-start
          288  23%                  syntax-ppss
           11   0%    + execute-extended-command
           44   3%   + byte-code
          171  13% - ...
          171  13%    Automatic GC
            5   0% + timer-event-handler

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 15 Aug 2022 18:42:08 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 15.08.2022 17:06, Eli Zaretskii wrote:
> >> Date: Sun, 14 Aug 2022 23:46:13 +0300
> >> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >>
> >> I've also run this test on master now, and M-> is not instant there
> >> either.
> > 
> > It's instantaneous here, so I'm not sure why you see something
> > different.
> 
> No idea. Size of file, or even the contents might the reason.
> 
> Here's the profiler output anyway:
> 
>          1067  85% - command-execute
>          1067  85%  - call-interactively
>          1023  82%   - funcall-interactively
>          1012  81%    - end-of-buffer
>          1008  81%     - recenter
>          1008  81%      - jit-lock-function

When did you last resync from Git?  'recenter' got "optimized"
yesterday for buffers with long lines.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 17:03, Eli Zaretskii wrote:

> I tested this branch with an unoptimized build and a 416MB file made
> of 23 copies of dictionary.json, after disabling show-paren-mode.
> 
> Observations:
> 
>    . With the default value of font-lock-large-files, which allows
>      font-lock to widen as it pleases, the initial M-> in takes 5 min
>      here.  That's way too long.  On master, this is instantaneous.
>    . Thereafter jumping into various random places inside the file is
>      much faster, but still takes between 1 min and 2.25 min -- again
>      too long, even if I divide by 10 to get an approximation to your
>      faster machine.  On master this takes maybe 1 sec, maybe less.
>    . Horizontal and vertical motion commands are as fast (or as slow)
>      as on master, which is not surprising.
> 
> Bottom line: with the default value of font-lock-large-files, this
> scales much worse than the master branch -- which again is no
> surprise.

So we change the default, then?

   (setq font-lock-large-files '(narrow . 5000))

> My conclusion is that we do need to prevent uncontrolled
> widening in fontification-functions, if we want a scalable solution.

Do you mean the widening inside font-lock-fontify-region that's 
controlled by font-lock-dont-widen?

> How small or large should the narrowing be is a separate issue.

Right.

> If we
> want to allow a more flexible control of that, we could introduce yet
> another variable exposed to Lisp, so that users and perhaps Lisp
> programs could tune that, instead of hard-coding the value as we do
> now on master.
> 
> I honestly don't see any reason to explore more sophisticated
> alternatives for where and how to narrow, until and unless major modes
> will on their side develop capabilities which will make such
> complications (both in code, but mainly on the user part) justified.

I'm pretty sure that doing all that in Lisp is a good thing, and would 
be conducive to whatever improvements we might want to add later...

> If you agree, I'll add on master a variable to control the narrowing
> when we call fontification-functions.

...but if you've sure in your decision, okay, please add that additional 
var which will decide the narrowing bounds for fontification-functions 
(and whether to narrow at all).

That should solve my immediate needs, and I'll try to avoid this subject 
from now on. Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 15 Aug 2022 18:52:41 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 15.08.2022 17:03, Eli Zaretskii wrote:
> 
> > I tested this branch with an unoptimized build and a 416MB file made
> > of 23 copies of dictionary.json, after disabling show-paren-mode.
> > 
> > Observations:
> > 
> >    . With the default value of font-lock-large-files, which allows
> >      font-lock to widen as it pleases, the initial M-> in takes 5 min
> >      here.  That's way too long.  On master, this is instantaneous.
> >    . Thereafter jumping into various random places inside the file is
> >      much faster, but still takes between 1 min and 2.25 min -- again
> >      too long, even if I divide by 10 to get an approximation to your
> >      faster machine.  On master this takes maybe 1 sec, maybe less.
> >    . Horizontal and vertical motion commands are as fast (or as slow)
> >      as on master, which is not surprising.
> > 
> > Bottom line: with the default value of font-lock-large-files, this
> > scales much worse than the master branch -- which again is no
> > surprise.
> 
> So we change the default, then?
> 
>    (setq font-lock-large-files '(narrow . 5000))

No, I want a simple variable that just gives the size of the narrowed
region, with nil meaning don't narrow at all.

> > My conclusion is that we do need to prevent uncontrolled
> > widening in fontification-functions, if we want a scalable solution.
> 
> Do you mean the widening inside font-lock-fontify-region that's 
> controlled by font-lock-dont-widen?

No, I mean _any_ widening in _any_ function placed on the
fontification-functions hook.

> > I honestly don't see any reason to explore more sophisticated
> > alternatives for where and how to narrow, until and unless major modes
> > will on their side develop capabilities which will make such
> > complications (both in code, but mainly on the user part) justified.
> 
> I'm pretty sure that doing all that in Lisp is a good thing, and would 
> be conducive to whatever improvements we might want to add later...

No, because quite a few packages use jit-lock-register to run stuff
that is not font-lock at all, and I don't want that to slow down
redisplay when long lines are involved.  And trying various values of
a variable exposed to Lisp is easy enough.

> > If you agree, I'll add on master a variable to control the narrowing
> > when we call fontification-functions.
> 
> ...but if you've sure in your decision, okay, please add that additional 
> var which will decide the narrowing bounds for fontification-functions 
> (and whether to narrow at all).

Will do soon.

Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 18:52, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2022 18:42:08 +0300
>> Cc:56682@debbugs.gnu.org,gregory@heytings.org,monnier@iro.umontreal.ca
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> On 15.08.2022 17:06, Eli Zaretskii wrote:
>>>> Date: Sun, 14 Aug 2022 23:46:13 +0300
>>>> Cc:56682@debbugs.gnu.org,gregory@heytings.org,monnier@iro.umontreal.ca
>>>> From: Dmitry Gutov<dgutov@yandex.ru>
>>>>
>>>> I've also run this test on master now, and M-> is not instant there
>>>> either.
>>> It's instantaneous here, so I'm not sure why you see something
>>> different.
>> No idea. Size of file, or even the contents might the reason.
>>
>> Here's the profiler output anyway:
>>
>>           1067  85% - command-execute
>>           1067  85%  - call-interactively
>>           1023  82%   - funcall-interactively
>>           1012  81%    - end-of-buffer
>>           1008  81%     - recenter
>>           1008  81%      - jit-lock-function
> When did you last resync from Git?  'recenter' got "optimized"
> yesterday for buffers with long lines.

Just today.

In any case, it doesn't look like recenter's problem, since the output 
says all (or vast majority) of its time is spent in jit-lock-function.

I just did 'make bootstrap' to be doubly sure: no change.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 15 Aug 2022 19:44:07 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> Here's the profiler output anyway:
> >>
> >>           1067  85% - command-execute
> >>           1067  85%  - call-interactively
> >>           1023  82%   - funcall-interactively
> >>           1012  81%    - end-of-buffer
> >>           1008  81%     - recenter
> >>           1008  81%      - jit-lock-function
> > When did you last resync from Git?  'recenter' got "optimized"
> > yesterday for buffers with long lines.
> 
> Just today.
> 
> In any case, it doesn't look like recenter's problem, since the output 
> says all (or vast majority) of its time is spent in jit-lock-function.

AFAIU 'recenter' shouldn't at all call jit-lock-function in a buffer
with long lines.  Is this "emacs -Q" without any changes from
defaults?

> I just did 'make bootstrap' to be doubly sure: no change.

What can I say? I'm stumped.  It's definitely instantaneous here, in
the long-line.xml file.  Even if I don't turn off show-paren-mode.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 19:05, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2022 18:52:41 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 15.08.2022 17:03, Eli Zaretskii wrote:
>>
>>> I tested this branch with an unoptimized build and a 416MB file made
>>> of 23 copies of dictionary.json, after disabling show-paren-mode.
>>>
>>> Observations:
>>>
>>>     . With the default value of font-lock-large-files, which allows
>>>       font-lock to widen as it pleases, the initial M-> in takes 5 min
>>>       here.  That's way too long.  On master, this is instantaneous.
>>>     . Thereafter jumping into various random places inside the file is
>>>       much faster, but still takes between 1 min and 2.25 min -- again
>>>       too long, even if I divide by 10 to get an approximation to your
>>>       faster machine.  On master this takes maybe 1 sec, maybe less.
>>>     . Horizontal and vertical motion commands are as fast (or as slow)
>>>       as on master, which is not surprising.
>>>
>>> Bottom line: with the default value of font-lock-large-files, this
>>> scales much worse than the master branch -- which again is no
>>> surprise.
>>
>> So we change the default, then?
>>
>>     (setq font-lock-large-files '(narrow . 5000))
> 
> No, I want a simple variable that just gives the size of the narrowed
> region, with nil meaning don't narrow at all.

OK.

>>> My conclusion is that we do need to prevent uncontrolled
>>> widening in fontification-functions, if we want a scalable solution.
>>
>> Do you mean the widening inside font-lock-fontify-region that's
>> controlled by font-lock-dont-widen?
> 
> No, I mean _any_ widening in _any_ function placed on the
> fontification-functions hook.
> 
>>> I honestly don't see any reason to explore more sophisticated
>>> alternatives for where and how to narrow, until and unless major modes
>>> will on their side develop capabilities which will make such
>>> complications (both in code, but mainly on the user part) justified.
>>
>> I'm pretty sure that doing all that in Lisp is a good thing, and would
>> be conducive to whatever improvements we might want to add later...
> 
> No, because quite a few packages use jit-lock-register to run stuff
> that is not font-lock at all, and I don't want that to slow down
> redisplay when long lines are involved.  And trying various values of
> a variable exposed to Lisp is easy enough.

It should also be possible to move the whole invocation of the 
fontification-functions up to Lisp, and apply the narrowing in the new 
function that does that.

Another reason I chose font-lock-fontify-region, though, is that it's 
called from different places. It's the common entry point for font-lock.

Like, from 'font-lock-ensure', for example (font-lock-turn-on-thing-lock 
sets font-lock-ensure-function to #'jit-lock-fontify-now).

It's called from Org (see all references to org-font-lock-ensure), for 
instance. mmm-mode calls font-lock-fontify-region directly (sometime 
we'll change it to use font-lock-ensure instead). Also see 
htmlfontify-buffer, occur-engine-line, describe-variable, 
diff-syntax-fontify-props.

It's also used in after-change-functions when font-lock-support-mode is 
not `jit-lock', but that's probably not important.

Some of these definitely can be dealing with long lines and/or large 
files. If font-lock-fontify-region decides the narrowing bounds, it can 
apply them correctly it each case.

I get what you're saying about alternative fontification-functions, 
though: we have multiple similar hooks: it, then jit-lock-functions, and 
font-lock-keywords are more or less used for the same purpose.

I've never seen fontification-functions contain anything but 
jit-lock-function, but jit-lock-functions is used by nlinum, for 
instance. And bug-reference-mode, and goto-address-mode. Perhaps we 
should ask Stefan's opinion on this.

As a practical matter, though, neither goto-addr, nor bug-reference 
should call syntax-ppss or do anything similarly costly in on long 
lines/in large files.

And nlinum might not appreciate being narrowed. It seems to be working 
okay without that in my 20 MB XML file. And in 200 MB one too.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 15 Aug 2022 21:17:23 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> I'm pretty sure that doing all that in Lisp is a good thing, and would
> >> be conducive to whatever improvements we might want to add later...
> > 
> > No, because quite a few packages use jit-lock-register to run stuff
> > that is not font-lock at all, and I don't want that to slow down
> > redisplay when long lines are involved.  And trying various values of
> > a variable exposed to Lisp is easy enough.
> 
> It should also be possible to move the whole invocation of the 
> fontification-functions up to Lisp

How do you do that, given that these functions should be called from
redisplay with the region of buffer text only redisplay knows about?

> Another reason I chose font-lock-fontify-region, though, is that it's 
> called from different places. It's the common entry point for font-lock.

I have no problem with doing something similar in font-lock, I'm just
saying that it cannot be the only place.

> I've never seen fontification-functions contain anything but 
> jit-lock-function, but jit-lock-functions is used by nlinum, for 
> instance. And bug-reference-mode, and goto-address-mode. Perhaps we 
> should ask Stefan's opinion on this.

Stefan was and is on the CC list for the entire discussion.

> And nlinum might not appreciate being narrowed. It seems to be working 
> okay without that in my 20 MB XML file. And in 200 MB one too.

Why is it important what nlinum does, when we have native line numbers
nowadays?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I've never seen fontification-functions contain anything but 
> jit-lock-function,
>

You did not read the thread carefully enough: two weeks ago it was 
mentioned that fontification-functions has at least two other users: in 
ELPA multi-mode.el sets fontification-functions to multi-fontify, and in 
MELPA poly-lock.el sets fontification-functions to poly-lock-function.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> No, I want a simple variable that just gives the size of the narrowed 
> region, with nil meaning don't narrow at all.
>

FWIW, I strongly object to the addition of such a variable (unless it is 
clearly stated that it is added temporarily and should not be used except 
for testing purposes).

Adding such a variable only two weeks after locked narrowing has been 
introduced means that modes will have little incentive to adapt to that 
stronger constraint, if they can "fix" whatever problems that constraint 
might cause by setting that variable to nil in their initialization hooks.

We did not have enough time yet to explore whether and how syntax-ppss can 
be improved.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 19:58, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2022 19:44:07 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> Here's the profiler output anyway:
>>>>
>>>>            1067  85% - command-execute
>>>>            1067  85%  - call-interactively
>>>>            1023  82%   - funcall-interactively
>>>>            1012  81%    - end-of-buffer
>>>>            1008  81%     - recenter
>>>>            1008  81%      - jit-lock-function
>>> When did you last resync from Git?  'recenter' got "optimized"
>>> yesterday for buffers with long lines.
>>
>> Just today.
>>
>> In any case, it doesn't look like recenter's problem, since the output
>> says all (or vast majority) of its time is spent in jit-lock-function.
> 
> AFAIU 'recenter' shouldn't at all call jit-lock-function in a buffer
> with long lines.  Is this "emacs -Q" without any changes from
> defaults?

It's a buffer without long lines. Simply a large file.

'emacs -Q', yes.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 22:25, Gregory Heytings wrote:
> 
>>
>> I've never seen fontification-functions contain anything but 
>> jit-lock-function,
>>
> 
> You did not read the thread carefully enough: two weeks ago it was 
> mentioned that fontification-functions has at least two other users: in 
> ELPA multi-mode.el sets fontification-functions to multi-fontify, and in 
> MELPA poly-lock.el sets fontification-functions to poly-lock-function.

Okay. But those should be easily adapted to use jit-lock-functions instead.

They'll get all the corresponding performance improvements (just-in-time 
fontification) as a bonus.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 22:36, Gregory Heytings wrote:
> 
>>
>> No, I want a simple variable that just gives the size of the narrowed 
>> region, with nil meaning don't narrow at all.
>>
> 
> FWIW, I strongly object to the addition of such a variable (unless it is 
> clearly stated that it is added temporarily and should not be used 
> except for testing purposes).
> 
> Adding such a variable only two weeks after locked narrowing has been 
> introduced means that modes will have little incentive to adapt to that 
> stronger constraint, if they can "fix" whatever problems that constraint 
> might cause by setting that variable to nil in their initialization hooks.

I don't think major modes would take advantage of that var. I hope not, 
at least.

> We did not have enough time yet to explore whether and how syntax-ppss 
> can be improved.

If you wanted the development to take this route, it would have really 
made more sense to hold off on applying the narrowing to 
fontification-functions, keeping font-lock slow-ish in large files. That 
would have encouraged direct work on speeding it up.

But anyway. Suppose we have some more success with syntax-ppss. Two options:

1) parse-partial-sexp is now faster by 10x. That just means we have 10x 
larger files to deal with.

2) We have learned to find a "safe position" in some file types, to 
avoid the full scan from the beginning. It's not hard to implement that 
for JSON and XML, as long as we can stand _some_ imprecision. But there 
will remain all other file types for which nobody has written such logic.

In both cases we fall back to something. And the preceding discussion 
has concluded that that something is narrowing.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov [2022-08-15 22:54:30] wrote:
> On 15.08.2022 22:25, Gregory Heytings wrote:
>>> I've never seen fontification-functions contain anything but
>>> jit-lock-function,
>>>
>> You did not read the thread carefully enough: two weeks ago it was
>> mentioned that fontification-functions has at least two other users: in
>> ELPA multi-mode.el sets fontification-functions to multi-fontify, and in
>> MELPA poly-lock.el sets fontification-functions to poly-lock-function.
>
> Okay.  But those should be easily adapted to use jit-lock-functions instead.

IIUC they use `fontification-functions` to wrap the "select the current
major mode" operation around the call to `jit-lock-function` so that
each submode can use its own set of jit-lock clients.

It's harder to hook into `jit-lock-functions` to do that because you
need to detect when `jit-lock-functions` is modified.

AFAICT mmm-mode does not try to do that at all.

> They'll get all the corresponding performance improvements (just-in-time
> fontification) as a bonus.

Not sure what you mean by that.  `fontification-functions` is already
called "just-in-time" (and those packages don't re-implement jit-lock on
their own, they just wrap themselves around jit-lock).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I don't think major modes would take advantage of that var. I hope not, 
> at least.
>

You're optimistic, let's say I'm less optimistic.

>
> If you wanted the development to take this route, it would have really 
> made more sense to hold off on applying the narrowing to 
> fontification-functions, keeping font-lock slow-ish in large files. That 
> would have encouraged direct work on speeding it up.
>

That's your opinion.  What I think would have most likely happened is what 
you yourself said: these are large files, it's unavoidable that font 
locking takes somes time (these are not your exact words, I hope they 
express what you said faithfully).

>
> But anyway. Suppose we have some more success with syntax-ppss. Two 
> options:
>

There are many other options besides the two options you listed.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 23:25, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
> Dmitry Gutov [2022-08-15 22:54:30] wrote:
>> On 15.08.2022 22:25, Gregory Heytings wrote:
>>>> I've never seen fontification-functions contain anything but
>>>> jit-lock-function,
>>>>
>>> You did not read the thread carefully enough: two weeks ago it was
>>> mentioned that fontification-functions has at least two other users: in
>>> ELPA multi-mode.el sets fontification-functions to multi-fontify, and in
>>> MELPA poly-lock.el sets fontification-functions to poly-lock-function.
>>
>> Okay.  But those should be easily adapted to use jit-lock-functions instead.
> 
> IIUC they use `fontification-functions` to wrap the "select the current
> major mode" operation around the call to `jit-lock-function` so that
> each submode can use its own set of jit-lock clients.
> 
> It's harder to hook into `jit-lock-functions` to do that because you
> need to detect when `jit-lock-functions` is modified.

Do they detect when fontification-functions are modified? Sounds like 
just as much of a rare occurrence.

> AFAICT mmm-mode does not try to do that at all.

It does the wrapping through custom font-lock-fontify-region-function 
and syntax-propertize-function. The major mode specific values are saved 
after the major mode is called. Any subsequent modifications are just 
ignored.

>> They'll get all the corresponding performance improvements (just-in-time
>> fontification) as a bonus.
> 
> Not sure what you mean by that.  `fontification-functions` is already
> called "just-in-time" (and those packages don't re-implement jit-lock on
> their own, they just wrap themselves around jit-lock).

Okay then.

Sounds like somewhere in there there's a duplication of design still.

But also: if they go on to call jit-lock anyway, then they would go 
through whatever application of narrowing that we put in there. Be that 
in font-lock-fontify-region, or in jit-lock-function.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 23:29, Gregory Heytings wrote:

>> If you wanted the development to take this route, it would have really 
>> made more sense to hold off on applying the narrowing to 
>> fontification-functions, keeping font-lock slow-ish in large files. 
>> That would have encouraged direct work on speeding it up.
>>
> 
> That's your opinion.  What I think would have most likely happened is 
> what you yourself said: these are large files, it's unavoidable that 
> font locking takes somes time (these are not your exact words, I hope 
> they express what you said faithfully).

I said those words just as I was working on improving font-lock's 
performance in JSON files. An effort which doesn't make sense if you 
narrow around font-lock anyway.

Also note that until now it has been impossible to improve redisplay of 
big files by working on font-lock (other bottlenecks were much more 
significant).

>> But anyway. Suppose we have some more success with syntax-ppss. Two 
>> options:
>>
> 
> There are many other options besides the two options you listed.

Perhaps. I don't think anybody's mentioned them here, though.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>
> I said those words just as I was working on improving font-lock's 
> performance in JSON files. An effort which doesn't make sense if you 
> narrow around font-lock anyway.
>
> Also note that until now it has been impossible to improve redisplay of 
> big files by working on font-lock (other bottlenecks were much more 
> significant).
>

Wrong.  The (much more common) case of big files _without_ long lines, in 
which the same font locking slowdowns already exist, has AFAICS not been a 
sufficient incentive to improve font locking.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 15.08.2022 23:53, Gregory Heytings wrote:
>>
>> I said those words just as I was working on improving font-lock's 
>> performance in JSON files. An effort which doesn't make sense if you 
>> narrow around font-lock anyway.
>>
>> Also note that until now it has been impossible to improve redisplay 
>> of big files by working on font-lock (other bottlenecks were much more 
>> significant).
>>
> 
> Wrong.  The (much more common) case of big files _without_ long lines, 
> in which the same font locking slowdowns already exist, has AFAICS not 
> been a sufficient incentive to improve font locking.

Okay, you're right about that.

There have been many more complaints about being unable to edit a file 
with a 5000+ line comfortably, than about editing 100+ MB files.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

On 15.08.2022 21:28, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2022 21:17:23 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> I'm pretty sure that doing all that in Lisp is a good thing, and would
>>>> be conducive to whatever improvements we might want to add later...
>>>
>>> No, because quite a few packages use jit-lock-register to run stuff
>>> that is not font-lock at all, and I don't want that to slow down
>>> redisplay when long lines are involved.  And trying various values of
>>> a variable exposed to Lisp is easy enough.
>>
>> It should also be possible to move the whole invocation of the
>> fontification-functions up to Lisp
> 
> How do you do that, given that these functions should be called from
> redisplay with the region of buffer text only redisplay knows about?

Something like in the attached patch.

>> Another reason I chose font-lock-fontify-region, though, is that it's
>> called from different places. It's the common entry point for font-lock.
> 
> I have no problem with doing something similar in font-lock, I'm just
> saying that it cannot be the only place.

If we apply narrowing in handle_fontified_prop (or like in the 
attachment), and also in font-lock-fontify-region, it seems like every 
significant (*) execution path will go through two repetitions of the 
same logic.

That's not ideal.

(*) Significant by the metric of having impact on Emacs's performance, 
in large files especially.

>> I've never seen fontification-functions contain anything but
>> jit-lock-function, but jit-lock-functions is used by nlinum, for
>> instance. And bug-reference-mode, and goto-address-mode. Perhaps we
>> should ask Stefan's opinion on this.
> 
> Stefan was and is on the CC list for the entire discussion.
> 
>> And nlinum might not appreciate being narrowed. It seems to be working
>> okay without that in my 20 MB XML file. And in 200 MB one too.
> 
> Why is it important what nlinum does, when we have native line numbers
> nowadays?

Just an example of a mode which uses jit-lock-functions. With an 
implication that other, less known modes, might have similar needs and 
behavior. There aren't too many known examples.

[-- Attachment #2: run-fontification-functions.diff --]
[-- Type: text/x-patch, Size: 2306 bytes --]

diff --git a/lisp/font-core.el b/lisp/font-core.el
index f70c42bb03..09e55bcfb1 100644
--- a/lisp/font-core.el
+++ b/lisp/font-core.el
@@ -266,6 +266,12 @@ turn-on-font-lock-if-desired
     (let (inhibit-quit)
       (turn-on-font-lock))))
 
+(defun run-fontification-functions (pos)
+  ;; XXX: Apply narrowing around here.
+  ;; (pcase-exhaustive font-lock-large-files
+  ;;   ...)
+  (run-hook-with-args 'fontification-functions pos))
+
 (define-globalized-minor-mode global-font-lock-mode
   font-lock-mode turn-on-font-lock-if-desired
   ;; What was this :extra-args thingy for?  --Stef
diff --git a/src/xdisp.c b/src/xdisp.c
index 1e8f70b2db..ce90aeefbc 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4381,53 +4381,18 @@ handle_fontified_prop (struct it *it)
 	  NILP (prop) && IT_CHARPOS (*it) < Z))
     {
       specpdl_ref count = SPECPDL_INDEX ();
-      Lisp_Object val;
       struct buffer *obuf = current_buffer;
       ptrdiff_t begv = BEGV, zv = ZV;
       bool old_clip_changed = current_buffer->clip_changed;
       bool saved_inhibit_flag = it->f->inhibit_clear_image_cache;
 
-      val = Vfontification_functions;
-      specbind (Qfontification_functions, Qnil);
-
       eassert (it->end_charpos == ZV);
 
       /* Don't allow Lisp that runs from 'fontification-functions'
 	 clear our face and image caches behind our back.  */
       it->f->inhibit_clear_image_cache = true;
 
-      if (!CONSP (val) || EQ (XCAR (val), Qlambda))
-	safe_call1 (val, pos);
-      else
-	{
-	  Lisp_Object fns, fn;
-
-	  fns = Qnil;
-
-	  for (; CONSP (val); val = XCDR (val))
-	    {
-	      fn = XCAR (val);
-
-	      if (EQ (fn, Qt))
-		{
-		  /* A value of t indicates this hook has a local
-		     binding; it means to run the global binding too.
-		     In a global value, t should not occur.  If it
-		     does, we must ignore it to avoid an endless
-		     loop.  */
-		  for (fns = Fdefault_value (Qfontification_functions);
-		       CONSP (fns);
-		       fns = XCDR (fns))
-		    {
-		      fn = XCAR (fns);
-		      if (!EQ (fn, Qt))
-			safe_call1 (fn, pos);
-		    }
-		}
-	      else
-		safe_call1 (fn, pos);
-	    }
-	}
+      safe_call1 (intern ("run-fontification-functions"), pos);
 
       it->f->inhibit_clear_image_cache = saved_inhibit_flag;
       unbind_to (count, Qnil);

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> FWIW, I strongly object to the addition of such a variable (unless it is
>> clearly stated that it is added temporarily and should not be used except
>> for testing purposes).
>> Adding such a variable only two weeks after locked narrowing has been
>> introduced means that modes will have little incentive to adapt to that
>> stronger constraint, if they can "fix" whatever problems that constraint
>> might cause by setting that variable to nil in their initialization hooks.
> I don't think major modes would take advantage of that var.
> I hope not, at least.

I wouldn't mind because if things get slow, we get to point the finger
squarely at the culprit.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> FWIW, I strongly object to the addition of such a variable (unless it 
>>> is clearly stated that it is added temporarily and should not be used 
>>> except for testing purposes).
>>>
>>> Adding such a variable only two weeks after locked narrowing has been 
>>> introduced means that modes will have little incentive to adapt to 
>>> that stronger constraint, if they can "fix" whatever problems that 
>>> constraint might cause by setting that variable to nil in their 
>>> initialization hooks.
>>
>> I don't think major modes would take advantage of that var. I hope not, 
>> at least.
>
> I wouldn't mind because if things get slow, we get to point the finger 
> squarely at the culprit.
>

If it is clearly stated that it should only be used for testing/debugging 
purposes, yes.  Otherwise it's just another variable that anyone can use 
as they see fit.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> IIUC they use `fontification-functions` to wrap the "select the current
>> major mode" operation around the call to `jit-lock-function` so that
>> each submode can use its own set of jit-lock clients.
>> It's harder to hook into `jit-lock-functions` to do that because you
>> need to detect when `jit-lock-functions` is modified.
>
> Do they detect when fontification-functions are modified? Sounds like just
> as much of a rare occurrence.

Could be.

>> AFAICT mmm-mode does not try to do that at all.
>
> It does the wrapping through custom font-lock-fontify-region-function and
> syntax-propertize-function. The major mode specific values are saved after
> the major mode is called. Any subsequent modifications are just ignored.

I'm not sure why they decided to wrap `fontification-functions`, but in
theory it lets them support things like a submode that's using
`nhexl-mode`, another using `nlinum-mode`, and yet another using
`glasses-mode`.

In any case, these multi mode packages always need to "look back" in
order to find which mode to use at a given position, so they inevitably
need some kind of widening to work correctly.

>>> They'll get all the corresponding performance improvements (just-in-time
>>> fontification) as a bonus.
>> Not sure what you mean by that.  `fontification-functions` is already
>> called "just-in-time" (and those packages don't re-implement jit-lock on
>> their own, they just wrap themselves around jit-lock).
> Okay then.
> Sounds like somewhere in there there's a duplication of design still.

I think the duplication is in the weird semantics of
`fontification-functions`: this supposedly holds a list of functions,
but it's difficult to make it work right with more than one function.
I think it should be replaced with a single `fontification-function` instead.

> But also: if they go on to call jit-lock anyway, then they would go through
> whatever application of narrowing that we put in there. Be that in
> font-lock-fontify-region, or in jit-lock-function.

Indeed.  But before doing so, they may(?) scan the whole buffer to find out
which submode to use at the current location.

I think it's OK to "break" those packages with some forced narrowing
until they get improved to do something sane in large files or
long lines.  But of course it should be possible to unbreak them without
recompiling Emacs.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> There have been many more complaints about being unable to edit a file with
> a 5000+ line comfortably, than about editing 100+ MB files.

Also because we have things like so-long to handle the large file case.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> There have been many more complaints about being unable to edit a file 
>> with a 5000+ line comfortably, than about editing 100+ MB files.
>
> Also because we have things like so-long to handle the large file case.
>

These things are expedients for want of anything better, aren't they?

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> If it is clearly stated that it should only be used for testing/debugging
> purposes, yes.  Otherwise it's just another variable that anyone can use as
> they see fit.

I don't think it should be restricted to testing/debugging.  But it
should clearly say that using it should be limited to code that has been
carefully written to take into account the potential
performance implications.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Otherwise it's just another variable that anyone can use as they see fit.

All Emacs variables and functions are like that.  It's not something to
deplore, on the contrary, it's what makes Emacs what it is.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-08-15 22:14:07] wrote:
>>> There have been many more complaints about being unable to edit a file
>>> with a 5000+ line comfortably, than about editing 100+ MB files.
>> Also because we have things like so-long to handle the large file case.
> These things are expedients for want of anything better, aren't they?

Yes, just like the narrowing your use in your long lines patch.

They're slightly different in that they don't try to prevent replacing
those solutions with something better (or worse), tho.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 16.08.2022 00:57, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:

>>> AFAICT mmm-mode does not try to do that at all.
>>
>> It does the wrapping through custom font-lock-fontify-region-function and
>> syntax-propertize-function. The major mode specific values are saved after
>> the major mode is called. Any subsequent modifications are just ignored.
> 
> I'm not sure why they decided to wrap `fontification-functions`, but in
> theory it lets them support things like a submode that's using
> `nhexl-mode`, another using `nlinum-mode`, and yet another using
> `glasses-mode`.

I'm guessing this could also be implemented through a wrapping 
jit-lock-functions element if mmm-mode, say, saved the value of this 
variable corresponding to each major mode. Similarly to 
mmm-fontify-region-function.

But doing it in a caller up a level looks a little cleaner, of course.

> In any case, these multi mode packages always need to "look back" in
> order to find which mode to use at a given position, so they inevitably
> need some kind of widening to work correctly.

Right. Even if they need to skip back just a little bit (usually), being 
able to look back is useful.

Although the "which mode" information is often available through an 
overlay or text property at point, so that need might be worked around 
in a lot of cases.

>>>> They'll get all the corresponding performance improvements (just-in-time
>>>> fontification) as a bonus.
>>> Not sure what you mean by that.  `fontification-functions` is already
>>> called "just-in-time" (and those packages don't re-implement jit-lock on
>>> their own, they just wrap themselves around jit-lock).
>> Okay then.
>> Sounds like somewhere in there there's a duplication of design still.
> 
> I think the duplication is in the weird semantics of
> `fontification-functions`: this supposedly holds a list of functions,
> but it's difficult to make it work right with more than one function.
> I think it should be replaced with a single `fontification-function` instead.

I would try to go further: either document that `fontification-function' 
is ultimately supposed to call jit-lock, or call jit-lock-function 
directly (from handle_fontified_props or whatever), forcing polymode and 
multi-mode to use around-advice on that function (will that work on a 
function directly called from C?).

If we clarify that fontification has to go through jit-lock-function 
either way, we can more confidently move the application of font-lock 
narrowing to jit-lock-fontify-now (that's where fontification-functions 
and font-lock-ensure meet).

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 15 Aug 2022 19:36:49 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Dmitry Gutov <dgutov@yandex.ru>, 56682@debbugs.gnu.org, 
>     monnier@iro.umontreal.ca
> 
> > No, I want a simple variable that just gives the size of the narrowed 
> > region, with nil meaning don't narrow at all.
> >
> 
> FWIW, I strongly object to the addition of such a variable (unless it is 
> clearly stated that it is added temporarily and should not be used except 
> for testing purposes).
> 
> Adding such a variable only two weeks after locked narrowing has been 
> introduced means that modes will have little incentive to adapt to that 
> stronger constraint, if they can "fix" whatever problems that constraint 
> might cause by setting that variable to nil in their initialization hooks.

Modes can do that already by changing long-line-threshold to nil,
don't they?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Mon, 15 Aug 2022 22:51:16 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 15.08.2022 19:58, Eli Zaretskii wrote:
> >> Date: Mon, 15 Aug 2022 19:44:07 +0300
> >> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >>
> >>>> Here's the profiler output anyway:
> >>>>
> >>>>            1067  85% - command-execute
> >>>>            1067  85%  - call-interactively
> >>>>            1023  82%   - funcall-interactively
> >>>>            1012  81%    - end-of-buffer
> >>>>            1008  81%     - recenter
> >>>>            1008  81%      - jit-lock-function
> >>> When did you last resync from Git?  'recenter' got "optimized"
> >>> yesterday for buffers with long lines.
> >>
> >> Just today.
> >>
> >> In any case, it doesn't look like recenter's problem, since the output
> >> says all (or vast majority) of its time is spent in jit-lock-function.
> > 
> > AFAIU 'recenter' shouldn't at all call jit-lock-function in a buffer
> > with long lines.  Is this "emacs -Q" without any changes from
> > defaults?
> 
> It's a buffer without long lines. Simply a large file.

??? Are we talking about long-line.xml, or are we talking about
some other file?  We did nothing to change the behavior in large files
without long lines.

If you are saying that 'recenter' takes too long in some file, and
therefore delays M-> for too long, it is probably a separate issue, so
please report that as a separate bug, preferably with the file that
causes it.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 16 Aug 2022 00:18:35 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> It should also be possible to move the whole invocation of the
> >> fontification-functions up to Lisp
> > 
> > How do you do that, given that these functions should be called from
> > redisplay with the region of buffer text only redisplay knows about?
> 
> Something like in the attached patch.

I fail to see the difference which would justify such a change.

> If we apply narrowing in handle_fontified_prop (or like in the 
> attachment), and also in font-lock-fontify-region, it seems like every 
> significant (*) execution path will go through two repetitions of the 
> same logic.
> 
> That's not ideal.

But not a catastrophe, either.

> >> And nlinum might not appreciate being narrowed. It seems to be working
> >> okay without that in my 20 MB XML file. And in 200 MB one too.
> > 
> > Why is it important what nlinum does, when we have native line numbers
> > nowadays?
> 
> Just an example of a mode which uses jit-lock-functions. With an 
> implication that other, less known modes, might have similar needs and 
> behavior. There aren't too many known examples.

Let's hear them hollering, and then see what are their problems.

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> However, I do insist on my statement that expecting major-mode to not
>> widen is a bug.
>
> No, it's a design decision, which makes it possible to use specific
> implementation techniques.  Org-mode's support for multiple major modes
> (org-babel) uses an implementation technique that does not require
> narrowing.  Both approaches have their virtues and downsides, but if the
> submode widens then the supermode has less freedom in terms of which
> implementation technique to use.

I agree that major-mode doing (widen) may be problematic. However, at
least Org mode does need to do it in order to parse the buffer.

>> There are valid cases when a major mode needs to examine
>> the whole buffer.
>
> Not if the whole buffer contains text which the major mode can't
> understand.  There are valid cases where a major mode needs to widen,
> indeed, but not when called from font-lock or from
> `indent-line-function`.

`org-indent-line' (the value of `indent-line-function' in Org mode) does
require invoking parser in order to determine the syntactic object at
point, which, in turn, needs to consider the full buffer contents. This
is prescribed by Org syntax. We cannot avoid this.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I have seen very long lines in Org. Specifically, in source block
>> output. (http query)
>
> How long is "very long" in those cases?

AFAIR, over a screen full of text when truncate-lines is set to nil.
It should be over 5k symbols in my screen.

I remember this because it triggered a significant fontification
slowdown caused by some (now fixed) Org regexp hitting too many matches
on that line.

Also, I just checked some of my org files and found a >15k chars long line.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Otherwise it's just another variable that anyone can use as they see 
>> fit.
>
> All Emacs variables and functions are like that.
>

Not all of them, no.  See the three examples I mentioned earlier: 
internal-interpreter-environment, max-specpdl-size, max-lisp-eval-depth. 
And I'm sure there are others.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>>> There have been many more complaints about being unable to edit a 
>>>> file with a 5000+ line comfortably, than about editing 100+ MB files.
>>>
>>> Also because we have things like so-long to handle the large file 
>>> case.
>>
>> These things are expedients for want of anything better, aren't they?
>
> Yes, just like the narrowing your use in your long lines patch.
>

That's... unfair.  These things turn off everything (in particular, the 
major mode and font locking) and provide a bare-bones editing experience. 
What we'd like now is to provide Emacs users a near-complete editing 
experience in all buffers.

>
> They're slightly different in that they don't try to prevent replacing 
> those solutions with something better (or worse), tho.
>

When one looks at things from far enough, everything is indeed "slightly 
different" from everything else.  For example, the Statue of Liberty is 
"slightly different" from Empire State Building.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>>>> There have been many more complaints about being unable to edit a 
>>>>> file with a 5000+ line comfortably, than about editing 100+ MB 
>>>>> files.
>>>> 
>>>> Also because we have things like so-long to handle the large file 
>>>> case.
>>> 
>>> These things are expedients for want of anything better, aren't they?
>> 
>> Yes, just like the narrowing your use in your long lines patch.
>
> That's... unfair.  These things turn off everything (in particular, the 
> major mode and font locking) and provide a bare-bones editing 
> experience. What we'd like now is to provide Emacs users a near-complete 
> editing experience in all buffers.
>

And, by the way, what could be a better solution?  Without assuming that 
computers have infinitely fast CPUs, that is?

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> No, I want a simple variable that just gives the size of the narrowed 
>>> region, with nil meaning don't narrow at all.
>>
>> FWIW, I strongly object to the addition of such a variable (unless it 
>> is clearly stated that it is added temporarily and should not be used 
>> except for testing purposes).
>>
>> Adding such a variable only two weeks after locked narrowing has been 
>> introduced means that modes will have little incentive to adapt to that 
>> stronger constraint, if they can "fix" whatever problems that 
>> constraint might cause by setting that variable to nil in their 
>> initialization hooks.
>
> Modes can do that already by changing long-line-threshold to nil, don't 
> they?
>

They can indeed, but that variable is documented as "for debugging 
purposes" only.  Moreover setting that variable to nil implies that they 
deactivate all long line optimizations, which is a higher price to pay.

The way I see this is that Emacs is doing efforts to solve editing 
slowdowns as best as possible, but that there is a "last mile" that cannot 
be solved without the collaboration from major and minor modes.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> I'm not sure I correctly understand what you mean, but it seems to me 
>> that the fact that the Org parser caches its results implies that Org 
>> doesn't need to access the whole buffer to make "local" decisions.  It 
>> can use the data in cache for everything that is outside of the current 
>> narrowing, and update the portion of the cache corresponding to the 
>> current narrowing. IOW, there is no need to maintain separate parser 
>> caches for each possible narrowing state.  Am I misunderstanding 
>> something?
>
> Cache existence is not guaranteed.
>

Can that cache not be created unconditionally when the Org file is opened?

>
> When cache does not exist, Org must widen. If not, the cache will be 
> valid only for the current narrowing.
>
> When cache do exist, Org may not need to widen. But that also mean that 
> widening has happened in the past when the cache was generated.
>
> At the end, Org does need to be able to widen safely. At least in some 
> cases.
>

That will always be possible.  What should better be avoided is to use 
widen (and to scan the whole buffer) in performance-critical places, such 
as fontification-functions.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Not if the whole buffer contains text which the major mode can't 
>> understand.  There are valid cases where a major mode needs to widen, 
>> indeed, but not when called from font-lock or from 
>> `indent-line-function`.
>
> `org-indent-line' (the value of `indent-line-function' in Org mode) does 
> require invoking parser in order to determine the syntactic object at 
> point, which, in turn, needs to consider the full buffer contents. This 
> is prescribed by Org syntax. We cannot avoid this.
>

Do you really mean that there is no way to avoid invoking the parser on 
the whole buffer, and to possibly process 20 MB of data, when 
org-indent-line is called?

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

>> Cache existence is not guaranteed.
>>
>
> Can that cache not be created unconditionally when the Org file is opened?

Without cache, the parser will have to process everything on every
request to determine object at point.

>> When cache does not exist, Org must widen. If not, the cache will be 
>> valid only for the current narrowing.
>>
>> When cache do exist, Org may not need to widen. But that also mean that 
>> widening has happened in the past when the cache was generated.
>>
>> At the end, Org does need to be able to widen safely. At least in some 
>> cases.
>>
>
> That will always be possible.  What should better be avoided is to use 
> widen (and to scan the whole buffer) in performance-critical places, such 
> as fontification-functions.

We tried to. In fact, public Org repository does use regexp search and
tries to scan as little as possible inside fontification-function.

However, that approach proved incorrect in a number of occasions. We
have a stable stream of bug reports related to incorrect fontification.
Some were fixed by creating more intricate heuristics, but we reached
the point when using the parser can be actually faster compared to the
fragile heuristic tower in our fontification code.

Our parser, in fact, does not really scan the whole buffer from the very
beginning to determine object at point. It only parses the minimal
possible portion that is sufficient to determine syntax object at the
requested position. This scan is very fast and becomes even faster with
caching. But it does need to look at the whole buffer because Org syntax
is context-dependent. Previous objects outside narrowing can alter the
syntax inside.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

>> `org-indent-line' (the value of `indent-line-function' in Org mode) does 
>> require invoking parser in order to determine the syntactic object at 
>> point, which, in turn, needs to consider the full buffer contents. This 
>> is prescribed by Org syntax. We cannot avoid this.
>>
>
> Do you really mean that there is no way to avoid invoking the parser on 
> the whole buffer, and to possibly process 20 MB of data, when 
> org-indent-line is called?

In the worst-case scenario, yes. However, it is usually not the case.
org-indent-line usually needs to look no further than the parent
headline - a few lines above in the majority of Org documents.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 16.08.2022 05:45, Eli Zaretskii wrote:
>> If we apply narrowing in handle_fontified_prop (or like in the
>> attachment), and also in font-lock-fontify-region, it seems like every
>> significant (*) execution path will go through two repetitions of the
>> same logic.
>>
>> That's not ideal.
> But not a catastrophe, either.

I suppose not.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: monnier@iro.umontreal.ca,  dgutov@yandex.ru,  gregory@heytings.org,
>   56682@debbugs.gnu.org
> Date: Tue, 16 Aug 2022 12:38:37 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I have seen very long lines in Org. Specifically, in source block
> >> output. (http query)
> >
> > How long is "very long" in those cases?
> 
> AFAIR, over a screen full of text when truncate-lines is set to nil.
> It should be over 5k symbols in my screen.
> 
> I remember this because it triggered a significant fontification
> slowdown caused by some (now fixed) Org regexp hitting too many matches
> on that line.
> 
> Also, I just checked some of my org files and found a >15k chars long line.

5k shouldn't present a problem, especially when some/most of it is
invisible (as frequently is the case in Org).  15k might be a problem,
indeed, depending on what's there.  However, since you never
complained about long lines until this day, I guess it isn't such a
big problem.

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Also, I just checked some of my org files and found a >15k chars long line.
>
> 5k shouldn't present a problem, especially when some/most of it is
> invisible (as frequently is the case in Org).  15k might be a problem,
> indeed, depending on what's there.  However, since you never
> complained about long lines until this day, I guess it isn't such a
> big problem.

You are likely right. Note that this branch of the discussion mostly
started from me clarifying data about some large Org files. I did not
imply that Org files commonly have very long lines. Simply large-sized
Org files are more common (5-20Mb).

Also, regarding most of the lines being invisible. Unless I miss
something, jit-lock actually tries to trigger fontification in the
invisible lines when we have something like:
-- <window top>
* headline
<Mbs of invisible text hidden using 'invisible text property>
* another headline
-- <window bottom>

I even had to write a dedicated filter in
org-fold-core.el:org-fold-core-fontify-region (on Org main branch) to
avoid fontification inside hidden text.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>> Cache existence is not guaranteed.
>>
>> Can that cache not be created unconditionally when the Org file is 
>> opened?
>
> Without cache, the parser will have to process everything on every 
> request to determine object at point.
>

I'm not sure I understand what you mean?  If the cache is created when the 
Org file is opened and updated when necessary, its existence becomes 
guaranteed, no?

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Gregory Heytings <gregory@heytings.org> writes:

>>>> Cache existence is not guaranteed.
>>>
>>> Can that cache not be created unconditionally when the Org file is 
>>> opened?
>>
>> Without cache, the parser will have to process everything on every 
>> request to determine object at point.
>>
>
> I'm not sure I understand what you mean?  If the cache is created when the 
> Org file is opened and updated when necessary, its existence becomes 
> guaranteed, no?

Cache (or parts of cache) may be invalidated when buffer is edited.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 16 Aug 2022 08:26:49 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> >> Adding such a variable only two weeks after locked narrowing has been 
> >> introduced means that modes will have little incentive to adapt to that 
> >> stronger constraint, if they can "fix" whatever problems that 
> >> constraint might cause by setting that variable to nil in their 
> >> initialization hooks.
> >
> > Modes can do that already by changing long-line-threshold to nil, don't 
> > they?
> 
> They can indeed, but that variable is documented as "for debugging 
> purposes" only.

That never stopped anyone, IME.

> Moreover setting that variable to nil implies that they 
> deactivate all long line optimizations, which is a higher price to pay.

They could do that only locally in one buffer, or temporarily, or just
set it to a much larger value, or ...

> The way I see this is that Emacs is doing efforts to solve editing 
> slowdowns as best as possible, but that there is a "last mile" that cannot 
> be solved without the collaboration from major and minor modes.

I'm not worried by that: we do control what code gets installed in the
modes that are part of Emacs, so we can always prevent any such modes
from (ab)using these variables.  And if third-party packages want to
do that, in effect harming their users, it's eventually their funeral.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: monnier@iro.umontreal.ca,  dgutov@yandex.ru,  gregory@heytings.org,
>   56682@debbugs.gnu.org
> Date: Tue, 16 Aug 2022 19:30:11 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Also, I just checked some of my org files and found a >15k chars long line.
> >
> > 5k shouldn't present a problem, especially when some/most of it is
> > invisible (as frequently is the case in Org).  15k might be a problem,
> > indeed, depending on what's there.  However, since you never
> > complained about long lines until this day, I guess it isn't such a
> > big problem.
> 
> You are likely right. Note that this branch of the discussion mostly
> started from me clarifying data about some large Org files. I did not
> imply that Org files commonly have very long lines. Simply large-sized
> Org files are more common (5-20Mb).
> 
> Also, regarding most of the lines being invisible. Unless I miss
> something, jit-lock actually tries to trigger fontification in the
> invisible lines when we have something like:
> -- <window top>
> * headline
> <Mbs of invisible text hidden using 'invisible text property>
> * another headline
> -- <window bottom>

That 's true, but:

  . jit-lock is called by the display engine, and the display engine
    skips invisible text
  . jit-lock calls the fontification functions with a relatively small
    chunk of buffer text

The result of these two factoids is that (a) jit-lock shouldn't
attempt fontifications, except when the un-fontified text starts
exactly at the beginning of invisible text; and (b) even if
fontifications are invoked on invisible text, it should be a
relatively small chunk of it.

More importantly, when I wrote the above, I wasn't thinking about
fontifications, I was thinking about the rest of the display code,
which always skips invisible characters, and skips them using
relatively fast search for text properties, not the much slower
iteration through the text one character at a time.

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Also, regarding most of the lines being invisible. Unless I miss
>> something, jit-lock actually tries to trigger fontification in the
>> invisible lines when we have something like:
>> -- <window top>
>> * headline
>> <Mbs of invisible text hidden using 'invisible text property>
>> * another headline
>> -- <window bottom>
>
> That 's true, but:
>
>   . jit-lock is called by the display engine, and the display engine
>     skips invisible text
>   . jit-lock calls the fontification functions with a relatively small
>     chunk of buffer text

Apparently large chunks of text were requested in my tests. In any case,
let's not go into this here. Let me re-check these observations again and
file a bug report if I can create a reproducer. This issue is unrelated
to the changes described in the current thread.

> More importantly, when I wrote the above, I wasn't thinking about
> fontifications, I was thinking about the rest of the display code,
> which always skips invisible characters, and skips them using
> relatively fast search for text properties, not the much slower
> iteration through the text one character at a time.

Just in case, please note that Org does not touch 'invisible text
property to hide text. Instead, we set char-property-alias-alist
remapping several other text properties with different non-nil values
into 'invisible. This also means that functions like
`next-single-property-change' may not skip the whole invisible chunk in
one go. Not sure if it is important.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>>>> Adding such a variable only two weeks after locked narrowing has been 
>>>> introduced means that modes will have little incentive to adapt to 
>>>> that stronger constraint, if they can "fix" whatever problems that 
>>>> constraint might cause by setting that variable to nil in their 
>>>> initialization hooks.
>>>
>>> Modes can do that already by changing long-line-threshold to nil, 
>>> don't they?
>>
>> They can indeed, but that variable is documented as "for debugging 
>> purposes" only.
>
> That never stopped anyone, IME.
>

At least they cannot tell the haven't been warned...

Can this change at least wait a few days?  I'm working on the next 
iteration of the code, I'll see if I can include such a feature.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: monnier@iro.umontreal.ca,  dgutov@yandex.ru,  gregory@heytings.org,
>   56682@debbugs.gnu.org
> Date: Tue, 16 Aug 2022 20:25:57 +0800
> 
> > More importantly, when I wrote the above, I wasn't thinking about
> > fontifications, I was thinking about the rest of the display code,
> > which always skips invisible characters, and skips them using
> > relatively fast search for text properties, not the much slower
> > iteration through the text one character at a time.
> 
> Just in case, please note that Org does not touch 'invisible text
> property to hide text. Instead, we set char-property-alias-alist
> remapping several other text properties with different non-nil values
> into 'invisible. This also means that functions like
> `next-single-property-change' may not skip the whole invisible chunk in
> one go. Not sure if it is important.

I don't think I understand what you are saying, but as long as
whatever you do causes the relevant text not to be displayed, it means
the display engine did understand the text is invisible and refrained
from displaying it.  Which means it skipped it, as it does with any
invisible text.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 16 Aug 2022 12:42:04 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> 
> >>>> Adding such a variable only two weeks after locked narrowing has been 
> >>>> introduced means that modes will have little incentive to adapt to 
> >>>> that stronger constraint, if they can "fix" whatever problems that 
> >>>> constraint might cause by setting that variable to nil in their 
> >>>> initialization hooks.
> >>>
> >>> Modes can do that already by changing long-line-threshold to nil, 
> >>> don't they?
> >>
> >> They can indeed, but that variable is documented as "for debugging 
> >> purposes" only.
> >
> > That never stopped anyone, IME.
> >
> 
> At least they cannot tell the haven't been warned...
> 
> Can this change at least wait a few days?

It will wait regardless, because I still have too many things on my
plate that are more important.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The way I see this is that Emacs is doing efforts to solve editing slowdowns
> as best as possible, but that there is a "last mile" that cannot be solved
> without the collaboration from major and minor modes.

The locked narrowing prevents that "last mile", sadly.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  56682@debbugs.gnu.org,  dgutov@yandex.ru
> Date: Tue, 16 Aug 2022 09:07:59 -0400
> 
> > The way I see this is that Emacs is doing efforts to solve editing slowdowns
> > as best as possible, but that there is a "last mile" that cannot be solved
> > without the collaboration from major and minor modes.
> 
> The locked narrowing prevents that "last mile", sadly.

Nonsense, it doesn't prevent any such things.

Let's not exaggerate when we try to make a point, okay?

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I agree that major-mode doing (widen) may be problematic. However, at
> least Org mode does need to do it in order to parse the buffer.

There's no doubt that a major mode may need to override a user's
narrowing in order to work correctly.  That doesn't mean that it
absolutely always needs to have access to the whole buffer.

>>> There are valid cases when a major mode needs to examine
>>> the whole buffer.
>> Not if the whole buffer contains text which the major mode can't
>> understand.  There are valid cases where a major mode needs to widen,
>> indeed, but not when called from font-lock or from
>> `indent-line-function`.
> `org-indent-line' (the value of `indent-line-function' in Org mode) does
> require invoking parser in order to determine the syntactic object at
> point, which, in turn, needs to consider the full buffer contents.

*All* indentation code needs to look at the context.  For that reason
`indent-according-to-mode` already widens as appropriate before calling
`indent-line-function`.  For that reason the function placed on
`indent-line-function` *never* needs to widen (and should never widen).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Also, regarding most of the lines being invisible. Unless I miss
> something, jit-lock actually tries to trigger fontification in the
> invisible lines when we have something like:
> -- <window top>
> * headline
> <Mbs of invisible text hidden using 'invisible text property>
> * another headline
> -- <window bottom>

Normally, jit-lock does not fontify invisible text except as a side
effect of fontifying visible text (because the fontification is made in
chunks of about 1kB, plus rounding up to wholelines and such).

So it's normal that it fontifies a couple kB of that MB of invisible
text, but if it fontifies all those MBs I'd consider it as a bug.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> There are valid cases when a major mode needs to examine
>>>> the whole buffer.
>>> Not if the whole buffer contains text which the major mode can't
>>> understand.  There are valid cases where a major mode needs to widen,
>>> indeed, but not when called from font-lock or from
>>> `indent-line-function`.
>> `org-indent-line' (the value of `indent-line-function' in Org mode) does
>> require invoking parser in order to determine the syntactic object at
>> point, which, in turn, needs to consider the full buffer contents.
>
> *All* indentation code needs to look at the context.  For that reason
> `indent-according-to-mode` already widens as appropriate before calling
> `indent-line-function`.  For that reason the function placed on
> `indent-line-function` *never* needs to widen (and should never widen).

Then, can Emacs throw an error if widen must not be allowed in certain
scenarios? In complex codebase like Org, we cannot guarantee that some
user-customized function does not widen. Or spurious advice. Debugging
absence of widening somewhere inside deep function calls will be a
nightmare.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>>>> There have been many more complaints about being unable to edit a file
>>>>> with a 5000+ line comfortably, than about editing 100+ MB files.
>>>> Also because we have things like so-long to handle the large file case.
>>> These things are expedients for want of anything better, aren't they?
>> Yes, just like the narrowing your use in your long lines patch.
> That's... unfair.

I disagree.  And I don't think it's something you should be ashamed of:
it's standard practice to use expedient solutions for want of anything
better.  It's how we make progress.

Those expedient solutions (i.e. those cheap solutions that still cover
most cases tolerably) might be "not ideal" but finding them is usually
not easy, so pride would be more appropriate.

>> They're slightly different in that they don't try to prevent replacing
>> those solutions with something better (or worse), tho.
> When one looks at things from far enough, everything is indeed "slightly
> different" from everything else.  For example, the Statue of Liberty is
> "slightly different" from Empire State Building.

I agree with you that "slightly different" is a euphemism when
describing whether an expedient solution can be overridden or not.
:-)


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>>>>> There have been many more complaints about being unable to edit a file
>>>>>> with a 5000+ line comfortably, than about editing 100+ MB files.
>>>>> Also because we have things like so-long to handle the large file case.
>>>> These things are expedients for want of anything better, aren't they?
>>> Yes, just like the narrowing your use in your long lines patch.
>> That's... unfair.  These things turn off everything (in particular, the
>> major mode and font locking) and provide a bare-bones editing
>> experience. What we'd like now is to provide Emacs users a near-complete
>> editing experience in all buffers.
> And, by the way, what could be a better solution?

A solution that is humble enough to leave an open door for ad-hoc
case-specific handling of the problem.

> Without assuming that computers have infinitely fast CPUs, that is?

If the font-lock rules to use in the buffer depend on whether the first
line of the buffer starts with #! or not, then your narrowing will break
them even tho they could work perfectly fine in a 1TB buffer without
requiring an infinitely fast CPU (since they don't need to scan the
whole buffer: just the region of interest plus the first 2 chars).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 56682@debbugs.gnu.org,  Eli Zaretskii <eliz@gnu.org>,  Dmitry Gutov
>  <dgutov@yandex.ru>
> Date: Tue, 16 Aug 2022 09:43:56 -0400
> 
> > And, by the way, what could be a better solution?
> 
> A solution that is humble enough to leave an open door for ad-hoc
> case-specific handling of the problem.

I don't get it: a way to provide such an open door was described here
many days ago.  It's not like you, Stefan, lack write access to the
repository.  What am I missing?

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 16.08.2022 16:09, Eli Zaretskii wrote:
>> From: Stefan Monnier<monnier@iro.umontreal.ca>
>> Cc: Eli Zaretskii<eliz@gnu.org>,56682@debbugs.gnu.org,dgutov@yandex.ru
>> Date: Tue, 16 Aug 2022 09:07:59 -0400
>>
>>> The way I see this is that Emacs is doing efforts to solve editing slowdowns
>>> as best as possible, but that there is a "last mile" that cannot be solved
>>> without the collaboration from major and minor modes.
>> The locked narrowing prevents that "last mile", sadly.
> Nonsense, it doesn't prevent any such things.
> 
> Let's not exaggerate when we try to make a point, okay?

Why nonsense?

I've made the same point a bunch of times: arbitrary narrowing without 
regard for context stops any potential major-mode specific code from 
looking for a "safe position" which might be outside of that narrowing.

The new variable (which will let us increase the font-lock narrowing 
radius) should get us 95% there, though.

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 16.08.2022 05:33, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2022 22:51:16 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 15.08.2022 19:58, Eli Zaretskii wrote:
>>>> Date: Mon, 15 Aug 2022 19:44:07 +0300
>>>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>>>> From: Dmitry Gutov <dgutov@yandex.ru>
>>>>
>>>>>> Here's the profiler output anyway:
>>>>>>
>>>>>>             1067  85% - command-execute
>>>>>>             1067  85%  - call-interactively
>>>>>>             1023  82%   - funcall-interactively
>>>>>>             1012  81%    - end-of-buffer
>>>>>>             1008  81%     - recenter
>>>>>>             1008  81%      - jit-lock-function
>>>>> When did you last resync from Git?  'recenter' got "optimized"
>>>>> yesterday for buffers with long lines.
>>>>
>>>> Just today.
>>>>
>>>> In any case, it doesn't look like recenter's problem, since the output
>>>> says all (or vast majority) of its time is spent in jit-lock-function.
>>>
>>> AFAIU 'recenter' shouldn't at all call jit-lock-function in a buffer
>>> with long lines.  Is this "emacs -Q" without any changes from
>>> defaults?
>>
>> It's a buffer without long lines. Simply a large file.
> 
> ??? Are we talking about long-line.xml, or are we talking about
> some other file?  We did nothing to change the behavior in large files
> without long lines.

I'm talking about a 20 MB file I created manually with some simple 
contents. Just no long lines.

long-line.xml is only 300 KB, but if you query-replace '</' with '^J</' 
(to break up long lines) and then copy its contents 100 times, and save, 
then kill and reopen and pres M->, you should see the same delay that 
I'm talking about, on master.

The conditions seem to be: no long lines, file size > 20MB.

The profiler output looks the same.

> If you are saying that 'recenter' takes too long in some file, and
> therefore delays M-> for too long, it is probably a separate issue, so
> please report that as a separate bug, preferably with the file that
> causes it.

It seems pertinent to me since ultimately the decision to narrow in 
handle_fontified_prop is driven by the idea that font-lock in large 
files ends up being a problem anyway. And here we have a large file 
where this doesn't help.

But I can file it separately, of course.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]


>
> And I don't think it's something you should be ashamed of: it's standard 
> practice to use expedient solutions for want of anything better.  It's 
> how we make progress.
>
> Those expedient solutions (i.e. those cheap solutions that still cover 
> most cases tolerably) might be "not ideal" but finding them is usually 
> not easy, so pride would be more appropriate.
>

I'm neither ashamed nor proud, I'm not Kim Kardashian 😉  I just try to 
have fun, for the programmer-specific value of "fun".  Still, I cannot see 
how an editor could provide a complete editing experience in all buffers 
without limiting in one way or another what is done, and what can be done, 
after each editing command.

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Without assuming that computers have infinitely fast CPUs, that is?
>
> If the font-lock rules to use in the buffer depend on whether the first 
> line of the buffer starts with #! or not, then your narrowing will break 
> them even tho they could work perfectly fine in a 1TB buffer without 
> requiring an infinitely fast CPU (since they don't need to scan the 
> whole buffer: just the region of interest plus the first 2 chars).
>

Why on earth should a font-lock rule check, inside each redisplay cycle, 
whether the first line of the buffer starts with #! or not?  Isn't that an 
information that can, and in fact should, be cached?

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 16 Aug 2022 16:58:15 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 16.08.2022 16:09, Eli Zaretskii wrote:
> >> From: Stefan Monnier<monnier@iro.umontreal.ca>
> >> Cc: Eli Zaretskii<eliz@gnu.org>,56682@debbugs.gnu.org,dgutov@yandex.ru
> >> Date: Tue, 16 Aug 2022 09:07:59 -0400
> >>
> >>> The way I see this is that Emacs is doing efforts to solve editing slowdowns
> >>> as best as possible, but that there is a "last mile" that cannot be solved
> >>> without the collaboration from major and minor modes.
> >> The locked narrowing prevents that "last mile", sadly.
> > Nonsense, it doesn't prevent any such things.
> > 
> > Let's not exaggerate when we try to make a point, okay?
> 
> Why nonsense?

Because it takes no effort to override it and then try changes in any
major mode.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> Date: Tue, 16 Aug 2022 17:00:18 +0300
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> It's a buffer without long lines. Simply a large file.
> > 
> > ??? Are we talking about long-line.xml, or are we talking about
> > some other file?  We did nothing to change the behavior in large files
> > without long lines.
> 
> I'm talking about a 20 MB file I created manually with some simple 
> contents. Just no long lines.

That's a different problem, one we didn't try to fix yet.

> > If you are saying that 'recenter' takes too long in some file, and
> > therefore delays M-> for too long, it is probably a separate issue, so
> > please report that as a separate bug, preferably with the file that
> > causes it.
> 
> It seems pertinent to me since ultimately the decision to narrow in 
> handle_fontified_prop is driven by the idea that font-lock in large 
> files ends up being a problem anyway. And here we have a large file 
> where this doesn't help.

Because it's a different problem, with likely a different solution.

> But I can file it separately, of course.

Yes, please.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii [2022-08-16 17:07:46] wrote:
>> >> The locked narrowing prevents that "last mile", sadly.
>> > Nonsense, it doesn't prevent any such things.
>> > Let's not exaggerate when we try to make a point, okay?
>> Why nonsense?
> Because it takes no effort to override it and then try changes in any
> major mode.

AFAIK it requires recompiling Emacs, which in practice means nobody will
be motivated to try it out since they can't really share the result with
anyone else.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  56682@debbugs.gnu.org,
>   gregory@heytings.org
> Date: Tue, 16 Aug 2022 10:11:57 -0400
> 
> Eli Zaretskii [2022-08-16 17:07:46] wrote:
> >> >> The locked narrowing prevents that "last mile", sadly.
> >> > Nonsense, it doesn't prevent any such things.
> >> > Let's not exaggerate when we try to make a point, okay?
> >> Why nonsense?
> > Because it takes no effort to override it and then try changes in any
> > major mode.
> 
> AFAIK it requires recompiling Emacs

No, it doesn't.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I'm neither ashamed nor proud, I'm not Kim Kardashian 😉  I just try to have
> fun, for the programmer-specific value of "fun".  Still, I cannot see how an
> editor could provide a complete editing experience in all buffers without
> limiting in one way or another what is done, and what can be done, after
> each editing command.

That's way too general a statement to be useful in deciding whether
un-overridable locked narrowing is appropriate, IMO.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Gregory Heytings]

>> Still, I cannot see how an editor could provide a complete editing 
>> experience in all buffers without limiting in one way or another what 
>> is done, and what can be done, after each editing command.
>
> That's way too general a statement to be useful in deciding whether 
> un-overridable locked narrowing is appropriate, IMO.
>

That wasn't a statement about "un-overridable locked narrowing".  Anyway, 
as I said to Eli an hour ago, you'll have what you want (a way to override 
locked narrowing) in a few days.  Don't panic.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-08-16 14:03:16] wrote:
>>> Without assuming that computers have infinitely fast CPUs, that is?
>> If the font-lock rules to use in the buffer depend on whether the first
>> line of the buffer starts with #! or not, then your narrowing will break
>> them even tho they could work perfectly fine in a 1TB buffer without
>> requiring an infinitely fast CPU (since they don't need to scan the whole
>> buffer: just the region of interest plus the first 2 chars).
> Why on earth should a font-lock rule check, inside each redisplay cycle,
> whether the first line of the buffer starts with #! or not?

[ Nitpick: font-lock is not triggered "inside each redisplay cycle".  ]
Because it's simple and efficient.

> Isn't that an information that can, and in fact should, be cached?

"Can" maybe, "should" probably not since managing the cache will be
more trouble than it's worth.

Also, in your case you'd need that info to always be up-to-date, in
which case it's not really a "cache" (since that word usually refers to
something that may be available but isn't guaranteed to be so, and is
re-filled lazily), but a redundant representation of the data (which is
usually more costly to manage).


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii [2022-08-16 17:14:35] wrote:
>> >> >> The locked narrowing prevents that "last mile", sadly.
>> >> > Nonsense, it doesn't prevent any such things.
>> >> > Let's not exaggerate when we try to make a point, okay?
>> >> Why nonsense?
>> > Because it takes no effort to override it and then try changes in any
>> > major mode.
>> AFAIK it requires recompiling Emacs
> No, it doesn't.

I missed it then: you're saying that I can make nlinum-mode override the
locked narrowing to find the real current line number without having to
recompile Emacs?


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-08-16 14:21:26] wrote:
> That wasn't a statement about "un-overridable locked narrowing".  Anyway, as
> I said to Eli an hour ago, you'll have what you want (a way to override
> locked narrowing) in a few days.

Thanks.


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 16.08.2022 17:10, Eli Zaretskii wrote:
>> Date: Tue, 16 Aug 2022 17:00:18 +0300
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> It's a buffer without long lines. Simply a large file.
>>>
>>> ??? Are we talking about long-line.xml, or are we talking about
>>> some other file?  We did nothing to change the behavior in large files
>>> without long lines.
>>
>> I'm talking about a 20 MB file I created manually with some simple
>> contents. Just no long lines.
> 
> That's a different problem, one we didn't try to fix yet.
> 
>>> If you are saying that 'recenter' takes too long in some file, and
>>> therefore delays M-> for too long, it is probably a separate issue, so
>>> please report that as a separate bug, preferably with the file that
>>> causes it.
>>
>> It seems pertinent to me since ultimately the decision to narrow in
>> handle_fontified_prop is driven by the idea that font-lock in large
>> files ends up being a problem anyway. And here we have a large file
>> where this doesn't help.
> 
> Because it's a different problem, with likely a different solution.
> 
>> But I can file it separately, of course.
> 
> Yes, please.

Bug#57245. Thanks.

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Tue, 16 Aug 2022 10:31:54 -0400
> 
> Eli Zaretskii [2022-08-16 17:14:35] wrote:
> >> >> >> The locked narrowing prevents that "last mile", sadly.
> >> >> > Nonsense, it doesn't prevent any such things.
> >> >> > Let's not exaggerate when we try to make a point, okay?
> >> >> Why nonsense?
> >> > Because it takes no effort to override it and then try changes in any
> >> > major mode.
> >> AFAIK it requires recompiling Emacs
> > No, it doesn't.
> 
> I missed it then: you're saying that I can make nlinum-mode override the
> locked narrowing to find the real current line number without having to
> recompile Emacs?

No, I'm saying that you can test improvements in nlinum-mode that
would make it work better with long lines without recompiling.

bug#56682: Fix the long lines font locking related slowdowns

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii [2022-08-16 18:51:41] wrote:
>> I missed it then: you're saying that I can make nlinum-mode override the
>> locked narrowing to find the real current line number without having to
>> recompile Emacs?
>
> No, I'm saying that you can test improvements in nlinum-mode that
> would make it work better with long lines without recompiling.

How?


        Stefan

bug#56682: Fix the long lines font locking related slowdowns

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Tue, 16 Aug 2022 13:15:04 -0400
> 
> Eli Zaretskii [2022-08-16 18:51:41] wrote:
> >> I missed it then: you're saying that I can make nlinum-mode override the
> >> locked narrowing to find the real current line number without having to
> >> recompile Emacs?
> >
> > No, I'm saying that you can test improvements in nlinum-mode that
> > would make it work better with long lines without recompiling.
> 
> How?

I'm sure you can figure it out if you just think for a few seconds.

But if you want me to do it for you, then please tell which changes do
you want to try and in what situations, because the technique depends
on that.

(Not that I understand where are you going with this.  Especially
since a feature to unlock the narrowing is just one simple commit
away.)

bug#56682: locked narrowing (was: bug#56682: Fix the long lines font locking related slowdowns)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii [2022-08-16 20:26:23] wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
>> Date: Tue, 16 Aug 2022 13:15:04 -0400
>> Eli Zaretskii [2022-08-16 18:51:41] wrote:
>> >> I missed it then: you're saying that I can make nlinum-mode override the
>> >> locked narrowing to find the real current line number without having to
>> >> recompile Emacs?
>> > No, I'm saying that you can test improvements in nlinum-mode that
>> > would make it work better with long lines without recompiling.
>> How?
> I'm sure you can figure it out if you just think for a few seconds.

I have no idea where to start because I can't see any way to circumvent
the locked narrowing.

> But if you want me to do it for you, then please tell which changes do
> you want to try and in what situations, because the technique depends
> on that.

I'd start with the patch below.

> (Not that I understand where are you going with this.  Especially
> since a feature to unlock the narrowing is just one simple commit
> away.)

I really appreciate Gregory's (and your) work to improve the handling of
long lines.  This was way overdue.  *BUT* the code for the locked
narrowing just .... sucks rocks.
[ Sorry, Gregory.  But see below for more concrete/constructive criticism.  ]

So, given that you (Eli) accepted it into `master` and have not voiced
concerns over it even after Dmitry's and my complaints, I feel that you
do support it enough that I'm not in a position to just install
"one simple commit".

The most glaring problems I see with it (beside the impossibility to
override the locking, obviously):

- `widen` will silently fail.  Maybe this is the right thing to do,
  maybe not, but it's definitely not clear (e.g. it leads to inf-loops
  in CC-mode, which doesn't seem much better than the excruciating
  slowdowns that locked narrowing is intended to fix).  It *should* be
  (or have been) discussed.
- Not only you can't `widen` but you can't narrow either
  (`narrow-to-region` just does nothing when the restriction is locked,
  regardless if the call tries to grow or shrink the restriction).
  AFAIK this is a plain bug, but given the fundamental disagreement
  about what is "right" in this discussion, I can't be sure.
- The locking is global: any `widen` or `narrow-to-region` will fail
  when a locked narrowing is in effect, even when it's applied to
  an unrelated buffer.  Again, it looks like a plain bug, but...
- ...the plural in the name "restrictions-locked" suggests that the global
  effect is on purpose, tho I can't see any justification for it.
- The doc for `restrictions-locked` says:

      This happens when `narrow-to-region', which see, is called from Lisp
      with an optional argument LOCK non-nil.

  but the `narrow-to-region` function doesn't have any such LOCK argument.
- The `narrow_to_region_internal` function, when called with `lock=true`
  oddly includes the functionality of save-restriction to restore the
  restriction and point but not to restore the `restrictions-locked` state.
- In that same function the two branches of the `if (lock)` have a lot
  of obvious redundancy.

So, yes, I could go and write a patch which I think fixes those
problems, but I don't understand how this code made it to `master` in
the first place, so I feel like I'm missing something and can't be sure
what is considered a bug and what is considered a feature.


        Stefan


diff --git a/nlinum.el b/nlinum.el
index 4f0e02fef1..3feaaca5c3 100644
--- a/nlinum.el
+++ b/nlinum.el
@@ -312,7 +312,7 @@ Only works right if point is at BOL."
   (if nlinum-widen
       (save-excursion
         (save-restriction
-          (widen)
+          (REALLY-widen)
           (forward-line 0)              ;In case (point-min) was not at BOL.
           (let ((nlinum-widen nil))
             (nlinum--line-number-at-pos))))

bug#56682: locked narrowing

[dick]

SM> the code for the locked narrowing just .... sucks rocks.

To the extent, however oblique, my foray into long lines
(https://youtu.be/5C3wB3xiMmU) instigated this rift, I am contented.

bug#56682: locked narrowing (was: bug#56682: Fix the long lines font locking related slowdowns)

[Gregory Heytings]

>
> [ Sorry, Gregory.  But see below for more concrete/constructive 
> criticism. ]
>

There's no need to be sorry.  This is WIP, any (constructive) criticism is 
welcome.  And I do not understand why you did not raise some of the points 
below earlier.  Perhaps I misuderstood or overlooked something in what you 
said earlier, but to me your objection to that part of the code until now 
was mainly/purely "philosophical".

>
> - Not only you can't `widen` but you can't narrow either 
> (`narrow-to-region` just does nothing when the restriction is locked, 
> regardless if the call tries to grow or shrink the restriction). AFAIK 
> this is a plain bug, but given the fundamental disagreement about what 
> is "right" in this discussion, I can't be sure.
>

This (allowing narrow-to-region and widen to work within the bounds of the 
locked narrowing) is currently being implemented, after an off-list 
discussion with Alan.

>
> - The locking is global: any `widen` or `narrow-to-region` will fail 
> when a locked narrowing is in effect, even when it's applied to an 
> unrelated buffer.  Again, it looks like a plain bug, but...
>

It's not global, given that it is controlled by symbol that is bound 
within a function called inside a specific buffer.  But indeed, if inside 
that function we use narrow-to-region/widen in another buffer, that would 
fail.  It seems easy to correct that bug, though: if 
(current_buffer->long_line_optimizations_p && ! NILP 
(Vrestrictions_locked)) ...

>
> - The doc for `restrictions-locked` says:
>
>  This happens when `narrow-to-region', which see, is called from Lisp
>  with an optional argument LOCK non-nil.
>
>  but the `narrow-to-region` function doesn't have any such LOCK 
> argument.
>

That should have been removed, indeed.  In a previous iteration of the 
code a LOCK argument was added to that function, and later removed.

>
> - The `narrow_to_region_internal` function, when called with `lock=true` 
> oddly includes the functionality of save-restriction to restore the 
> restriction and point but not to restore the `restrictions-locked` 
> state.
>

This I do not understand.  The comment above that function explains that 
it should be followed by "specbind (Qrestrictions_locked, Qt)".  There is 
no need to restore the restrictions-locked state, or am I misunderstanding 
something?

>
> - In that same function the two branches of the `if (lock)` have a lot 
> of obvious redundancy.
>

Not "a lot", but indeed there's some redundancy there that could have been 
avoided.

I'll fix those problems soon.

bug#56682: locked narrowing (was: bug#56682: Fix the long lines font locking related slowdowns)

[Gregory Heytings]

Another comment:

>
> diff --git a/nlinum.el b/nlinum.el
> index 4f0e02fef1..3feaaca5c3 100644
> --- a/nlinum.el
> +++ b/nlinum.el
> @@ -312,7 +312,7 @@ Only works right if point is at BOL."
>   (if nlinum-widen
>       (save-excursion
>         (save-restriction
> -          (widen)
> +          (REALLY-widen)
>           (forward-line 0)              ;In case (point-min) was not at BOL.
>           (let ((nlinum-widen nil))
>             (nlinum--line-number-at-pos))))
>

FWIW, that's not a good example, because it's a bad idea to calculate line 
numbers in buffers with long lines or with too many lines: it takes too 
much time.

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-08-16 19:39:21] wrote:
>> [ Sorry, Gregory.  But see below for more concrete/constructive
>>   criticism. ]
> Perhaps I misuderstood or overlooked something in what you said
> earlier, but to me your objection to that part of the code until now
> was mainly/purely "philosophical".

My *main* objection is in the not-overridable part of the design, yes
(i.e. philosophical).  The rest is just a minor question of coding.

>> - The locking is global: any `widen` or `narrow-to-region` will fail when
>> a locked narrowing is in effect, even when it's applied to an unrelated
>> buffer.  Again, it looks like a plain bug, but...
> It's not global, given that it is controlled by symbol that is bound within
> a function called inside a specific buffer.

Which buffer is current when the let-binding is installed has no
influence on the behavior because the variable is not buffer-local
(i.e. it's global, which is why the effect is global).
The dynamic scope means that the effect is limited *in time* (and only
applies to the current thread, admittedly), but it's still global in the
sense that it affects all the code that is run during this time, no
matter where it is executed (or were it was written).

But, yes, dynamically scoped variables are often described as being
"locally bound", even though the effect of that binding is global in the
sense that it applies everywhere while active, so there's plenty of
opportunities for confusion with this terminology :-(

> But indeed, if inside that
> function we use narrow-to-region/widen in another buffer, that would fail.
> It seems easy to correct that bug, though: if
> (current_buffer->long_line_optimizations_p && ! NILP (Vrestrictions_locked))

If `current_buffer` is not the one that's currently being narrowed by
redisplay, then `->long_line_optimizations_p` doesn't say that
`current_buffer` has been narrowed by "us", so such a fix might work but
I think a better fix is to `Fmake_local_variable` before calling
`specbind` so that it really states clearly that this lock applies only
to the narrowing installed in this particular buffer.

>> - The doc for `restrictions-locked` says:
>>
>>  This happens when `narrow-to-region', which see, is called from Lisp
>>  with an optional argument LOCK non-nil.
>>
>>  but the `narrow-to-region` function doesn't have any such LOCK argument.
>
> That should have been removed, indeed.  In a previous iteration of the code
> a LOCK argument was added to that function, and later removed.

Adding such a LOCK argument might be a good idea, actually, since ELisp
packages may also want to impose restrictions on widening (see my recent
message to `emacs-devel` about ELisp support for locked narrowing).

>> - The `narrow_to_region_internal` function, when called with `lock=true`
>> oddly includes the functionality of save-restriction to restore the
>> restriction and point but not to restore the `restrictions-locked` state.
>
> This I do not understand.  The comment above that function explains that it
> should be followed by "specbind (Qrestrictions_locked, Qt)".  There is no
> need to restore the restrictions-locked state, or am
> I misunderstanding something?

Why leave the `specbind (Qrestrictions_locked, Qt)` out of the function
when you've already included the 3 calls to `record_unwind_protect`?
The 4 share the same requirements (because `specbind` pushes on the
`specpdl`, just like `record_unwind_protect`).  They naturally
belong together.

Also, a function that sometimes pushes to `specpdl` (and hence requires
the caller to do the `SPECPDL_INDEX + unbind_to` dance) and sometimes
not, depending on some value of its argument is a bit unnatural (it
complicates the reasoning when we try and convince ourselves that the
stack discipline of the specpdl is obeyed).  So I'd recommend you split
the function into two: a first one that only does the narrowing itself
and a second one which calls the first plus unconditionally does the
`specbind+record_unwind_protect`.

>> - In that same function the two branches of the `if (lock)` have a lot of
>> obvious redundancy.
> Not "a lot", but indeed there's some redundancy there that could have
> been avoided.

AFAICT it's more than half of the code of each branch.


        Stefan

bug#56682: locked narrowing

[Gregory Heytings]

>
> Adding such a LOCK argument might be a good idea, actually, since ELisp 
> packages may also want to impose restrictions on widening (see my recent 
> message to `emacs-devel` about ELisp support for locked narrowing).
>

It might be a good idea indeed.  I did not have an immediate need for it, 
and did not see how to do it (because of the unbalanced specbind).

>
> Why leave the `specbind (Qrestrictions_locked, Qt)` out of the function 
> when you've already included the 3 calls to `record_unwind_protect`? The 
> 4 share the same requirements (because `specbind` pushes on the 
> `specpdl`, just like `record_unwind_protect`).  They naturally belong 
> together.
>

They do indeed, and that's actually what the original code did, but I was 
told that it was better to not do that.

>
> Also, a function that sometimes pushes to `specpdl` (and hence requires 
> the caller to do the `SPECPDL_INDEX + unbind_to` dance) and sometimes 
> not, depending on some value of its argument is a bit unnatural (it 
> complicates the reasoning when we try and convince ourselves that the 
> stack discipline of the specpdl is obeyed).  So I'd recommend you split 
> the function into two: a first one that only does the narrowing itself 
> and a second one which calls the first plus unconditionally does the 
> `specbind+record_unwind_protect`.
>

Okay, thanks.  That's probably another way to do it, indeed.

>> Not "a lot", but indeed there's some redundancy there that could have 
>> been avoided.
>
> AFAICT it's more than half of the code of each branch.
>

That's still not "a lot", there are 12 statements in total in the two 
branches.  And they have 3 statements in common...

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Gregory Heytings [2022-08-16 20:23:46] wrote:
> Another comment:
>> diff --git a/nlinum.el b/nlinum.el
>> index 4f0e02fef1..3feaaca5c3 100644
>> --- a/nlinum.el
>> +++ b/nlinum.el
>> @@ -312,7 +312,7 @@ Only works right if point is at BOL."
>>   (if nlinum-widen
>>       (save-excursion
>>         (save-restriction
>> -          (widen)
>> +          (REALLY-widen)
>>           (forward-line 0)              ;In case (point-min) was not at BOL.
>>           (let ((nlinum-widen nil))
>>             (nlinum--line-number-at-pos))))
> FWIW, that's not a good example, because it's a bad idea to calculate line
> numbers in buffers with long lines or with too many lines: it takes too
> much time.

In my tests, it works fast enough for pretty large buffers and pretty
long lines, so while the above might be too crude, the limits we should
use for it should be much lower than `syntax-wholeline-max` and even
probably larger than `large-file-warning-threshold`.


        Stefan

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Monnier [2022-08-16 17:42:45] wrote:
> Gregory Heytings [2022-08-16 20:23:46] wrote:
>> Another comment:
>>> diff --git a/nlinum.el b/nlinum.el
>>> index 4f0e02fef1..3feaaca5c3 100644
>>> --- a/nlinum.el
>>> +++ b/nlinum.el
>>> @@ -312,7 +312,7 @@ Only works right if point is at BOL."
>>>   (if nlinum-widen
>>>       (save-excursion
>>>         (save-restriction
>>> -          (widen)
>>> +          (REALLY-widen)
>>>           (forward-line 0)              ;In case (point-min) was not at BOL.
>>>           (let ((nlinum-widen nil))
>>>             (nlinum--line-number-at-pos))))
>> FWIW, that's not a good example, because it's a bad idea to calculate line
>> numbers in buffers with long lines or with too many lines: it takes too
>> much time.
> In my tests, it works fast enough for pretty large buffers and pretty
> long lines,

And actually, it doesn't seem to be slowed by long lines, only by the
buffer size.  So its `widen` should definitely override the
line-length-induced narrowing (tho maybe not the buffer-size-induced
narrowing if/when we add such a thing).


        Stefan

bug#56682: locked narrowing (was: bug#56682: Fix the long lines font locking related slowdowns)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru, 56682@debbugs.gnu.org, gregory@heytings.org
> Date: Tue, 16 Aug 2022 15:18:23 -0400
> 
> diff --git a/nlinum.el b/nlinum.el
> index 4f0e02fef1..3feaaca5c3 100644
> --- a/nlinum.el
> +++ b/nlinum.el
> @@ -312,7 +312,7 @@ Only works right if point is at BOL."
>    (if nlinum-widen
>        (save-excursion
>          (save-restriction
> -          (widen)
> +          (REALLY-widen)
>            (forward-line 0)              ;In case (point-min) was not at BOL.
>            (let ((nlinum-widen nil))
>              (nlinum--line-number-at-pos))))

If a package wants to count lines in a function registered with
jit-lock, I think it makes no sense for it to do that in Lisp.  It
would be much better to let such packages use the built-in
capabilities that we use all the time, both on the mode line and under
display-line-numbers mode.  There are at least two advantages to that:
(1) it will be at least slightly faster, sometimes more than slightly
(due to caching); and (2) the problem of narrowing would be seamlessly
resolved.

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> If a package wants to count lines in a function registered with
> jit-lock, I think it makes no sense for it to do that in Lisp.  It
> would be much better to let such packages use the built-in
> capabilities that we use all the time, both on the mode line and under
> display-line-numbers mode.  There are at least two advantages to that:
> (1) it will be at least slightly faster, sometimes more than slightly
> (due to caching); and (2) the problem of narrowing would be seamlessly
> resolved.

FWIW, I agree.  I don't think this information is cleanly&reliably
exposed to ELisp, tho, which is why nlinum-mode does it "by hand" (with
its own caching).


        Stefan

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Wed, 17 Aug 2022 09:06:21 -0400
> 
> > If a package wants to count lines in a function registered with
> > jit-lock, I think it makes no sense for it to do that in Lisp.  It
> > would be much better to let such packages use the built-in
> > capabilities that we use all the time, both on the mode line and under
> > display-line-numbers mode.  There are at least two advantages to that:
> > (1) it will be at least slightly faster, sometimes more than slightly
> > (due to caching); and (2) the problem of narrowing would be seamlessly
> > resolved.
> 
> FWIW, I agree.  I don't think this information is cleanly&reliably
> exposed to ELisp

It isn't currently.  I was suggesting to do so.

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> FWIW, I agree.  I don't think this information is cleanly&reliably
>> exposed to ELisp
>
> It isn't currently.  I was suggesting to do so.

Last time I looked at it, I couldn't quite understand how the cache
works there, so I wasn't sure how to make sure it returns valid
information even if called outside of redisplay.


        Stefan

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Wed, 17 Aug 2022 09:50:59 -0400
> 
> >> FWIW, I agree.  I don't think this information is cleanly&reliably
> >> exposed to ELisp
> >
> > It isn't currently.  I was suggesting to do so.
> 
> Last time I looked at it, I couldn't quite understand how the cache
> works there, so I wasn't sure how to make sure it returns valid
> information even if called outside of redisplay.

Looking at what maybe_produce_line_number does doesn't help?

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Last time I looked at it, I couldn't quite understand how the cache
>> works there, so I wasn't sure how to make sure it returns valid
>> information even if called outside of redisplay.
> Looking at what maybe_produce_line_number does doesn't help?

I see two problems:

- The problem I already mentioned: the cache seems to be
  maintained/flushed by the redisplay code, so when the function is
  called outside of redisplay I don't know if `w->base_line_number/pos`
  is still valid.
- The fact that this uses line numbers counted from BEGV whereas nlinum
  counts from BEG.


        Stefan

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Wed, 17 Aug 2022 10:11:28 -0400
> 
> >> Last time I looked at it, I couldn't quite understand how the cache
> >> works there, so I wasn't sure how to make sure it returns valid
> >> information even if called outside of redisplay.
> > Looking at what maybe_produce_line_number does doesn't help?
> 
> I see two problems:
> 
> - The problem I already mentioned: the cache seems to be
>   maintained/flushed by the redisplay code, so when the function is
>   called outside of redisplay I don't know if `w->base_line_number/pos`
>   is still valid.

So you are saying that any command which uses vertical-motion or more
generally any of the move_it_* functions will work incorrectly when
line numbers are on display?  I'm not aware of any such problems.

> - The fact that this uses line numbers counted from BEGV whereas nlinum
>   counts from BEG.

I think you missed the display-line-numbers-widen variable and its
effect.

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> >> Last time I looked at it, I couldn't quite understand how the cache
>> >> works there, so I wasn't sure how to make sure it returns valid
>> >> information even if called outside of redisplay.
>> > Looking at what maybe_produce_line_number does doesn't help?
>> 
>> I see two problems:
>> 
>> - The problem I already mentioned: the cache seems to be
>>   maintained/flushed by the redisplay code, so when the function is
>>   called outside of redisplay I don't know if `w->base_line_number/pos`
>>   is still valid.
>
> So you are saying that any command which uses vertical-motion or more
> generally any of the move_it_* functions will work incorrectly when
> line numbers are on display?  I'm not aware of any such problems.

No, I'm sure the existing code somehow handles it right.  I just don't
know how.  I can't see anything in the code of `insert` (say) which
flushes `w->base_line_number` when needed, so there needs to be code
somewhere which flushes it more lazily before using that value.  I just
don't know where that code is and which data it uses to flush it.
For this reason I don't know under which condition I can make use of
`w->base_line_number`.

>> - The fact that this uses line numbers counted from BEGV whereas nlinum
>>   counts from BEG.
>
> I think you missed the display-line-numbers-widen variable and its
> effect.

Ah, indeed, thanks, that helps.
[ BTW, another hurdle if that the existing cache is linked to windows,
  whereas the actual info requested doesn't care about windows (as long
  as we count logical lines rather than screen lines) and would also
  make sense in a buffer that's not displayed at all.  This should
  usually not be a problem for nlinum-mode, tho it could happen
  when called from `font-lock-ensure`.  ]


        Stefan

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Wed, 17 Aug 2022 11:58:32 -0400
> 
> >> - The problem I already mentioned: the cache seems to be
> >>   maintained/flushed by the redisplay code, so when the function is
> >>   called outside of redisplay I don't know if `w->base_line_number/pos`
> >>   is still valid.
> >
> > So you are saying that any command which uses vertical-motion or more
> > generally any of the move_it_* functions will work incorrectly when
> > line numbers are on display?  I'm not aware of any such problems.
> 
> No, I'm sure the existing code somehow handles it right.  I just don't
> know how.  I can't see anything in the code of `insert` (say) which
> flushes `w->base_line_number` when needed, so there needs to be code
> somewhere which flushes it more lazily before using that value.  I just
> don't know where that code is and which data it uses to flush it.
> For this reason I don't know under which condition I can make use of
> `w->base_line_number`.

Why can't you simply use the same code maybe_produce_line_number uses
for that?

> [ BTW, another hurdle if that the existing cache is linked to windows,
>   whereas the actual info requested doesn't care about windows (as long
>   as we count logical lines rather than screen lines) and would also
>   make sense in a buffer that's not displayed at all.  This should
>   usually not be a problem for nlinum-mode, tho it could happen
>   when called from `font-lock-ensure`.  ]

Why does it matter which line numbers you compute if they are never
displayed?

bug#56682: locked narrowing

[Eli Zaretskii]

> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, dgutov@yandex.ru
> Date: Wed, 17 Aug 2022 19:52:54 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > > So you are saying that any command which uses vertical-motion or more
> > > generally any of the move_it_* functions will work incorrectly when
> > > line numbers are on display?  I'm not aware of any such problems.
> > 
> > No, I'm sure the existing code somehow handles it right.  I just don't
> > know how.  I can't see anything in the code of `insert` (say) which
> > flushes `w->base_line_number` when needed, so there needs to be code
> > somewhere which flushes it more lazily before using that value.  I just
> > don't know where that code is and which data it uses to flush it.
> > For this reason I don't know under which condition I can make use of
> > `w->base_line_number`.
> 
> Why can't you simply use the same code maybe_produce_line_number uses
> for that?

Or even just always use display_count_lines, if you don't trust the
cache.

And another thought: did you try using format-mode-line as the means
to get the line number from which you could start?

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> >> - The problem I already mentioned: the cache seems to be
>> >>   maintained/flushed by the redisplay code, so when the function is
>> >>   called outside of redisplay I don't know if `w->base_line_number/pos`
>> >>   is still valid.
>> >
>> > So you are saying that any command which uses vertical-motion or more
>> > generally any of the move_it_* functions will work incorrectly when
>> > line numbers are on display?  I'm not aware of any such problems.
>> 
>> No, I'm sure the existing code somehow handles it right.  I just don't
>> know how.  I can't see anything in the code of `insert` (say) which
>> flushes `w->base_line_number` when needed, so there needs to be code
>> somewhere which flushes it more lazily before using that value.  I just
>> don't know where that code is and which data it uses to flush it.
>> For this reason I don't know under which condition I can make use of
>> `w->base_line_number`.
>
> Why can't you simply use the same code maybe_produce_line_number uses
> for that?

Because I can't find that code.  Apparently `maybe_produce_line_number`
just uses `w->base_line_number` whenever it's not 0, so apparently it's
set to 0 elsewhere.

[ BTW, apparently when `display_line_numbers_widen` is in use and the
  buffer is narrowed, we don't use that cache at all for
  `maybe_produce_line_number` (according to xdisp.c:24217).  IOW that
  cache always counts from BEGV, so we also need to check that BEGV is
  still the same as when the cache was filled.  I also can't see in
  `decode_mode_spec` where we check that the narrowing has changed
  before using the cached value.  All in all, I just don't understand
  how that cache is maintained.  ]

>> [ BTW, another hurdle if that the existing cache is linked to windows,
>>   whereas the actual info requested doesn't care about windows (as long
>>   as we count logical lines rather than screen lines) and would also
>>   make sense in a buffer that's not displayed at all.  This should
>>   usually not be a problem for nlinum-mode, tho it could happen
>>   when called from `font-lock-ensure`.  ]
> Why does it matter which line numbers you compute if they are never
> displayed?

They may be displayed at some other time: nlinum-mode won't be invoked
if that same portion of the buffer becomes visible later on (it's just
like font-lock: the highlighting applied stays until something like the
after-change-function causes it to be refreshed).


        Stefan

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Wed, 17 Aug 2022 13:58:21 -0400
> 
> > Why can't you simply use the same code maybe_produce_line_number uses
> > for that?
> 
> Because I can't find that code.

What do you mean? it's in maybe_produce_line_number, obviously...

> Apparently `maybe_produce_line_number`
> just uses `w->base_line_number` whenever it's not 0, so apparently it's
> set to 0 elsewhere.

It's set to zero in several places in xdisp.c.

But again, if you don't want to trust the cached values, just use
display_count_lines starting from BOB.

> > Why does it matter which line numbers you compute if they are never
> > displayed?
> 
> They may be displayed at some other time: nlinum-mode won't be invoked
> if that same portion of the buffer becomes visible later on (it's just
> like font-lock: the highlighting applied stays until something like the
> after-change-function causes it to be refreshed).

Then invalidate the information on that portion when the buffer
becomes displayed, and let the code count again.

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Wed, 17 Aug 2022 13:58:21 -0400
> 
> [ BTW, apparently when `display_line_numbers_widen` is in use and the
>   buffer is narrowed, we don't use that cache at all for
>   `maybe_produce_line_number` (according to xdisp.c:24217).  IOW that
>   cache always counts from BEGV, so we also need to check that BEGV is
>   still the same as when the cache was filled.  I also can't see in
>   `decode_mode_spec` where we check that the narrowing has changed
>   before using the cached value.  All in all, I just don't understand
>   how that cache is maintained.  ]

Search in xdisp.c for "->base_line_number = 0", and you will see that
we invalidate the cache when the narrowing changes or when other
similar calamities happen.

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Because I can't find that code.
> What do you mean? it's in maybe_produce_line_number, obviously...

My intuition tells me it should be there, but I can't find it.

>> Apparently `maybe_produce_line_number`
>> just uses `w->base_line_number` whenever it's not 0, so apparently it's
>> set to 0 elsewhere.
> It's set to zero in several places in xdisp.c.

I can't seem to find even one of them related to the fact that the
buffer was modified.

When I test the code, I can see that it works correctly, but I can't see
in the code how the cache gets flushed/ignored/refreshed when the
buffer's contents is modified (e.g. by inserting a newline at BOB).

> But again, if you don't want to trust the cached values, just use
> display_count_lines starting from BOB.

That's already what nlinum does (via `line-number-at-pos`).


        Stefan

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Search in xdisp.c for "->base_line_number = 0", and you will see that
> we invalidate the cache when the narrowing changes or when other
> similar calamities happen.

OK, after another search I found (in `redisplay_window` and
`try_scrolling`) that we do:

	  if (!just_this_one_p
	      || current_buffer->clip_changed
	      || BEG_UNCHANGED < CHARPOS (startp))
	    /* Forget any recorded base line for line number display.  */
	    w->base_line_number = 0;

So IIUC `w->base_line_number` can be used if:

          w->base_line_number != 0
          && !current_buffer->clip_changed
          && BEG_UNCHANGED < w->base_line_pos

?

Why do we flush the cache when `just_this_one_p` is false?


        Stefan

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Wed, 17 Aug 2022 15:09:22 -0400
> 
> >> Because I can't find that code.
> > What do you mean? it's in maybe_produce_line_number, obviously...
> 
> My intuition tells me it should be there, but I can't find it.

Really?  I meant this:

      if (!last_line)
	{
	  /* If possible, reuse data cached by line-number-mode.  */
	  if (it->w->base_line_number > 0
	      && it->w->base_line_pos > 0
	      && it->w->base_line_pos <= IT_CHARPOS (*it)
	      /* line-number-mode always displays narrowed line
		 numbers, so we cannot use its data if the user wants
		 line numbers that disregard narrowing, or if the
		 buffer's narrowing has just changed.  */
	      && !(line_numbers_wide
		   && (BEG_BYTE != BEGV_BYTE || Z_BYTE != ZV_BYTE))
	      && !current_buffer->clip_changed)
	    {
	      start_from = CHAR_TO_BYTE (it->w->base_line_pos);
	      last_line = it->w->base_line_number - 1;
	    }
	  else
	    start_from = beg_byte;
	  if (!it->lnum_bytepos)
	    first_time = true;
	}
      else
	start_from = it->lnum_bytepos;

      /* Paranoia: what if someone changes the narrowing since the
	 last time display_line was called?  Shouldn't really happen,
	 but who knows what some crazy Lisp invoked by :eval could do?  */
      if (!(beg_byte <= start_from && start_from <= z_byte))
	{
	  last_line = 0;
	  start_from = beg_byte;
	}

      this_line =
	last_line + display_count_lines_logically (start_from,
						   IT_BYTEPOS (*it),
						   IT_CHARPOS (*it), &bytepos);

> >> Apparently `maybe_produce_line_number`
> >> just uses `w->base_line_number` whenever it's not 0, so apparently it's
> >> set to 0 elsewhere.
> > It's set to zero in several places in xdisp.c.
> 
> I can't seem to find even one of them related to the fact that the
> buffer was modified.

This:

	  if (!just_this_one_p
	      || current_buffer->clip_changed
	      || BEG_UNCHANGED < CHARPOS (startp))
	    /* Forget any recorded base line for line number display.  */
	    w->base_line_number = 0;

And this:

  /* Forget any previously recorded base line for line number display.  */
  if (!buffer_unchanged_p)
    w->base_line_number = 0;

> When I test the code, I can see that it works correctly, but I can't see
> in the code how the cache gets flushed/ignored/refreshed when the
> buffer's contents is modified (e.g. by inserting a newline at BOB).

Of course, this works: it's about the oldest code in the display
engine.

> > But again, if you don't want to trust the cached values, just use
> > display_count_lines starting from BOB.
> 
> That's already what nlinum does (via `line-number-at-pos`).

Then maybe all this discussion is just a waste of breath.

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Wed, 17 Aug 2022 15:18:57 -0400
> 
> Why do we flush the cache when `just_this_one_p` is false?

Because it means many things could have changed, and we aren't sure we
can trust the cache.

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> And another thought: did you try using format-mode-line as the means
> to get the line number from which you could start?

Yes, but in my tests, it wasn't any faster.

BTW, while I have now found which code flushes the base_line cache
during redisplay (there are a few different places, in
`redisplay_window` and `try_scrolling` (not sure why it needs to be
done at more than one place, tho)),
I still haven't found where that cache gets flushed or ignored for
the case where it's used for `format-mode-line`.  That code doesn't
seem to pay attention to BEG_UNCHANGED nor even to the changes
in narrowing.

Oh... wait... aha!  I still don't know how it handles buffer
modifications, but I now know that it just fails to pay attention to
changes to the narrowing: so, it's no surprised that I couldn't find the
corresponding flushing code, it's just missing:

    emacs -Q lisp/subr.el
    M->
    M-: (list (format-mode-line "%l")
              (save-restriction (narrow-to-region (+ (point-min) 4000) (point-max))
                                (format-mode-line "%l")))

returns twice the same line number.


        Stefan

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> >> Because I can't find that code.
>> > What do you mean? it's in maybe_produce_line_number, obviously...
>> My intuition tells me it should be there, but I can't find it.
>
> Really?  I meant this:
>
>       if (!last_line)
> 	{
> 	  /* If possible, reuse data cached by line-number-mode.  */
> 	  if (it->w->base_line_number > 0
> 	      && it->w->base_line_pos > 0
> 	      && it->w->base_line_pos <= IT_CHARPOS (*it)
> 	      /* line-number-mode always displays narrowed line
> 		 numbers, so we cannot use its data if the user wants
> 		 line numbers that disregard narrowing, or if the
> 		 buffer's narrowing has just changed.  */
> 	      && !(line_numbers_wide
> 		   && (BEG_BYTE != BEGV_BYTE || Z_BYTE != ZV_BYTE))
> 	      && !current_buffer->clip_changed)
> 	    {
> 	      start_from = CHAR_TO_BYTE (it->w->base_line_pos);
> 	      last_line = it->w->base_line_number - 1;
> 	    }
> 	  else
> 	    start_from = beg_byte;
> 	  if (!it->lnum_bytepos)
> 	    first_time = true;
> 	}
>       else
> 	start_from = it->lnum_bytepos;
>
>       /* Paranoia: what if someone changes the narrowing since the
> 	 last time display_line was called?  Shouldn't really happen,
> 	 but who knows what some crazy Lisp invoked by :eval could do?  */
>       if (!(beg_byte <= start_from && start_from <= z_byte))
> 	{
> 	  last_line = 0;
> 	  start_from = beg_byte;
> 	}
>
>       this_line =
> 	last_line + display_count_lines_logically (start_from,
> 						   IT_BYTEPOS (*it),
> 						   IT_CHARPOS (*it), &bytepos);

This checks for changes in narrowing (via !current_buffer->clip_changed),
but not for changes in the buffer contents (these are apparently
checked in `redisplay_window` and `try_scrolling` instead).

>> >> Apparently `maybe_produce_line_number`
>> >> just uses `w->base_line_number` whenever it's not 0, so apparently it's
>> >> set to 0 elsewhere.
>> > It's set to zero in several places in xdisp.c.
>> I can't seem to find even one of them related to the fact that the
>> buffer was modified.
>
> This:
>
> 	  if (!just_this_one_p
> 	      || current_buffer->clip_changed
> 	      || BEG_UNCHANGED < CHARPOS (startp))
> 	    /* Forget any recorded base line for line number display.  */
> 	    w->base_line_number = 0;
>
> And this:
>
>   /* Forget any previously recorded base line for line number display.  */
>   if (!buffer_unchanged_p)
>     w->base_line_number = 0;

Right, I did find them in the end.
Note that these are in code which is not used for `format-mode-line`.

>> > But again, if you don't want to trust the cached values, just use
>> > display_count_lines starting from BOB.
>>
>> That's already what nlinum does (via `line-number-at-pos`).
>
> Then maybe all this discussion is just a waste of breath.

It'll help me improve the code's doc, tho.


        Stefan

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Why do we flush the cache when `just_this_one_p` is false?
> Because it means many things could have changed,

I can't see which ones that could be.

> and we aren't sure we can trust the cache.

:-(


        Stefan

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  56682@debbugs.gnu.org,  gregory@heytings.org
> Date: Wed, 17 Aug 2022 16:09:09 -0400
> 
> >> Why do we flush the cache when `just_this_one_p` is false?
> > Because it means many things could have changed,
> 
> I can't see which ones that could be.

just_this_one_p is true when we are only redisplaying the selected
window.

bug#56682: locked narrowing

[Gregory Heytings]

I pushed a (hopefully) improved version of the the locked narrowing 
feature in the feature/improved-locked-narrowing branch.  It is now 
possible:

1. to unlock a locked narrowing,

2. to use narrow-to-region and widen within the bounds of a locked 
narrowing,

3. to enter and leave a locked narrowing within a locked narrowing, and

4. to use that feature in Lisp code (either directly or with a helper 
macro).

As far as I understand, given that it is possible to unlock a locked 
narrowing, adding the possibility to choose how the bounds of the locked 
narrowings are computed has become unnecessary.

Comments/feedback welcome.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Sat, 20 Aug 2022 16:20:57 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, 56682@debbugs.gnu.org, 
>     dgutov@yandex.ru
> 
> 
> I pushed a (hopefully) improved version of the the locked narrowing 
> feature in the feature/improved-locked-narrowing branch.  It is now 
> possible:
> 
> 1. to unlock a locked narrowing,
> 
> 2. to use narrow-to-region and widen within the bounds of a locked 
> narrowing,
> 
> 3. to enter and leave a locked narrowing within a locked narrowing, and
> 
> 4. to use that feature in Lisp code (either directly or with a helper 
> macro).
> 
> As far as I understand, given that it is possible to unlock a locked 
> narrowing, adding the possibility to choose how the bounds of the locked 
> narrowings are computed has become unnecessary.
> 
> Comments/feedback welcome.

Thanks.

The problem with narrowing around the calls to pre- and post-command
hooks (see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57207#107)
still exists, AFAICT, and needs to be resolved.

bug#56682: locked narrowing

[Eli Zaretskii]

Hmm, and I also don't see the change you promised in

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57207#89

bug#56682: locked narrowing

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 261 bytes --]


>
> Hmm, and I also don't see the change you promised in
>
>  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=57207#89
>

Yes, this changes is focused on the "locked narrowing" issue only, which 
is complex enough.  I'll keep my promise, don't worry 😉

bug#56682: locked narrowing

[Dmitry Gutov]

(Sorry, resending after unarchiving the bug)

Hi Gregory,

I don't mean to hurry you up, but:

On 20.08.2022 19:20, Gregory Heytings wrote:
> 
> I pushed a (hopefully) improved version of the the locked narrowing 
> feature in the feature/improved-locked-narrowing branch.  It is now 
> possible:
> 
> 1. to unlock a locked narrowing,
> 
> 2. to use narrow-to-region and widen within the bounds of a locked 
> narrowing,
> 
> 3. to enter and leave a locked narrowing within a locked narrowing, and
> 
> 4. to use that feature in Lisp code (either directly or with a helper 
> macro).
> 
> As far as I understand, given that it is possible to unlock a locked 
> narrowing, adding the possibility to choose how the bounds of the locked 
> narrowings are computed has become unnecessary.

Does that mean that you concluded that font-lock will be able (and 
should) unlock the narrowing before doing its thing?

Which will be equivalent to not applying the narrowing in 
handle_fontified_prop to begin with.

bug#56682: locked narrowing

[Gregory Heytings]

>
> Does that mean that you concluded that font-lock will be able (and 
> should) unlock the narrowing before doing its thing?
>

No, obviously that's not what it means.  It only means that font locking 
routines have the possibility to unlock the narrowing.

I'll be back on track on this topic next week.

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I finally found the time to work on this again.  I pushed a number of
> further improvements in the feature/improved-locked-narrowing branch.

Thanks.  I'll take a look ASAP.


        Stefan

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Sat, 26 Nov 2022 00:30:53 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, dgutov@yandex.ru
> 
> 
> Hi Eli & Stefan,
> 
> I finally found the time to work on this again.  I pushed a number of 
> further improvements in the feature/improved-locked-narrowing branch.
> 
> In particular, Stefan, I used the suggestions you made in 
> jwvtu623mi9.fsf-monnier+emacs@gnu.org on August 23rd (I can't believe so 
> much time has passed!), and added comments to make the code (hopefully) 
> easier to read.
> 
> Comments and feedback welcome.

Below:

> +(defun with-narrowing-1 (start end tag body)
> +  "Helper function for `with-narrowing', which see."
> +  (save-restriction
> +    (progn
> +      (narrow-to-region start end)
> +      (narrowing-lock tag)
> +      (funcall body))))
> +
> +(defun with-narrowing-2 (start end body)
> +  "Helper function for `with-narrowing', which see."
> +  (save-restriction
> +    (progn
> +      (narrow-to-region start end)
> +      (funcall body))))
> +

Should these two helpers be internal functions?

> +/* Retrieve one of the BEGV/ZV bounds of a narrowing in BUF from the
> +   narrowing_locks alist.  When OUTERMOST is true, the bounds that
> +   were set by the user and that are visible on display are returned.
> +   Otherwise the innermost locked narrowing bounds are returned.  */
> +static ptrdiff_t
> +narrowing_lock_get_bound (Lisp_Object buf, bool begv, bool outermost)
> +{
> +  if (NILP (Fbuffer_live_p (buf)))
> +    return 0;

Zero is a strange value to return from a function the is documented to
return buffer positions.  It is also not documented in the commentary.

Also, I'm not sure I understand why we need the buffer-live-p test here.
This is an internal C subroutine, so it's up to the callers to make sure BUF
is a live buffer.

> +  buffer_locks = Fcar (Fcdr (buffer_locks));

Please avoid calling Fcar and Fcdr from C -- that just burns up CPU cycles
for no good reason.  Use CAR and CDR instead.

In general, if some Lisp primitive has a subroutine that is specifically for
calls from C, always prefer to use the subroutine, since that's why those
subroutines are there in the first place.  They typically avoid some
safeguards and checks that are necessary when calling from Lisp, but not
when calling from C.

> +  Lisp_Object marker = begv ? Fcar (bounds) : Fcar (Fcdr (bounds));
> +  eassert (MARKERP (marker));
> +  Lisp_Object pos = Fmarker_position (marker);
> +  eassert (! NILP (pos));

Why isn't there an assertion that the marker belongs to the buffer BUF?  I
think this is a more probable problem than position being nil.

> +/* Remove the innermost lock in BUF from the narrowing_lock alist.  */
>  static void
> -unwind_locked_zv (Lisp_Object point_max)
> +narrowing_lock_pop (Lisp_Object buf)
>  {
> -  SET_BUF_ZV (current_buffer, XFIXNUM (point_max));
> +  Lisp_Object buffer_locks = assq_no_quit (buf, narrowing_locks);
> +  eassert (! NILP (buffer_locks));

Why this assertion?  There's no similar assertions in other functions that
deal with narrowing_locks.

> +  if (EQ (narrowing_lock_peek_tag (buf), Qoutermost_narrowing))
> +    narrowing_locks = Fdelq (Fassoc (buf, narrowing_locks, Qnil),
> +			     narrowing_locks);
> +  else
> +    Fsetcdr (buffer_locks, list1 (Fcdr (Fcar (Fcdr (buffer_locks)))));

Please use XSETCDR instead of Fsetcdr.

> +  begv = narrowing_lock_get_bound (buf, true, false);
> +  zv = narrowing_lock_get_bound (buf, false, false);
> +  if (begv && zv)

This goes back to the question about returning zero for buffer positions:
zero is an unusual value, I'd prefer to use -1 instead.

>      {
> -      EMACS_INT tem = s; s = e; e = tem;
> +      SET_BUF_BEGV (XBUFFER (buf), begv);
> +      SET_BUF_ZV (XBUFFER (buf), zv);

The values you keep in narrowing_locks are markers, so they include the byte
position.  By returning only the character position, you've lost that useful
information, and thus SET_BUF_BEGV/SET_BUF_ZV will have to recompute it.  I
think it's better to return the marker there, and then you can use the byte
positions here and call SET_BUF_BEGV_BOTH and SET_BUF_ZV_BOTH.

> +/* When redisplay is called in a function executed while a locked
> +   narrowing is in effect, restore the narrowing bounds that were set
> +   by the user, and restore the bounds of the locked narrowing when
> +   returning from redisplay.  */
> +void
> +reset_outermost_narrowings (void)

This function doesn't care about redisplay, it will do its job for any
caller.  So the commentary should describe what it does regardless of
redisplay, and then say that it is called from redisplay when so-and-so.

> +  for (val = narrowing_locks; CONSP (val); val = XCDR (val))
>      {
> -      if (!(BEGV <= s && s <= e && e <= ZV))
> -	args_out_of_range (start, end);
> +      buf = Fcar (Fcar (val));
> +      eassert (BUFFERP (buf));
> +      ptrdiff_t begv = narrowing_lock_get_bound (buf, true, true);
> +      ptrdiff_t zv = narrowing_lock_get_bound (buf, false, true);
> +      SET_BUF_BEGV (XBUFFER (buf), begv);
> +      SET_BUF_ZV (XBUFFER (buf), zv);

No checking of values of begv and zv?

> +      record_unwind_protect (unwind_reset_outermost_narrowing, buf);
> +    }
> +}
>  
> -      if (BEGV != s || ZV != e)
> -	current_buffer->clip_changed = 1;
> +/* Helper functions to save and restore the narrowing locks of the
> +   current buffer in save-restriction.  */
> +static Lisp_Object
> +narrowing_locks_save (void)
> +{
> +  Lisp_Object buf = Fcurrent_buffer ();
> +  Lisp_Object locks = assq_no_quit (buf, narrowing_locks);
> +  if (NILP (locks))
> +    return Qnil;
> +  locks = Fcar (Fcdr (locks));
> +  return Fcons (buf, Fcopy_sequence (locks));
> +}
> +
> +static void
> +narrowing_locks_restore (Lisp_Object buf_and_saved_locks)
> +{
> +  if (NILP (buf_and_saved_locks))
> +    return;
> +  Lisp_Object buf = Fcar (buf_and_saved_locks);
> +  eassert (BUFFERP (buf));
> +  Lisp_Object saved_locks = Fcdr (buf_and_saved_locks);
> +  eassert (! NILP (saved_locks));

Again, I don't understand the need for an assertion here.  Just return if
saved_locks is nil.

What about some tests?

Thanks.

bug#56682: locked narrowing

[Gregory Heytings]

Hi Dmitry,

I pushed another change to the feature/improved-locked-narrowing branch, 
which makes the locked narrowing around low-level hooks (and therefore 
around fontification-functions) configurable.  I hope that addresses your 
concerns.

bug#56682: locked narrowing

[Gregory Heytings]

Thanks for your detailed review, and for the useful suggestions!

A few comments/questions below:

>> +(defun with-narrowing-1 (start end tag body)
>> +(defun with-narrowing-2 (start end body)
>
> Should these two helpers be internal functions?
>

You mean, they should be called with-narrowing--{1,2} (or perhaps 
with--narrowing-{1,2})?

>> +/* Remove the innermost lock in BUF from the narrowing_lock alist.  */
>>  static void
>> -unwind_locked_zv (Lisp_Object point_max)
>> +narrowing_lock_pop (Lisp_Object buf)
>>  {
>> -  SET_BUF_ZV (current_buffer, XFIXNUM (point_max));
>> +  Lisp_Object buffer_locks = assq_no_quit (buf, narrowing_locks);
>> +  eassert (! NILP (buffer_locks));
>
> Why this assertion?  There's no similar assertions in other functions 
> that deal with narrowing_locks.
>

Because 'pop' on an empty stack is an error, and (more importantly) it 
would mean that the narrowing_locks alist is corrupted.  The cdr's in that 
alist should never be nil.

>> +static void
>> +narrowing_locks_restore (Lisp_Object buf_and_saved_locks)
>> +{
>> +  if (NILP (buf_and_saved_locks))
>> +    return;
>> +  Lisp_Object buf = Fcar (buf_and_saved_locks);
>> +  eassert (BUFFERP (buf));
>> +  Lisp_Object saved_locks = Fcdr (buf_and_saved_locks);
>> +  eassert (! NILP (saved_locks));
>
> Again, I don't understand the need for an assertion here.  Just return 
> if saved_locks is nil.
>

Strictly speaking there is no need for an assertion, but again it would 
mean that something that isn't supposed to happen happened.  In this case 
it would mean that narrowing_locks_restore is called with a list of length 
1, containing only a buffer, whereas it is supposed to be called with the 
return value of narrowing_locks_save, which is either nil or a list of 
length >= 2.

>
> The values you keep in narrowing_locks are markers, so they include the 
> byte position.  By returning only the character position, you've lost 
> that useful information, and thus SET_BUF_BEGV/SET_BUF_ZV will have to 
> recompute it.  I think it's better to return the marker there, and then 
> you can use the byte positions here and call SET_BUF_BEGV_BOTH and 
> SET_BUF_ZV_BOTH.
>

Thanks, I didn't think of that possibility, it seems better indeed.

>
> What about some tests?
>

I'd have to add some tests, indeed.  Not sure I'll have time to do that 
today.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Sat, 26 Nov 2022 16:15:34 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: monnier@iro.umontreal.ca, 56682@debbugs.gnu.org, dgutov@yandex.ru
> 
> >> +(defun with-narrowing-1 (start end tag body)
> >> +(defun with-narrowing-2 (start end body)
> >
> > Should these two helpers be internal functions?
> >
> 
> You mean, they should be called with-narrowing--{1,2} (or perhaps 
> with--narrowing-{1,2})?

I actually think these should not be called with-SOMETHING, since they
aren't macros.  And yes, the function names should include "--".

> >> +/* Remove the innermost lock in BUF from the narrowing_lock alist.  */
> >>  static void
> >> -unwind_locked_zv (Lisp_Object point_max)
> >> +narrowing_lock_pop (Lisp_Object buf)
> >>  {
> >> -  SET_BUF_ZV (current_buffer, XFIXNUM (point_max));
> >> +  Lisp_Object buffer_locks = assq_no_quit (buf, narrowing_locks);
> >> +  eassert (! NILP (buffer_locks));
> >
> > Why this assertion?  There's no similar assertions in other functions 
> > that deal with narrowing_locks.
> >
> 
> Because 'pop' on an empty stack is an error

You don't need to pop, just return without doing anything.

> and (more importantly) it would mean that the narrowing_locks alist is
> corrupted.  The cdr's in that alist should never be nil.

A comment to that effect should be in order, then.

> >> +static void
> >> +narrowing_locks_restore (Lisp_Object buf_and_saved_locks)
> >> +{
> >> +  if (NILP (buf_and_saved_locks))
> >> +    return;
> >> +  Lisp_Object buf = Fcar (buf_and_saved_locks);
> >> +  eassert (BUFFERP (buf));
> >> +  Lisp_Object saved_locks = Fcdr (buf_and_saved_locks);
> >> +  eassert (! NILP (saved_locks));
> >
> > Again, I don't understand the need for an assertion here.  Just return 
> > if saved_locks is nil.
> >
> 
> Strictly speaking there is no need for an assertion, but again it would 
> mean that something that isn't supposed to happen happened.  In this case 
> it would mean that narrowing_locks_restore is called with a list of length 
> 1, containing only a buffer, whereas it is supposed to be called with the 
> return value of narrowing_locks_save, which is either nil or a list of 
> length >= 2.

I don't think I understand (you avoid the assertion in other cases which
look similar to me), but if there are good reasons, please add comments
there explaining the rationale.

> > What about some tests?
> 
> I'd have to add some tests, indeed.  Not sure I'll have time to do that 
> today.

We can add the tests after the branch is cut, I just wanted to be sure we
will have them eventually.

Thanks.

bug#56682: locked narrowing

[Eli Zaretskii]

> Cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> Date: Sat, 26 Nov 2022 18:51:37 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> We can add the tests after the branch is cut, I just wanted to be sure we
> will have them eventually.

Btw, one more general comment: we will need to tell in NEWS what we expect
from developers of modes to do to comply with these protocols, when they
adapt to Emacs 29 their code related to font-lock and anything else that can
run from redisplay.  The actual text for NEWS can be added later, after the
branch is cut, but maybe thinking about that aspect will affect your code
and/or the doc strings.

bug#56682: locked narrowing

[Gregory Heytings]

>
> I actually think these should not be called with-SOMETHING, since they 
> aren't macros.  And yes, the function names should include "--".
>

Done, I went for internal--with-narrowing.

>> Because 'pop' on an empty stack is an error
>
> You don't need to pop, just return without doing anything.
>
>> and (more importantly) it would mean that the narrowing_locks alist is 
>> corrupted.  The cdr's in that alist should never be nil.
>
> A comment to that effect should be in order, then.
>

Sorry, I was confused, in fact just return'ing in that case is fine.

>> Strictly speaking there is no need for an assertion, but again it would 
>> mean that something that isn't supposed to happen happened.  In this 
>> case it would mean that narrowing_locks_restore is called with a list 
>> of length 1, containing only a buffer, whereas it is supposed to be 
>> called with the return value of narrowing_locks_save, which is either 
>> nil or a list of length >= 2.
>
> I don't think I understand (you avoid the assertion in other cases which 
> look similar to me), but if there are good reasons, please add comments 
> there explaining the rationale.
>

I'm just being extra cautious here.  I added a comment to explain that 
assertion (and the previous one).

>
> We can add the tests after the branch is cut, I just wanted to be sure 
> we will have them eventually.
>

I'll do that.

bug#56682: locked narrowing

[Gregory Heytings]

>> I don't think I understand (you avoid the assertion in other cases 
>> which look similar to me), but if there are good reasons, please add 
>> comments there explaining the rationale.
>
> I'm just being extra cautious here.  I added a comment to explain that 
> assertion (and the previous one).
>

In the end, I removed that assertion, there are indeed no "good reasons" 
to keep it.

bug#56682: locked narrowing

[Dmitry Gutov]

Hi Gregory,

On 26/11/2022 16:14, Gregory Heytings wrote:

> I pushed another change to the feature/improved-locked-narrowing branch, 
> which makes the locked narrowing around low-level hooks (and therefore 
> around fontification-functions) configurable.  I hope that addresses 
> your concerns.

Thank you, this looks like a step in the right direction.

But I tried to see the effects of this new variable 
(long-line-locked-narrowing-region-size), and it was a little puzzling.

With the default value (500000) I open dictionary.json and start hitting 
PgDn a lot. Syntax highlighting starts to look broken at the buffer 
position that's around half that: at my last attempt it was at point 259999.

Looking at the code, there is indeed some halving going on, so maybe it 
would be more easier to understand if the variable was called 
locked-narrowing-radius, and used as such -- without division.

Either way, though, if I set the var to a very large value, the syntax 
highlighting looks okay.

Another thing that bothered me when testing: going from eob to bob in 
dictionary.json took ~4 seconds every time, independent of buffer being 
edited or not. And that delay didn't show up anywhere in the profiler. 
(setq bidi-inhibit-bpa nil) didn't help at all, but (setq 
bidi-display-reordering nil) eliminated that delay. So I'm still kind of 
puzzled why we bother to restrict font-lock, but didn't do that with 
this significantly more costly computation. The value of 
long-line-locked-narrowing-region-size (big or small) has no effect on it.

Going back higher level, I wanted to repeat that don't understand why we 
apply narrowing around font-lock in buffers with long lines, but not in 
large buffers without long lines. Or vice versa.

And we'll probably come back to this question sometime later, but here's 
something that looks like a regression. Editing dictionary.json is 
snappier than dictionary-pp.json (same file but pretty-printed so 
without long lines).

Even with with common setup (and emacs -Q):

1. (setq long-line-locked-narrowing-region-size 50000)
2. (setq bidi-display-reordering nil)
3. Toggle show-paren-mode off, just in case.
4. electric-pair-mode off, same.

typing in the latter file exhibits random pauses in redisplay up to 0.5s 
(and sometimes 1s+). I haven't managed to catch the exact source of 
those pauses (and they're longer with my personal config), but even 
regular editing is slower: evaluating

   (benchmark 1 '(progn (insert " ") (redisplay t)))

in dictionary.json reports something like 0.038906s, but in 
dictionary-pp.json it prints ~0.136387s.

Commit 89a10ffc (before the last merge) behaves the same, but going 
further back, this commit from July behaves differently: b283e36cf19.

Emacs built from that version reports ~0.033022s in dictionary-pp.json 
and 0.114781s at bob in dictionary.json (scrolling to the end 
predictably freezes that Emacs build).

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Tue, 29 Nov 2022 05:20:30 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>,
>  56682@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> Another thing that bothered me when testing: going from eob to bob in 
> dictionary.json took ~4 seconds every time, independent of buffer being 
> edited or not. And that delay didn't show up anywhere in the profiler. 
> (setq bidi-inhibit-bpa nil) didn't help at all, but (setq 
> bidi-display-reordering nil) eliminated that delay. So I'm still kind of 
> puzzled why we bother to restrict font-lock, but didn't do that with 
> this significantly more costly computation.

Why do you think we didn't restrict them?  Limitations on costly
bidi-related stuff is in Emacs since the very beginning, in v24, AFAIR.  And
no code in bidi.c or xdisp.c ever goes outside the current narrowing.

> Even with with common setup (and emacs -Q):
> 
> 1. (setq long-line-locked-narrowing-region-size 50000)
> 2. (setq bidi-display-reordering nil)
> 3. Toggle show-paren-mode off, just in case.
> 4. electric-pair-mode off, same.
> 
> typing in the latter file exhibits random pauses in redisplay up to 0.5s 
> (and sometimes 1s+). I haven't managed to catch the exact source of 
> those pauses (and they're longer with my personal config), but even 
> regular editing is slower: evaluating
> 
>    (benchmark 1 '(progn (insert " ") (redisplay t)))
> 
> in dictionary.json reports something like 0.038906s, but in 
> dictionary-pp.json it prints ~0.136387s.

It is wrong to discuss here issues that are evidently unrelated to long
lines and the measures to make Emacs more efficient with long lines.  Please
file a separate bug report about the delays you see, and please post there
the file dictionary-pp.json you used in these tests.  Bonus points for
including in the report buffer positions where "typing exhibits random
pauses in redisplay", to make search for those more efficient.

Last, but not least: the "optimizations" (I prefer to call them "shortcuts")
which we installed for making Emacs more responsive with long lines are just
that: shortcuts.  They deliberately ignore certain possible complications in
buffer text and properties/overlays, and bypass the potentially costly code
which handles them, in order to make the display code significantly faster,
at a price of being less accurate and correct.  So I'm not surprised that
you sometimes see faster redisplay with long lines: it could well be due to
these simplifications -- and the price is that in some situations the
display will be incorrect.  So it could turn out that there's nothing wrong
with the "unoptimized" redisplay: it simply sometimes takes time to do all
that it has to do to produce correct display on the screen.  The resulting
loss of accuracy and/or correctness is IMO justified when lines are so long
as to make Emacs unusable without these shortcuts, but not when Emacs can
cope with the complete and accurate display perfectly well, albeit somewhat
slower.  So you might be basically comparing apples with oranges here.

But that is theory: we need specific recipes to investigate the causes for
the delay.  

> Commit 89a10ffc (before the last merge) behaves the same, but going 
> further back, this commit from July behaves differently: b283e36cf19.
> 
> Emacs built from that version reports ~0.033022s in dictionary-pp.json 
> and 0.114781s at bob in dictionary.json (scrolling to the end 
> predictably freezes that Emacs build).

I don't understand what exactly you timed here (since it cannot be scrolling
to eob).  Is it the "insert one SPC" or is it something else?

In any case, 0.114781s just 19% (and 22 msec) slower than 0.136387s, so is
this something serious to talk about?  There could be any number of reasons
for this, including some changes to the JSON mode, perhaps?

bug#56682: locked narrowing

[Dmitry Gutov]

On 29/11/2022 14:47, Eli Zaretskii wrote:
>> Date: Tue, 29 Nov 2022 05:20:30 +0200
>> Cc: Eli Zaretskii <eliz@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>,
>>   56682@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> Another thing that bothered me when testing: going from eob to bob in
>> dictionary.json took ~4 seconds every time, independent of buffer being
>> edited or not. And that delay didn't show up anywhere in the profiler.
>> (setq bidi-inhibit-bpa nil) didn't help at all, but (setq
>> bidi-display-reordering nil) eliminated that delay. So I'm still kind of
>> puzzled why we bother to restrict font-lock, but didn't do that with
>> this significantly more costly computation.
> 
> Why do you think we didn't restrict them?  Limitations on costly
> bidi-related stuff is in Emacs since the very beginning, in v24, AFAIR.  And
> no code in bidi.c or xdisp.c ever goes outside the current narrowing.

Just because it was really slow. Apologies, I seem to be unable to 
reproduce this effect now. I'll get back to this if I see it again.

>> Even with with common setup (and emacs -Q):
>>
>> 1. (setq long-line-locked-narrowing-region-size 50000)
>> 2. (setq bidi-display-reordering nil)
>> 3. Toggle show-paren-mode off, just in case.
>> 4. electric-pair-mode off, same.
>>
>> typing in the latter file exhibits random pauses in redisplay up to 0.5s
>> (and sometimes 1s+). I haven't managed to catch the exact source of
>> those pauses (and they're longer with my personal config), but even
>> regular editing is slower: evaluating
>>
>>     (benchmark 1 '(progn (insert " ") (redisplay t)))
>>
>> in dictionary.json reports something like 0.038906s, but in
>> dictionary-pp.json it prints ~0.136387s.
> 
> It is wrong to discuss here issues that are evidently unrelated to long
> lines and the measures to make Emacs more efficient with long lines.  Please
> file a separate bug report about the delays you see, and please post there
> the file dictionary-pp.json you used in these tests.  Bonus points for
> including in the report buffer positions where "typing exhibits random
> pauses in redisplay", to make search for those more efficient.

My theory was that it's likely related, since there haven't been any 
other major changes to the display engine later. Of course I could be 
mistaken.

If you don't see the problem right away, sure I can file this separately.

dictionary-pp.json is produced from dictionary.json using 'M-x 
json-pretty-print-buffer'. It's 27 MB long, do you want me to upload it 
to some file sharing platform?

Positions: any near bob.

The profiler output looks like this:

          526  87% - command-execute
          516  85%  - call-interactively
          378  62%   - funcall-interactively
          370  61%    - eval-expression
          370  61%     - eval
          370  61%      - benchmark
          370  61%       - benchmark-call
          194  32%        - #<lambda 0x91bc787b0d851>
          194  32%         - progn
          194  32%            redisplay
          160  26%        - #<lambda 0x91bc787b0d851>
          160  26%         - progn
          160  26%            redisplay
           16   2%        - #<lambda 0x91bc787b0d851>
           16   2%         - progn
           16   2%            redisplay

> Last, but not least: the "optimizations" (I prefer to call them "shortcuts")
> which we installed for making Emacs more responsive with long lines are just
> that: shortcuts.  They deliberately ignore certain possible complications in
> buffer text and properties/overlays, and bypass the potentially costly code
> which handles them, in order to make the display code significantly faster,
> at a price of being less accurate and correct.  So I'm not surprised that
> you sometimes see faster redisplay with long lines: it could well be due to
> these simplifications -- and the price is that in some situations the
> display will be incorrect.  So it could turn out that there's nothing wrong
> with the "unoptimized" redisplay: it simply sometimes takes time to do all
> that it has to do to produce correct display on the screen.  The resulting
> loss of accuracy and/or correctness is IMO justified when lines are so long
> as to make Emacs unusable without these shortcuts, but not when Emacs can
> cope with the complete and accurate display perfectly well, albeit somewhat
> slower.  So you might be basically comparing apples with oranges here.
> 
> But that is theory: we need specific recipes to investigate the causes for
> the delay.
> 
>> Commit 89a10ffc (before the last merge) behaves the same, but going
>> further back, this commit from July behaves differently: b283e36cf19.
>>
>> Emacs built from that version reports ~0.033022s in dictionary-pp.json
>> and 0.114781s at bob in dictionary.json (scrolling to the end
>> predictably freezes that Emacs build).
> 
> I don't understand what exactly you timed here (since it cannot be scrolling
> to eob).  Is it the "insert one SPC" or is it something else?

"insert one SPC and redisplay".

The 'insert' call by itself is very fast.

> In any case, 0.114781s just 19% (and 22 msec) slower than 0.136387s, so is
> this something serious to talk about?  There could be any number of reasons
> for this, including some changes to the JSON mode, perhaps?

It's 0.114 vs 0.033 (in dictionary-pp.json, file without long lines).

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Tue, 29 Nov 2022 15:21:12 +0200
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Why do you think we didn't restrict them?  Limitations on costly
> > bidi-related stuff is in Emacs since the very beginning, in v24, AFAIR.  And
> > no code in bidi.c or xdisp.c ever goes outside the current narrowing.
> 
> Just because it was really slow. Apologies, I seem to be unable to 
> reproduce this effect now. I'll get back to this if I see it again.

What is not reproducible? the 0.5sec delays or something else?

Is it still worth our while to look at the dictionary-pp.json file and the
slow redisplay with it?

> dictionary-pp.json is produced from dictionary.json using 'M-x 
> json-pretty-print-buffer'. It's 27 MB long, do you want me to upload it 
> to some file sharing platform?

If you compress it with xz, doesn't become small enough to post here?

If not, please upload somewhere and tell where.

> The profiler output looks like this:
> 
>           526  87% - command-execute
>           516  85%  - call-interactively
>           378  62%   - funcall-interactively
>           370  61%    - eval-expression
>           370  61%     - eval
>           370  61%      - benchmark
>           370  61%       - benchmark-call
>           194  32%        - #<lambda 0x91bc787b0d851>
>           194  32%         - progn
>           194  32%            redisplay
>           160  26%        - #<lambda 0x91bc787b0d851>
>           160  26%         - progn
>           160  26%            redisplay
>            16   2%        - #<lambda 0x91bc787b0d851>
>            16   2%         - progn
>            16   2%            redisplay

This seems to say redisplay takes about 1/3rd of the time?

> > In any case, 0.114781s just 19% (and 22 msec) slower than 0.136387s, so is
> > this something serious to talk about?  There could be any number of reasons
> > for this, including some changes to the JSON mode, perhaps?
> 
> It's 0.114 vs 0.033 (in dictionary-pp.json, file without long lines).

I thought you were saying that Emacs at commit 89a10ffc and at current HEAD
took 0.136387 sec, which is slower than 0.114 sec at commit b283e36cf19
(from July)?  The 0.033 figure is with long-line optimizations, no?  So it's
a very different buffer text and different code paths.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Tue, 29 Nov 2022 17:26:46 +0200
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> It's 3 MB compressed. Let's see if it's small enough as an attachment.

Thanks, got it.

> The figure of 0.114 sec is for dictionary.json (in July Emacs). Which is 
> the same file but with all contents on one line.

Ah, yes.  Too many dictionaries and too little coffee.

Thanks, will look at this soon.

bug#56682: locked narrowing

[Gregory Heytings]

>
> But I tried to see the effects of this new variable 
> (long-line-locked-narrowing-region-size), and it was a little puzzling.
>
> With the default value (500000) I open dictionary.json and start hitting 
> PgDn a lot. Syntax highlighting starts to look broken at the buffer 
> position that's around half that: at my last attempt it was at point 
> 259999.
>
> Looking at the code, there is indeed some halving going on, so maybe it 
> would be more easier to understand if the variable was called 
> locked-narrowing-radius, and used as such -- without division.
>

I considered naming it with "radius", but decided not to do it.  This is 
an implementation detail.  And the docstring says "region around point", 
from which an attentive reader can infer that half of the region will be 
before and half of it will be after point.  It does not seem worth the 
price to introduce separate cases for point at or near BOB and point at or 
near EOB.

>
> Another thing that bothered me when testing: going from eob to bob in 
> dictionary.json took ~4 seconds every time, independent of buffer being 
> edited or not. And that delay didn't show up anywhere in the profiler. 
> (setq bidi-inhibit-bpa nil) didn't help at all, but (setq 
> bidi-display-reordering nil) eliminated that delay.
>

Your analysis is not correct: the delay you see is due to show-paren-mode. 
Turn it off, and you'll see that going from EOB to BOB is instantaneous.

>
> And we'll probably come back to this question sometime later, but here's 
> something that looks like a regression. Editing dictionary.json is 
> snappier than dictionary-pp.json (same file but pretty-printed so 
> without long lines).
>

Unless I misunderstand what you mean, this cannot be a regression: editing 
dictionary.json was not possible with Emacs 28, editing dictionary-pp.json 
was.  Even if the former were now snappier than the latter, that would not 
be a problem in itself.

>
> Even with with common setup (and emacs -Q):
>
> 1. (setq long-line-locked-narrowing-region-size 50000)
> 2. (setq bidi-display-reordering nil)
> 3. Toggle show-paren-mode off, just in case.
> 4. electric-pair-mode off, same.
>
> typing in the latter file exhibits random pauses in redisplay up to 0.5s 
> (and sometimes 1s+). I haven't managed to catch the exact source of 
> those pauses (and they're longer with my personal config), but even 
> regular editing is slower: evaluating
>
> (benchmark 1 '(progn (insert " ") (redisplay t)))
>
> in dictionary.json reports something like 0.038906s, but in 
> dictionary-pp.json it prints ~0.136387s.
>

I cannot reproduce that.  With emacs -Q, with dictionary-pp.json and with 
show-paren-mode disabled, type C-s aan zich RET.  Now type some text: in 
both cases the effect is near instantaneous here, and I do not see any 
random pauses.

With your benchmark and with dictionary-pp.json:

- on current master, after C-s eman RET, I see ~100 ms, and after C-s aan 
zich RET, I see ~100 ms

- at ff57f30bee (30 July), after C-s eman RET, I see ~20 ms, and after C-s 
aan zich RET, I see ~2700 (!) ms

- at c59b8dfefa (30 Jun), after C-s eman RET, I see ~20 ms, and after C-s 
aan zich RET, I see ~2700 (!) ms

- with Emacs 28, after C-s eman RET, I see ~20 ms, and after C-s aan zich 
RET, I see ~3000 (!) ms

With your benchmark and with dictionary.json:

- with current master, after C-s eman RET, I see ~70 ms, and after C-s aan 
zich RET, I see ~70 ms

So there is no regression as far as I can see.

bug#56682: locked narrowing

[Dmitry Gutov]

On 29/11/2022 19:10, Gregory Heytings wrote:

>> Looking at the code, there is indeed some halving going on, so maybe 
>> it would be more easier to understand if the variable was called 
>> locked-narrowing-radius, and used as such -- without division.
>>
> 
> I considered naming it with "radius", but decided not to do it.  This is 
> an implementation detail.  And the docstring says "region around point", 
> from which an attentive reader can infer that half of the region will be 
> before and half of it will be after point.  It does not seem worth the 
> price to introduce separate cases for point at or near BOB and point at 
> or near EOB.

All right. It would have been easier for me to understand, but it's 
nothing critical.

>> Another thing that bothered me when testing: going from eob to bob in 
>> dictionary.json took ~4 seconds every time, independent of buffer 
>> being edited or not. And that delay didn't show up anywhere in the 
>> profiler. (setq bidi-inhibit-bpa nil) didn't help at all, but (setq 
>> bidi-display-reordering nil) eliminated that delay.
>>
> 
> Your analysis is not correct: the delay you see is due to 
> show-paren-mode. Turn it off, and you'll see that going from EOB to BOB 
> is instantaneous.

show-paren-mode does seem to have a tiny effect here, but nothing 
comparable. And when it's on, it seems to blink red (the "mismatch" 
face), indicating that its scan probably stops earlier before it reaches 
the EOB.

Anyway, I'll let you know with more details when I see this again. 
Previously, I would see this effect (multi-second delays) every time I 
test the feature, now it is just gone. Maybe 'make bootstrap' helped.

>> And we'll probably come back to this question sometime later, but 
>> here's something that looks like a regression. Editing dictionary.json 
>> is snappier than dictionary-pp.json (same file but pretty-printed so 
>> without long lines).
>>
> 
> Unless I misunderstand what you mean, this cannot be a regression: 
> editing dictionary.json was not possible with Emacs 28, editing 
> dictionary-pp.json was.  Even if the former were now snappier than the 
> latter, that would not be a problem in itself.

I'm mostly talking about how editing dictionary-pp.json became slower.

The fact that editing dictionary.json is now faster than that is, while 
counter-intuitive, definitely not something to complain about by itself.

>> Even with with common setup (and emacs -Q):
>>
>> 1. (setq long-line-locked-narrowing-region-size 50000)
>> 2. (setq bidi-display-reordering nil)
>> 3. Toggle show-paren-mode off, just in case.
>> 4. electric-pair-mode off, same.
>>
>> typing in the latter file exhibits random pauses in redisplay up to 
>> 0.5s (and sometimes 1s+). I haven't managed to catch the exact source 
>> of those pauses (and they're longer with my personal config), but even 
>> regular editing is slower: evaluating
>>
>> (benchmark 1 '(progn (insert " ") (redisplay t)))
>>
>> in dictionary.json reports something like 0.038906s, but in 
>> dictionary-pp.json it prints ~0.136387s.
>>
> 
> I cannot reproduce that.  With emacs -Q, with dictionary-pp.json and 
> with show-paren-mode disabled, type C-s aan zich RET.  Now type some 
> text: in both cases the effect is near instantaneous here, and I do not 
> see any random pauses.

Right. "aan zich" is not far from EOB.

My scenarios are all near BOB.

> With your benchmark and with dictionary-pp.json:
> 
> - on current master, after C-s eman RET, I see ~100 ms, and after C-s 

Note the 100 ms.

> aan zich RET, I see ~100 ms

"eman" is near BOB for you as well, right?

> - at ff57f30bee (30 July), after C-s eman RET, I see ~20 ms, and after 
> C-s aan zich RET, I see ~2700 (!) ms
> 
> - at c59b8dfefa (30 Jun), after C-s eman RET, I see ~20 ms, and after 
> C-s aan zich RET, I see ~2700 (!) ms

Note the 20 ms.

> - with Emacs 28, after C-s eman RET, I see ~20 ms, and after C-s aan 
> zich RET, I see ~3000 (!) ms

Yup, the long delays near EOB are expected, I fixed them in js.el not 
too long ago (one in font-lock rules, and anothing by creating 
js-json-mode).

> With your benchmark and with dictionary.json:
> 
> - with current master, after C-s eman RET, I see ~70 ms, and after C-s 
> aan zich RET, I see ~70 ms
> 
> So there is no regression as far as I can see.

Your observations seem to match mine.

Look at the timings for dictionary-pp.json. After C-s eman:

master: ~100 ms
ff57f30bee, or c59b8dfefa, or Emacs 28: ~20 ms

That's the regression.

But like Eli said, it might be unrelated your code, it just seems to 
belong to the same general area. If my guess was wrong, I apologize.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Tue, 29 Nov 2022 17:10:27 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca>, 
>     56682@debbugs.gnu.org
> 
> > Looking at the code, there is indeed some halving going on, so maybe it 
> > would be more easier to understand if the variable was called 
> > locked-narrowing-radius, and used as such -- without division.
> 
> I considered naming it with "radius", but decided not to do it.  This is 
> an implementation detail.

Maybe long-line-locked-narrowing-half-size?

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Tue, 29 Nov 2022 19:56:30 +0200
> Cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> Look at the timings for dictionary-pp.json. After C-s eman:
> 
> master: ~100 ms
> ff57f30bee, or c59b8dfefa, or Emacs 28: ~20 ms
> 
> That's the regression.

The faster ones use a different major-mode, btw.

bug#56682: locked narrowing

[Dmitry Gutov]

On 29/11/2022 20:19, Eli Zaretskii wrote:
>> Date: Tue, 29 Nov 2022 19:56:30 +0200
>> Cc:56682@debbugs.gnu.org, Eli Zaretskii<eliz@gnu.org>,
>>   Stefan Monnier<monnier@iro.umontreal.ca>
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>> Look at the timings for dictionary-pp.json. After C-s eman:
>>
>> master: ~100 ms
>> ff57f30bee, or c59b8dfefa, or Emacs 28: ~20 ms
>>
>> That's the regression.
> The faster ones use a different major-mode, btw.

The old (faster) revision uses js-mode, the new one -- the optimized 
js-json-mode. I'm sure I could have made a mistake doing that 
optimization, but the behavior doesn't indicate that: the delays are 
higher near BOB and seem absent near EOB (in master/emacs-29), and they 
don't show up in the profiler in separate leaf nodes (the previous ones 
-- which I worked on optimizing -- did).

You can also copy the definition of js-json-mode to a file and evaluate 
it in the July Emacs, and use it with that file. Or enable js-mode 
instead in the emacs-29 Emacs. The measurements do not change here.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Tue, 29 Nov 2022 21:29:59 +0200
> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> That's the regression.
> > The faster ones use a different major-mode, btw.
> 
> The old (faster) revision uses js-mode, the new one -- the optimized 
> js-json-mode. I'm sure I could have made a mistake doing that 
> optimization, but the behavior doesn't indicate that: the delays are 
> higher near BOB and seem absent near EOB (in master/emacs-29), and they 
> don't show up in the profiler in separate leaf nodes (the previous ones 
> -- which I worked on optimizing -- did).

I didn't say the new mode was the culprit (you could have assumed I tried
the old mode as soon as I discovered this change since July).  I just
mentioned this for completeness, so we are aware what we are comparing.

Anyway, do you see any effect if you set long-line-threshold to nil?  Here
it makes the redisplay after insertion instantaneous (and this is an
unoptimized build, where without setting that variable to nil each insertion
in dictionary-pp.json takes about half a second).

bug#56682: locked narrowing

[Gregory Heytings]

>> - on current master, after C-s eman RET, I see ~100 ms, and after C-s
>
> Note the 100 ms.
>
>> aan zich RET, I see ~100 ms
>
> "eman" is near BOB for you as well, right?
>

It's the same file, so yes.

>> - at ff57f30bee (30 July), after C-s eman RET, I see ~20 ms, and after 
>> C-s aan zich RET, I see ~2700 (!) ms
>> 
>> - at c59b8dfefa (30 Jun), after C-s eman RET, I see ~20 ms, and after 
>> C-s aan zich RET, I see ~2700 (!) ms
>
> Note the 20 ms.
>

I did.

>> - with Emacs 28, after C-s eman RET, I see ~20 ms, and after C-s aan 
>> zich RET, I see ~3000 (!) ms
>
> Yup, the long delays near EOB are expected, I fixed them in js.el not 
> too long ago (one in font-lock rules, and anothing by creating 
> js-json-mode).
>

So what?  We had in that file 20 ms at BOB and 3000 ms at EOB, which means 
an average of 1500 ms (in that file).  Now we have 100 ms at BOB and EOB, 
which means an average of 100 ms (in that file).  How can that be a 
regression?

bug#56682: locked narrowing

[Dmitry Gutov]

On 29/11/2022 21:51, Eli Zaretskii wrote:
>> Date: Tue, 29 Nov 2022 21:29:59 +0200
>> Cc: 56682@debbugs.gnu.org, gregory@heytings.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>>>> That's the regression.
>>> The faster ones use a different major-mode, btw.
>>
>> The old (faster) revision uses js-mode, the new one -- the optimized
>> js-json-mode. I'm sure I could have made a mistake doing that
>> optimization, but the behavior doesn't indicate that: the delays are
>> higher near BOB and seem absent near EOB (in master/emacs-29), and they
>> don't show up in the profiler in separate leaf nodes (the previous ones
>> -- which I worked on optimizing -- did).
> 
> I didn't say the new mode was the culprit (you could have assumed I tried
> the old mode as soon as I discovered this change since July).  I just
> mentioned this for completeness, so we are aware what we are comparing.

For completeness we could also mention that js-mode also had one 
significant change in font-lock rules (also an optimization) which is 
not as easily transplantable.

> Anyway, do you see any effect if you set long-line-threshold to nil?  Here
> it makes the redisplay after insertion instantaneous (and this is an
> unoptimized build, where without setting that variable to nil each insertion
> in dictionary-pp.json takes about half a second).

Yep, that helps.

So the delay is due to it looking for long lines in the buffer, I guess?

Curious why it doesn't happen when near EOB. The benchmark reports a 
delay once, but if you run it multiple times, the timing go down to 
"instant".

bug#56682: locked narrowing

[Dmitry Gutov]

On 29/11/2022 22:11, Gregory Heytings wrote:

>>> - with Emacs 28, after C-s eman RET, I see ~20 ms, and after C-s aan 
>>> zich RET, I see ~3000 (!) ms
>>
>> Yup, the long delays near EOB are expected, I fixed them in js.el not 
>> too long ago (one in font-lock rules, and anothing by creating 
>> js-json-mode).
>>
> 
> So what?  We had in that file 20 ms at BOB and 3000 ms at EOB, which 
> means an average of 1500 ms (in that file).  Now we have 100 ms at BOB 
> and EOB, which means an average of 100 ms (in that file).  How can that 
> be a regression?

But the improvement near EOB in this scenario down from 1500 ms is not 
necessarily from the long-lines feature.

Like Eli pointed out, try (setq long-line-threshold nil).

The result will be that the benchmark will report ~30ms both near BOB 
and near EOB. So the long-lines-threshold thingy adds a regression here.

Again, not trying to criticize or anything, but a 100ms is usually 
considered as something that brings an operation from appearing 
"instantaneous" down to "noticeable delay". Since we're trying to make 
editing large files easier and snappier, that seems worth fixing.

Not to mention that the user might have other features enabled that can 
add a multiplier on top of this delay -- which is apparently the case in 
my personal configuration.

bug#56682: locked narrowing

[Gregory Heytings]

>
> The result will be that the benchmark will report ~30ms both near BOB 
> and near EOB. So the long-lines-threshold thingy adds a regression here.
>

It's not a regression, it's the price we agreed to pay to detect whether 
the buffer contains long lines.  That detection happens dynamically, to 
catch cases when long lines are inserted in a buffer.

>
> Again, not trying to criticize or anything, but a 100ms is usually 
> considered as something that brings an operation from appearing 
> "instantaneous" down to "noticeable delay".
>

I don't think your benchmark is representative.  While (benchmark-run 1 
(progn (insert "a") (redisplay t))) does take 100 ms, Emacs remains 
responsive when I type as fast as I can (which is faster than 10 
characters per second), and I do not see any delays.

bug#56682: locked narrowing

[Gregory Heytings]

>> The result will be that the benchmark will report ~30ms both near BOB 
>> and near EOB. So the long-lines-threshold thingy adds a regression 
>> here.
>
> It's not a regression, it's the price we agreed to pay to detect whether 
> the buffer contains long lines.  That detection happens dynamically, to 
> catch cases when long lines are inserted in a buffer.
>

After looking at this closely, it turns out that the change in 1c837c42c2 
doesn't do what it was meant to do.  Sigh...  Dmitry, can you confirm that 
the following change fixes this?

diff --git a/src/xdisp.c b/src/xdisp.c
index b09aa6ec96..cdd6a7f5fe 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -19535,7 +19535,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
    /* Check whether the buffer to be displayed contains long lines.  */
    if (!NILP (Vlong_line_threshold)
        && !current_buffer->long_line_optimizations_p
-      && CHARS_MODIFF - CHARS_UNCHANGED_MODIFIED > 8)
+      && CHARS_MODIFF - UNCHANGED_MODIFIED > 8)
      {
        ptrdiff_t cur, next, found, max = 0, threshold;
        threshold = XFIXNUM (Vlong_line_threshold);

bug#56682: locked narrowing

[Dmitry Gutov]

On 30/11/2022 02:15, Gregory Heytings wrote:
> 
>>> The result will be that the benchmark will report ~30ms both near BOB 
>>> and near EOB. So the long-lines-threshold thingy adds a regression here.
>>
>> It's not a regression, it's the price we agreed to pay to detect 
>> whether the buffer contains long lines.  That detection happens 
>> dynamically, to catch cases when long lines are inserted in a buffer.
>>
> 
> After looking at this closely, it turns out that the change in 
> 1c837c42c2 doesn't do what it was meant to do.  Sigh...  Dmitry, can you 
> confirm that the following change fixes this?
> 
> diff --git a/src/xdisp.c b/src/xdisp.c
> index b09aa6ec96..cdd6a7f5fe 100644
> --- a/src/xdisp.c
> +++ b/src/xdisp.c
> @@ -19535,7 +19535,7 @@ redisplay_window (Lisp_Object window, bool 
> just_this_one_p)
>     /* Check whether the buffer to be displayed contains long lines.  */
>     if (!NILP (Vlong_line_threshold)
>         && !current_buffer->long_line_optimizations_p
> -      && CHARS_MODIFF - CHARS_UNCHANGED_MODIFIED > 8)
> +      && CHARS_MODIFF - UNCHANGED_MODIFIED > 8)
>       {
>         ptrdiff_t cur, next, found, max = 0, threshold;
>         threshold = XFIXNUM (Vlong_line_threshold);
> 

Looking good. Thanks, Gregory!

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Wed, 30 Nov 2022 00:15:39 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>, 
>     Stefan Monnier <monnier@iro.umontreal.ca>
> 
> 
> >> The result will be that the benchmark will report ~30ms both near BOB 
> >> and near EOB. So the long-lines-threshold thingy adds a regression 
> >> here.
> >
> > It's not a regression, it's the price we agreed to pay to detect whether 
> > the buffer contains long lines.  That detection happens dynamically, to 
> > catch cases when long lines are inserted in a buffer.
> >
> 
> After looking at this closely, it turns out that the change in 1c837c42c2 
> doesn't do what it was meant to do.  Sigh...  Dmitry, can you confirm that 
> the following change fixes this?

I think this is just one aspect of the issue.  The loop under this condition
"punishes" large buffers with no long lines because we search the entire
buffer top to bottom.  To make this scale better, I think we should only
search in some predefined vicinity of the window, perhaps the same region to
which we narrow the buffer when we do detect long lines.  We can then repeat
the search if point moves far away from its last value.

There's no need to search the whole buffer, that will cause delays in very
large buffers for no good reason.

We've talked about this a few months ago, and you said fixing this was part
of your todo.  I think now is the time.

bug#56682: locked narrowing

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 100 bytes --]


>
> Looking good. Thanks, Gregory!
>

Thanks for checking!

Eli, is the attached okay for emacs-29?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Actually-improve-detection-of-long-lines.patch --]
[-- Type: text/x-diff; name=Actually-improve-detection-of-long-lines.patch, Size: 4237 bytes --]

From a900a41e10ca928ede13dab16e2b3c95e0e3d80c Mon Sep 17 00:00:00 2001
From: Gregory Heytings <gregory@heytings.org>
Date: Wed, 30 Nov 2022 09:58:44 +0000
Subject: [PATCH] Actually improve detection of long lines

* src/xdisp.c (redisplay_window): Update condition.

* src/xdisp.c (mark_window_display_accurate_1):
* src/pdumper.c (dump_buffer):
* src/buffer.h (BUF_CHARS_UNCHANGED_MODIFIED):
(struct buffer_text):
* src/buffer.c (Fget_buffer_create):
Revert 1c837c42c2, which was misguided.
---
 src/buffer.c  |  1 -
 src/buffer.h  | 11 -----------
 src/pdumper.c |  1 -
 src/xdisp.c   |  3 +--
 4 files changed, 1 insertion(+), 15 deletions(-)

diff --git a/src/buffer.c b/src/buffer.c
index 71be7ed9e1..69f27c9f47 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -604,7 +604,6 @@ DEFUN ("get-buffer-create", Fget_buffer_create, Sget_buffer_create, 1, 2, 0,
   set_buffer_intervals (b, NULL);
   BUF_UNCHANGED_MODIFIED (b) = 1;
   BUF_OVERLAY_UNCHANGED_MODIFIED (b) = 1;
-  BUF_CHARS_UNCHANGED_MODIFIED (b) = 1;
   BUF_END_UNCHANGED (b) = 0;
   BUF_BEG_UNCHANGED (b) = 0;
   *(BUF_GPT_ADDR (b)) = *(BUF_Z_ADDR (b)) = 0; /* Put an anchor '\0'.  */
diff --git a/src/buffer.h b/src/buffer.h
index dded0cd98c..80b3fd81e0 100644
--- a/src/buffer.h
+++ b/src/buffer.h
@@ -149,18 +149,12 @@ #define BUF_OVERLAY_UNCHANGED_MODIFIED(buf) \
 #define BUF_BEG_UNCHANGED(buf) ((buf)->text->beg_unchanged)
 #define BUF_END_UNCHANGED(buf) ((buf)->text->end_unchanged)
 
-#define BUF_CHARS_UNCHANGED_MODIFIED(buf) \
-  ((buf)->text->chars_unchanged_modified)
-
 #define UNCHANGED_MODIFIED \
   BUF_UNCHANGED_MODIFIED (current_buffer)
 #define OVERLAY_UNCHANGED_MODIFIED \
   BUF_OVERLAY_UNCHANGED_MODIFIED (current_buffer)
 #define BEG_UNCHANGED BUF_BEG_UNCHANGED (current_buffer)
 #define END_UNCHANGED BUF_END_UNCHANGED (current_buffer)
-
-#define CHARS_UNCHANGED_MODIFIED \
-  BUF_CHARS_UNCHANGED_MODIFIED (current_buffer)
 \f
 /* Functions to set PT in the current buffer, or another buffer.  */
 
@@ -274,11 +268,6 @@ #define FETCH_BYTE(n) *(BYTE_POS_ADDR ((n)))
        end_unchanged contain no useful information.  */
     modiff_count overlay_unchanged_modified;
 
-    /* CHARS_MODIFF as of last redisplay that finished.  It's used
-       when we only care about changes in actual buffer text, not in
-       any other kind of changes, like properties etc.  */
-    modiff_count chars_unchanged_modified;
-
     /* Properties of this buffer's text.  */
     INTERVAL intervals;
 
diff --git a/src/pdumper.c b/src/pdumper.c
index fedcd3e404..263343f60b 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -2811,7 +2811,6 @@ dump_buffer (struct dump_context *ctx, const struct buffer *in_buffer)
       DUMP_FIELD_COPY (out, buffer, own_text.end_unchanged);
       DUMP_FIELD_COPY (out, buffer, own_text.unchanged_modified);
       DUMP_FIELD_COPY (out, buffer, own_text.overlay_unchanged_modified);
-      DUMP_FIELD_COPY (out, buffer, own_text.chars_unchanged_modified);
       if (buffer->own_text.intervals)
         dump_field_fixup_later (ctx, out, buffer, &buffer->own_text.intervals);
       dump_field_lv_rawptr (ctx, out, buffer, &buffer->own_text.markers,
diff --git a/src/xdisp.c b/src/xdisp.c
index b09aa6ec96..67c07725c4 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -17264,7 +17264,6 @@ mark_window_display_accurate_1 (struct window *w, bool accurate_p)
 
       BUF_UNCHANGED_MODIFIED (b) = BUF_MODIFF (b);
       BUF_OVERLAY_UNCHANGED_MODIFIED (b) = BUF_OVERLAY_MODIFF (b);
-      BUF_CHARS_UNCHANGED_MODIFIED (b) = BUF_CHARS_MODIFF (b);
       BUF_BEG_UNCHANGED (b) = BUF_GPT (b) - BUF_BEG (b);
       BUF_END_UNCHANGED (b) = BUF_Z (b) - BUF_GPT (b);
 
@@ -19535,7 +19534,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
   /* Check whether the buffer to be displayed contains long lines.  */
   if (!NILP (Vlong_line_threshold)
       && !current_buffer->long_line_optimizations_p
-      && CHARS_MODIFF - CHARS_UNCHANGED_MODIFIED > 8)
+      && CHARS_MODIFF - UNCHANGED_MODIFIED > 8)
     {
       ptrdiff_t cur, next, found, max = 0, threshold;
       threshold = XFIXNUM (Vlong_line_threshold);
-- 
2.35.1

bug#56682: locked narrowing

[Gregory Heytings]

>
> I think this is just one aspect of the issue.  The loop under this 
> condition "punishes" large buffers with no long lines because we search 
> the entire buffer top to bottom.  To make this scale better, I think we 
> should only search in some predefined vicinity of the window, perhaps 
> the same region to which we narrow the buffer when we do detect long 
> lines.  We can then repeat the search if point moves far away from its 
> last value.
>
> There's no need to search the whole buffer, that will cause delays in 
> very large buffers for no good reason.
>
> We've talked about this a few months ago, and you said fixing this was 
> part of your todo.  I think now is the time.
>

Yes, it was and still is part of my todo.  At that point of time I was not 
at all expecting that Emacs 29 would be released three months later. 
What I had and still have in mind is too large a change for Emacs 29, so 
at first glance I think for now we'll have to live with the current 
solution.

Note that, apart from Dmitry, nobody reported a problem with that 
detection mechanism, and Dmitry's recipe with which he spotted that 
problem does not reflect actual usage.  And note that with the patch I 
just sent even Dmitry's recipe is fixed.

I'll see if I can design something smaller which would be suitable for 
Emacs 29, however.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Wed, 30 Nov 2022 00:15:39 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>, 
>     Stefan Monnier <monnier@iro.umontreal.ca>
> 
> After looking at this closely, it turns out that the change in 1c837c42c2 
> doesn't do what it was meant to do.  Sigh...  Dmitry, can you confirm that 
> the following change fixes this?

??? You are reverting what I believe was a fix for bug#57207, right?  I
don't think we want that problem back, do we?

How did you reach the conclusion that the change doesn't do what it was
supposed to? please tell more about what you saw and your conclusions.

bug#56682: locked narrowing

[Gregory Heytings]

>> After looking at this closely, it turns out that the change in 
>> 1c837c42c2 doesn't do what it was meant to do.  Sigh...  Dmitry, can 
>> you confirm that the following change fixes this?
>
> ??? You are reverting what I believe was a fix for bug#57207, right?
>

No, I'm not reverting, I'm fixing.  The condition before 1c837c42c2 was 
"MODIFF - UNCHANGED_MODIFIED > 8", with 1c837c42c2 it became "CHARS_MODIFF 
- CHARS_UNCHANGED_MODIFIED > 8", which appeared correct at that time but 
is not, with the patch it becomes "CHARS_MODIFF - UNCHANGED_MODIFIED > 8".

>
> How did you reach the conclusion that the change doesn't do what it was 
> supposed to? please tell more about what you saw and your conclusions.
>

The first condition triggered the long line detection too much: it was 
triggered when the the only changes in the buffer were fontification 
changes.  The second condition fixes that problem, but still triggers the 
long line detection too much: it is still triggered when the main (but not 
_only_) changes in the buffer are fontification changes, IOW, it is 
triggered even if a single character is inserted and "enough" 
fontification changes are made.  The condition in the patch is correct: 
the long line detection mechanism is triggered only when many characters 
are inserted.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Wed, 30 Nov 2022 10:11:17 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> > There's no need to search the whole buffer, that will cause delays in 
> > very large buffers for no good reason.
> >
> > We've talked about this a few months ago, and you said fixing this was 
> > part of your todo.  I think now is the time.
> >
> 
> Yes, it was and still is part of my todo.  At that point of time I was not 
> at all expecting that Emacs 29 would be released three months later. 
> What I had and still have in mind is too large a change for Emacs 29, so 
> at first glance I think for now we'll have to live with the current 
> solution.

I don't think I understand why it should be problematic or unsafe to limit
the same loop we have to a smaller region of the buffer.

> Note that, apart from Dmitry, nobody reported a problem with that 
> detection mechanism, and Dmitry's recipe with which he spotted that 
> problem does not reflect actual usage.  And note that with the patch I 
> just sent even Dmitry's recipe is fixed.

It should be clear that triggering such long searches should not be too
hard.  The search itself should scale better, similar to the general idea of
narrowing the display code, whose scaling is excellent, precisely because
the region doesn't grow with the buffer size.

> I'll see if I can design something smaller which would be suitable for 
> Emacs 29, however.

Well, I think what I proposed is safe enough already.  But maybe I'm missing
something.

bug#56682: locked narrowing

[Gregory Heytings]

>
> I don't think I understand why it should be problematic or unsafe to 
> limit the same loop we have to a smaller region of the buffer.
>

I see that you already changed the loop to scan from BEG to Z to only scan 
from BEGV to ZV; somehow I missed that change.  I don't think even that is 
correct: a command could insert text in the buffer outside of BEGV/ZV. 
While limiting the loop to a smaller region of the buffer looks like a 
reasonable thing to do, the question is: how can we determine that region? 
And I don't see how this can be done, because the detection code is called 
only when a buffer is about to be displayed, and at that moment we cannot 
know where characters were inserted in the buffer.  What the code does now 
is "if enough characters have been inserted in the buffer since last 
redisplay, check again whether the buffer contains long lines".

>
> It should be clear that triggering such long searches should not be too 
> hard.
>

The search is not that long, and it is not triggered for regular editing 
commands, e.g. while typing or when killing / yanking a line in a buffer.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Wed, 30 Nov 2022 13:52:48 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> No, I'm not reverting, I'm fixing.  The condition before 1c837c42c2 was 
> "MODIFF - UNCHANGED_MODIFIED > 8", with 1c837c42c2 it became "CHARS_MODIFF 
> - CHARS_UNCHANGED_MODIFIED > 8", which appeared correct at that time but 
> is not, with the patch it becomes "CHARS_MODIFF - UNCHANGED_MODIFIED > 8".
> 
> >
> > How did you reach the conclusion that the change doesn't do what it was 
> > supposed to? please tell more about what you saw and your conclusions.
> >
> 
> The first condition triggered the long line detection too much: it was 
> triggered when the the only changes in the buffer were fontification 
> changes.  The second condition fixes that problem, but still triggers the 
> long line detection too much: it is still triggered when the main (but not 
> _only_) changes in the buffer are fontification changes, IOW, it is 
> triggered even if a single character is inserted and "enough" 
> fontification changes are made.  The condition in the patch is correct: 
> the long line detection mechanism is triggered only when many characters 
> are inserted.

Ah, okay.  But let's take this opportunity to limit the search region as
well, OK?  I think it will be a better, more thorough fix, especially for
very large buffers without long lines.

bug#56682: locked narrowing

[Dmitry Gutov]

On 30/11/2022 12:11, Gregory Heytings wrote:
> Note that, apart from Dmitry, nobody reported a problem with that 
> detection mechanism, and Dmitry's recipe with which he spotted that 
> problem does not reflect actual usage.

FWIW, my recipe was a way to isolate a more pronounced issue that I have 
experienced in practice.

Even when editing with 'emacs -Q' I could notice an occasional stutter, 
but with https://github.com/kangjianbin/highlight-tail and/or 
https://github.com/Malabarba/smart-mode-line installed the problem got 
much more noticeable (apparently because both cause additional 
redisplays or whatever).

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Wed, 30 Nov 2022 14:40:36 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> > I don't think I understand why it should be problematic or unsafe to 
> > limit the same loop we have to a smaller region of the buffer.
> >
> 
> I see that you already changed the loop to scan from BEG to Z to only scan 
> from BEGV to ZV; somehow I missed that change.  I don't think even that is 
> correct: a command could insert text in the buffer outside of BEGV/ZV. 

It could (although that would be an unusual thing to do), but that's
immaterial: the display engine will never look outside of the restriction,
and so searching beyond that is not useful.  Moreover, safe code in Emacs
cannot look beyond BEGV..ZV without calling 'widen', because various
primitives and subroutines don't expect that and will barf sooner or later;
that's why I made that change in the loop.

> While limiting the loop to a smaller region of the buffer looks like a 
> reasonable thing to do, the question is: how can we determine that region? 
> And I don't see how this can be done, because the detection code is called 
> only when a buffer is about to be displayed, and at that moment we cannot 
> know where characters were inserted in the buffer.  What the code does now 
> is "if enough characters have been inserted in the buffer since last 
> redisplay, check again whether the buffer contains long lines".

If we are going to narrow the buffer to PT ± N characters, then searching
inside slightly more than this region (say, twice more or 5 times more)
should be enough, I think.  It might fail in some borderline cases, but most
files with very long lines have almost all of their lines long, so I see no
problem here.

OTOH, "punishing" large buffers that have no long lines at all doesn't
strike me as a good balance.  Your proposed patch will solve a particular
use case, but it cannot solve all of them.  When modiff becomes large
enough, we will search again, and the threshold of 8 is not so large to
prevent it from happening frequently enough to be an annoyance.  That
annoyance will be for no good reason in buffers that have no long lines.

If you are still unhappy with such a simple heuristics, we could go with
something slightly more complex, like change the threshold of modiff
dynamically based on the buffer size and perhaps the number of times we
already searched without finding long lines -- the larger these two are, the
higher we could bump the threshold.  But I wonder whether this is justified.
I'd say let's find a fixed value that works in all the examples we know
about (we have about a dozen, I think), and leave it at that until someone
reports a real-life problem with it.

bug#56682: locked narrowing

[Juri Linkov]

To be able to fix bug#56815 I started to test the locked narrowing,
and everything works smoothly with:

  (setq treesit-max-buffer-size 1200000000)
  (push '(js-json-mode . json-ts-mode) major-mode-remap-alist)
  (add-hook 'find-file-hook 'toggle-truncate-lines)

Navigating 1 GB hugedictionary.json exhibits no delays
(the recipe uses treesit to get the correct highlighting).

However, 2 problems encountered:

1. after 'M-g TAB' (move-to-column) to 214748364 or more,
   the display is not updated anymore: moving point to the left
   from this position shows the cursor, moving point to the right
   has no visible effect.  Is it a hard limit in the display engine?
   Its hex value is #xccccccc.

2. after starting Isearch at a large column number,
   Emacs hangs up indefinitely, e.g. with
   'M-g TAB 10000000 RET C-s' then even C-g doesn't get out.
   Debugging shows that the problem is in 'isearch-update'
   where the call to 'pos-visible-in-window-group-p' doesn't return.
   When this call is removed, the search is instantaneous.
   (Optimizing lazy-highlight is a separate problem in bug#56815.)

PS: it seems these problems are not related to the locked narrowing,
rather the locked narrowing helped to expose them, so maybe they
should be reported in a new separate bug report?

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  56682@debbugs.gnu.org,  Eli Zaretskii
>  <eliz@gnu.org>,  Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 30 Nov 2022 18:34:15 +0200
> 
> 1. after 'M-g TAB' (move-to-column) to 214748364 or more,
>    the display is not updated anymore: moving point to the left
>    from this position shows the cursor, moving point to the right
>    has no visible effect.  Is it a hard limit in the display engine?
>    Its hex value is #xccccccc.

Sounds like a possible bug.  Does point move?  What does "C-x =" say about
point if you move beyond column 214748364?

If you window is auto-hscrolled as result, then there is indeed hard limit:
the X coordinate of a screen line is an 'int', so MAX_INT divided by the
pixel-width of your default font is as far as we can go.

> 2. after starting Isearch at a large column number,
>    Emacs hangs up indefinitely, e.g. with
>    'M-g TAB 10000000 RET C-s' then even C-g doesn't get out.
>    Debugging shows that the problem is in 'isearch-update'
>    where the call to 'pos-visible-in-window-group-p' doesn't return.
>    When this call is removed, the search is instantaneous.
>    (Optimizing lazy-highlight is a separate problem in bug#56815.)

I thought we agreed that calling pos-visible-in-window-p is not a good idea
in this situation, since it will always think any position is visible?

> PS: it seems these problems are not related to the locked narrowing,
> rather the locked narrowing helped to expose them, so maybe they
> should be reported in a new separate bug report?

It is unrelated, because handling lines that are both very long and
truncated on display uses a separate set of display shortcuts, and locked
narrowing has almost no effect on that.

bug#56682: locked narrowing

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 3673 bytes --]


>
> It could (although that would be an unusual thing to do), but that's 
> immaterial: the display engine will never look outside of the 
> restriction, and so searching beyond that is not useful.  Moreover, safe 
> code in Emacs cannot look beyond BEGV..ZV without calling 'widen', 
> because various primitives and subroutines don't expect that and will 
> barf sooner or later; that's why I made that change in the loop.
>

I had to refresh my memory about this, and I think there is a 
misunderstanding here: we check whether a buffer contains long lines only 
when it is about to be redisplayed and its contents have changed enough 
since the last time it was displayed.  What happened between these two 
redisplays can be anything, e.g. it could be a buffer that has not been on 
display during an hour and that has been filled by a process in the 
meantime.  Using BEGV/ZV is wrong, again because we don't know what 
happened between these two redisplays, it could have been filled by a 
process and narrowed, and if after switching to that buffer the user 
widens it, Emacs would become unresponsive.  Or it could be a buffer that 
has been emptied in the background by a command, filled with the contents 
of a file, and narrowed to some portion of that file (e.g. a single hunk 
of a diff file); again if the user widens it to look at the whole file, 
Emacs will hang.  The only safe way to be certain that a buffer does not 
contain long lines is to check the whole buffer.

>
> If we are going to narrow the buffer to PT ± N characters, then 
> searching inside slightly more than this region (say, twice more or 5 
> times more) should be enough, I think.  It might fail in some borderline 
> cases, but most files with very long lines have almost all of their 
> lines long, so I see no problem here.
>

The intention when this detection mechanism was designed was explicitly to 
also support non-file visiting buffers, e.g. a shell buffer in which a 
user cat's a file with long lines.

>
> OTOH, "punishing" large buffers that have no long lines at all doesn't 
> strike me as a good balance.  Your proposed patch will solve a 
> particular use case, but it cannot solve all of them.
>

Indeed, by definition no heuristic can solve all problems.

>
> When modiff becomes large enough, we will search again, and the 
> threshold of 8 is not so large to prevent it from happening frequently 
> enough to be an annoyance.
>

Please, let's not throw away something that was designed with care and 
about which literally no one complained since it was introduced, for the 
sole reason that an artificial benchmark shows a slowdown of ~80 
milliseconds in a 30 MB buffer.  And note that it's not "modiff becomes 
large enough", it is "modiff changed enough between two redisplays".

>
> That annoyance will be for no good reason in buffers that have no long 
> lines.
>

It's not "for no good reason", it's for an excellent reason: it's the 
price to pay to ensure that Emacs remains responsive in as many cases as 
possible.  And it's not an annoyance, for all basic editing commands the 
detection loop will not run.

>
> If you are still unhappy with such a simple heuristics, we could go with 
> something slightly more complex, like change the threshold of modiff 
> dynamically based on the buffer size and perhaps the number of times we 
> already searched without finding long lines -- the larger these two are, 
> the higher we could bump the threshold.
>

I don't see how such a heuristic could catch a case such as a user cat'ing 
a file with long lines in a shell buffer.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Wed, 30 Nov 2022 22:09:59 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> I had to refresh my memory about this, and I think there is a 
> misunderstanding here: we check whether a buffer contains long lines only 
> when it is about to be redisplayed and its contents have changed enough 
> since the last time it was displayed.  What happened between these two 
> redisplays can be anything, e.g. it could be a buffer that has not been on 
> display during an hour and that has been filled by a process in the 
> meantime.  Using BEGV/ZV is wrong, again because we don't know what 
> happened between these two redisplays, it could have been filled by a 
> process and narrowed, and if after switching to that buffer the user 
> widens it, Emacs would become unresponsive.  Or it could be a buffer that 
> has been emptied in the background by a command, filled with the contents 
> of a file, and narrowed to some portion of that file (e.g. a single hunk 
> of a diff file); again if the user widens it to look at the whole file, 
> Emacs will hang.  The only safe way to be certain that a buffer does not 
> contain long lines is to check the whole buffer.

I still don't understand why we need to look beyond the current narrowing:
we will never display anything outside of that.  Why should we care about
what happens in parts of the buffer that we don't display, and thus cannot
affect redisplay performance?

And again, there are technical problems with doing that without a call to
'widen' -- this is why I made that change: it caused trouble in some
situation that I no longer recall.  If you want to argue for looking at the
entire buffer, you will have to insert a call to 'widen' there, otherwise
the code which does that will not be safe.

And in any case, the BEG vs BEGV issue is secondary here.  My main worry is
valid for buffers without any narrowing at all -- in fact, especially for
those.  So let's leave the BEGV issue alone, or discuss it separately.  here
it is just a tangent.

> > If we are going to narrow the buffer to PT ± N characters, then 
> > searching inside slightly more than this region (say, twice more or 5 
> > times more) should be enough, I think.  It might fail in some borderline 
> > cases, but most files with very long lines have almost all of their 
> > lines long, so I see no problem here.
> 
> The intention when this detection mechanism was designed was explicitly to 
> also support non-file visiting buffers, e.g. a shell buffer in which a 
> user cat's a file with long lines.

This takes a minor and insignificant aspect of the idea I presented and
ignores the main part.  Please respond to the main ideas, not to inaccurate
wording.

And most buffers with very long lines in real life do come from files, not
from shell commands.  So even here you take a relatively rare use case and
make it more significant than it is.

> > OTOH, "punishing" large buffers that have no long lines at all doesn't 
> > strike me as a good balance.  Your proposed patch will solve a 
> > particular use case, but it cannot solve all of them.
> 
> Indeed, by definition no heuristic can solve all problems.

I'm proposing what should be a better heuristic.

> > When modiff becomes large enough, we will search again, and the 
> > threshold of 8 is not so large to prevent it from happening frequently 
> > enough to be an annoyance.
> 
> Please, let's not throw away something that was designed with care and 
> about which literally no one complained since it was introduced, for the 
> sole reason that an artificial benchmark shows a slowdown of ~80 
> milliseconds in a 30 MB buffer.

That benchmark is an example of many use cases that can happen in real life,
in a large buffer with no long lines and a lot of editing activity.  I'm
surprised you don't see it.

> And note that it's not "modiff becomes large enough", it is "modiff
> changed enough between two redisplays".

Once again, a useless argument about a problem with inaccurate wording.  Do
you really think I don't understand how modiff works in this case?  Why
waste energy and bandwidth on such tangents?  Let's focus on the main issue
at hand.

The main issue at hand is how to avoid needless scanning of the entire
buffer for long lines, something which doesn't scale well with buffer size.
For very large buffers without any long lines, this is a regression in Emacs
29, because Emacs 28 didn't do that.  IOW, we make "good" buffers suffer
because of the (rare) danger presented by potentially "bad" buffers.  We
need to look for ways of making this degradation as small and as rare as
possible.

> > That annoyance will be for no good reason in buffers that have no long 
> > lines.
> 
> It's not "for no good reason", it's for an excellent reason: it's the 
> price to pay to ensure that Emacs remains responsive in as many cases as 
> possible.

It will be very hard to explain this "price" to users who never have to deal
with very long lines.  We must look for every way of making the price lower.

> And it's not an annoyance, for all basic editing commands the detection
> loop will not run.

It will run whenever redisplay discovers that a window needs an update on
display, and there was enough changes in buffer text since last redisplay.
It should be clear that with low enough threshold for text changes, this
could happen quite frequently.

> > If you are still unhappy with such a simple heuristics, we could go with 
> > something slightly more complex, like change the threshold of modiff 
> > dynamically based on the buffer size and perhaps the number of times we 
> > already searched without finding long lines -- the larger these two are, 
> > the higher we could bump the threshold.
> 
> I don't see how such a heuristic could catch a case such as a user cat'ing 
> a file with long lines in a shell buffer.

The modiff value will change by a large delta, because it basically counts
the number of characters inserted since last redisplay, right?  So the
difference

   CHARS_MODIFF - UNCHANGED_MODIFIED

will change by a large value, and will exceed even a higher threshold.
AFAIU, this should solve the case you present here.

bug#56682: locked narrowing

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 4761 bytes --]


>
> I still don't understand why we need to look beyond the current 
> narrowing: we will never display anything outside of that.  Why should 
> we care about what happens in parts of the buffer that we don't display, 
> and thus cannot affect redisplay performance?
>

My plain English explanation is apparently not clear enough, so here is a 
recipe:

1. M-: (let ((large-file-warning-threshold nil)) (find-file "dictionary.json") (narrow-to-region 4000000 4004000)) RET
2. C-x n w
3. Kaboom!

>>> If we are going to narrow the buffer to PT ± N characters, then 
>>> searching inside slightly more than this region (say, twice more or 5 
>>> times more) should be enough, I think.  It might fail in some 
>>> borderline cases, but most files with very long lines have almost all 
>>> of their lines long, so I see no problem here.
>>
>> The intention when this detection mechanism was designed was explicitly 
>> to also support non-file visiting buffers, e.g. a shell buffer in which 
>> a user cat's a file with long lines.
>
> This takes a minor and insignificant aspect of the idea I presented and 
> ignores the main part.  Please respond to the main ideas, not to 
> inaccurate wording.
>

I'm sorry to hear you read it that way.  It was not my intention, and it's 
not insignificant: the point I wanted to make is that it's important to 
catch cases in which long lines are added into buffers that do not yet 
contain long lines.  So it's important to keep in mind that we're not only 
discussing _files_ with long lines, which is a case that could have been 
easily caught by adding a check at the moment the file is read in.

>
> And most buffers with very long lines in real life do come from files, 
> not from shell commands.  So even here you take a relatively rare use 
> case and make it more significant than it is.
>

That was a case that was explicitly discussed three months ago.  We cannot 
suddenly decide that it's an insignificant one, or a not significant 
enough one.  And I do agree that it's less common in real life to have 
long lines in comint buffers than to open files with long lines.  Another 
example is an interpreter outputting a (large enough) bignum.

>>> OTOH, "punishing" large buffers that have no long lines at all doesn't 
>>> strike me as a good balance.  Your proposed patch will solve a 
>>> particular use case, but it cannot solve all of them.
>>
>> Indeed, by definition no heuristic can solve all problems.
>
> I'm proposing what should be a better heuristic.
>

As far as I can tell, there are two possible ways to catch such 
situations:

1. The first one is to _not_ use a heuristic, and to check in _each_ 
redisplay cycle whether a certain portion of the buffer around point 
contains long lines.  This adds a fixed cost to all calls to redisplay, 
which four months ago did not seem TRT to do.

2. The second one is to use a heuristic, which is what we do now, and to 
check from time to time, when the buffer contents have changed "enough", 
whether the buffer now contains long lines.  When the buffer is edited 
with small changes between each redisplay cycle, which is what happens in 
the vast majority of cases, using that heuristic adds _no_ cost whatsoever 
to calls to redisplay.  The cost is added to only a few calls to 
redisplay, and it is not fixed anymore, it is more important for a large 
buffer than for a small one.  Doing that seemed TRT four months ago.

>
> That benchmark is an example of many use cases that can happen in real 
> life, in a large buffer with no long lines and a lot of editing 
> activity.  I'm surprised you don't see it.
>

I don't see it, indeed.  And I'm surprised that nobody complained about it 
during the last four months.  Do you have a recipe with which I could see 
that effect?  I just tried to edit dictionary-pp.json "heavily" (large 
kills and yanks), and Emacs remained responsive, I did not experience any 
hiccups.  Neither with emacs -Q nor with my config.  (Of course, that was 
with the patch applied.)

>
> The main issue at hand is how to avoid needless scanning of the entire 
> buffer for long lines, something which doesn't scale well with buffer 
> size. For very large buffers without any long lines, this is a 
> regression in Emacs 29, because Emacs 28 didn't do that.  IOW, we make 
> "good" buffers suffer because of the (rare) danger presented by 
> potentially "bad" buffers.  We need to look for ways of making this 
> degradation as small and as rare as possible.
>

Yes, it's a compromise.  As I explained above, the only other possible 
compromise that is safe enough is to make _all_ buffers "suffer" (but 
suffer less).

bug#56682: locked narrowing

[Gregory Heytings]

>
> And I do agree that it's less common in real life to have long lines in 
> comint buffers than to open files with long lines.
>

I meant: I do not agree.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Thu, 01 Dec 2022 20:49:18 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> 1. M-: (let ((large-file-warning-threshold nil)) (find-file "dictionary.json") (narrow-to-region 4000000 4004000)) RET
> 2. C-x n w
> 3. Kaboom!

By "Kaboom!" you mean what? a crash?  Because it doesn't crash here.  This
is a build from the latest emacs-29 branch.

> As far as I can tell, there are two possible ways to catch such 
> situations:
> 
> 1. The first one is to _not_ use a heuristic, and to check in _each_ 
> redisplay cycle whether a certain portion of the buffer around point 
> contains long lines.  This adds a fixed cost to all calls to redisplay, 
> which four months ago did not seem TRT to do.
> 
> 2. The second one is to use a heuristic, which is what we do now, and to 
> check from time to time, when the buffer contents have changed "enough", 
> whether the buffer now contains long lines.  When the buffer is edited 
> with small changes between each redisplay cycle, which is what happens in 
> the vast majority of cases, using that heuristic adds _no_ cost whatsoever 
> to calls to redisplay.  The cost is added to only a few calls to 
> redisplay, and it is not fixed anymore, it is more important for a large 
> buffer than for a small one.  Doing that seemed TRT four months ago.

This is all beyond argument.  We do want the heuristic.  I just want it to
be cheaper than it is now, especially for buffers without any long lines,
where each time we run this loop we waste CPU cycles.  So I'm looking for
ways of wasting less of them.

> > That benchmark is an example of many use cases that can happen in real 
> > life, in a large buffer with no long lines and a lot of editing 
> > activity.  I'm surprised you don't see it.
> 
> I don't see it, indeed.  And I'm surprised that nobody complained about it 
> during the last four months.

Very large buffers are relatively rare, so I'm not surprised we didn't hear
about this until now.  But it is very easy to come up with a situation where
this happens, so we don't need to wait for complaints to know that they can
exist.

> Do you have a recipe with which I could see that effect?  I just tried to
> edit dictionary-pp.json "heavily" (large kills and yanks), and Emacs
> remained responsive, I did not experience any hiccups.

Edit large enough buffer, and the time it takes to run the loop which scans
the entire buffer will eventually cross the threshold of human perception.
dictionary-pp.json is just 28MB, which is not large enough.  Make a 500MB
file or even 2GB file, and I think you will see the effect.  Better yet,
time the loop for a large enough buffer, and we will be able to estimate the
time it takes for arbitrary buffer sizes with "reasonable" line length,
because that loop scales approximately linearly with the buffer size and the
number of newlines.

> > The main issue at hand is how to avoid needless scanning of the entire 
> > buffer for long lines, something which doesn't scale well with buffer 
> > size. For very large buffers without any long lines, this is a 
> > regression in Emacs 29, because Emacs 28 didn't do that.  IOW, we make 
> > "good" buffers suffer because of the (rare) danger presented by 
> > potentially "bad" buffers.  We need to look for ways of making this 
> > degradation as small and as rare as possible.
> 
> Yes, it's a compromise.  As I explained above, the only other possible 
> compromise that is safe enough is to make _all_ buffers "suffer" (but 
> suffer less).

I'm trying to find a better compromise, in the hope that one exists.  No
more, no less.

bug#56682: locked narrowing

[Gregory Heytings]

>
> Yes, it's a compromise.  As I explained above, the only other possible 
> compromise that is safe enough is to make _all_ buffers "suffer" (but 
> suffer less).
>

That's perhaps not clear enough, it would be better phrased as follows: 
"the only other possible compromise that is safe enough is to make _all_ 
buffers "suffer" (suffer more often, but suffer less than large buffers 
suffer with the alternative)".

bug#56682: locked narrowing

[Gregory Heytings]

>> 1. M-: (let ((large-file-warning-threshold nil)) (find-file "dictionary.json") (narrow-to-region 4000000 4004000)) RET
>> 2. C-x n w
>> 3. Kaboom!
>
> By "Kaboom!" you mean what? a crash?  Because it doesn't crash here. 
> This is a build from the latest emacs-29 branch.
>

I mean that Emacs is trapped in the long line problem again, don't you see 
it?

>
> This is all beyond argument.  We do want the heuristic.  I just want it 
> to be cheaper than it is now, especially for buffers without any long 
> lines, where each time we run this loop we waste CPU cycles.  So I'm 
> looking for ways of wasting less of them.
>

There are none, or at least none that I can see.  Your proposed heuristic 
to limit the scan to some portion of the buffer around point cannot work, 
precisely because the loop is not called during each redisplay cycle, but 
only from time to time.  See the example above.  Again, we cannot know 
what happened in the buffer between these two moments, which could be far 
away in time. We can either use a heuristic on the whole buffer, or not 
use a heuristic and scan only a portion of the buffer.  We cannot combine 
both, and use a heuristic on a portion of the buffer.  It would simply be 
too unsafe.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Thu, 01 Dec 2022 21:36:03 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> 
> >> 1. M-: (let ((large-file-warning-threshold nil)) (find-file "dictionary.json") (narrow-to-region 4000000 4004000)) RET
> >> 2. C-x n w
> >> 3. Kaboom!
> >
> > By "Kaboom!" you mean what? a crash?  Because it doesn't crash here. 
> > This is a build from the latest emacs-29 branch.
> >
> 
> I mean that Emacs is trapped in the long line problem again, don't you see 
> it?

No, not in particular.  Maybe you use a different file?

But anyway, what exactly does this prove, and how?  I asked why we need to
look beyond the narrowing, so how does the above answer that question? what
am I missing?

> > This is all beyond argument.  We do want the heuristic.  I just want it 
> > to be cheaper than it is now, especially for buffers without any long 
> > lines, where each time we run this loop we waste CPU cycles.  So I'm 
> > looking for ways of wasting less of them.
> 
> There are none, or at least none that I can see.  Your proposed heuristic 
> to limit the scan to some portion of the buffer around point cannot work, 
> precisely because the loop is not called during each redisplay cycle, but 
> only from time to time.  See the example above.  Again, we cannot know 
> what happened in the buffer between these two moments, which could be far 
> away in time. We can either use a heuristic on the whole buffer, or not 
> use a heuristic and scan only a portion of the buffer.  We cannot combine 
> both, and use a heuristic on a portion of the buffer.  It would simply be 
> too unsafe.

I give up.  I will probably try to find time to code this myself, since you
don't even want to try.

bug#56682: locked narrowing

[dick]

EZ> I give up.  I will probably try to find time to code this myself,
EZ> since you don't even want to try.

That's unfair.  All narrowing guy does is try.

bug#56682: locked narrowing

[Gregory Heytings]

>> I mean that Emacs is trapped in the long line problem again, don't you 
>> see it?
>
> No, not in particular.  Maybe you use a different file?
>

???  Did you try to edit the file after widening, or even just to move the 
cursor?  I'm using the same dictionary.json file we've been using.

>
> But anyway, what exactly does this prove, and how?  I asked why we need 
> to look beyond the narrowing, so how does the above answer that 
> question? what am I missing?
>

It proves that the long line detection loop must be executed on the whole 
buffer.  In this case there are no modifications to the buffer when it is 
widened, so the detection loop is not triggered, and because there was no 
long line in BEGV/ZV before widening Emacs did not activate the long line 
optimizations.

>
> I give up.  I will probably try to find time to code this myself, since 
> you don't even want to try.
>

Of course I'm willing to try.  But I already tried, I already explored the 
possibility you now consider, and concluded that it was not possible to do 
that.  I'm just trying to explain why and how I reached that conclusion. 
If you have other ideas, I'm all ears.

bug#56682: locked narrowing

[Gregory Heytings]

>> I give up.  I will probably try to find time to code this myself, since 
>> you don't even want to try.
>
> Of course I'm willing to try.  But I already tried, I already explored 
> the possibility you now consider, and concluded that it was not possible 
> to do that.  I'm just trying to explain why and how I reached that 
> conclusion. If you have other ideas, I'm all ears.
>

What we could perhaps do is to use something like

CHARS_MODIFF - UNCHANGED_MODIFIED > 8 + log10 (buffer-size) / 2

That would mean that the detection loop is triggered less for larger 
buffers than for smaller ones, or IOW that bigger changes are needed to 
trigger the loop in larger buffers.  It's a bit unsafe, but probably safe 
enough in practice.

Would that fit the bill?

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Maybe another option is to do the scan more lazily, keeping track more
precisely of what was scanned and/or needs rescanning.

Do an initial scan on the whole buffer and after that keep a BEG..END
limit approximating the region that has been modified since the last scan.

As long as

   END - BEG < LINE_LENGTH_THRESHOLD - MAX_SEEN_LINE_LENGTH

then we know we don't need to rescan anything at all.  And when we do
need to rescan we can limit the scan to

   BEG - MAX_SEEN_LINE_LENGTH ... END + MAX_SEEN_LINE_LENGTH

I assume here that we would remember the MAX_SEEN_LINE_LENGTH.


        Stefan

bug#56682: locked narrowing

[Gregory Heytings]

>
> Maybe another option is to do the scan more lazily, keeping track more 
> precisely of what was scanned and/or needs rescanning.
>

Thanks.  It's an option indeed, the problem here is how to keep track of 
what was scanned or needs rescanning.  I think doing that would probably 
need two markers for each buffer, say beg_scan_marker and end_scan_marker. 
An initial scan would set beg_scan_marker to Z and end_scan_marker to BEG. 
Then each operation in insdel.c would move beg_scan_marker downwards (if 
the buffer position at which the operation takes place is before the 
current value of beg_scan_marker) and end_scan_marker upwards (if the 
buffer position at which the operation takes place is after the current 
value of beg_scan_marker).  Then we would only need to scan 
beg_scan_marker - some_constant .. end_scan_marker + some_constant again.

That being said, I'm not sure the added complexity, to avoid small delays 
when performing large editing operations in huge buffers, is worth the 
price.

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Maybe another option is to do the scan more lazily, keeping track more
>> precisely of what was scanned and/or needs rescanning.
> Thanks.  It's an option indeed, the problem here is how to keep track of
> what was scanned or needs rescanning.

My suggestion was to have as "steady state" that "everything is scanned
except for a region between BEG...END and no line was found to be larger
than MAX_SEEN_LINE_LENGTH".  So it requires keeping track of a BEG..END
(BEG can be an integer but END would likely be an (insert-before)
marker) plus an integer keeping track of MAX_SEEN_LINE_LENGTH.
Initially BEG is 1 and END is Z.

I can't think of a good way to detect when MAX_SEEN_LINE_LENGTH can be
made smaller, tho, so we might still need to rescan the whole buffer
every once in a blue moon.


        Stefan

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Thu, 01 Dec 2022 22:23:38 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> > But anyway, what exactly does this prove, and how?  I asked why we need 
> > to look beyond the narrowing, so how does the above answer that 
> > question? what am I missing?
> 
> It proves that the long line detection loop must be executed on the whole 
> buffer.

No, it doesn't.  It shows that the conditions to re-examine buffer text are
incomplete and need to be augmented.

> In this case there are no modifications to the buffer when it is 
> widened, so the detection loop is not triggered, and because there was no 
> long line in BEGV/ZV before widening Emacs did not activate the long line 
> optimizations.

You didn't explain the problem clearly enough, so I didn't understand it
originally.  The actual problem is not that the restriction changes since
the last redisplay, the problem is that the restriction changes _and_ the
number of unsaved buffer text modifications is still below the threshold of
8.

Once I understood the scenario, the fix was a simple one-liner, which I
installed yesterday night.

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  56682@debbugs.gnu.org,  dgutov@yandex.ru
> Date: Thu, 01 Dec 2022 18:48:26 -0500
> 
> Do an initial scan on the whole buffer and after that keep a BEG..END
> limit approximating the region that has been modified since the last scan.

We currently don't have any way of telling what was modified since the last
scan.

> As long as
> 
>    END - BEG < LINE_LENGTH_THRESHOLD - MAX_SEEN_LINE_LENGTH
> 
> then we know we don't need to rescan anything at all.  And when we do
> need to rescan we can limit the scan to
> 
>    BEG - MAX_SEEN_LINE_LENGTH ... END + MAX_SEEN_LINE_LENGTH
> 
> I assume here that we would remember the MAX_SEEN_LINE_LENGTH.

I hope we can come up with a less complicated scheme.

We only need to know whether there are long lines in a region around point
that is large enough to make sure redisplay won't look beyond that.
(Gregory disagrees, but I have yet to see a reason why he would be right and
I would be wrong in this matter.)  We know where the window's point was at
the time of the last redisplay, so we can detect when point moves "too far",
and trigger rescanning of the region around the new location of point --
this should be a condition in addition to those that currently guard the
rescanning loop.

Given the above, if the region we actually scan is not the entire BEGV..ZV
one, but has its size limited from above by some fixed value, the scanning
should scale much better for very large buffers, because the scanning loop
will never run more than a fixed upper bound of time.  The limit on the
region can be large enough to be safe, because the scan is very fast.  So
only extremely large buffers will not have all of their accessible portion
scanned, which is consistent with other measures we take to limit
potentially long processing using more or less arbitrary limits, like
syntax-wholeline-max, MAX_PARAGRAPH_SEARCH in bidi.c etc.  The result is
potential inaccuracy and/or degradation, but only in very rare cases.

bug#56682: locked narrowing

[Gregory Heytings]

>> It proves that the long line detection loop must be executed on the 
>> whole buffer.
>
> No, it doesn't.  It shows that the conditions to re-examine buffer text 
> are incomplete and need to be augmented.
>

It does, and it's just an example of the general point I'm (apparently 
unsuccessfully) trying to convey: when redisplay is called, any kind of 
changes may have been made in the buffer at any point since last 
redisplay, and we cannot know where these changes occurred.  Stefan's 
proposed approach looks promising, however.

>> In this case there are no modifications to the buffer when it is 
>> widened, so the detection loop is not triggered, and because there was 
>> no long line in BEGV/ZV before widening Emacs did not activate the long 
>> line optimizations.
>
> You didn't explain the problem clearly enough, so I didn't understand it 
> originally.  The actual problem is not that the restriction changes 
> since the last redisplay, the problem is that the restriction changes 
> _and_ the number of unsaved buffer text modifications is still below the 
> threshold of 8.
>
> Once I understood the scenario, the fix was a simple one-liner, which I 
> installed yesterday night.
>

That fix is wrong, sorry.  Now the detection loop is triggered each time 
the user changes the narrowing, in the vast majority of cases for no good 
reason.

bug#56682: locked narrowing

[Gregory Heytings]

>
> We only need to know whether there are long lines in a region around 
> point that is large enough to make sure redisplay won't look beyond 
> that.
>

Sorry, but no, we don't.

>
> (Gregory disagrees, but I have yet to see a reason why he would be right 
> and I would be wrong in this matter.)
>

Consider this other scenario.  A buffer is buried, and modified in the 
background by some process.  That process adds a long line in that buffer, 
followed by enough non-long lines, with point after these non-long lines. 
Now the user asks for that buffer to be displayed again.  With your 
proposed approach, we would search around point if there are long lines. 
There aren't, so we do not activate long line optimizations.  Now the user 
scrolls the buffer.  The buffer has not changed in any way, so the 
detection is not triggered, and we reach the long line.  Kaboom!

bug#56682: locked narrowing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 56682@debbugs.gnu.org,  Eli Zaretskii <eliz@gnu.org>,  dgutov@yandex.ru
> Date: Thu, 01 Dec 2022 21:52:03 -0500
> 
> My suggestion was to have as "steady state" that "everything is scanned
> except for a region between BEG...END and no line was found to be larger
> than MAX_SEEN_LINE_LENGTH".  So it requires keeping track of a BEG..END
> (BEG can be an integer but END would likely be an (insert-before)
> marker) plus an integer keeping track of MAX_SEEN_LINE_LENGTH.
> Initially BEG is 1 and END is Z.

I don't understand how you keep track of BEG and END, in general.  E.g.,
deletion of a single character can in some cases reset BEG to 1 and END to
Z, right?

> I can't think of a good way to detect when MAX_SEEN_LINE_LENGTH can be
> made smaller, tho, so we might still need to rescan the whole buffer
> every once in a blue moon.

That, too, is a complication.  Once again, there's absolutely no reason to
scan more than a limited region around point where redisplay is likely to
look.  Scanning the entire buffer is a waste, and can potentially degrade
performance in very large buffers without any long lines.

bug#56682: locked narrowing

[Gregory Heytings]

>
> Once again, there's absolutely no reason to scan more than a limited 
> region around point where redisplay is likely to look.
>

That's correct if, and only if, we do that scan each time redisplay is 
called.  If we do that only from time to time, based on some heuristics, 
it is not correct anymore, as I explained repeatedly.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Fri, 02 Dec 2022 07:56:06 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: 56682@debbugs.gnu.org, monnier@iro.umontreal.ca, dgutov@yandex.ru
> 
> >> It proves that the long line detection loop must be executed on the 
> >> whole buffer.
> >
> > No, it doesn't.  It shows that the conditions to re-examine buffer text 
> > are incomplete and need to be augmented.
> 
> It does, and it's just an example of the general point I'm (apparently 
> unsuccessfully) trying to convey: when redisplay is called, any kind of 
> changes may have been made in the buffer at any point since last 
> redisplay, and we cannot know where these changes occurred.

Not _any_ kind of changes, no.  Only a few that we know about and track,
because redisplay needs to know about them.

> Stefan's proposed approach looks promising, however.

I explained why in my other messages why it sounds too complicated.

> > Once I understood the scenario, the fix was a simple one-liner, which I 
> > installed yesterday night.
> 
> That fix is wrong, sorry.  Now the detection loop is triggered each time 
> the user changes the narrowing, in the vast majority of cases for no good 
> reason.

The same will happen when the user types several characters.  Which happens
more frequently than changes of narrowing, in most use patterns.

And once again, you simply _cannot_ access buffer text beyond BEGV..ZV
safely.  It's a ticking time bomb if you do.

Anyway, let's stop here and agree to disagree.  I'm sorry, but I cannot
afford this endless argument; I'm swamped with too many tasks that no one
else except myself seems to be doing, and this argument already took an
inordinate amount of my time and energy.  Please install your
UNCHANGED_MODIFIED patch, and I will get to improving the scalability of the
scanning loop when I have time.  Please don't install any other changes in
this regard except that one patch.

Thanks.

bug#56682: locked narrowing

[Gregory Heytings]

>>> Maybe another option is to do the scan more lazily, keeping track more
>>> precisely of what was scanned and/or needs rescanning.
>>
>> Thanks.  It's an option indeed, the problem here is how to keep track 
>> of what was scanned or needs rescanning.
>
> My suggestion was to have as "steady state" that "everything is scanned 
> except for a region between BEG...END and no line was found to be larger 
> than MAX_SEEN_LINE_LENGTH".  So it requires keeping track of a BEG..END 
> (BEG can be an integer but END would likely be an (insert-before) 
> marker) plus an integer keeping track of MAX_SEEN_LINE_LENGTH. Initially 
> BEG is 1 and END is Z.
>

I'm not sure I understand how that would work, concretely.  After the 
initial scan, we would have beg=BEG / end=Z.  Now the buffer is modified, 
say 1000 characters are added at position 1000 and 10000 characters are 
added at position 10000.  What would be the values of beg and end after 
these two operations?  The two portions of the buffer that did not change 
are BEG .. 1000 and 20000 .. Z, so it seems to me that it's better to keep 
track of the portion of the buffer that did change, in this case 1000 .. 
20000.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Fri, 02 Dec 2022 08:04:06 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, 56682@debbugs.gnu.org, 
>     dgutov@yandex.ru
> 
> Consider this other scenario.  A buffer is buried, and modified in the 
> background by some process.  That process adds a long line in that buffer, 
> followed by enough non-long lines, with point after these non-long lines. 
> Now the user asks for that buffer to be displayed again.  With your 
> proposed approach, we would search around point if there are long lines. 
> There aren't, so we do not activate long line optimizations.  Now the user 
> scrolls the buffer.  The buffer has not changed in any way, so the 
> detection is not triggered, and we reach the long line.  Kaboom!

No "Kaboom!": when the user scrolls the buffer, point moves, so we can
detect that and trigger rescanning.

bug#56682: locked narrowing

[Gregory Heytings]

>> Stefan's proposed approach looks promising, however.
>
> I explained why in my other messages why it sounds too complicated.
>

Yet it's AFAICS the only possible approach that was proposed so far.

>> That fix is wrong, sorry.  Now the detection loop is triggered each 
>> time the user changes the narrowing, in the vast majority of cases for 
>> no good reason.
>
> The same will happen when the user types several characters.  Which 
> happens more frequently than changes of narrowing, in most use patterns.
>

The change is wrong nonetheless.  It's a local fix to circumvent an 
example scenario I gave, which was just that: an example.  Many other 
scenarios remain.

>
> And once again, you simply _cannot_ access buffer text beyond BEGV..ZV 
> safely.  It's a ticking time bomb if you do.
>

I already said I agree with that.  I only said (before Stefan proposed a 
potential solution) that I did not see a better way to handle that 
situation.

>
> I'm sorry, but I cannot afford this endless argument; I'm swamped with 
> too many tasks that no one else except myself seems to be doing, and 
> this argument already took an inordinate amount of my time and energy.
>
> Please install your UNCHANGED_MODIFIED patch, and I will get to 
> improving the scalability of the scanning loop when I have time. Please 
> don't install any other changes in this regard except that one patch.
>

Are you serious???  I mean, I've been spending an inordinate amount of my 
time to fix that long line problem, I think I now have a certain expertise 
in that area, I'm trying to find a way to improve these heuristics, you 
dismiss what I'm telling you, and now you tell me that you have too many 
tasks on your plate and that you nonetheless want to improve these 
heuristics yourself and that I should stop working on this???

bug#56682: locked narrowing

[Gregory Heytings]

>> Consider this other scenario.  A buffer is buried, and modified in the 
>> background by some process.  That process adds a long line in that 
>> buffer, followed by enough non-long lines, with point after these 
>> non-long lines. Now the user asks for that buffer to be displayed 
>> again.  With your proposed approach, we would search around point if 
>> there are long lines. There aren't, so we do not activate long line 
>> optimizations.  Now the user scrolls the buffer.  The buffer has not 
>> changed in any way, so the detection is not triggered, and we reach the 
>> long line.  Kaboom!
>
> No "Kaboom!": when the user scrolls the buffer, point moves, so we can 
> detect that and trigger rescanning.
>

In which case we are back to the alternative solution, which is to trigger 
rescanning on each redisplay cycle.  Which might, after all, be the 
simplest and safest solution.

bug#56682: locked narrowing

[Eli Zaretskii]

> Date: Fri, 02 Dec 2022 09:16:08 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: monnier@iro.umontreal.ca, 56682@debbugs.gnu.org, dgutov@yandex.ru
> 
> In which case we are back to the alternative solution, which is to trigger 
> rescanning on each redisplay cycle.  Which might, after all, be the 
> simplest and safest solution.

If the region we scan is limited by some reasonably large size, that could
indeed be the best solution.

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> My suggestion was to have as "steady state" that "everything is scanned
>> except for a region between BEG...END and no line was found to be larger
>> than MAX_SEEN_LINE_LENGTH".  So it requires keeping track of a BEG..END
>> (BEG can be an integer but END would likely be an (insert-before)
>> marker) plus an integer keeping track of MAX_SEEN_LINE_LENGTH.
>> Initially BEG is 1 and END is Z.
>
> I don't understand how you keep track of BEG and END, in general.

Every buffer modification updates it.

> E.g., deletion of a single character can in some cases reset BEG to
> 1 and END to Z, right?

No, why would you think so?
Any change at position POS would set BEG to (min BEG POS) and END to (max
END POS) (modulo marker-position updates for END).

I now realize that a deletion can increase line length by removing LF
chars, so the condition I proposed to decide when a rescan is needed
would need to be adjusted, but the basic idea still stands.

>> I can't think of a good way to detect when MAX_SEEN_LINE_LENGTH can be
>> made smaller, tho, so we might still need to rescan the whole buffer
>> every once in a blue moon.
> That, too, is a complication.

For the 99% of the buffers the MAX_SEEN_LINE_LENGTH should stay small
"for ever" so it's probably not a big issue.

> Once again, there's absolutely no reason to scan more than a limited
> region around point where redisplay is likely to look.

Indeed, that's another possible approach.

> Scanning the entire buffer is a waste, and can potentially degrade
> performance in very large buffers without any long lines.

The full-buffer scan would only be needed once when inserting the file
into the buffer, so it's not clear it would degrade performance in any
significant way, but if needed we can make it more lazy by keeping track
(in addition to the "beg..end of not yet scanned changes") of the
"beg..end of region already scanned" so we only scan the part of the
buffer actually visited (which will still degenerate to "the whole
buffer" if we display BOB and then EOB, of course, as usual).


        Stefan

bug#56682: locked narrowing

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I'm not sure I understand how that would work, concretely.  After the
> initial scan, we would have beg=BEG / end=Z.

No, after the initial the BEG..END region would be empty (since
BEG..END represents the part of the buffer *not yet* scanned).

> Now the buffer is modified,
> say 1000 characters are added at position 1000 and 10000 characters are
> added at position 10000.  What would be the values of beg and end after
> these two operations?

BEG=1000 and END=20000.

> The two portions of the buffer that did not change are BEG .. 1000 and
> 20000 .. Z, so it seems to me that it's better to keep track of the
> portion of the buffer that did change, in this case 1000 .. 20000.

That's exactly what I'm suggesting.


        Stefan

bug#56682: locked narrowing

[Juri Linkov]

> 2. after starting Isearch at a large column number,
>    Emacs hangs up indefinitely, e.g. with
>    'M-g TAB 10000000 RET C-s' then even C-g doesn't get out.
>    Debugging shows that the problem is in 'isearch-update'
>    where the call to 'pos-visible-in-window-group-p' doesn't return.
>    When this call is removed, the search is instantaneous.

There is another side-effect: after evaluating manually
'M-: (pos-visible-in-window-p) RET' at the position 1000000,
point moves to the position 749873 afterwards.  It seems
this is related to locked narrowing, because point doesn't jump
after customizing long-line-locked-narrowing-region-size to 0.

bug#56682: locked narrowing

[Gregory Heytings]

>
> There is another side-effect: after evaluating manually 'M-: 
> (pos-visible-in-window-p) RET' at the position 1000000, point moves to 
> the position 749873 afterwards.  It seems this is related to locked 
> narrowing, because point doesn't jump after customizing 
> long-line-locked-narrowing-region-size to 0.
>

I cannot reproduce that.  With the following recipe:

emacs -Q
C-x C-f dictionary.json RET
M-g c 1000000 RET
M-: (pos-visible-in-window-p) RET

point does not move.  Is there something missing in that recipe?

bug#56682: locked narrowing

[Juri Linkov]

>> There is another side-effect: after evaluating manually 'M-:
>> (pos-visible-in-window-p) RET' at the position 1000000, point moves to
>> the position 749873 afterwards.  It seems this is related to locked
>> narrowing, because point doesn't jump after customizing
>> long-line-locked-narrowing-region-size to 0.
>
> I cannot reproduce that.  With the following recipe:
>
> emacs -Q
> C-x C-f dictionary.json RET
> M-g c 1000000 RET
> M-: (pos-visible-in-window-p) RET
>
> point does not move.  Is there something missing in that recipe?

The prerequisite is this setting:

  (add-hook 'find-file-hook 'toggle-truncate-lines)