>
> I still don't understand why we need to look beyond the current 
> narrowing: we will never display anything outside of that.  Why should 
> we care about what happens in parts of the buffer that we don't display, 
> and thus cannot affect redisplay performance?
>

My plain English explanation is apparently not clear enough, so here is a 
recipe:

1. M-: (let ((large-file-warning-threshold nil)) (find-file "dictionary.json") (narrow-to-region 4000000 4004000)) RET
2. C-x n w
3. Kaboom!

>>> If we are going to narrow the buffer to PT ± N characters, then 
>>> searching inside slightly more than this region (say, twice more or 5 
>>> times more) should be enough, I think.  It might fail in some 
>>> borderline cases, but most files with very long lines have almost all 
>>> of their lines long, so I see no problem here.
>>
>> The intention when this detection mechanism was designed was explicitly 
>> to also support non-file visiting buffers, e.g. a shell buffer in which 
>> a user cat's a file with long lines.
>
> This takes a minor and insignificant aspect of the idea I presented and 
> ignores the main part.  Please respond to the main ideas, not to 
> inaccurate wording.
>

I'm sorry to hear you read it that way.  It was not my intention, and it's 
not insignificant: the point I wanted to make is that it's important to 
catch cases in which long lines are added into buffers that do not yet 
contain long lines.  So it's important to keep in mind that we're not only 
discussing _files_ with long lines, which is a case that could have been 
easily caught by adding a check at the moment the file is read in.

>
> And most buffers with very long lines in real life do come from files, 
> not from shell commands.  So even here you take a relatively rare use 
> case and make it more significant than it is.
>

That was a case that was explicitly discussed three months ago.  We cannot 
suddenly decide that it's an insignificant one, or a not significant 
enough one.  And I do agree that it's less common in real life to have 
long lines in comint buffers than to open files with long lines.  Another 
example is an interpreter outputting a (large enough) bignum.

>>> OTOH, "punishing" large buffers that have no long lines at all doesn't 
>>> strike me as a good balance.  Your proposed patch will solve a 
>>> particular use case, but it cannot solve all of them.
>>
>> Indeed, by definition no heuristic can solve all problems.
>
> I'm proposing what should be a better heuristic.
>

As far as I can tell, there are two possible ways to catch such 
situations:

1. The first one is to _not_ use a heuristic, and to check in _each_ 
redisplay cycle whether a certain portion of the buffer around point 
contains long lines.  This adds a fixed cost to all calls to redisplay, 
which four months ago did not seem TRT to do.

2. The second one is to use a heuristic, which is what we do now, and to 
check from time to time, when the buffer contents have changed "enough", 
whether the buffer now contains long lines.  When the buffer is edited 
with small changes between each redisplay cycle, which is what happens in 
the vast majority of cases, using that heuristic adds _no_ cost whatsoever 
to calls to redisplay.  The cost is added to only a few calls to 
redisplay, and it is not fixed anymore, it is more important for a large 
buffer than for a small one.  Doing that seemed TRT four months ago.

>
> That benchmark is an example of many use cases that can happen in real 
> life, in a large buffer with no long lines and a lot of editing 
> activity.  I'm surprised you don't see it.
>

I don't see it, indeed.  And I'm surprised that nobody complained about it 
during the last four months.  Do you have a recipe with which I could see 
that effect?  I just tried to edit dictionary-pp.json "heavily" (large 
kills and yanks), and Emacs remained responsive, I did not experience any 
hiccups.  Neither with emacs -Q nor with my config.  (Of course, that was 
with the patch applied.)

>
> The main issue at hand is how to avoid needless scanning of the entire 
> buffer for long lines, something which doesn't scale well with buffer 
> size. For very large buffers without any long lines, this is a 
> regression in Emacs 29, because Emacs 28 didn't do that.  IOW, we make 
> "good" buffers suffer because of the (rare) danger presented by 
> potentially "bad" buffers.  We need to look for ways of making this 
> degradation as small and as rare as possible.
>

Yes, it's a compromise.  As I explained above, the only other possible 
compromise that is safe enough is to make _all_ buffers "suffer" (but 
suffer less).