bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

To reproduce, visit any file with very long lines, turn on
truncate-lines, and invoke Isearch. after typing some search string
for which there are matches in the buffer, Isearch will highlight many
matches that are not visible in the window.  In an extreme case where
a file has a single very long line, all the matches in the buffer will
be highlighted.  This both takes long time, and slows down many
display-related commands, because the large number of overlays created
for the highlighting slows down redisplay.

This happens because isearch.el uses window-end as the limit beyond
which to stop highlighting, but that is a poor indicator of visibility
in buffers under truncate-lines.

If isearch.el cannot find a better criterion for stopping highlighting
beyond some buffer position, it should at least use some reasonable
fixed (or customizable) limit.  Another possibility is to turn off
lazy-highlight by default when truncate-lines is turned on.

This issue is related to poor Emacs performance in buffers with very
long lines, and its fix is essential for making Emacs more usable with
such buffers.

In GNU Emacs 29.0.50 (build 1562, i686-pc-mingw32)
 of 2022-07-28 built on HOME-C4E4A596F7
Repository revision: 5999dc1cd925ddf8af0b893432124af7904a6918
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY
W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32
ls-lisp disp-table term/w32-win w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer nadvice seq
simple cl-generic indonesian philippine cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process emacs)

Memory information:
((conses 16 42081 12220)
 (symbols 48 6231 0)
 (strings 16 16687 2024)
 (string-bytes 1 454721)
 (vectors 16 9209)
 (vector-slots 8 143790 14221)
 (floats 8 23 45)
 (intervals 40 268 7)
 (buffers 888 10))

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

> To reproduce, visit any file with very long lines, turn on
> truncate-lines, and invoke Isearch. after typing some search string
> for which there are matches in the buffer, Isearch will highlight many
> matches that are not visible in the window.  In an extreme case where
> a file has a single very long line, all the matches in the buffer will
> be highlighted.  This both takes long time, and slows down many
> display-related commands, because the large number of overlays created
> for the highlighting slows down redisplay.
>
> This happens because isearch.el uses window-end as the limit beyond
> which to stop highlighting, but that is a poor indicator of visibility
> in buffers under truncate-lines.
>
> If isearch.el cannot find a better criterion for stopping highlighting
> beyond some buffer position, it should at least use some reasonable
> fixed (or customizable) limit.  Another possibility is to turn off
> lazy-highlight by default when truncate-lines is turned on.
>
> This issue is related to poor Emacs performance in buffers with very
> long lines, and its fix is essential for making Emacs more usable with
> such buffers.

I didn't follow the new design of long lines handling.
Does it just narrow each long truncated line?
When narrowing is in effect then lazy-highlight
should already do the right thing since it can't go
outside of the narrowed regions.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: 56815@debbugs.gnu.org
> Date: Thu, 28 Jul 2022 23:08:23 +0300
> 
> > If isearch.el cannot find a better criterion for stopping highlighting
> > beyond some buffer position, it should at least use some reasonable
> > fixed (or customizable) limit.  Another possibility is to turn off
> > lazy-highlight by default when truncate-lines is turned on.
> >
> > This issue is related to poor Emacs performance in buffers with very
> > long lines, and its fix is essential for making Emacs more usable with
> > such buffers.
> 
> I didn't follow the new design of long lines handling.
> Does it just narrow each long truncated line?

I don't think it matters, because that narrowing is in effect only as
long as redisplay runs.  Isearch's lazy-highlight runs off a timer,
not from redisplay.

I think the main point here is that under truncate-lines, whether a
given buffer position is or isn't visible cannot be easily deduced
from just its position and the window-end position.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Lars Ingebrigtsen]

This reminds me of something I wondered about when futzing about with
inhibiting lazy-highlights w.r.t. text covered by a `display' property
(which was fixed a different way):

Would it be possibly to make the isearch lazy-highlight just skip the
portions of the buffer that aren't visible?

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

> This reminds me of something I wondered about when futzing about with
> inhibiting lazy-highlights w.r.t. text covered by a `display' property
> (which was fixed a different way):
>
> Would it be possibly to make the isearch lazy-highlight just skip the
> portions of the buffer that aren't visible?

Indeed what is worth trying is to add pos-visible-in-window-group-p
either to isearch-filter-predicate, or directly to isearch-lazy-highlight-search.
Isearch.el already has some uses of pos-visible-in-window-group-p,
but currently only in isearch-update.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Gregory Heytings]

>> Would it be possibly to make the isearch lazy-highlight just skip the 
>> portions of the buffer that aren't visible?
>
> Indeed what is worth trying is to add pos-visible-in-window-group-p 
> either to isearch-filter-predicate, or directly to 
> isearch-lazy-highlight-search.
>

That wouldn't work alas.

The problem is how "visible" is defined here (by pos-visible-in-window-p). 
With truncated lines, all buffer positions between window-start and 
window-end are "visible", or IOW, all buffer positions on the left or 
right of the actually visible line portion are "visible".

Moreover pos-visible-in-window-p is a slow function, so it would be 
terribly slow to call it for each candidate position in a buffer with 
(tens of) thousands of candidate positions.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 18:37:55 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Lars Ingebrigtsen <larsi@gnus.org>, Eli Zaretskii <eliz@gnu.org>, 
>     56815@debbugs.gnu.org
> 
> > Indeed what is worth trying is to add pos-visible-in-window-group-p 
> > either to isearch-filter-predicate, or directly to 
> > isearch-lazy-highlight-search.
> 
> That wouldn't work alas.
> 
> The problem is how "visible" is defined here (by pos-visible-in-window-p). 
> With truncated lines, all buffer positions between window-start and 
> window-end are "visible", or IOW, all buffer positions on the left or 
> right of the actually visible line portion are "visible".

One could use posn-at-point instead.

> Moreover pos-visible-in-window-p is a slow function, so it would be 
> terribly slow to call it for each candidate position in a buffer with 
> (tens of) thousands of candidate positions.

I think posn-at-point is much faster, because it never actually looks
beyond the right edge of the window (it considers every position
beyond the rightmost shown one to be at the edge of the window, either
on the right-fringe, if fringes are shown, or at the last_visible_x
coordinate).

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Gregory Heytings]

>> The problem is how "visible" is defined here (by 
>> pos-visible-in-window-p). With truncated lines, all buffer positions 
>> between window-start and window-end are "visible", or IOW, all buffer 
>> positions on the left or right of the actually visible line portion are 
>> "visible".
>
> One could use posn-at-point instead.
>

One could, but...

>> Moreover pos-visible-in-window-p is a slow function, so it would be 
>> terribly slow to call it for each candidate position in a buffer with 
>> (tens of) thousands of candidate positions.
>
> I think posn-at-point is much faster,
>

No, it is slower.  On my laptop, with long-line.xml and truncate-lines:

(benchmark-run 1000 (pos-visible-in-window-p 310000)) takes 4 seconds

(benchmark-run 1000 (numberp (cadr (posn-at-point 310000)))) takes 6 seconds

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 19:11:10 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: juri@linkov.net, larsi@gnus.org, 56815@debbugs.gnu.org
> 
> > I think posn-at-point is much faster,
> >
> 
> No, it is slower.  On my laptop, with long-line.xml and truncate-lines:
> 
> (benchmark-run 1000 (pos-visible-in-window-p 310000)) takes 4 seconds
> 
> (benchmark-run 1000 (numberp (cadr (posn-at-point 310000)))) takes 6 seconds

Why does it matter how long pos-visible-in-window-p takes when it
cannot tell whether the position is visible?

As for posn-at-point, 6 msec per position is not bad, if we in
addition limit the number of matches by some reasonable number, like
100, say.

Moreover, isearch.el could use some special-purpose algorithm for
looking for visible matches in this case.  For example, as soon as
posn-at-point tells us a match is to the right of the window's right
edge, isearch.el could go to the beginning of the next line and
continue from there, instead of checking all the subsequent matches on
the same line (with a result that is known in advance).  This way, we
will probably have even fewer matches, and the search will be faster,
except for very large values of hscroll.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> Cc: larsi@gnus.org, 56815@debbugs.gnu.org, juri@linkov.net
> Date: Fri, 29 Jul 2022 22:31:22 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > (benchmark-run 1000 (pos-visible-in-window-p 310000)) takes 4 seconds
> > 
> > (benchmark-run 1000 (numberp (cadr (posn-at-point 310000)))) takes 6 seconds
> 
> Why does it matter how long pos-visible-in-window-p takes when it
> cannot tell whether the position is visible?

Actually, it can: you just need to call it with the 3rd argument
non-nil.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Gregory Heytings]

>
> Moreover, isearch.el could use some special-purpose algorithm for 
> looking for visible matches in this case.  For example, as soon as 
> posn-at-point tells us a match is to the right of the window's right 
> edge, isearch.el could go to the beginning of the next line and continue 
> from there, instead of checking all the subsequent matches on the same 
> line (with a result that is known in advance).  This way, we will 
> probably have even fewer matches, and the search will be faster, except 
> for very large values of hscroll.
>

And you really think it's TRT to add such a special-purpose algorithm for 
that very specific "long-and-truncated lines" case everywhere in Emacs? 
Because isearch is just one example, there are many other places where you 
will similarly need to reduce the portion of the buffer that will be 
considered by the command to the portion of the buffer that is actually 
visible on screen.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 19:52:08 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: juri@linkov.net, larsi@gnus.org, 56815@debbugs.gnu.org
> 
> > Moreover, isearch.el could use some special-purpose algorithm for 
> > looking for visible matches in this case.  For example, as soon as 
> > posn-at-point tells us a match is to the right of the window's right 
> > edge, isearch.el could go to the beginning of the next line and continue 
> > from there, instead of checking all the subsequent matches on the same 
> > line (with a result that is known in advance).  This way, we will 
> > probably have even fewer matches, and the search will be faster, except 
> > for very large values of hscroll.
> 
> And you really think it's TRT to add such a special-purpose algorithm for 
> that very specific "long-and-truncated lines" case everywhere in Emacs? 
> Because isearch is just one example, there are many other places where you 
> will similarly need to reduce the portion of the buffer that will be 
> considered by the command to the portion of the buffer that is actually 
> visible on screen.

Which other places?

In this bug we are talking specifically about isearch.el adding
thousands of overlays to a buffer, 99.99% of which are not visible.
If there are other features that do something similar, please name
them.

And yes, if we want truncate-lines to work reasonably well, we need to
fix any features which behave silly in that display mode.  We will not
remove truncate-lines from Emacs, and we will not make it
dysfunctional.  But let's begin by discovering those features, so that
this discussion could be less academical.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Gregory Heytings]

>
> And yes, if we want truncate-lines to work reasonably well, we need to 
> fix any features which behave silly in that display mode.
>

If we want truncate-lines to work reasonably well without fixing all such 
features one-by-one, we need to fix the root cause of that dysfunction, 
which is not inside these individual features, but is the fact that (- 
(window-end) (window-start)) is huge.

>
> We will not remove truncate-lines from Emacs, and we will not make it 
> dysfunctional.
>

Nobody is asking or suggesting to remove truncate-lines from Emacs, or to 
make is dysfunctional.  Admitting that it doesn't work together with long 
lines, which has always been the case, is something entirely different.

But I feel you will tell me that I'm pressuring you, so I bow out of this 
discussion.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> Date: Fri, 29 Jul 2022 20:26:00 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: larsi@gnus.org, 56815@debbugs.gnu.org, juri@linkov.net
> 
> > And yes, if we want truncate-lines to work reasonably well, we need to 
> > fix any features which behave silly in that display mode.
> 
> If we want truncate-lines to work reasonably well without fixing all such 
> features one-by-one, we need to fix the root cause of that dysfunction, 
> which is not inside these individual features, but is the fact that (- 
> (window-end) (window-start)) is huge.

I'm not convinced that this is the root cause, nor that there is
indeed a single cause.  So far we have found only one feature where
this is true, and even that is only because Isearch uses that
particular method of deciding when to produce the highlight overlays.

It is true that the narrowing optimizations work less well under
truncate-lines, for these reasons, but there are other places where we
can make improvements and other methods of doing that.

There's no reason to believe that the general issue of long lines can
be solved by a small number of changes that narrow the buffer.
Whatever is left after that we should try solving by other methods.

> > We will not remove truncate-lines from Emacs, and we will not make it 
> > dysfunctional.
> >
> 
> Nobody is asking or suggesting to remove truncate-lines from Emacs, or to 
> make is dysfunctional.  Admitting that it doesn't work together with long 
> lines, which has always been the case, is something entirely different.

I admit that it's a tougher nut that currently is not yet cracked,
yes.  But if improvements can be made in that mode, we should at least
try making them, even if those improvements call for some
complications in our code.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

>> And yes, if we want truncate-lines to work reasonably well, we need to
>> fix any features which behave silly in that display mode.
>>
>
> If we want truncate-lines to work reasonably well without fixing all such
> features one-by-one, we need to fix the root cause of that dysfunction,
> which is not inside these individual features, but is the fact that (-
> (window-end) (window-start)) is huge.

Do you think introducing the rectangular narrowing could be a more general fix?
We already have rectangular regions generalized as non-contiguous regions.
Could non-contiguous narrowing help to restrict the accessible buffer area
to the visible screen rectangle?

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Gregory Heytings]

>>> And yes, if we want truncate-lines to work reasonably well, we need to 
>>> fix any features which behave silly in that display mode.
>>
>> If we want truncate-lines to work reasonably well without fixing all 
>> such features one-by-one, we need to fix the root cause of that 
>> dysfunction, which is not inside these individual features, but is the 
>> fact that (- (window-end) (window-start)) is huge.
>
> Do you think introducing the rectangular narrowing could be a more 
> general fix? We already have rectangular regions generalized as 
> non-contiguous regions. Could non-contiguous narrowing help to restrict 
> the accessible buffer area to the visible screen rectangle?
>

Yes, I think so.  But I also think it would be wrong to do that.  That 
would be an enormous change to the code (wherever we use BEGV and ZV we 
would have to use instead two arrays BEGV[] and ZV[]) for a microscopic 
use case, and it wouldn't even solve the problem completely, because the 
largest possible line width is limited to about 200 MB anyway (because its 
width in pixels is stored in an int).

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,  larsi@gnus.org,  56815@debbugs.gnu.org
> Date: Sun, 31 Jul 2022 22:40:22 +0300
> 
> >> And yes, if we want truncate-lines to work reasonably well, we need to
> >> fix any features which behave silly in that display mode.
> >>
> >
> > If we want truncate-lines to work reasonably well without fixing all such
> > features one-by-one, we need to fix the root cause of that dysfunction,
> > which is not inside these individual features, but is the fact that (-
> > (window-end) (window-start)) is huge.
> 
> Do you think introducing the rectangular narrowing could be a more general fix?
> We already have rectangular regions generalized as non-contiguous regions.
> Could non-contiguous narrowing help to restrict the accessible buffer area
> to the visible screen rectangle?

I don't understand how such a restriction will work on the level of
buffer text.  E.g., what happens when you insert a single character
before the beginning of one of the narrowed regions? does the region
move with the text or does it stay on the same buffer positions?

Basically, as long as Emacs represents buffer text as a single long C
string (aside of the gap), with no additional structure, I don't see
how implement this with any reasonable complexity.

And I agree with Gregory that the gains in this case will be too low,
relative to the complexity.

Let's keep this in its correct proportions: we "just" want isearch to
behave reasonably well in long-and-truncated lines, because IMO
isearch is one of the few features that we cannot break or disable in
such buffers -- the ability to search is too fundamental to editing,
even if that editing means just viewing the text.

So if no good ideas arise that are simple enough to implement, I'm
okay with simply disabling isearch-lazy-highlight in such buffers, as
I wrote in the original bug report.  (Doing that will probably require
exposing the "long-lines" flag to Lisp, but that's fine by me.)

Thanks.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

> So if no good ideas arise that are simple enough to implement, I'm
> okay with simply disabling isearch-lazy-highlight in such buffers, as
> I wrote in the original bug report.  (Doing that will probably require
> exposing the "long-lines" flag to Lisp, but that's fine by me.)

Exposing the "long-lines" flag to Lisp is exactly what is needed.
So when lazy-highlight will detect a special value of a new
buffer-local variable when narrowing is in effect due to long lines,
then lazy-highlight can switch to a different algorithm
like you described earlier.  Currently isearch-lazy-highlight-new-loop
checks for window-group-start and window-group-end.  But when
the "long-lines" flag is set, it should also check for window-hscroll
because narrowed lazy-highlighting should be updated on hscrolling.

This "long-lines" flag will be also useful for pre-command-hook
and post-command-hook to convey the information about narrowing
to hooks.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Gregory Heytings]

>> So if no good ideas arise that are simple enough to implement, I'm okay 
>> with simply disabling isearch-lazy-highlight in such buffers, as I 
>> wrote in the original bug report.  (Doing that will probably require 
>> exposing the "long-lines" flag to Lisp, but that's fine by me.)
>
> Exposing the "long-lines" flag to Lisp is exactly what is needed.
>

My current plan is to run a hook when long lines are detected, in which 
modes such as isearch will be free to add any function they like.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> Date: Tue, 02 Aug 2022 07:31:16 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, 56815@debbugs.gnu.org, larsi@gnus.org
> 
> 
> >> So if no good ideas arise that are simple enough to implement, I'm okay 
> >> with simply disabling isearch-lazy-highlight in such buffers, as I 
> >> wrote in the original bug report.  (Doing that will probably require 
> >> exposing the "long-lines" flag to Lisp, but that's fine by me.)
> >
> > Exposing the "long-lines" flag to Lisp is exactly what is needed.
> >
> 
> My current plan is to run a hook when long lines are detected, in which 
> modes such as isearch will be free to add any function they like.

I'm not sure I agree that a hook is a better interface.  A hook is
only useful to features that are already turned on when the hook is
called; features that are turned on later will miss it.

By contrast, an accessor function can be called at any time, and for
an attribute that, once set remains set, I think it's a more
convenient way of adapting features and packages to this condition.

Or am I missing something?

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: gregory@heytings.org,  larsi@gnus.org,  56815@debbugs.gnu.org
> Date: Mon, 01 Aug 2022 21:09:05 +0300
> 
> > So if no good ideas arise that are simple enough to implement, I'm
> > okay with simply disabling isearch-lazy-highlight in such buffers, as
> > I wrote in the original bug report.  (Doing that will probably require
> > exposing the "long-lines" flag to Lisp, but that's fine by me.)
> 
> Exposing the "long-lines" flag to Lisp is exactly what is needed.
> So when lazy-highlight will detect a special value of a new
> buffer-local variable when narrowing is in effect due to long lines,
> then lazy-highlight can switch to a different algorithm
> like you described earlier.  Currently isearch-lazy-highlight-new-loop
> checks for window-group-start and window-group-end.  But when
> the "long-lines" flag is set, it should also check for window-hscroll
> because narrowed lazy-highlighting should be updated on hscrolling.
> 
> This "long-lines" flag will be also useful for pre-command-hook
> and post-command-hook to convey the information about narrowing
> to hooks.

You can now use the new long-line-optimizations-p predicate for that.
Although from your description above, perhaps Isearch should behave
like that in a buffer with line-truncation regardless of the long
lines.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

> You can now use the new long-line-optimizations-p predicate for that.
> Although from your description above, perhaps Isearch should behave
> like that in a buffer with line-truncation regardless of the long
> lines.

This means using the same condition like in 'line-move'?

                  (and
                   (or truncate-lines
                       (and (integerp truncate-partial-width-windows)
                            (< (window-total-width)
                               truncate-partial-width-windows))
                       (and truncate-partial-width-windows
                            (not (integerp truncate-partial-width-windows))
                            (not (window-full-width-p))))
                   ;; ...or if lines are truncated, this buffer
                   ;; doesn't have very long lines.
                   (long-line-optimizations-p))

I suggest to leave 'long-line-optimizations-p' at the end because
in such cases when 'lazy-highlight-cleanup' is customized to nil
the users might want to leave highlighting on truncated lines
after exiting Isearch to see matches when scrolling horizontally
(when lines are not long).

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: gregory@heytings.org,  larsi@gnus.org,  56815@debbugs.gnu.org
> Date: Sun, 21 Aug 2022 19:32:14 +0300
> 
> > You can now use the new long-line-optimizations-p predicate for that.
> > Although from your description above, perhaps Isearch should behave
> > like that in a buffer with line-truncation regardless of the long
> > lines.
> 
> This means using the same condition like in 'line-move'?
> 
>                   (and
>                    (or truncate-lines
>                        (and (integerp truncate-partial-width-windows)
>                             (< (window-total-width)
>                                truncate-partial-width-windows))
>                        (and truncate-partial-width-windows
>                             (not (integerp truncate-partial-width-windows))
>                             (not (window-full-width-p))))
>                    ;; ...or if lines are truncated, this buffer
>                    ;; doesn't have very long lines.
>                    (long-line-optimizations-p))

Yes, this is the (unfortunately complicated) way of determining
whether the window has truncated lines.

> I suggest to leave 'long-line-optimizations-p' at the end because

It _is_ at the end, isn't it?

> in such cases when 'lazy-highlight-cleanup' is customized to nil
> the users might want to leave highlighting on truncated lines
> after exiting Isearch to see matches when scrolling horizontally
> (when lines are not long).

Ideally, when the window is hscrolled, the matches that come into view
should be lazily highlighted, and those which become invisible should
be de-highlighted.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

>> This means using the same condition like in 'line-move'?
>>
>>                   (and
>>                    (or truncate-lines
>>                        (and (integerp truncate-partial-width-windows)
>>                             (< (window-total-width)
>>                                truncate-partial-width-windows))
>>                        (and truncate-partial-width-windows
>>                             (not (integerp truncate-partial-width-windows))
>>                             (not (window-full-width-p))))
>>                    ;; ...or if lines are truncated, this buffer
>>                    ;; doesn't have very long lines.
>>                    (long-line-optimizations-p))
>
> Yes, this is the (unfortunately complicated) way of determining
> whether the window has truncated lines.

Would it be possible to refactor this to a separate function?
I guess it could be in simple.el.  But what would be a good name?
Maybe 'truncate-lines-p'?

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: gregory@heytings.org,  larsi@gnus.org,  56815@debbugs.gnu.org
> Date: Sun, 21 Aug 2022 22:03:48 +0300
> 
> >> This means using the same condition like in 'line-move'?
> >>
> >>                   (and
> >>                    (or truncate-lines
> >>                        (and (integerp truncate-partial-width-windows)
> >>                             (< (window-total-width)
> >>                                truncate-partial-width-windows))
> >>                        (and truncate-partial-width-windows
> >>                             (not (integerp truncate-partial-width-windows))
> >>                             (not (window-full-width-p))))
> >>                    ;; ...or if lines are truncated, this buffer
> >>                    ;; doesn't have very long lines.
> >>                    (long-line-optimizations-p))
> >
> > Yes, this is the (unfortunately complicated) way of determining
> > whether the window has truncated lines.
> 
> Would it be possible to refactor this to a separate function?

Sure.

> I guess it could be in simple.el.  But what would be a good name?
> Maybe 'truncate-lines-p'?

lines-truncated-p, I guess.  Or window-lines-truncated-p.  And I think
it should be in window.el.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

>> I guess it could be in simple.el.  But what would be a good name?
>> Maybe 'truncate-lines-p'?
>
> lines-truncated-p, I guess.  Or window-lines-truncated-p.  And I think
> it should be in window.el.

To keep the prefix like in all truncate-related names,
truncated-partial-width-window-p could serve as an example:

  (truncated-partial-width-window-p &optional WINDOW)
  (truncated-lines-p &optional window)

Or can truncated-partial-width-window-p be called directly?  By just

  (or truncate-lines (truncated-partial-width-window-p))

Anyway, here is the patch that probably not needed
when truncated-partial-width-window-p can be used instead:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: truncated-lines-p.patch --]
[-- Type: text/x-diff, Size: 3227 bytes --]

diff --git a/lisp/simple.el b/lisp/simple.el
index d862c0e9a8..8f7440a3bb 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -7700,13 +7700,7 @@ line-move
                  ;; Lines are not truncated...
                  (not
                   (and
-                   (or truncate-lines
-                       (and (integerp truncate-partial-width-windows)
-                            (< (window-total-width)
-                               truncate-partial-width-windows))
-                       (and truncate-partial-width-windows
-                            (not (integerp truncate-partial-width-windows))
-                            (not (window-full-width-p))))
+                   (truncated-lines-p)
                    ;; ...or if lines are truncated, this buffer
                    ;; doesn't have very long lines.
                    (long-line-optimizations-p)))
@@ -7716,14 +7710,7 @@ line-move
 	       ;; Display-based column are incompatible with goal-column.
 	       (not goal-column)
                ;; Lines aren't truncated.
-               (not
-                (or truncate-lines
-                    (and (integerp truncate-partial-width-windows)
-                         (< (window-width)
-                            truncate-partial-width-windows))
-                    (and truncate-partial-width-windows
-                         (not (integerp truncate-partial-width-windows))
-                         (not (window-full-width-p)))))
+               (not (truncated-lines-p))
 	       ;; When the text in the window is scrolled to the left,
 	       ;; display-based motion doesn't make sense (because each
 	       ;; logical line occupies exactly one screen line).
diff --git a/lisp/window.el b/lisp/window.el
index 4d88ffa903..aecac8c8d0 100644
--- a/lisp/window.el
+++ b/lisp/window.el
@@ -9012,6 +9023,18 @@ shrink-window-horizontally
   (interactive "p")
   (shrink-window delta t))
 
+(defun truncated-lines-p (&optional window)
+  "Return non-nil when lines are truncated.
+This depends on the values of `truncate-lines' and
+`truncate-partial-width-windows'."
+  (or truncate-lines
+      (and (integerp truncate-partial-width-windows)
+           (< (window-total-width window)
+              truncate-partial-width-windows))
+      (and truncate-partial-width-windows
+           (not (integerp truncate-partial-width-windows))
+           (not (window-full-width-p window)))))
+
 (defun count-screen-lines (&optional beg end count-final-newline window)
   "Return the number of screen lines in the region.
 The number of screen lines may be different from the number of actual lines,
@@ -9044,10 +9067,7 @@ count-screen-lines
       ;; vertical-motion returns a number that is 1 larger than it
       ;; should.  We need to fix that.
       (setq end-invisible-p
-            (and (or truncate-lines
-                     (and (natnump truncate-partial-width-windows)
-                          (< (window-total-width window)
-                             truncate-partial-width-windows)))
+            (and (truncated-lines-p window)
                  (save-excursion
                    (goto-char finish)
                    (> (- (current-column) (window-hscroll window))

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: gregory@heytings.org,  larsi@gnus.org,  56815@debbugs.gnu.org
> Date: Mon, 22 Aug 2022 09:48:47 +0300
> 
> >> I guess it could be in simple.el.  But what would be a good name?
> >> Maybe 'truncate-lines-p'?
> >
> > lines-truncated-p, I guess.  Or window-lines-truncated-p.  And I think
> > it should be in window.el.
> 
> To keep the prefix like in all truncate-related names,
> truncated-partial-width-window-p could serve as an example:

It has "window" in its name.  Your proposal doesn't, so the name is
too general, IMO.

> Or can truncated-partial-width-window-p be called directly?  By just
> 
>   (or truncate-lines (truncated-partial-width-window-p))

Fine by me.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 184 bytes --]

>> Or can truncated-partial-width-window-p be called directly?  By just
>>
>>   (or truncate-lines (truncated-partial-width-window-p))
>
> Fine by me.

Then I'll push a better patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: truncated-partial-width-window-p.patch --]
[-- Type: text/x-diff, Size: 2526 bytes --]

diff --git a/lisp/simple.el b/lisp/simple.el
index 460aff8bd8..72674d3a1d 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -7700,13 +7700,7 @@ line-move
                  ;; Lines are not truncated...
                  (not
                   (and
-                   (or truncate-lines
-                       (and (integerp truncate-partial-width-windows)
-                            (< (window-total-width)
-                               truncate-partial-width-windows))
-                       (and truncate-partial-width-windows
-                            (not (integerp truncate-partial-width-windows))
-                            (not (window-full-width-p))))
+                   (or truncate-lines (truncated-partial-width-window-p))
                    ;; ...or if lines are truncated, this buffer
                    ;; doesn't have very long lines.
                    (long-line-optimizations-p)))
@@ -7716,14 +7710,7 @@ line-move
 	       ;; Display-based column are incompatible with goal-column.
 	       (not goal-column)
                ;; Lines aren't truncated.
-               (not
-                (or truncate-lines
-                    (and (integerp truncate-partial-width-windows)
-                         (< (window-width)
-                            truncate-partial-width-windows))
-                    (and truncate-partial-width-windows
-                         (not (integerp truncate-partial-width-windows))
-                         (not (window-full-width-p)))))
+               (not (or truncate-lines (truncated-partial-width-window-p)))
 	       ;; When the text in the window is scrolled to the left,
 	       ;; display-based motion doesn't make sense (because each
 	       ;; logical line occupies exactly one screen line).
diff --git a/lisp/window.el b/lisp/window.el
index 4d88ffa903..98c14e7bbc 100644
--- a/lisp/window.el
+++ b/lisp/window.el
@@ -9044,10 +9055,7 @@ count-screen-lines
       ;; vertical-motion returns a number that is 1 larger than it
       ;; should.  We need to fix that.
       (setq end-invisible-p
-            (and (or truncate-lines
-                     (and (natnump truncate-partial-width-windows)
-                          (< (window-total-width window)
-                             truncate-partial-width-windows)))
+            (and (or truncate-lines (truncated-partial-width-window-p window))
                  (save-excursion
                    (goto-char finish)
                    (> (- (current-column) (window-hscroll window))

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: gregory@heytings.org,  larsi@gnus.org,  56815@debbugs.gnu.org
> Date: Tue, 23 Aug 2022 10:41:21 +0300
> 
> >>   (or truncate-lines (truncated-partial-width-window-p))
> >
> > Fine by me.
> 
> Then I'll push a better patch:

Btw, I didn't use window-total-width instead of window-width.  What I
did mimics what the display engine does when it determines whether to
truncate lines in a window.  So I think
truncated-partial-width-window-p should be fixed to use
window-total-width instead.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

>> >>   (or truncate-lines (truncated-partial-width-window-p))
>> >
>> > Fine by me.
>> 
>> Then I'll push a better patch:
>
> Btw, I didn't use window-total-width instead of window-width.  What I
> did mimics what the display engine does when it determines whether to
> truncate lines in a window.  So I think
> truncated-partial-width-window-p should be fixed to use
> window-total-width instead.

Now this patch fixes truncated-partial-width-window-p to use
window-total-width.  Also it fixes the recent code for
long_line_optimizations_p in scan_for_column to better mimic
what the display engine does:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: truncated-partial-width-window-p.patch --]
[-- Type: text/x-diff, Size: 4822 bytes --]

diff --git a/lisp/simple.el b/lisp/simple.el
index 460aff8bd8..6e8cd798e0 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -7700,13 +7700,7 @@ line-move
                  ;; Lines are not truncated...
                  (not
                   (and
-                   (or truncate-lines
-                       (and (integerp truncate-partial-width-windows)
-                            (< (window-total-width)
-                               truncate-partial-width-windows))
-                       (and truncate-partial-width-windows
-                            (not (integerp truncate-partial-width-windows))
-                            (not (window-full-width-p))))
+                   (or truncate-lines (truncated-partial-width-window-p))
                    ;; ...or if lines are truncated, this buffer
                    ;; doesn't have very long lines.
                    (long-line-optimizations-p)))
@@ -7716,14 +7710,7 @@ line-move
 	       ;; Display-based column are incompatible with goal-column.
 	       (not goal-column)
                ;; Lines aren't truncated.
-               (not
-                (or truncate-lines
-                    (and (integerp truncate-partial-width-windows)
-                         (< (window-width)
-                            truncate-partial-width-windows))
-                    (and truncate-partial-width-windows
-                         (not (integerp truncate-partial-width-windows))
-                         (not (window-full-width-p)))))
+               (not (or truncate-lines (truncated-partial-width-window-p)))
 	       ;; When the text in the window is scrolled to the left,
 	       ;; display-based motion doesn't make sense (because each
 	       ;; logical line occupies exactly one screen line).
@@ -7983,7 +7970,7 @@ line-move-finish
 
 	;; Move to the desired column.
         (if (and line-move-visual
-                 (not (or truncate-lines truncate-partial-width-windows)))
+                 (not (or truncate-lines (truncated-partial-width-window-p))))
             ;; Under line-move-visual, goal-column should be
             ;; interpreted in units of the frame's canonical character
             ;; width, which is exactly what vertical-motion does.
diff --git a/lisp/window.el b/lisp/window.el
index 4d88ffa903..a2559a2591 100644
--- a/lisp/window.el
+++ b/lisp/window.el
@@ -9044,10 +9055,7 @@ count-screen-lines
       ;; vertical-motion returns a number that is 1 larger than it
       ;; should.  We need to fix that.
       (setq end-invisible-p
-            (and (or truncate-lines
-                     (and (natnump truncate-partial-width-windows)
-                          (< (window-total-width window)
-                             truncate-partial-width-windows)))
+            (and (or truncate-lines (truncated-partial-width-window-p window))
                  (save-excursion
                    (goto-char finish)
                    (> (- (current-column) (window-hscroll window))
@@ -10140,7 +10148,7 @@ scroll-command--goto-goal-column
   (when goal-column
     ;; Move to the desired column.
     (if (and line-move-visual
-             (not (or truncate-lines truncate-partial-width-windows)))
+             (not (or truncate-lines (truncated-partial-width-window-p))))
         ;; Under line-move-visual, goal-column should be
         ;; interpreted in units of the frame's canonical character
         ;; width, which is exactly what vertical-motion does.
@@ -10449,7 +10457,7 @@ truncated-partial-width-window-p
     (let ((t-p-w-w (buffer-local-value 'truncate-partial-width-windows
 				       (window-buffer window))))
       (if (integerp t-p-w-w)
-	  (< (window-width window) t-p-w-w)
+	  (< (window-total-width window) t-p-w-w)
         t-p-w-w))))
 
 \f
diff --git a/src/indent.c b/src/indent.c
index cb368024d9..aa905f387b 100644
--- a/src/indent.c
+++ b/src/indent.c
@@ -577,12 +577,15 @@ scan_for_column (ptrdiff_t *endpos, EMACS_INT *goalcol,
 
       if (!NILP (BVAR (current_buffer, truncate_lines)))
 	lines_truncated = true;
-      else if (w && FIXNUMP (Vtruncate_partial_width_windows))
-	lines_truncated =
-	  w->total_cols < XFIXNAT (Vtruncate_partial_width_windows);
-      else if (w && !NILP (Vtruncate_partial_width_windows))
-	lines_truncated =
-	  w->total_cols < FRAME_COLS (XFRAME (WINDOW_FRAME (w)));
+      else if (!NILP (Vtruncate_partial_width_windows) && w
+	       && w->total_cols < FRAME_COLS (XFRAME (WINDOW_FRAME (w))))
+	{
+	  if (FIXNUMP (Vtruncate_partial_width_windows))
+	    lines_truncated =
+	      w->total_cols < XFIXNAT (Vtruncate_partial_width_windows);
+	  else
+	    lines_truncated = true;
+	}
       /* Special optimization for buffers with long and truncated
 	 lines: assumes that each character is a single column.  */
       if (lines_truncated)

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

>>> >>   (or truncate-lines (truncated-partial-width-window-p))
>>> >
>>> > Fine by me.
>>> 
>>> Then I'll push a better patch:
>>
>> Btw, I didn't use window-total-width instead of window-width.  What I
>> did mimics what the display engine does when it determines whether to
>> truncate lines in a window.  So I think
>> truncated-partial-width-window-p should be fixed to use
>> window-total-width instead.
>
> Now this patch fixes truncated-partial-width-window-p to use
> window-total-width.  Also it fixes the recent code for
> long_line_optimizations_p in scan_for_column to better mimic
> what the display engine does:

Now pushed to master.

bug#56815: 29.0.50; Isearch lazy-highlight highlights too much when truncate-lines is in effect

[Juri Linkov]

While waiting when buffer-local current_buffer->long_line_optimizations_p
will be exposed to Lisp, I noticed another need for lazy-highlight optimization:
in large buffers (but without long lines) some customization makes replacement
too slow when lazy-highlight tries to rehighlight the whole buffer
after every replacement.  For query-replace it should be enough to
highlight matches only on the displayed screen, therefore this patch
fixes the problem:

```
diff --git a/lisp/replace.el b/lisp/replace.el
index cac0edf43a..9a1f6f22e0 100644
--- a/lisp/replace.el
+++ b/lisp/replace.el
@@ -2744,7 +2750,9 @@ replace-highlight
 	    (isearch-case-fold-search case-fold)
 	    (isearch-forward (not backward))
 	    (isearch-other-end match-beg)
-	    (isearch-error nil))
+	    (isearch-error nil)
+	    (isearch-lazy-count nil)
+	    (lazy-highlight-buffer nil))
 	(isearch-lazy-highlight-new-loop range-beg range-end))))
```