Speedup of display of long and truncated lines

Speedup of display of long and truncated lines

[Eli Zaretskii]

I've installed a few changes intended to speed up redisplay of buffers
with long lines when lines are truncated.  This is WIP.  With the
changes I installed, navigation via C-f/C-b should be much faster, and
likewise C-e and C-a (although C-a still has one problem I see that
needs to be fixed).  Vertical cursor motion commands (C-n/C-p) are
still slow and need to be sped up; that's on my TODO.

I didn't yet look at C-v/M-v, mainly because I don't have examples of
files with long lines that have enough long lines to justify movement
by window-full.  (I could, of course, produce such files myself, but
the question is: do such files exist in Real Life, enough to make the
scrolling commands important in these cases?  Examples of such files,
if they exist, are welcome.)

I'm sure these changes break some use cases, or even cause crashes.
Reports of such issues are therefore welcome, as are examples of files
with long lines which exhibit slow redisplay after these changes
(under line-truncation).

Thanks in advance, and happy hacking!

Re: Speedup of display of long and truncated lines

[Stefan Monnier]

> I didn't yet look at C-v/M-v, mainly because I don't have examples of
> files with long lines that have enough long lines to justify movement
> by window-full.  (I could, of course, produce such files myself, but
> the question is: do such files exist in Real Life, enough to make the
> scrolling commands important in these cases?  Examples of such files,
> if they exist, are welcome.)

I think we used to get such things occasionally in backtraces where you
might get a few hundred lines were some of those lines can be
pathologically long.  Nowadays we're more careful to elide parts of
each line of the backtrace, tho (not just for redisplay's sake but also
to speed up the printing itself), so it probably doesn't count as a real
life example any more.


        Stefan

Re: Speedup of display of long and truncated lines

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sat, 13 Aug 2022 10:28:45 -0400
> 
> > I didn't yet look at C-v/M-v, mainly because I don't have examples of
> > files with long lines that have enough long lines to justify movement
> > by window-full.  (I could, of course, produce such files myself, but
> > the question is: do such files exist in Real Life, enough to make the
> > scrolling commands important in these cases?  Examples of such files,
> > if they exist, are welcome.)
> 
> I think we used to get such things occasionally in backtraces where you
> might get a few hundred lines were some of those lines can be
> pathologically long.

Longer than 10,000 characters?  That's the threshold I use for now.

Re: Speedup of display of long and truncated lines

[Gregory Heytings]

>
> I've installed a few changes intended to speed up redisplay of buffers 
> with long lines when lines are truncated.
>

Thanks!

>
> I didn't yet look at C-v/M-v, mainly because I don't have examples of 
> files with long lines that have enough long lines to justify movement by 
> window-full.  (I could, of course, produce such files myself, but the 
> question is: do such files exist in Real Life, enough to make the 
> scrolling commands important in these cases?  Examples of such files, if 
> they exist, are welcome.)
>

Yes, such files exist in Real Life.  An example are database dumps, in 
which you can have hundreds of very long lines.

Re: Speedup of display of long and truncated lines

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 15:34:49 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: emacs-devel@gnu.org
> 
> > I didn't yet look at C-v/M-v, mainly because I don't have examples of 
> > files with long lines that have enough long lines to justify movement by 
> > window-full.  (I could, of course, produce such files myself, but the 
> > question is: do such files exist in Real Life, enough to make the 
> > scrolling commands important in these cases?  Examples of such files, if 
> > they exist, are welcome.)
> 
> Yes, such files exist in Real Life.  An example are database dumps, in 
> which you can have hundreds of very long lines.

Thanks, I'd appreciate any pointers to where I could download examples
of such files.

Re: Speedup of display of long and truncated lines

[Gregory Heytings]

>>> I didn't yet look at C-v/M-v, mainly because I don't have examples of 
>>> files with long lines that have enough long lines to justify movement 
>>> by window-full.  (I could, of course, produce such files myself, but 
>>> the question is: do such files exist in Real Life, enough to make the 
>>> scrolling commands important in these cases?  Examples of such files, 
>>> if they exist, are welcome.)
>>
>> Yes, such files exist in Real Life.  An example are database dumps, in 
>> which you can have hundreds of very long lines.
>
> Thanks, I'd appreciate any pointers to where I could download examples 
> of such files.
>

I cannot share such an example myself alas, the real-life databases I have 
access to contain business-sensitive data.  But you can easily create a 
similar file with the following script.  Its two parameters are the number 
of tables in the database and the number of elements in each table.  100 
and 10000 are a good starting point (for what is still a small database).

#!/bin/bash
for i in $(seq 1 $1)
do
   echo -n "INSERT INTO TABLE_$i VALUES "
   for j in $(seq 1 $2)
   do
     echo -n "(10,'100','QWERTY','Foo Bar Baz',50,4852,$j)"
     (($j == $2)) && echo -n ';' || echo -n ','
   done
   echo
done

Re: Speedup of display of long and truncated lines

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 17:10:44 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> I've installed a few changes intended to speed up redisplay of buffers
> with long lines when lines are truncated.  This is WIP.  With the
> changes I installed, navigation via C-f/C-b should be much faster, and
> likewise C-e and C-a (although C-a still has one problem I see that
> needs to be fixed).  Vertical cursor motion commands (C-n/C-p) are
> still slow and need to be sped up; that's on my TODO.

Vertical cursor motion commands have been sped up now.  If someone
sees cases of unreasonably slow vertical cursor motion, please report
them as bugs (and please tell how much time they took, so we could be
sure that our notions of "unreasonably slow" are similar ;-).

'recenter' was also sped up, which should make M-> faster, especially
when lines are truncated.

C-v/M-v are next.

Re: Speedup of display of long and truncated lines

[Eli Zaretskii]

> Date: Sat, 13 Aug 2022 15:52:56 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: emacs-devel@gnu.org
> 
> 
> >>> I didn't yet look at C-v/M-v, mainly because I don't have examples of 
> >>> files with long lines that have enough long lines to justify movement 
> >>> by window-full.  (I could, of course, produce such files myself, but 
> >>> the question is: do such files exist in Real Life, enough to make the 
> >>> scrolling commands important in these cases?  Examples of such files, 
> >>> if they exist, are welcome.)
> >>
> >> Yes, such files exist in Real Life.  An example are database dumps, in 
> >> which you can have hundreds of very long lines.
> >
> > Thanks, I'd appreciate any pointers to where I could download examples 
> > of such files.
> >
> 
> I cannot share such an example myself alas, the real-life databases I have 
> access to contain business-sensitive data.  But you can easily create a 
> similar file with the following script.  Its two parameters are the number 
> of tables in the database and the number of elements in each table.  100 
> and 10000 are a good starting point (for what is still a small database).
> 
> #!/bin/bash
> for i in $(seq 1 $1)
> do
>    echo -n "INSERT INTO TABLE_$i VALUES "
>    for j in $(seq 1 $2)
>    do
>      echo -n "(10,'100','QWERTY','Foo Bar Baz',50,4852,$j)"
>      (($j == $2)) && echo -n ';' || echo -n ','
>    done
>    echo
> done

Which major mode is used for visiting such files?

Re: Speedup of display of long and truncated lines

[Gregory Heytings]

>> #!/bin/bash
>> for i in $(seq 1 $1)
>> do
>>    echo -n "INSERT INTO TABLE_$i VALUES "
>>    for j in $(seq 1 $2)
>>    do
>>      echo -n "(10,'100','QWERTY','Foo Bar Baz',50,4852,$j)"
>>      (($j == $2)) && echo -n ';' || echo -n ','
>>    done
>>    echo
>> done
>
> Which major mode is used for visiting such files?
>

sql-mode

Re: Speedup of display of long and truncated lines

[Dmitry Gutov]

On 20.08.2022 19:22, Gregory Heytings wrote:
> 
>>> #!/bin/bash
>>> for i in $(seq 1 $1)
>>> do
>>>    echo -n "INSERT INTO TABLE_$i VALUES "
>>>    for j in $(seq 1 $2)
>>>    do
>>>      echo -n "(10,'100','QWERTY','Foo Bar Baz',50,4852,$j)"
>>>      (($j == $2)) && echo -n ';' || echo -n ','
>>>    done
>>>    echo
>>> done
>>
>> Which major mode is used for visiting such files?
>>
> 
> sql-mode

FWIW, my brief testing seems to show sql-mode's font-lock has similar 
performance to js-json-mode. Maybe like 1.5x slower in syntax-ppss (due 
to syntax-propertize-function being present), but that's it.

Tested with font-lock without narrowing, of course.

Re: Speedup of display of long and truncated lines

[Eli Zaretskii]

> Date: Sun, 21 Aug 2022 14:22:15 +0300
> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > sql-mode
> 
> FWIW, my brief testing seems to show sql-mode's font-lock has similar 
> performance to js-json-mode. Maybe like 1.5x slower in syntax-ppss (due 
> to syntax-propertize-function being present), but that's it.
> 
> Tested with font-lock without narrowing, of course.

I think I'm done with this task now.  I installed today two changes in
the scrolling commands and in the display code that make C-v/M-v
significantly faster when lines are very long and truncated on
display.

So, barring crashes and other bugs, I think this issue can be closed.
But if you find situations where Emacs is unbearably sluggish with
long and truncated lines, please report that as bugs, so that they
could be fixed.