Very long lines in shell-mode

Very long lines in shell-mode

[Romain Francoise]

We recently fixed a similar issue in sgml-mode so we might want to have
a look at this one, it's been around for ever: very very long lines in
shell-mode cause Emacs to slow down considerably, eating CPU constantly.

For example, try:

  emacs -Q -f shell
  perl -e 'print "a" x 2000000; print "\n"'

This is probably due to the font-lock keywords set up by shell mode.
Unfortunately, they probably can't be anchored properly.  (And they're
very useful, so we can't just get rid of them...)

(This is not a regression compared to Emacs 21.)

-- 
Romain Francoise <romain@orebokech.com> | The sea! the sea! the open
it's a miracle -- http://orebokech.com/ | sea! The blue, the fresh, the
                                        | ever free! --Bryan W. Procter

Re: Very long lines in shell-mode

[Stefan Monnier]

> We recently fixed a similar issue in sgml-mode so we might want to have
> a look at this one, it's been around for ever: very very long lines in
> shell-mode cause Emacs to slow down considerably, eating CPU constantly.

This may be due to something completely different: font-lock works on
a line-by-line basis.

>   emacs -Q -f shell
>   perl -e 'print "a" x 2000000; print "\n"'

Most likely the 2MB line will be inserted in the buffer in many little
chunks of maybe 4KB or so: for each chunk font-lock may decide to refontify
the whole line.  Each fontification step takes time proportional to the
length of the line, so you get O(n^2) behavior.

And this is without even considering the regexps involved which may very
likely add a factor of n to that.

Maybe for such cases, font-lock should have a "sanity check" and either not
fontify the line at all, or fontify only a part of it at a time.

In either case it's probably not a good idea to try such a thing at
this stage.


        Stefan

Re: Very long lines in shell-mode

[Richard Stallman]

    Most likely the 2MB line will be inserted in the buffer in many little
    chunks of maybe 4KB or so: for each chunk font-lock may decide to refontify
    the whole line.  Each fontification step takes time proportional to the
    length of the line, so you get O(n^2) behavior.

    And this is without even considering the regexps involved which may very
    likely add a factor of n to that.

    Maybe for such cases, font-lock should have a "sanity check" and either not
    fontify the line at all, or fontify only a part of it at a time.

A possible simple solution: in Shell mode, don't fontify a line that
doesn't end in a newline.  If that solves the problem, it would be
simple enough to install now.

Re: Very long lines in shell-mode

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

> A possible simple solution: in Shell mode, don't fontify a line that
> doesn't end in a newline.  If that solves the problem, it would be
> simple enough to install now.

What about prompt lines ?  They never end in a newline.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Very long lines in shell-mode

[Richard Stallman]

    > A possible simple solution: in Shell mode, don't fontify a line that
    > doesn't end in a newline.  If that solves the problem, it would be
    > simple enough to install now.

    What about prompt lines ?  They never end in a newline.

We would have to make an exception for them, but that may not be hard
either.