bug#16875: python, comint-mode: Large output makes Emacs freeze

[fgallina@gnu.org (Fabián Ezequiel Gallina)]

Andreas Röhler <andreas.roehler@easy-emacs.de> writes:

>> Indeed, "pprint" should solve it already. Maybe make that the default?
>> IMO looks better anyway.

That's changing the Python shell default behavior and I that's out of
the scope of python.el.

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13675
>> To a certain extend.
>> There some more operations done by comint, which aren't needed when
>> executing Python code from source but may slow down it.
>
> I expect the main slowdown comes from font-lock.  But indeed, it''d be
> good to try and profile it to see where the time is spent.
>

From profiling, a lot of time spent on font-locking while another bunch
is spent at `comint-postoutput-scroll-to-bottom' (default member of
`comint-output-filter-functions').

After disabling both, the output is sent pretty quickly and Emacs, while
it's a bit slower, it's still responsive.

WRT the font-lock slowness I must say I'm not happy the way it's
implemented, there should be a better way to just fontify the text after
the current prompt.  That should avoid any extreme fontification cases.

>
> If the problem is really in "unneeded comint functionality" (or call it
> "bloat"), that's an option, but of course if we don't know where the
> performance problem comes, we may end up with the same problem anyway.
>

Now about the `comint-postoutput-scroll-to-bottom', this alone can also
make Emacs be extremely irresponsive (pretty much like with the
font-locking enabled).  The real culprit of it's slowness is `recenter'
function which seems to get really slow with long lines.  Here are the
elp-results with font-lock disabled (with a smaller sample of output):

comint-output-filter                             103         29.795948749  0.2892810558
comint-postoutput-scroll-to-bottom               104         28.636891969  0.2753547304
recenter                                         104         28.635311547  0.2753395341
python-pdbtrack-comint-output-filter-function    104         0.990129131   0.0095204724
comint-watch-for-password-prompt                 104         0.0924025480  0.0008884860

> Maybe comint.el could also be changed so that it "wraps" lines if they
> get past some arbitrary maximum length (like 10K chars, for example).
> It wouldn't be a great solution, but if the performance sucks really bad
> past 10K chars, wrapping the line might be a lesser evil.
>

Disabling font-lock and avoiding the `recenter' calls works pretty well,
so I rather focus on improving those than modifying comint to workaround
this.

As a final related note, to avoid unnecessary filters, I think the
`inferior-python-mode' should set the `comint-output-filter-functions'
to a minimum and point users to append custom desired filters using the
`inferior-python-mode-hook'.



Regards,
Fabián