* bug#16875: python, comint-mode: Large output makes Emacs freeze @ 2014-02-25 9:21 Andreas Röhler 2014-06-25 1:19 ` Fabián Ezequiel Gallina 2014-07-27 2:22 ` Fabián Ezequiel Gallina 0 siblings, 2 replies; 12+ messages in thread From: Andreas Röhler @ 2014-02-25 9:21 UTC (permalink / raw To: 16875 May confirm what is reported here: http://stackoverflow.com/questions/20128425/can-i-stop-the-repl-locking-up-when-using-emacs-and-python Form sent is [[[False] * 200 for i in range(3)] for j in range(200)] Does only occur with plain Python-shell, not if IPython is running. IIUC that's because IPython return is passed through a kind of pretty-print, inserting newlines. Comints dealing with long lines seems the cause. With process-send-string it works. ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-02-25 9:21 bug#16875: python, comint-mode: Large output makes Emacs freeze Andreas Röhler @ 2014-06-25 1:19 ` Fabián Ezequiel Gallina 2014-06-25 5:27 ` Andreas Röhler 2014-07-27 2:22 ` Fabián Ezequiel Gallina 1 sibling, 1 reply; 12+ messages in thread From: Fabián Ezequiel Gallina @ 2014-06-25 1:19 UTC (permalink / raw To: 16875 This looks related to: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13675 Some ways to attenuate irresponsiveness: 1. Disable python shell font-locking: (setq python-shell-enable-font-lock nil) 2. Use Python's pprint to print such expressions, this makes it behave pretty much like iPython: >>> import pprint >>> pprint.pprint([[[False] * 200 for i in range(3)] for j in range(200)]) From the python.el side there's not much we can do. Fabián ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-06-25 1:19 ` Fabián Ezequiel Gallina @ 2014-06-25 5:27 ` Andreas Röhler 2014-06-25 14:26 ` Stefan Monnier 0 siblings, 1 reply; 12+ messages in thread From: Andreas Röhler @ 2014-06-25 5:27 UTC (permalink / raw To: 16875 On 25.06.2014 03:19, Fabián Ezequiel Gallina wrote: > > This looks related to: > http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13675 > To a certain extend. There some more operations done by comint, which aren't needed when executing Python code from source but may slow down it. Thus solution in python-mode.el is to surpass comint-mode completely and use "start-process" and "process-send-string" See py-fast-process. > Some ways to attenuate irresponsiveness: > > 1. Disable python shell font-locking: > (setq python-shell-enable-font-lock nil) > > 2. Use Python's pprint to print such expressions, this makes it behave > pretty much like iPython: > > >>> import pprint > >>> pprint.pprint([[[False] * 200 for i in range(3)] for j in range(200)]) > Indeed, "pprint" should solve it already. Maybe make that the default? IMO looks better anyway. Andreas ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-06-25 5:27 ` Andreas Röhler @ 2014-06-25 14:26 ` Stefan Monnier 2014-06-25 15:28 ` Andreas Röhler 2014-07-20 22:27 ` Fabián Ezequiel Gallina 0 siblings, 2 replies; 12+ messages in thread From: Stefan Monnier @ 2014-06-25 14:26 UTC (permalink / raw To: Andreas Röhler; +Cc: 16875 >> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13675 > To a certain extend. > There some more operations done by comint, which aren't needed when > executing Python code from source but may slow down it. I expect the main slowdown comes from font-lock. But indeed, it''d be good to try and profile it to see where the time is spent. Last time I tried to speed up M-x grep, I changed compile.el so that it only processes (font-lock and friends) the output up to the last \n. This way, when a very long line is received in 100 chunks, it doesn't get re-processed 100 times. Maybe comint.el could do the same (especially since it already assumes that "text between the last \n and EOB is a prompt"). > Thus solution in python-mode.el is to surpass comint-mode completely and use > "start-process" and "process-send-string" If the problem is really in "unneeded comint functionality" (or call it "bloat"), that's an option, but of course if we don't know where the performance problem comes, we may end up with the same problem anyway. > Indeed, "pprint" should solve it already. Maybe make that the default? > IMO looks better anyway. [ I don't use Python, so I don't have an opinion on that, but if it works, fine by me. ] Maybe comint.el could also be changed so that it "wraps" lines if they get past some arbitrary maximum length (like 10K chars, for example). It wouldn't be a great solution, but if the performance sucks really bad past 10K chars, wrapping the line might be a lesser evil. Stefan ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-06-25 14:26 ` Stefan Monnier @ 2014-06-25 15:28 ` Andreas Röhler 2014-07-20 22:27 ` Fabián Ezequiel Gallina 1 sibling, 0 replies; 12+ messages in thread From: Andreas Röhler @ 2014-06-25 15:28 UTC (permalink / raw To: Stefan Monnier; +Cc: 16875 On 25.06.2014 16:26, Stefan Monnier wrote: [ ... ] > Maybe comint.el could also be changed so that it "wraps" lines if they > get past some arbitrary maximum length (like 10K chars, for example). > It wouldn't be a great solution, but if the performance sucks really bad > past 10K chars, wrapping the line might be a lesser evil. > > > Stefan > Sounds promising, thanks! As comint is built for human interaction, readability of output matters. Andreas ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-06-25 14:26 ` Stefan Monnier 2014-06-25 15:28 ` Andreas Röhler @ 2014-07-20 22:27 ` Fabián Ezequiel Gallina 2014-07-23 6:20 ` Andreas Röhler 1 sibling, 1 reply; 12+ messages in thread From: Fabián Ezequiel Gallina @ 2014-07-20 22:27 UTC (permalink / raw To: Stefan Monnier; +Cc: 16875 Andreas Röhler <andreas.roehler@easy-emacs.de> writes: >> Indeed, "pprint" should solve it already. Maybe make that the default? >> IMO looks better anyway. That's changing the Python shell default behavior and I that's out of the scope of python.el. Stefan Monnier <monnier@iro.umontreal.ca> writes: >>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13675 >> To a certain extend. >> There some more operations done by comint, which aren't needed when >> executing Python code from source but may slow down it. > > I expect the main slowdown comes from font-lock. But indeed, it''d be > good to try and profile it to see where the time is spent. > From profiling, a lot of time spent on font-locking while another bunch is spent at `comint-postoutput-scroll-to-bottom' (default member of `comint-output-filter-functions'). After disabling both, the output is sent pretty quickly and Emacs, while it's a bit slower, it's still responsive. WRT the font-lock slowness I must say I'm not happy the way it's implemented, there should be a better way to just fontify the text after the current prompt. That should avoid any extreme fontification cases. > > If the problem is really in "unneeded comint functionality" (or call it > "bloat"), that's an option, but of course if we don't know where the > performance problem comes, we may end up with the same problem anyway. > Now about the `comint-postoutput-scroll-to-bottom', this alone can also make Emacs be extremely irresponsive (pretty much like with the font-locking enabled). The real culprit of it's slowness is `recenter' function which seems to get really slow with long lines. Here are the elp-results with font-lock disabled (with a smaller sample of output): comint-output-filter 103 29.795948749 0.2892810558 comint-postoutput-scroll-to-bottom 104 28.636891969 0.2753547304 recenter 104 28.635311547 0.2753395341 python-pdbtrack-comint-output-filter-function 104 0.990129131 0.0095204724 comint-watch-for-password-prompt 104 0.0924025480 0.0008884860 > Maybe comint.el could also be changed so that it "wraps" lines if they > get past some arbitrary maximum length (like 10K chars, for example). > It wouldn't be a great solution, but if the performance sucks really bad > past 10K chars, wrapping the line might be a lesser evil. > Disabling font-lock and avoiding the `recenter' calls works pretty well, so I rather focus on improving those than modifying comint to workaround this. As a final related note, to avoid unnecessary filters, I think the `inferior-python-mode' should set the `comint-output-filter-functions' to a minimum and point users to append custom desired filters using the `inferior-python-mode-hook'. Regards, Fabián ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-07-20 22:27 ` Fabián Ezequiel Gallina @ 2014-07-23 6:20 ` Andreas Röhler 0 siblings, 0 replies; 12+ messages in thread From: Andreas Röhler @ 2014-07-23 6:20 UTC (permalink / raw To: Fabián Ezequiel Gallina, Stefan Monnier; +Cc: 16875 On 21.07.2014 00:27, Fabián Ezequiel Gallina wrote: > [ ... ] > Disabling font-lock and avoiding the `recenter' calls works pretty well, > so I rather focus on improving those than modifying comint to workaround > this. > Given font-locking is an issue only when triggered while output arrives, maybe there is a way to disable it alongside with RET, but enabling it as soon no output is pending? Andreas ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-02-25 9:21 bug#16875: python, comint-mode: Large output makes Emacs freeze Andreas Röhler 2014-06-25 1:19 ` Fabián Ezequiel Gallina @ 2014-07-27 2:22 ` Fabián Ezequiel Gallina 2014-07-27 4:03 ` Eli Zaretskii 2014-07-27 10:20 ` Stefan Monnier 1 sibling, 2 replies; 12+ messages in thread From: Fabián Ezequiel Gallina @ 2014-07-27 2:22 UTC (permalink / raw To: 16875-done Fixed in revno 117582 in trunk Emacs is still slow, but at least responds to commands. I feel that everything that could be optimized on the comint side is pretty much there and that the slowness being experienced is in fact related to the long-lines slowness bug mentioned before[0]. Here's the current output for elp-results: comint-output-filter 206 3.5420630980 0.0171944810 python-pdbtrack-comint-output-filter-function 207 2.5283979559 0.0122144828 ansi-color-filter-apply 618 1.5662161150 0.0025343302 python-comint-postoutput-scroll-to-bottom 207 0.400739811 0.0019359411 comint-postoutput-scroll-to-bottom 1 0.365398304 0.365398304 recenter 1 0.365378877 0.365378877 python-shell-comint-end-of-output-p 412 0.0554427629 0.0001345698 python-shell-font-lock-comint-output-filter-function 207 0.032279105 0.0001559377 ansi-color-process-output 207 0.0107249640 5.181...e-05 ansi-color-apply-on-region 207 0.0098468030 4.756...e-05 comint-carriage-motion 206 0.0014299259 6.941...e-06 python-shell-font-lock-post-command-hook 3 0.00034127 0.0001137566 process-mark 828 0.000202256 2.442...e-07 process-kill-buffer-query-function 206 0.0001694799 8.227...e-07 comint-send-input 1 0.000149412 0.000149412 ansi-color--find-face 207 8.6848e-05 4.195...e-07 process-status 208 7.752...e-05 3.727...e-07 process-buffer 207 7.367...e-05 3.558...e-07 ansi-color-apply-overlay-face 207 6.695...e-05 3.234...e-07 python-util-text-properties-replace-name 1 4.8394e-05 4.8394e-05 comint-simple-send 1 3.8879e-05 3.8879e-05 python-shell-font-lock-cleanup-buffer 1 2.344e-05 2.344e-05 comint-send-string 1 2.2201e-05 2.2201e-05 process-send-string 1 1.5571e-05 1.5571e-05 comint-add-to-input-history 1 1.2823e-05 1.2823e-05 comint-adjust-window-point 1 4.201e-06 4.201e-06 comint-preinput-scroll-to-bottom 3 2.708e-06 9.026...e-07 python-util-comint-last-prompt 6 2.490...e-06 4.150...e-07 comint-snapshot-last-prompt 2 2.462e-06 1.231e-06 [0] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=13675 Cheers, Fabián ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-07-27 2:22 ` Fabián Ezequiel Gallina @ 2014-07-27 4:03 ` Eli Zaretskii 2014-07-28 21:35 ` Fabián Ezequiel Gallina 2014-07-27 10:20 ` Stefan Monnier 1 sibling, 1 reply; 12+ messages in thread From: Eli Zaretskii @ 2014-07-27 4:03 UTC (permalink / raw To: Fabián Ezequiel Gallina; +Cc: 16875 > From: fgallina@gnu.org (Fabián Ezequiel Gallina) > Date: Sat, 26 Jul 2014 23:22:41 -0300 > > > Emacs is still slow, but at least responds to commands. I feel that > everything that could be optimized on the comint side is pretty much > there and that the slowness being experienced is in fact related to the > long-lines slowness bug mentioned before[0]. That bug is about displaying long lines. Is the code that is slow involved in displaying such lines? If there's no display involved, then the discussion you point to is not relevant. Also, please tell how long are your lines, on the average. ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-07-27 4:03 ` Eli Zaretskii @ 2014-07-28 21:35 ` Fabián Ezequiel Gallina 2014-07-29 7:32 ` Eli Zaretskii 0 siblings, 1 reply; 12+ messages in thread From: Fabián Ezequiel Gallina @ 2014-07-28 21:35 UTC (permalink / raw To: Eli Zaretskii; +Cc: 16875 Eli Zaretskii <eliz@gnu.org> writes: >> From: fgallina@gnu.org (Fabián Ezequiel Gallina) >> Date: Sat, 26 Jul 2014 23:22:41 -0300 >> >> >> Emacs is still slow, but at least responds to commands. I feel that >> everything that could be optimized on the comint side is pretty much >> there and that the slowness being experienced is in fact related to the >> long-lines slowness bug mentioned before[0]. > > That bug is about displaying long lines. Is the code that is slow > involved in displaying such lines? If there's no display involved, > then the discussion you point to is not relevant. > > Also, please tell how long are your lines, on the average. > Yes, such code outputs a long line consisting of 841601 chars. One way to get that output in a file for testing is executing the following shell command: python -c 'print ([[[False] * 200 for i in range(3)] for j in range(200)])' > /tmp/out.log Even opening that file in fundamental-mode makes Emacs slow. For instance calling `previous-line' from the end of buffer several times takes 1.1 seconds on average to complete. previous-line 30 33.263101639 1.1087700546 Next line is a bit more responsive though: next-line 30 14.028573178 0.4676191059 Operations seem to take longer and longer as point moves away from the beginning of buffer. Cheers, Fabián ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-07-28 21:35 ` Fabián Ezequiel Gallina @ 2014-07-29 7:32 ` Eli Zaretskii 0 siblings, 0 replies; 12+ messages in thread From: Eli Zaretskii @ 2014-07-29 7:32 UTC (permalink / raw To: Fabián Ezequiel Gallina; +Cc: 16875 > From: fgallina@gnu.org (Fabián Ezequiel Gallina) > Cc: 16875@debbugs.gnu.org, andreas.roehler@easy-emacs.de > Date: Mon, 28 Jul 2014 18:35:09 -0300 > > Eli Zaretskii <eliz@gnu.org> writes: > > >> From: fgallina@gnu.org (Fabián Ezequiel Gallina) > >> Date: Sat, 26 Jul 2014 23:22:41 -0300 > >> > >> > >> Emacs is still slow, but at least responds to commands. I feel that > >> everything that could be optimized on the comint side is pretty much > >> there and that the slowness being experienced is in fact related to the > >> long-lines slowness bug mentioned before[0]. > > > > That bug is about displaying long lines. Is the code that is slow > > involved in displaying such lines? If there's no display involved, > > then the discussion you point to is not relevant. > > > > Also, please tell how long are your lines, on the average. > > > > Yes, such code outputs a long line consisting of 841601 chars. In that case, yes, that's the "long line display slowness" problem. If you profile Emacs with 'profiler-start' and 'profiler-report', you should see that most of the time is spent in redisplay. > Operations seem to take longer and longer as point moves away from the > beginning of buffer. That's expected, although the time should level out after some number of lines. ^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#16875: python, comint-mode: Large output makes Emacs freeze 2014-07-27 2:22 ` Fabián Ezequiel Gallina 2014-07-27 4:03 ` Eli Zaretskii @ 2014-07-27 10:20 ` Stefan Monnier 1 sibling, 0 replies; 12+ messages in thread From: Stefan Monnier @ 2014-07-27 10:20 UTC (permalink / raw To: 16875 > Emacs is still slow, but at least responds to commands. I feel that > everything that could be optimized on the comint side is pretty much > there and that the slowness being experienced is in fact related to the > long-lines slowness bug mentioned before[0]. Have you tried to ensure that font-lock is only applied once a line is complete (i.e. after seeing the corresponding \n)? Stefan ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2014-07-29 7:32 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-02-25 9:21 bug#16875: python, comint-mode: Large output makes Emacs freeze Andreas Röhler 2014-06-25 1:19 ` Fabián Ezequiel Gallina 2014-06-25 5:27 ` Andreas Röhler 2014-06-25 14:26 ` Stefan Monnier 2014-06-25 15:28 ` Andreas Röhler 2014-07-20 22:27 ` Fabián Ezequiel Gallina 2014-07-23 6:20 ` Andreas Röhler 2014-07-27 2:22 ` Fabián Ezequiel Gallina 2014-07-27 4:03 ` Eli Zaretskii 2014-07-28 21:35 ` Fabián Ezequiel Gallina 2014-07-29 7:32 ` Eli Zaretskii 2014-07-27 10:20 ` Stefan Monnier
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.