Strange shell mode performance

Strange shell mode performance

[Herman, Géza]

Hi all,

I'm experiencing strange shell (comint) mode performance. If I cat a 10 
MB file, it takes 1:20 to cat it. Emacs uses only 30% cpu. But, if I 
press enter after I send the cat command, it speeds up, and it only 
takes 8 sec (emacs cpu usage goes up to 100%).

Same thing: if I execute "seq 100000" in a small emacs window, it takes 
~3 seconds. But, if I press enter after executing seq, it takes only 0.4 
sec (this is not 100% reproducible, sometimes pressing enter doesn't 
make a difference).
(I executed comint-clear-buffer before measurements)

It happens with 26.3 and ~3-week-old master branch as well, without any 
extra config (emacs -Q).

Any ideas what causes this? Would it be possible to always have the 
better performance without needing to press enter?

Thanks,
Geza

Re: Strange shell mode performance

[Stefan Monnier]

> Any ideas what causes this?  Would it be possible to always have the better
> performance without needing to press enter?

I suggest you `M-x report-emacs-bug` since it seems to be
a performance bug (that doesn't guarantee that it will be fixed, tho:
it's sometimes quite difficult to fix those problems).


        Stefan

Re: Strange shell mode performance

[Robert Thorpe]

I suspect that this is about memory allocation.

I've noticed strange things like this before with large shell buffers.
I don't think that pressing enter is really what makes the difference.

I expect it's the memory that Emacs has allocated.  When you have a
small shell buffer you're usually reusing memory that has already been
allocated to Emacs.  That's because Emacs is constantly allocating and
deallocating memory within it's own memory management.  AFAIK it never
actually gives memory back to the OS (modern OSes don't really need that anyway).

However, when the size of the shell buffer rises above the largest size
that the Emacs process have ever been things change.  Then new memory
has to be allocated to Emacs.

I may be wrong about this though, it was ages since I looked at the
code.

BR,
Robert Thorpe

Herman, Géza <geza.herman@gmail.com> writes:

> Hi all,
>
> I'm experiencing strange shell (comint) mode performance. If I cat a 10 
> MB file, it takes 1:20 to cat it. Emacs uses only 30% cpu. But, if I 
> press enter after I send the cat command, it speeds up, and it only 
> takes 8 sec (emacs cpu usage goes up to 100%).
>
> Same thing: if I execute "seq 100000" in a small emacs window, it takes 
> ~3 seconds. But, if I press enter after executing seq, it takes only 0.4 
> sec (this is not 100% reproducible, sometimes pressing enter doesn't 
> make a difference).
> (I executed comint-clear-buffer before measurements)
>
> It happens with 26.3 and ~3-week-old master branch as well, without any 
> extra config (emacs -Q).
>
> Any ideas what causes this? Would it be possible to always have the 
> better performance without needing to press enter?
>
> Thanks,
> Geza

Re: Strange shell mode performance

[Herman, Géza]

The bug is caused by process-adaptive-read-buffering.

I analyzed the issue. For my use case, I made shell ~60x faster by 
simple modifications (details: 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=44007).

On 2020-10-21 03:27, Robert Thorpe wrote:
> I suspect that this is about memory allocation.
>
> I've noticed strange things like this before with large shell buffers.
> I don't think that pressing enter is really what makes the difference.
>
> I expect it's the memory that Emacs has allocated.  When you have a
> small shell buffer you're usually reusing memory that has already been
> allocated to Emacs.  That's because Emacs is constantly allocating and
> deallocating memory within it's own memory management.  AFAIK it never
> actually gives memory back to the OS (modern OSes don't really need that anyway).
>
> However, when the size of the shell buffer rises above the largest size
> that the Emacs process have ever been things change.  Then new memory
> has to be allocated to Emacs.
>
> I may be wrong about this though, it was ages since I looked at the
> code.
>
> BR,
> Robert Thorpe
>
> Herman, Géza <geza.herman@gmail.com> writes:
>
>> Hi all,
>>
>> I'm experiencing strange shell (comint) mode performance. If I cat a 10
>> MB file, it takes 1:20 to cat it. Emacs uses only 30% cpu. But, if I
>> press enter after I send the cat command, it speeds up, and it only
>> takes 8 sec (emacs cpu usage goes up to 100%).
>>
>> Same thing: if I execute "seq 100000" in a small emacs window, it takes
>> ~3 seconds. But, if I press enter after executing seq, it takes only 0.4
>> sec (this is not 100% reproducible, sometimes pressing enter doesn't
>> make a difference).
>> (I executed comint-clear-buffer before measurements)
>>
>> It happens with 26.3 and ~3-week-old master branch as well, without any
>> extra config (emacs -Q).
>>
>> Any ideas what causes this? Would it be possible to always have the
>> better performance without needing to press enter?
>>
>> Thanks,
>> Geza