Is it possible to change the recv buffer size dynamically for network process or subprocess?

Is it possible to change the recv buffer size dynamically for network process or subprocess?

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 378 bytes --]

For example, I have binary message format like this:

1. It has a four-bytes header and payload.
2. The four-bytes header is the payload size of the message.

First I want to set the size of the recv buffer to 4. After the four bytes arrived, I parse them and get the size of the payload. Then I set the size of the recv buffer to the size of the payload to recv the payload.



[-- Attachment #2: Type: text/html, Size: 874 bytes --]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Robert Pluim]

>>>>> On Sat, 21 Mar 2020 11:01:05 +0800, HaiJun Zhang <netjune@outlook.com> said:

    HaiJun> For example, I have binary message format like this:
    HaiJun> 1. It has a four-bytes header and payload.
    HaiJun> 2. The four-bytes header is the payload size of the message.

    HaiJun> First I want to set the size of the recv buffer to 4. After the four
    HaiJun> bytes arrived, I parse them and get the size of the payload. Then I
    HaiJun> set the size of the recv buffer to the size of the payload to recv the
    HaiJun> payload.

Why bother? Youʼre going to have to handle partial recv for the
payload anyway, so you might as well do it from the start.

Robert

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

在 2020年3月21日 +0800 PM8:10，Robert Pluim <rpluim@gmail.com>，写道：
> > > > > > On Sat, 21 Mar 2020 11:01:05 +0800, HaiJun Zhang <netjune@outlook.com> said:
>
> HaiJun> For example, I have binary message format like this:
> HaiJun> 1. It has a four-bytes header and payload.
> HaiJun> 2. The four-bytes header is the payload size of the message.
>
> HaiJun> First I want to set the size of the recv buffer to 4. After the four
> HaiJun> bytes arrived, I parse them and get the size of the payload. Then I
> HaiJun> set the size of the recv buffer to the size of the payload to recv the
> HaiJun> payload.
>
> Why bother? Youʼre going to have to handle partial recv for the
> payload anyway, so you might as well do it from the start.
>
> Robert

I want to do flow control, because too many data may cause UI unresponsive. Set the recv buffer size to 4, and if I don’t fetch the four bytes, please don’t receive more data. If the size (In the four bytes header) is too big, ask the user if the message should be received.


[-- Attachment #2: Type: text/html, Size: 2234 bytes --]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Robert Pluim]

>>>>> On Sun, 22 Mar 2020 09:26:37 +0800, HaiJun Zhang <netjune@outlook.com> said:

    HaiJun> I want to do flow control, because too many data may cause UI
    HaiJun> unresponsive. Set the recv buffer size to 4, and if I don’t fetch the
    HaiJun> four bytes, please don’t receive more data. If the size (In the four
    HaiJun> bytes header) is too big, ask the user if the message should be
    HaiJun> received.

I donʼt see how this is any different from just doing recv, checking
the 4 bytes, and then asking the user. In any case, if the remote end
has sent you the message, it will be in the local kernel buffers
anyway, so you'll have to read it at some point anyway.

Anyway, emacs-27 has read-process-output-max, which I guess you could
set to 4, but I suspect that will cause all sorts of other issues.

Robert

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Sun, 22 Mar 2020 16:12:31 +0100
> Cc: emacs-devel@gnu.org
> 
> Anyway, emacs-27 has read-process-output-max, which I guess you could
> set to 4, but I suspect that will cause all sorts of other issues.

Almost certainly.  I won't recommend any measure like that without
some serious profiling that suggests this is indeed a significant
factor (and I'd be surprised if it is).

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

在 2020年3月22日 +0800 PM11:12，Robert Pluim <rpluim@gmail.com>，写道：
> > > > > > On Sun, 22 Mar 2020 09:26:37 +0800, HaiJun Zhang <netjune@outlook.com> said:
>
> HaiJun> I want to do flow control, because too many data may cause UI
> HaiJun> unresponsive. Set the recv buffer size to 4, and if I don’t fetch the
> HaiJun> four bytes, please don’t receive more data. If the size (In the four
> HaiJun> bytes header) is too big, ask the user if the message should be
> HaiJun> received.
>
> I donʼt see how this is any different from just doing recv, checking
> the 4 bytes, and then asking the user. In any case, if the remote end
> has sent you the message, it will be in the local kernel buffers
> anyway, so you’ll have to read it at some point anyway.
>
With flow control, we can block the sender(for tcp or subprocess) for sometime to keep good responsiveness.


[-- Attachment #2: Type: text/html, Size: 2023 bytes --]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Stefan Monnier]

> I want to do flow control, because too many data may cause UI
> unresponsive. Set the recv buffer size to 4, and if I don’t fetch the four
> bytes, please don’t receive more data. If the size (In the four bytes
> header) is too big, ask the user if the message should be received.

I think I understand your description, and I think I understand somewhat
how Emacs's code works in this respect, but I really don't see how the
two relate.

Presumably you're writing Elisp code using Emacs's process filters.
What is the concrete problem you're seeing that you're trying to solve
(your description seems to focus on *how* to solve your problem)?

Does Emacs take too long to react to some events?  Does its memory use
grow unreasonably?  Is the rest of the system affected?


        Stefan

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

在 2020年3月23日 +0800 AM2:13，Stefan Monnier <monnier@iro.umontreal.ca>，写道：
> > I want to do flow control, because too many data may cause UI
> > unresponsive. Set the recv buffer size to 4, and if I don’t fetch the four
> > bytes, please don’t receive more data. If the size (In the four bytes
> > header) is too big, ask the user if the message should be received.
>
> I think I understand your description, and I think I understand somewhat
> how Emacs's code works in this respect, but I really don't see how the
> two relate.
>
> Presumably you're writing Elisp code using Emacs's process filters.
> What is the concrete problem you're seeing that you're trying to solve
> (your description seems to focus on *how* to solve your problem)?
>
> Does Emacs take too long to react to some events? Does its memory use
> grow unreasonably? Is the rest of the system affected?
>
>
> Stefan
>

If too many data arrives, emacs may be unresponsive. I want to block the data sender(subprocess) sometimes to keep good responsiveness.


[-- Attachment #2: Type: text/html, Size: 1653 bytes --]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Eli Zaretskii]

> Date: Tue, 24 Mar 2020 12:30:00 +0800
> From: HaiJun Zhang <netjune@outlook.com>
> Cc: Robert Pluim <rpluim@gmail.com>, emacs-devel@gnu.org
> 
>  Does Emacs take too long to react to some events? Does its memory use
>  grow unreasonably? Is the rest of the system affected?
> 
>  Stefan
> 
> If too many data arrives, emacs may be unresponsive.

Unresponsive due to what processing?  Are you saying that just
_receiving_ the data takes most of the time?  I think what takes most
of the time is the processing by the filter function, in which case
making the filter function process just the few first characters will
do.

But even if you think I'm wrong, and most of the time is taken by
receiving the data, please show the profile to back that up.  It is
IME wrong to start optimizing code without measuring it first, because
bottlenecks usually happen in unexpected places, and you may well end
up gaining nothing from the optimization that is not backed by
measurements.

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 858 bytes --]

在 2020年3月24日 +0800 PM10:01，Eli Zaretskii <eliz@gnu.org>，写道：
> > Date: Tue, 24 Mar 2020 12:30:00 +0800
> > From: HaiJun Zhang <netjune@outlook.com>
> > Cc: Robert Pluim <rpluim@gmail.com>, emacs-devel@gnu.org
> >
> > Does Emacs take too long to react to some events? Does its memory use
> > grow unreasonably? Is the rest of the system affected?
> >
> > Stefan
> >
> > If too many data arrives, emacs may be unresponsive.
>
> Unresponsive due to what processing? Are you saying that just
> _receiving_ the data takes most of the time? I think what takes most
> of the time is the processing by the filter function, in which case
> making the filter function process just the few first characters will
> do.
>

Yes. I want to block the sender for some time. It is like to limit the bandwidth to 500KB/s or 10 packets/s.


[-- Attachment #2: Type: text/html, Size: 1463 bytes --]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Eli Zaretskii]

> Date: Thu, 26 Mar 2020 21:43:31 +0800
> From: HaiJun Zhang <netjune@outlook.com>
> Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, emacs-devel@gnu.org
> 
>  If too many data arrives, emacs may be unresponsive.
> 
>  Unresponsive due to what processing? Are you saying that just
>  _receiving_ the data takes most of the time? I think what takes most
>  of the time is the processing by the filter function, in which case
>  making the filter function process just the few first characters will
>  do.
> 
> Yes. I want to block the sender for some time. It is like to limit the bandwidth to 500KB/s or 10 packets/s.

I don't see how this will help anything, sorry.

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

For example, when I run find or grep in a directory with plenty of files, the output of the subprocess may continue for 1 minute, while I can’t stop it.
在 2020年3月26日 +0800 PM10:38，Eli Zaretskii <eliz@gnu.org>，写道：
> > Date: Thu, 26 Mar 2020 21:43:31 +0800
> > From: HaiJun Zhang <netjune@outlook.com>
> > Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, emacs-devel@gnu.org
> >
> > If too many data arrives, emacs may be unresponsive.
> >
> > Unresponsive due to what processing? Are you saying that just
> > _receiving_ the data takes most of the time? I think what takes most
> > of the time is the processing by the filter function, in which case
> > making the filter function process just the few first characters will
> > do.
> >
> > Yes. I want to block the sender for some time. It is like to limit the bandwidth to 500KB/s or 10 packets/s.
>
> I don't see how this will help anything, sorry.

[-- Attachment #2: Type: text/html, Size: 1513 bytes --]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Eli Zaretskii]

> Date: Sun, 29 Mar 2020 09:44:41 +0800
> From: HaiJun Zhang <netjune@outlook.com>
> Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, emacs-devel@gnu.org
> 
> For example, when I run find or grep in a directory with plenty of files, the output of the subprocess may
> continue for 1 minute, while I can’t stop it.

That doesn't use network communications, so it is not relevant to the
issue at hand.

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

在 2020年3月29日 +0800 AM10:38，Eli Zaretskii <eliz@gnu.org>，写道：
> > Date: Sun, 29 Mar 2020 09:44:41 +0800
> > From: HaiJun Zhang <netjune@outlook.com>
> > Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, emacs-devel@gnu.org
> >
> > For example, when I run find or grep in a directory with plenty of files, the output of the subprocess may
> > continue for 1 minute, while I can’t stop it.
>
> That doesn't use network communications, so it is not relevant to the
> issue at hand.

Something like that. It seems that emacs treat them the same.

One of my use case:
I plan to hack lsp-mode to limit the bandwidth and the message size.

When I use lsp-mode for dart language, It happened that 10000 lines were sent to emacs for one completion which cause emacs unresponsive for several seconds.

When I use lsp-mode for go language, It happened that 400 messages were sent to emacs during one second which cause emacs unresponsive for a while.


[-- Attachment #2: Type: text/html, Size: 1945 bytes --]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Eli Zaretskii]

> Date: Sat, 4 Apr 2020 20:17:10 +0800
> From: HaiJun Zhang <netjune@outlook.com>
> Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, emacs-devel@gnu.org
> 
> One of my use case:
> I plan to hack lsp-mode to limit the bandwidth and the message size.
> 
> When I use lsp-mode for dart language, It happened that 10000 lines were sent to emacs for one completion
> which cause emacs unresponsive for several seconds.
> 
> When I use lsp-mode for go language, It happened that 400 messages were sent to emacs during one
> second which cause emacs unresponsive for a while.

Emacs becomes unresponsive in these cases because it processes the
stuff that arrives in its entirety, not because it receives it.  The
code which processes the received stuff can easily stop after some
amount has been processed, and leave the rest for the next cycle.

IOW, it is better to modify the process filter than make the receive
buffer smaller.

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[yyoncho]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

Hi HaiJun,

I second Eli's response - if this is really the case we could stop
processing messages at some point and continue later(e. g. we may check if
there is pending user input) . I kindly suggest reporting these issues on
lsp-mode side before jumping into emacs-devel.

Thanks,
Ivan

On Sat, Apr 4, 2020 at 3:18 PM HaiJun Zhang <netjune@outlook.com> wrote:

> 在 2020年3月29日 +0800 AM10:38，Eli Zaretskii <eliz@gnu.org>，写道：
>
> Date: Sun, 29 Mar 2020 09:44:41 +0800
> From: HaiJun Zhang <netjune@outlook.com>
> Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, emacs-devel@gnu.org
>
> For example, when I run find or grep in a directory with plenty of files,
> the output of the subprocess may
> continue for 1 minute, while I can’t stop it.
>
>
> That doesn't use network communications, so it is not relevant to the
> issue at hand.
>
>
> Something like that. It seems that emacs treat them the same.
>
> One of my use case:
> I plan to hack lsp-mode to limit the bandwidth and the message size.
>
> When I use lsp-mode for dart language, It happened that 10000 lines were
> sent to emacs for one completion which cause emacs unresponsive for several
> seconds.
>
> When I use lsp-mode for go language, It happened that 400 messages were
> sent to emacs during one second which cause emacs unresponsive for a
> while.
>
>

[-- Attachment #2: Type: text/html, Size: 2519 bytes --]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For example, when I run find or grep in a directory with plenty of files, the output of the subprocess may continue for 1 minute, while I can’t stop it.

In ITS I put limits on the size of output buffers in interactive remote login
so as to prevent that sort of problem.  The small buffer ensured that if you
typed C-c or C-z, you wouldn't have to wait as lots of output arrived
before the program stopped generating more output.

In Emacs, we want large buffers for the situation where a Lisp program
is running and goes for some time without allowing subprocesses output
to enter.

Perhaps if we study the cases where we need a large buffer and the
cases where we don't want a large buffer, we could find a criterion to
distinguish.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Stefan Monnier]

> Yes.  I want to block the sender for some time.  It is like to limit the
> bandwidth to 500KB/s or 10 packets/s.

[ Again, a concrete scenario where you show a practical need for this
  would help a lot in the discussion.  ]

There is no way currently for Emacs to "throttle" the incoming data,
indeed (other than by keeping itself busy).

We should maybe add a way to "suspend" processing the output of
a specific process (and calling `accept-process-output` with an
explicit process argument would override it).

It would be convenient to do that while debugging the execution of
a process filter.


        Stefan

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Noam Postavsky]

On Thu, 26 Mar 2020 at 12:14, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> We should maybe add a way to "suspend" processing the output of
> a specific process (and calling `accept-process-output` with an
> explicit process argument would override it).

Doesn't setting the process-filter to t do this? (except for the
override part, I guess)

(set-process-filter PROCESS FILTER)

Give PROCESS the filter function FILTER; nil means default.
A value of t means stop accepting output from the process.
[....]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Stefan Monnier]

> Doesn't setting the process-filter to t do this? (except for the
> override part, I guess)
>
> (set-process-filter PROCESS FILTER)
>
> Give PROCESS the filter function FILTER; nil means default.
> A value of t means stop accepting output from the process.
> [....]

Ah, indeed you're right.

It's a bit clunky (it means you need to temporarily store the filter
elsewhere and other code that may want to change the filter needs to
know where to find it), but that should be good enough for a start.

HaiJun?


        Stefan

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[HaiJun Zhang]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

在 2020年3月27日 +0800 AM1:34，Stefan Monnier <monnier@iro.umontreal.ca>，写道：
> > Doesn't setting the process-filter to t do this? (except for the
> > override part, I guess)
> >
> > (set-process-filter PROCESS FILTER)
> >
> > Give PROCESS the filter function FILTER; nil means default.
> > A value of t means stop accepting output from the process.
> > [....]
>
> Ah, indeed you're right.
>
> It's a bit clunky (it means you need to temporarily store the filter
> elsewhere and other code that may want to change the filter needs to
> know where to find it), but that should be good enough for a start.
>
> HaiJun?
>
>
> Stefan
>
Yes. Thanks.  @Noam Postavsky

[-- Attachment #2: Type: text/html, Size: 1226 bytes --]

Re: Is it possible to change the recv buffer size dynamically for network process or subprocess?

[Stefan Monnier]

> If too many data arrives, emacs may be unresponsive. I want to block the
> data sender(subprocess) sometimes to keep good responsiveness.

So you want a mechanism to control when the process-filter is run,
basically (since AFAIK the moment when we consume (at the C/OS level)
our input is the same as the moment we run the process filter).

Usually this is done implicitly as a side-effect of Emacs being busy,
but IIUC you want to temporarily suspend processing your process's input
even tho Emacs is not busy?

Can you give a concrete scenario you have seen, so we can better
understand what could be done?


        Stefan