bug#10815: counterintuitive results with process-send-string

[Stefan Monnier <monnier@iro.umontreal.ca>]

severity 10815 important
thanks

> Here is a very short extract from my code (exact code, no simplification):

>   (message "sending command %s" c)
>   (process-send-string connection
>                        (concat request-start "\n" c "\n" end-of-message
> "\n"))
>   (message "command sent:   %s" c)

> and an extract of the resulting *Messages* (I have replaced the real
> "command" contents by timestamps):

> sending command 1
> sending command 2
> command sent:   2
> sending command 3
> command sent:   3
> command sent:   1

This output is not a problem in itself, it can be explained as follows:
- (message "sending command %s" 1)
- (process-send-string connection .... 1)
- after sending the command we check timers and process filters; turns
  out we received an answer from the process so we run the filter.
- (message "sending command %s" 2)
- (process-send-string connection .... 2)
- (message "command sent: %s" 2)
- before returning from process-send-string, we check timers and filters
  once more and find another answer, so we run the filter again.
- (message "sending command %s" 3)
- (process-send-string connection .... 3)
- (message "command sent: %s" 3)
- finally the timers and filters seem all processed, we can return.
- (message "command sent: %s" 1)

> The other end of the socket connection receives the messages in the order 2,
> 3, 1, i.e., the same as indicated by "comment sent" debug messages.

That is more problematic.  I haven't looked enough at the code to see
exactly why it happens, but I agree it's undesirable.

> It seems that process-send-string,although it is blocking (until sent data
> is acknowledged), may allow execution of other code (which in this case
> calls process-send-string again).  This seems to be allowed by its
> specification: "Output from processes can arrive in between bunches.",

Indeed, but it doesn't justify the kind of out-of-order-sending you're
seeing (although it may explain it).

> except that in my setting, I am almost sure than no input can be available
> at this moment, at least from this connection.

Are you sure: consider the case where the OS forces Emacs to yield right
after sending the bytes, such that the other process gets to reply
before Emacs finishes the process-send-string?

Of course, rather than process filters, the culprit might be a timer.

> In fact, the calls to process-send-string are initiated by
> after-modify hooks, originating from a single user command (which
> performs several modifications).

Can they also be triggered from timers?  I.e. is there some timer code
modifying the buffer?  Or does the process-filter itself modify the
buffer (seems unlikely since it would lead to inf-loops)?

> But what is really annoying, is that the inner call to process-send-string
> takes priority on the pending one: the second message is actually sent
> before the first, while I would expect the messages to be queued in the
> right order.

Yes, that seems to be a bug/misfeature.


        Stefan