Connection drops data if it sent too quickly

Connection drops data if it sent too quickly

[Kai Großjohann]

Tramp calls, say, ssh via start-process.  This results in an
interactive shell from the remote host.  Then Tramp sends commands to
this shell.  For example, it might send:

    mimencode -u -b <<'EOF'

Then Tramp uses process-send-region to send base64 encoded data to
this ssh process.  Then Tramp sends the string EOF.

The problem is that the connection is dropping bytes when the data is
sent too quickly.  I've changed it to send chunks of 500 bytes, then
wait 0.1 seconds.  This works.

This happens using for Michael, a codeveloper, using GNU Emacs 20.7.1
(hppa1.1-hp-hpux10.20, Motif) to connect to localhost.  The file in
question is between 300k and 400k (so the base64 encoded data is
correspondingly bigger).  It also happens to other people, but I don't
have a reference.  The remote end only sees 150 bytes (!) or so of it.
Removing the (sleep-for 0.1) but keeping the send-in-chunks mechanism
lets 250k (or so) arrive on the remote end.

There seems to be code in process-send-region (or a function called
from it) that tries to avoid sending data too quickly, but I don't
grok it.

Just unconditionally throttling the speed of data transmission seems
kludgy.  But what to do?

(Maybe Michael could try what happens when he does

  mimencode -b /tmp/largefile | \
    ssh jrl@localhost /bin/sh -c 'mimencode -u -b > /tmp/outfile'

outside of Emacs.  Maybe the problem is in ssh.  But nevertheless,
Tramp needs to work around it.)

Thanks for any advice.
-- 
A turnip curses Elvis

Re: Connection drops data if it sent too quickly

[Michael Albinus]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> (Maybe Michael could try what happens when he does
> 
>   mimencode -b /tmp/largefile | \
>     ssh jrl@localhost /bin/sh -c 'mimencode -u -b > /tmp/outfile'
> 
> outside of Emacs.  Maybe the problem is in ssh.  But nevertheless,
> Tramp needs to work around it.)

ssh seems to be OK:

albinus@slbwba:[1512] ll /home/albinus/aaa ; md5 /home/albinus/aaa
-rw-rw-r--   1 albinus  smc3       376588 Jan 20 13:44 /home/albinus/aaa
MD5(/home/albinus/aaa)= 835f87fcd02a568f1c7a4454e7c2a0db

albinus@slbwba:[1513] mimencode -b /home/albinus/aaa | ssh slbwba '/appli/pub/bin/mimencode -u -b -o /tmp/aaa'

albinus@slbwba:[1514] ll /tmp/aaa ; md5 /tmp/aaa
-rw-r--r--   1 albinus  smc3       376588 Feb  4 11:46 /tmp/aaa
MD5(/tmp/aaa)= 835f87fcd02a568f1c7a4454e7c2a0db

Best regards, Michael.

Re: Connection drops data if it sent too quickly

[Richard Stallman]

    The problem is that the connection is dropping bytes when the data is
    sent too quickly.  I've changed it to send chunks of 500 bytes, then
    wait 0.1 seconds.  This works.

Perhaps we should put that into process-send-string so that Lisp code
won't need to concern itself with this issue.  But I wonder
whether the bug is in ssh.  If so, that's the place to fix it.

If Emacs is talking to some other program instead, does the same
bug happen?

    There seems to be code in process-send-region (or a function called
    from it) that tries to avoid sending data too quickly, but I don't
    grok it.

I probably don't understand it either, but I don't think the problem
being solved was one of losing data.  I think that the buffer would
get full and even worse things would happen.

Re: Connection drops data if it sent too quickly

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

> Perhaps we should put that into process-send-string so that Lisp code
> won't need to concern itself with this issue.  But I wonder
> whether the bug is in ssh.  If so, that's the place to fix it.
> 
> If Emacs is talking to some other program instead, does the same
> bug happen?

I've written some few lines which reproduce the problem under HP-UX
with all Emacsen available for me (20.7, 21.2, 21.2.93):

(with-temp-buffer
  (let ((bytes 1000)
	(proc (start-process (buffer-name) (current-buffer) "wc" "-c")))
    (process-send-string proc (make-string bytes ?x))
    (process-send-eof proc)
    (process-send-eof proc)
    (accept-process-output proc 1)
    (goto-char (point-min))
    (re-search-forward "\\w+")
    (message "Bytes sent: %s\tBytes received: %s" bytes (match-string 0))))

No problem under other Unices, like Solaris.

Best regards, Michael.

Re: Connection drops data if it sent too quickly

[Kim F. Storm]

Michael Albinus <Michael.Albinus@alcatel.de> writes:

> Richard Stallman <rms@gnu.org> writes:
> 
> > Perhaps we should put that into process-send-string so that Lisp code
> > won't need to concern itself with this issue.  But I wonder
> > whether the bug is in ssh.  If so, that's the place to fix it.
> > 
> > If Emacs is talking to some other program instead, does the same
> > bug happen?
> 
> I've written some few lines which reproduce the problem under HP-UX
> with all Emacsen available for me (20.7, 21.2, 21.2.93):
> 
> (with-temp-buffer
>   (let ((bytes 1000)
> 	(proc (start-process (buffer-name) (current-buffer) "wc" "-c")))
>     (process-send-string proc (make-string bytes ?x))
>     (process-send-eof proc)
>     (process-send-eof proc)
>     (accept-process-output proc 1)
>     (goto-char (point-min))
>     (re-search-forward "\\w+")
>     (message "Bytes sent: %s\tBytes received: %s" bytes (match-string 0))))
> 
> No problem under other Unices, like Solaris.

So this sounds like a HP-UX specific problem.

As such, it would be great if the current 0.1 second chunk delay (and
similar delay in tramp-send-linewise) was a defvar which could be set
to 0.0 on non-HPUX (and non-Tru64) platforms.  

That would make tramp run MUCH faster.  I tried this on GNU/Linux, and
it makes a huge difference!

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Connection drops data if it sent too quickly

[Michael Albinus]

storm@cua.dk (Kim F. Storm) writes:

> > I've written some few lines which reproduce the problem under HP-UX
> > with all Emacsen available for me (20.7, 21.2, 21.2.93):
> > 
> > (with-temp-buffer
> >   (let ((bytes 1000)
> > 	(proc (start-process (buffer-name) (current-buffer) "wc" "-c")))
> >     (process-send-string proc (make-string bytes ?x))
> >     (process-send-eof proc)
> >     (process-send-eof proc)
> >     (accept-process-output proc 1)
> >     (goto-char (point-min))
> >     (re-search-forward "\\w+")
> >     (message "Bytes sent: %s\tBytes received: %s" bytes (match-string 0))))
> > 
> > No problem under other Unices, like Solaris.
> 
> So this sounds like a HP-UX specific problem.

Not a native HP-UX problem, but a problem of Emacs process handling
under HP-UX. The same code works without any problem from XEmacs.

> As such, it would be great if the current 0.1 second chunk delay (and
> similar delay in tramp-send-linewise) was a defvar which could be set
> to 0.0 on non-HPUX (and non-Tru64) platforms.  
> 
> That would make tramp run MUCH faster.  I tried this on GNU/Linux, and
> it makes a huge difference!

Yes, this would make sense. Maybe an autodetection during startup of
Tramp, based on the code above.

Kai?

Best regards, Michael.

Re: Connection drops data if it sent too quickly

[Kai Großjohann]

Michael Albinus <Michael.Albinus@alcatel.de> writes:

> Yes, this would make sense. Maybe an autodetection during startup of
> Tramp, based on the code above.

Yes, this seems like a good idea.  Yesterday I was a bit confused,
but now I think it depends on the machine that Emacs is running on,
so checking on system-type might be the right thing to do.

What's the system-type under HP-UX?

I think that the run-time check you're proposing might be too
expensive, even though it is of course the right solution.

Can/will process-send-string (or friends) be fixed on HP-UX?
-- 
A turnip curses Elvis

Re: Connection drops data if it sent too quickly

[Richard Stallman]

    As such, it would be great if the current 0.1 second chunk delay (and
    similar delay in tramp-send-linewise) was a defvar which could be set
    to 0.0 on non-HPUX (and non-Tru64) platforms.  

Could someone possibly implement this at the C level in
process-send-string?  That is the clean place to do it,
so that Lisp programs need not worry about this issue.

Re: Connection drops data if it sent too quickly

[Richard Stallman]

    I've written some few lines which reproduce the problem under HP-UX
    with all Emacsen available for me (20.7, 21.2, 21.2.93):

Maybe this is a bug in HPUX.  If so, it would be nice to have code
inside process-send-string to work around the bug.  It could be under
a HPUX conditional.

But someone would have to write the code.