bug#4750: 23.1; copied files via dired/tramp/ssh garbled

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Andreas Politz]

Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs@gnu.org mailing list,
and to the gnu.emacs.bug news group.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

emacs -Q

I open 2 dired buffers, one local, one remote via ssh.  Marking a (any)
file in the local buffer and pressing 'C RET' starts to copy it to the
remote directory.

Only a part of the file reaches it's destination. Neither dired nor
tramp report any problems. 

Maybe it's a debian problem.  Anyways it must have appeared in one of
the last updates.

-ap

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/usr/share/emacs/23.1/etc/DEBUG for instructions.


In GNU Emacs 23.1.1 (x86_64-pc-linux-gnu, GTK+ Version 2.16.5)
 of 2009-08-03 on nautilus, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.10604000
configured using `configure  '--build=x86_64-linux-gnu' '--host=x86_64-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.1/site-lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/23.1/leim' '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 'build_alias=x86_64-linux-gnu' 'host_alias=x86_64-linux-gnu' 'CFLAGS=-DDEBIAN -g -O2' 'LDFLAGS=-g' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: C
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: nil
  default-enable-multibyte-characters: t

Major mode: Dired by name

Minor modes in effect:
  easy-repeat-mode: t
  recentf-mode: t
  ido-hacks-mode: t
  window-numbering-mode: t
  show-paren-mode: t
  icomplete-mode: t
  global-hi-lock-mode: t
  hi-lock-mode: t
  display-time-mode: t
  savehist-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
@ s s h . f h - t r i e r . d e : <return> C-. i C-h 
i C-h r i t r a m p <return> C-s C-g C-x d / s s h 
: p o l i t z a @ f f <backspace> <backspace> s s h 
. f h - t r i e r l <backspace> . d e : ~ <return> 
C-x 2 C-x d ~ / m a s t e r / b u k / u e b <tab> b 
u k . p <backspace> <backspace> <backspace> <backspace> 
<backspace> <return> C-x s C-x d C-a C-k C-/ M-d M-d 
M-d M-d M-d M-d M-d M-d M-d M-d ~ <return> C-s p d 
f <return> M-2 C-x o C-x o C-s m a s <return> <return> 
<return> R C-g D y e s <return> g C-x o C <return> 
C C-a C-k / s s M-p M-n C-p C-p h M-p M-n C-p C-p C-n 
C-p : p o l i t z a @ s s h - f <backspace> <backspace> 
. f h - t r i e r . d e : ~ / m a s t e r / b u k / 
<return> C-x o g C-M-b M-b M-b M-b M-b M-b M-f M-f 
g M-: C-g M-b C-M-b C-M-b C-M-b C-M-b C-M-SPC M-w M-: 
( * SPC 2 SPC C-y ) <return> M-x r e p o <tab> r t 
- e m a c s - b u f C-g M-x l o a d - f i l e <return> 
C-a C-k ~ / . e m a c s . d / i n i t . e l <return> 
M-x r e p o r <return>

Recent messages:
Loading /home/politza/.emacs.d/major-modes/axiom.el (source)...done
Loading /home/politza/.emacs.d/major-modes/haskell-mode-2.4/haskell-site-file.el (source)...done
Ido mode enabled
Source file `/home/politza/.emacs.d/minor-modes/ido-hacks.el' newer than byte-compiled file
Loading /home/politza/.recentf...done
Cleaning up the recentf list...
File /home/politza/master/buk/uebungen/uebungen.tex removed from the recentf list
File /home/politza/master/buk/uebungen/_region_.tex removed from the recentf list
Cleaning up the recentf list...done (2 removed)
Loading /home/politza/.emacs.d/init.el (source)...done

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Michael Albinus]

Andreas Politz <politza@fh-trier.de> writes:

> I open 2 dired buffers, one local, one remote via ssh.  Marking a (any)
> file in the local buffer and pressing 'C RET' starts to copy it to the
> remote directory.
>
> Only a part of the file reaches it's destination. Neither dired nor
> tramp report any problems. 

Could you, please, set `tramp-verbose' to 8, and repeat the test?
Afterwards, I would like to see the Tramp debg buffer.

> -ap

Best regards, Michael.

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Andreas Politz]

[-- Attachment #1: Type: text/plain, Size: 756 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> Andreas Politz <politza@fh-trier.de> writes:
>
>> I open 2 dired buffers, one local, one remote via ssh.  Marking a (any)
>> file in the local buffer and pressing 'C RET' starts to copy it to the
>> remote directory.
>>
>> Only a part of the file reaches it's destination. Neither dired nor
>> tramp report any problems. 
>
> Could you, please, set `tramp-verbose' to 8, and repeat the test?
> Afterwards, I would like to see the Tramp debg buffer.
>
>> -ap
>
> Best regards, Michael.

I copied the 2 shell functions into a script and executed it locally, on
the remote machine and via `ssh host command'.  No problems there, all
this methods generate the complete file.

Anyway here is the debug log.



[-- Attachment #2: tramp.debug --]
[-- Type: application/octet-stream, Size: 135580 bytes --]

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Michael Albinus]

Andreas Politz <politza@fh-trier.de> writes:

>>> I open 2 dired buffers, one local, one remote via ssh.  Marking a (any)
>>> file in the local buffer and pressing 'C RET' starts to copy it to the
>>> remote directory.
>>>
>>> Only a part of the file reaches it's destination. Neither dired nor
>>> tramp report any problems. 
>>
>> Could you, please, set `tramp-verbose' to 8, and repeat the test?
>> Afterwards, I would like to see the Tramp debg buffer.
>>
>>> -ap
>>
>> Best regards, Michael.
>
> I copied the 2 shell functions into a script and executed it locally, on
> the remote machine and via `ssh host command'.  No problems there, all
> this methods generate the complete file.
>
> Anyway here is the debug log.

OK, the problem is clear I believe.

> 12:29:01 tramp-send-command (6) # echo \"`uname -sr`\" 2>/dev/null; echo tramp_exit_status $?
> 12:29:01 tramp-wait-for-regexp (6) #
> "AIX 3"
> tramp_exit_status 0
>
> ///643be83973c0506db98f651a756e0efc

Your remote hosts runs AIX. Honestly, this is one of the systems we run
into trouble from time to time ...

> 12:29:44 tramp-send-command (6) # tramp_perl_decode >/u/f0/students/politza/buk.log <<'EOF'
> VGhpcyBpcyBwZGZUZVhrLCBWZXJzaW9uIDMuMTQxNTkyLTEuNDAuMyAoV2ViMkMgNy41LjYpIChm
> b3JtYXQ9cGRmbGF0ZXggMjAwOS4xMC4zKSAgMTggT0NUIDIwMDkgMjA6NTkKZW50ZXJpbmcgZXh0

...

> MDAwIChtYXguIDEwMDAwMDAwKQoK
> EOF

I've taken the whole encoded string from your message, and I've applied
"M-x base64-decode-region" on the region. It decoded fine, to 33024
bytes. But ...

> 12:29:49 tramp-send-command (6) # /usr/bin/ls -d -Al -d buk.log
> 12:29:49 tramp-wait-for-regexp (6) #
> -rwxrwxrwx   1 politza  infstd     26940 Oct 18 18:59 buk.log
>
> ///643be83973c0506db98f651a756e0efc

... there have been copied less bytes.

There is a known bug in `process-send-string'. For some remote systems,
it doesn't send the whole string. You seem to suffer from this.

You might try a workaround, using `tramp-chunksize'. Read the docstring
of that variable, in order to find a way how to set it for your case.

Furthermore, I recommend you to set the variable `file-precious-flag' to
t. In the Tramp case, it adds additional checks, whether a file has been
written correctly.

Best regards, Michael.

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Andreas Politz]

Michael Albinus <michael.albinus@gmx.de> writes:

> Andreas Politz <politza@fh-trier.de> writes:
>
>>>> I open 2 dired buffers, one local, one remote via ssh.  Marking a (any)
>>>> file in the local buffer and pressing 'C RET' starts to copy it to the
>>>> remote directory.
>>>>
>>>> Only a part of the file reaches it's destination. Neither dired nor
>>>> tramp report any problems. 
>>>
>>
[...]
>> Anyway here is the debug log.
>
> OK, the problem is clear I believe.
>
[...]
> Your remote hosts runs AIX. Honestly, this is one of the systems we run
> into trouble from time to time ...
>
[...]
> ... there have been copied less bytes.
>
> There is a known bug in `process-send-string'. For some remote systems,
> it doesn't send the whole string. You seem to suffer from this.
>
> You might try a workaround, using `tramp-chunksize'. Read the docstring
> of that variable, in order to find a way how to set it for your case.

The method described there finds no flaws, up to 5000 bytes anyway.
Nevertheless I tested `tramp-chunksize' with some values
(10,1000,4000,10000), but no change.

It seems to constantly fail the very first time after a connection is
established.  
>
> Furthermore, I recommend you to set the variable `file-precious-flag' to
> t. In the Tramp case, it adds additional checks, whether a file has been
> written correctly.

At least there will be no surprises anymore,

thanks.

A.Politz

>
> Best regards, Michael.

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Andreas Politz]

Michael Albinus <michael.albinus@gmx.de> writes:

> Andreas Politz <politza@fh-trier.de> writes:
>
>>>> I open 2 dired buffers, one local, one remote via ssh.  Marking a (any)
>>>> file in the local buffer and pressing 'C RET' starts to copy it to the
>>>> remote directory.
>>>>
>>>> Only a part of the file reaches it's destination. Neither dired nor
>>>> tramp report any problems. 
>>>
>>
[...]
>> Anyway here is the debug log.
>
> OK, the problem is clear I believe.
>
[...]
> Your remote hosts runs AIX. Honestly, this is one of the systems we run
> into trouble from time to time ...
>
[...]
> ... there have been copied less bytes.
>
> There is a known bug in `process-send-string'. For some remote systems,
> it doesn't send the whole string. You seem to suffer from this.
>
> You might try a workaround, using `tramp-chunksize'. Read the docstring
> of that variable, in order to find a way how to set it for your case.

The method described there finds no flaws, up to 5000 bytes anyway.
Nevertheless I tested `tramp-chunksize' with some values
(10,1000,4000,10000), but no change.

It seems to constantly fail the very first time after a connection is
established.  
>
> Furthermore, I recommend you to set the variable `file-precious-flag' to
> t. In the Tramp case, it adds additional checks, whether a file has been
> written correctly.

At least there will be no surprises anymore,

thanks.

A.Politz

>
> Best regards, Michael.

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Michael Albinus]

Andreas Politz <politza@fh-trier.de> writes:

> It seems to constantly fail the very first time after a connection is
> established.  

Does it mean, it happens *only* the very first time, and it copies then
correctly? That would be very strange ...

> thanks.
>
> A.Politz

Best regards, Michael.

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Andreas Politz]

Michael Albinus <michael.albinus@gmx.de> writes:

> Andreas Politz <politza@fh-trier.de> writes:
>
>> It seems to constantly fail the very first time after a connection is
>> established.  
>
> Does it mean, it happens *only* the very first time, and it copies then
> correctly? That would be very strange ...

No, after the very first fail it's rather undeterministic.

It could be, that it was just ,luck' (I tested about 10 times.) and has
nothing to do with it.

>
>> thanks.
>>
>> A.Politz
>
> Best regards, Michael.

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Michael Albinus]

Andreas Politz <politza@fh-trier.de> writes:

>> Does it mean, it happens *only* the very first time, and it copies then
>> correctly? That would be very strange ...
>
> No, after the very first fail it's rather undeterministic.
>
> It could be, that it was just ,luck' (I tested about 10 times.) and has
> nothing to do with it.

OK. What you can do is to use "scp" (or "scpc") instead of "ssh". This
uses scp for copying the files, instead of applying an internal
encoding/decoding.

Due to optimization reasons, "scp" also applies internal encoding/
decoding for small files. This can be changed by decreasing the value of
`tramp-copy-size-limit'. A value of 0 disables this optimization
completely.

Best regards, Michael.

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Michael Albinus]

Andreas Politz <politza@fh-trier.de> writes:

>> Does it mean, it happens *only* the very first time, and it copies then
>> correctly? That would be very strange ...
>
> No, after the very first fail it's rather undeterministic.
>
> It could be, that it was just ,luck' (I tested about 10 times.) and has
> nothing to do with it.

This is a very old bug report. Do you still see this problem?

Best regards, Michael.

bug#4750: 23.1; copied files via dired/tramp/ssh garbled

[Michael Albinus]

Andreas Politz <politza@hochschule-trier.de> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> This is a very old bug report. Do you still see this problem?
>
> No, but my university switched os awhile go. No more AIX.

Well, so we have no chance to retest. AIX has always been problematic
for Tramp, and some AIX specific tweaks were added after you have
submitted the report. Maybe it is fixed now; I haven't seen more related
problem reports.

Closing the bug.

> -ap

Best regards, Michael.