TRAMP corrupting files

TRAMP corrupting files

[Will Willis]

I'm using Tramp from my Mac to edit remote files. Twice in the last
few days I've experienced Tramp corrupting my files.

The first corruption happened mid way through a PHP file, looking like this:

...
 <option value="C"<?=$uresult['store_cluster']=="C"?"
selected":""?>>Cluster C</option>
			  <opv˷⌀ “䛆需戀܀㜀䜀阀

I think the characters might have been encoded above. What I see in
emacs is this

...
 <option value="C"<?=$uresult['store_cluster']=="C"?"
selected":""?>>Cluster C</option>
			  <opv\367^B#\340\240\220\223\306...

This corruption goes on for 35,202 characters (all on one line) then
the file ends.


The second time I had a file get corrupted, interestingly also in a
PHP file inside of a <select> option group, There were not funky
characters, just a few dozen lines of missing from my file.

It's really starting to scare me as I'm not sure I can trust this
method of remote file editing... My hosting company was able to assist
me once in recovering a backup... I'd have for this to happen again..

Is TRAMP known to do this sort of thing?

Are there Emacs alternatives to remote file editing (w/o corruption)?

Thanks.

Carbon Emacs  "22.2.1"
Tramp "2.0.57"

Re: TRAMP corrupting files

[Anselm Helbig]

Hi!

> I'm using Tramp from my Mac to edit remote files. Twice in the last
> few days I've experienced Tramp corrupting my files.
> 
> The first corruption happened mid way through a PHP file, looking like this:
> 
> ...
>  <option value="C"<?=$uresult['store_cluster']=="C"?"
> selected":""?>>Cluster C</option>
> 			  <opv˷⌀ “䛆需戀܀㜀䜀阀
> 
> I think the characters might have been encoded above. What I see in
> emacs is this
> 
> ...
>  <option value="C"<?=$uresult['store_cluster']=="C"?"
> selected":""?>>Cluster C</option>
> 			  <opv\367^B#\340\240\220\223\306...
> 
> This corruption goes on for 35,202 characters (all on one line) then
> the file ends.
> 
> 
> The second time I had a file get corrupted, interestingly also in a
> PHP file inside of a <select> option group, There were not funky
> characters, just a few dozen lines of missing from my file.
> 
> It's really starting to scare me as I'm not sure I can trust this
> method of remote file editing... My hosting company was able to assist
> me once in recovering a backup... I'd have for this to happen again..
> 
> Is TRAMP known to do this sort of thing?
> 
> Are there Emacs alternatives to remote file editing (w/o corruption)?
> 
> Thanks.
> 
> Carbon Emacs  "22.2.1"
> Tramp "2.0.57"

I didn't have these kinds of problem. But TRAMP can definitely be
fragile under some circumstances - it's a complex thing to handle all
possible remote hosts, terminal settings and connection methods. 

There's a lot of configuration options for TRAMP you could try, i.e.
use the scp/sftp external transfer methods. This slows things down a
bit unless you're using OpenSSHs ControlMaster feature. 

Another option might be the experimental "fish" connection method. 

TRAMP relies on the mimencode and uuencode/uudecode commands on the
remote host for transferring your files. What remote host are you
connecting to? What OS does it run? Is mimencode present? Can you have
it installed on the remote host? Maybe this will solve your problem. 

For alternatives to TRAMP, see

  http://www.emacswiki.org/emacs/CategoryRemoteEmacs

You could also mount the remote directory via fuse with sshfs on
Linux. For the Mac there's macfuse:

  http://code.google.com/p/macfuse/

You're losing a bit of emacs integration here: i.e. if you're using
eshell and cd to a remote TRAMP directory, all commands that you issue
are executed are really sent to the remote host. This can not work
anymore with fuse. Also there's integration with version control and
TRAMP.

You could also change your workflow completely, set up version control
and a development system on your local computer and push changes to
the remote host in a more controlled fashion. 

HTH,

Anselm


-- 
Anselm Helbig 
mailto:anselm.helbig+news2009@googlemail.com

Re: TRAMP corrupting files

[Will Willis]

Thanks for all the suggestions.

I'm on a Mac, and the remote host is BSD. I'll give Fuse a try, as I
don't really issue commands via eshell on the remote host.

Thanks again,
Will

On Fri, May 8, 2009 at 3:21 AM, Anselm Helbig
<anselm.helbig+news2009@googlemail.com> wrote:
> Hi!
>
>> I'm using Tramp from my Mac to edit remote files. Twice in the last
>> few days I've experienced Tramp corrupting my files.
>>
>> The first corruption happened mid way through a PHP file, looking like this:
>>
>> ...
>>  <option value="C"<?=$uresult['store_cluster']=="C"?"
>> selected":""?>>Cluster C</option>
>>                         <opv˷⌀   “䛆需戀܀㜀䜀阀
>>
>> I think the characters might have been encoded above. What I see in
>> emacs is this
>>
>> ...
>>  <option value="C"<?=$uresult['store_cluster']=="C"?"
>> selected":""?>>Cluster C</option>
>>                         <opv\367^B#\340\240\220\223\306...
>>
>> This corruption goes on for 35,202 characters (all on one line) then
>> the file ends.
>>
>>
>> The second time I had a file get corrupted, interestingly also in a
>> PHP file inside of a <select> option group, There were not funky
>> characters, just a few dozen lines of missing from my file.
>>
>> It's really starting to scare me as I'm not sure I can trust this
>> method of remote file editing... My hosting company was able to assist
>> me once in recovering a backup... I'd have for this to happen again..
>>
>> Is TRAMP known to do this sort of thing?
>>
>> Are there Emacs alternatives to remote file editing (w/o corruption)?
>>
>> Thanks.
>>
>> Carbon Emacs  "22.2.1"
>> Tramp "2.0.57"
>
> I didn't have these kinds of problem. But TRAMP can definitely be
> fragile under some circumstances - it's a complex thing to handle all
> possible remote hosts, terminal settings and connection methods.
>
> There's a lot of configuration options for TRAMP you could try, i.e.
> use the scp/sftp external transfer methods. This slows things down a
> bit unless you're using OpenSSHs ControlMaster feature.
>
> Another option might be the experimental "fish" connection method.
>
> TRAMP relies on the mimencode and uuencode/uudecode commands on the
> remote host for transferring your files. What remote host are you
> connecting to? What OS does it run? Is mimencode present? Can you have
> it installed on the remote host? Maybe this will solve your problem.
>
> For alternatives to TRAMP, see
>
>  http://www.emacswiki.org/emacs/CategoryRemoteEmacs
>
> You could also mount the remote directory via fuse with sshfs on
> Linux. For the Mac there's macfuse:
>
>  http://code.google.com/p/macfuse/
>
> You're losing a bit of emacs integration here: i.e. if you're using
> eshell and cd to a remote TRAMP directory, all commands that you issue
> are executed are really sent to the remote host. This can not work
> anymore with fuse. Also there's integration with version control and
> TRAMP.
>
> You could also change your workflow completely, set up version control
> and a development system on your local computer and push changes to
> the remote host in a more controlled fashion.
>
> HTH,
>
> Anselm
>
>
> --
> Anselm Helbig
> mailto:anselm.helbig+news2009@googlemail.com
>

Re: TRAMP corrupting files

[Ian Eure]

On May 7, 2009, at 10:44 PM, Will Willis wrote:

> I'm using Tramp from my Mac to edit remote files. Twice in the last
> few days I've experienced Tramp corrupting my files.
>
> The first corruption happened mid way through a PHP file, looking  
> like this:
>
> ...
> <option value="C"<?=$uresult['store_cluster']=="C"?"
> selected":""?>>Cluster C</option>
> 			  <opv˷⌀ “䛆需戀܀㜀䜀阀
>
> I think the characters might have been encoded above. What I see in
> emacs is this
>
> ...
> <option value="C"<?=$uresult['store_cluster']=="C"?"
> selected":""?>>Cluster C</option>
> 			  <opv\367^B#\340\240\220\223\306...
>
> This corruption goes on for 35,202 characters (all on one line) then
> the file ends.
>
>
> The second time I had a file get corrupted, interestingly also in a
> PHP file inside of a <select> option group, There were not funky
> characters, just a few dozen lines of missing from my file.
>
> It's really starting to scare me as I'm not sure I can trust this
> method of remote file editing... My hosting company was able to assist
> me once in recovering a backup... I'd have for this to happen again..
>
> Is TRAMP known to do this sort of thing?
>
For what it's worth, I live and breathe Tramp. The vast majority of my  
editing happens remotely via tramp, and I have _never_ experienced  
anything like what you have. Which is not to say it can never happen,  
but I don't think it should in the normal course of events.

I'm currently using CVS snapshots of Carbon Emacs 23 on two Macs.  
Before that, I was using Aquamacs, and XEmacs 20/21 on Debian before  
that. The remote hosts are a mix of Lenny and Etch Debian boxes. I've  
used the rsync and ssh Tramp methods.

  - Ian

Re: TRAMP corrupting files

[Will Willis]

On Mon, May 18, 2009 at 5:35 PM, Ian Eure <ian@digg.com> wrote:
> On May 7, 2009, at 10:44 PM, Will Willis wrote:
>
>> I'm using Tramp from my Mac to edit remote files. Twice in the last
>> few days I've experienced Tramp corrupting my files.
>>
>> The first corruption happened mid way through a PHP file, looking like
>> this:
>>
>> ...
>> <option value="C"<?=$uresult['store_cluster']=="C"?"
>> selected":""?>>Cluster C</option>
>>                          <opv˷⌀   “䛆需戀܀㜀䜀阀
>>
>> I think the characters might have been encoded above. What I see in
>> emacs is this
>>
>> ...
>> <option value="C"<?=$uresult['store_cluster']=="C"?"
>> selected":""?>>Cluster C</option>
>>                          <opv\367^B#\340\240\220\223\306...
>>
>> This corruption goes on for 35,202 characters (all on one line) then
>> the file ends.
>>
>>
>> The second time I had a file get corrupted, interestingly also in a
>> PHP file inside of a <select> option group, There were not funky
>> characters, just a few dozen lines of missing from my file.
>>
>> It's really starting to scare me as I'm not sure I can trust this
>> method of remote file editing... My hosting company was able to assist
>> me once in recovering a backup... I'd have for this to happen again..
>>
>> Is TRAMP known to do this sort of thing?
>>
> For what it's worth, I live and breathe Tramp. The vast majority of my
> editing happens remotely via tramp, and I have _never_ experienced anything
> like what you have. Which is not to say it can never happen, but I don't
> think it should in the normal course of events.
>
> I'm currently using CVS snapshots of Carbon Emacs 23 on two Macs. Before
> that, I was using Aquamacs, and XEmacs 20/21 on Debian before that. The
> remote hosts are a mix of Lenny and Etch Debian boxes. I've used the rsync
> and ssh Tramp methods.
>
>  - Ian

I'm going to start using rsync to see if that helps.. I wish I were
making this up. I've had files corrupt on me as many as 3 times in one
night. I'd love to upgrade to Carbon Emacs 23, but I'm still on OS X
10.4, holding out for snow leopard.

-Will

Re: TRAMP corrupting files (it's OS X's fault)

[Daniel Colascione]

On May 18, 2009, at 9:40 PM, Will Willis wrote:
> I'm going to start using rsync to see if that helps.. I wish I were
> making this up. I've had files corrupt on me as many as 3 times in one

I ran into the same problem. It's actually OS X's fault: OS X (at  
least 10.5) doesn't give Emacs any locale environment variables. Emacs  
guesses it's running under 8859-1, which has disastrous consequences  
for tramp. (If you want to test it, run locale under eshell. In the  
terminal, you see the expected locale output because Terminal.app  
*itself* sets up the correct environment before running your shell.)

One solution is to just give Emacs the right environment variables.  
I've outlined how to do that here:

http://qtmstr.livejournal.com/19155.html

(Modify the specific environment variable values according to taste.)

Another approach is to add this to your Emacs initialization:

(when (eq system-type 'darwin)
     (prefer-coding-system 'utf-8))

These fixed the corruption issue *I* was having. YMMV.

> I'd love to upgrade to Carbon Emacs 23, but I'm still on OS X
> 10.4, holding out for snow leopard.

There is no CarbonEmacs 23. The OS X support will be provided via a  
new Cocoa port which, to put it kindly, is broken. You might want to  
stick with CarbonEmacs for now.

Re: TRAMP corrupting files (it's OS X's fault)

[Will Willis]

Thank you so much for the suggestion!!

-Will

On Tue, May 19, 2009 at 12:17 AM, Daniel Colascione
<danc@merrillpress.com> wrote:
> On May 18, 2009, at 9:40 PM, Will Willis wrote:
>>
>> I'm going to start using rsync to see if that helps.. I wish I were
>> making this up. I've had files corrupt on me as many as 3 times in one
>
> I ran into the same problem. It's actually OS X's fault: OS X (at least
> 10.5) doesn't give Emacs any locale environment variables. Emacs guesses
> it's running under 8859-1, which has disastrous consequences for tramp. (If
> you want to test it, run locale under eshell. In the terminal, you see the
> expected locale output because Terminal.app *itself* sets up the correct
> environment before running your shell.)
>
> One solution is to just give Emacs the right environment variables. I've
> outlined how to do that here:
>
> http://qtmstr.livejournal.com/19155.html
>
> (Modify the specific environment variable values according to taste.)
>
> Another approach is to add this to your Emacs initialization:
>
> (when (eq system-type 'darwin)
>    (prefer-coding-system 'utf-8))
>
> These fixed the corruption issue *I* was having. YMMV.
>
>> I'd love to upgrade to Carbon Emacs 23, but I'm still on OS X
>> 10.4, holding out for snow leopard.
>
> There is no CarbonEmacs 23. The OS X support will be provided via a new
> Cocoa port which, to put it kindly, is broken. You might want to stick with
> CarbonEmacs for now.
>

Re: TRAMP corrupting files (it's OS X's fault)

[Michael Albinus]

Daniel Colascione <danc@merrillpress.com> writes:

> On May 18, 2009, at 9:40 PM, Will Willis wrote:
>> I'm going to start using rsync to see if that helps.. I wish I were
>> making this up. I've had files corrupt on me as many as 3 times in one
>
> I ran into the same problem. It's actually OS X's fault: OS X (at
> least 10.5) doesn't give Emacs any locale environment variables. Emacs
> guesses it's running under 8859-1, which has disastrous consequences
> for tramp. (If you want to test it, run locale under eshell. In the
> terminal, you see the expected locale output because Terminal.app
> *itself* sets up the correct environment before running your shell.)

Tramp cannot solve the problem, as said. But it might be worth to know,
that Tramp supports the `file-precious-flag' variable. If it is set to
`t', Tramp signals an error, when the remote file is corrupted during
copying.

Best regards, Michael.

Re: TRAMP corrupting files (it's OS X's fault)

[Daniel Colascione]

On Tuesday 26 May 2009, Michael Albinus wrote:
> Tramp cannot solve the problem, as said. But it might be worth to know,
> that Tramp supports the `file-precious-flag' variable. If it is set to
> `t', Tramp signals an error, when the remote file is corrupted during
> copying.

Why does file-precious-flag even exist? Isn't the behavior you get when it's t the behavior you *always* want when the combination of the various backup-by-* variables allows it?

Re: TRAMP corrupting files (it's OS X's fault)

[Michael Albinus]

Daniel Colascione <danc@merrillpress.com> writes:

> On Tuesday 26 May 2009, Michael Albinus wrote:
>> Tramp cannot solve the problem, as said. But it might be worth to know,
>> that Tramp supports the `file-precious-flag' variable. If it is set to
>> `t', Tramp signals an error, when the remote file is corrupted during
>> copying.
>
> Why does file-precious-flag even exist? Isn't the behavior you get
> when it's t the behavior you *always* want when the combination of the
> various backup-by-* variables allows it?

Yes, you can have the same effect with backup-by-* variables. If you
read the code of backup-buffer, you see

      (if (or file-precious-flag
	      backup-by-copying
	      (and backup-by-copying-when-linked
		   (> (file-nlinks real-file-name) 1))
	      (and (or backup-by-copying-when-mismatch
		       (integerp backup-by-copying-when-privileged-mismatch))
              ...

I believe, there are historical reasons for the existence of that
variable. Tramp just has occupied the variable because it exists,
avoiding introduction of another customer option.

Best regards, Michael.