bug#7383: 24.0.50; end-of-line style on remote files

bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

It seems that Emacs doesn't respect the end-of-line style when dealing
with remote files.

Try this:
1.- Start Emacs (emacs -Q)
2.- Open a remote file, which has DOS end-of-line style (<cr><lf>).
3.- Make a change to the file and save it.
4.- The modeline still says that the end-of-line style is DOS (char
"\" on Windows version), but in fact it isn't. It has changed to UNIX
style (<lf>).

...and even worse:
5.- M-x revert-buffer.
6.- The modeline keeps unchanged (like in step 4). In order to get the
correct modeline flag, you have to kill the buffer and re-visit the
file.

---------------------------------------
Dani.


In GNU Emacs 24.0.50.1 (i386-mingw-nt5.1.2600)
 of 2010-11-09 on 3249CTO
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (4.4) --no-opt --cflags
-Ic:/imagesupport/include'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ESP
  value of $XMODIFIERS: nil
  locale-coding-system: cp1252
  default enable-multibyte-characters: t

Major mode: Conf[Space]

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

Ping!

Has anyone take a look at this bug report?

I'm trying to use GNU Emacs for all my editing at my workplace, and
this bug is really annoying, because I often have to edit remote files
with both UNIX and DOS EOL format, and whenever I save a remote
DOS-format file, that file ends up having UNIX EOL format, which is
unacceptable.

I'd very much appreciate a solution for that.

TIA.

-- 
Dani Moncayo

bug#7383: 24.0.50; end-of-line style on remote files

[Michael Albinus]

Dani Moncayo <dmoncayo@gmail.com> writes:

> Ping!
>
> Has anyone take a look at this bug report?

Oops, I've overlooked this. Thanks for the reminder.

> It seems that Emacs doesn't respect the end-of-line style when dealing
> with remote files.
>
> Try this:
> 1.- Start Emacs (emacs -Q)
> 2.- Open a remote file, which has DOS end-of-line style (<cr><lf>).
> 3.- Make a change to the file and save it.
> 4.- The modeline still says that the end-of-line style is DOS (char
> "\" on Windows version), but in fact it isn't. It has changed to UNIX
> style (<lf>).
>
> ...and even worse:
> 5.- M-x revert-buffer.
> 6.- The modeline keeps unchanged (like in step 4). In order to get the
> correct modeline flag, you have to kill the buffer and re-visit the
> file.

Your local Emacs runs on Windows XP. Which connection method do you use
to connect to the remote machine (plink? pscp?)?

Reading tramp-sh.el, Tramp seems to set eol conversion for the process
communication. For the transferred files, I couldn't see any eol
conversion settings in the code.

I'll continue to dig.

> TIA.

Best regards, Michael.

bug#7383: Fwd: bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

[I'm forwarding the bellow message to the mailing list, because I
unintentionally replied privately.]


---------- Forwarded message ----------
From: Dani Moncayo <dmoncayo@gmail.com>
Date: Thu, Apr 28, 2011 at 12:53
Subject: Re: bug#7383: 24.0.50; end-of-line style on remote files
To: Michael Albinus <michael.albinus@gmx.de>


On Thu, Apr 28, 2011 at 12:17, Michael Albinus <michael.albinus@gmx.de> wrote:
>
> Your local Emacs runs on Windows XP.

Yes.


> Which connection method do you use to connect to the remote machine (plink? pscp?)?

I don't know. I'm using the latest pre-compiled version for windows
published by Sean [1], and I've not set up anything special beyond
that.

[1] http://alpha.gnu.org/gnu/emacs/windows/


>
> I'll continue to dig.
>

Thanks a lot.

-- 
Dani Moncayo

bug#7383: Fwd: bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

[I'm forwarding some messages to the mailing list, because I
unintentionally replied privately.]


---------- Forwarded message ----------
From: Dani Moncayo <dmoncayo@gmail.com>
Date: Thu, Apr 28, 2011 at 12:59
Subject: Re: bug#7383: 24.0.50; end-of-line style on remote files
To: Michael Albinus <michael.albinus@gmx.de>


On Thu, Apr 28, 2011 at 12:53, Dani Moncayo <dmoncayo@gmail.com> wrote:
> On Thu, Apr 28, 2011 at 12:17, Michael Albinus <michael.albinus@gmx.de> wrote:
>
>> Which connection method do you use to connect to the remote machine (plink? pscp?)?
>

Sorry, did you mean "which TRAMP method"? If so, I use the default:
 C-x C-f /user@host:/path-to-file


-- 
Dani Moncayo

bug#7383: Fwd: bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

[I'm forwarding some messages to the mailing list, because I
unintentionally replied privately.]


---------- Forwarded message ----------
From: Michael Albinus <michael.albinus@gmx.de>
Date: Thu, Apr 28, 2011 at 13:17
Subject: Re: bug#7383: 24.0.50; end-of-line style on remote files
To: Dani Moncayo <dmoncayo@gmail.com>


Dani Moncayo <dmoncayo@gmail.com> writes:

> Sorry, did you mean "which TRAMP method"? If so, I use the default:
>   C-x C-f /user@host:/path-to-file

This defaults to "pscp". You could check it with "C-h v tramp-default-method".

If this is the case, could you please eval

(setq tramp-copy-size-limit nil)

and check afterwards, whether the problem still happens?

Best regards, Michael.

bug#7383: Fwd: bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

[I'm forwarding some messages to the mailing list, because I
unintentionally replied privately.]


---------- Forwarded message ----------
From: Dani Moncayo <dmoncayo@gmail.com>
Date: Thu, Apr 28, 2011 at 15:36
Subject: Re: bug#7383: 24.0.50; end-of-line style on remote files
To: Michael Albinus <michael.albinus@gmx.de>


On Thu, Apr 28, 2011 at 13:17, Michael Albinus <michael.albinus@gmx.de> wrote:
> Dani Moncayo <dmoncayo@gmail.com> writes:
>
>> Sorry, did you mean "which TRAMP method"? If so, I use the default:
>>   C-x C-f /user@host:/path-to-file
>
> This defaults to "pscp". You could check it with "C-h v tramp-default-method".
>

Its value is "ftp"


> If this is the case, could you please eval
>
> (setq tramp-copy-size-limit nil)
>
> and check afterwards, whether the problem still happens?

Yes, it still happens after doing that.


Here is another example of bad behavior:
 1. Create a new buffer (on Windows, it will have DOS format).
 2. Save the buffer to a remote location (C-x C-s /user@host:/path-to-file).

Then, I see the same two problems described in the OP:
 a.- The modeline indicator is wrong: It says that the file has DOS
format (in fact has UNIX format).
 b.- The new file has been created with UNIX format (it should be DOS format).


--
Dani Moncayo

bug#7383: Fwd: bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

[I'm forwarding some messages to the mailing list, because I
unintentionally replied privately.]


---------- Forwarded message ----------
From: Michael Albinus <michael.albinus@gmx.de>
Date: Thu, Apr 28, 2011 at 15:54
Subject: Re: bug#7383: 24.0.50; end-of-line style on remote files
To: Dani Moncayo <dmoncayo@gmail.com>


Dani Moncayo <dmoncayo@gmail.com> writes:

> On Thu, Apr 28, 2011 at 13:17, Michael Albinus <michael.albinus@gmx.de> wrote:
>> Dani Moncayo <dmoncayo@gmail.com> writes:
>>
>>> Sorry, did you mean "which TRAMP method"? If so, I use the default:
>>>   C-x C-f /user@host:/path-to-file
>>
>> This defaults to "pscp". You could check it with "C-h v tramp-default-method".
>
> Its value is "ftp"

Then we are speaking about ange-ftp, and not about Tramp.

Does the same effect (EOL conversion) happens, when you copy files with
your local FTP client?

Best regards, Michael.

bug#7383: Fwd: bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

[I'm forwarding some messages to the mailing list, because I
unintentionally replied privately.]


---------- Forwarded message ----------
From: Dani Moncayo <dmoncayo@gmail.com>
Date: Thu, Apr 28, 2011 at 16:30
Subject: Re: bug#7383: 24.0.50; end-of-line style on remote files
To: Michael Albinus <michael.albinus@gmx.de>


On Thu, Apr 28, 2011 at 15:54, Michael Albinus <michael.albinus@gmx.de> wrote:
>
> Does the same effect (EOL conversion) happens, when you copy files with
> your local FTP client?
>

I don't know which ftp client is Emacs using under the hood. I guess
it's the standard windows "ftp" program. In that case, I've done a
quick test and found that, if I set binary mode on, the file got by
ftp is the exact same as the original (which doesn't happen without
setting binary mode on).

--
Dani Moncayo

bug#7383: Fwd: bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

[I'm forwarding some messages to the mailing list, because I
unintentionally replied privately.]


---------- Forwarded message ----------
From: Michael Albinus <michael.albinus@gmx.de>
Date: Sun, May 1, 2011 at 12:43
Subject: Re: bug#7383: 24.0.50; end-of-line style on remote files
To: Dani Moncayo <dmoncayo@gmail.com>


Dani Moncayo <dmoncayo@gmail.com> writes:

> I don't know which ftp client is Emacs using under the hood. I guess
> it's the standard windows "ftp" program. In that case, I've done a
> quick test and found that, if I set binary mode on, the file got by
> ftp is the exact same as the original (which doesn't happen without
> setting binary mode on).

From ange-ftp.el:

;; Binary file transfers:
;;
;; By default ange-ftp transfers files in ASCII mode.  If a file being
;; transferred matches the value of ange-ftp-binary-file-name-regexp then
;; binary mode is used for that transfer.

Maybe you play with `ange-ftp-binary-file-name-regexp' settings, for
example set it to "".

Best regards, Michael.

bug#7383: Fwd: bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

[I'm forwarding some messages to the mailing list, because I
unintentionally replied privately.]


---------- Forwarded message ----------
From: Dani Moncayo <dmoncayo@gmail.com>
Date: Wed, May 4, 2011 at 11:32
Subject: Re: bug#7383: 24.0.50; end-of-line style on remote files
To: Michael Albinus <michael.albinus@gmx.de>


On Sun, May 1, 2011 at 12:43, Michael Albinus <michael.albinus@gmx.de> wrote:
>
> Maybe you play with `ange-ftp-binary-file-name-regexp' settings, for
> example set it to "".
>

Sorry for the delay. These days I had no access to the Windows XP box
where I reproduce the problem.  (BTW, at home I was unable to
reproduce it using a Windows 7 client and  Ubuntu 10.10 ftp server).

Well, if I do what you said ((setq ange-ftp-binary-file-name-regexp
"")), then the problem seem to disappear in the quick test I've done.

So now I wonder two things:

1. In what cases do we want to have text-mode ftp transfers when
working with remote files? Because, IMO TRT would be to always use
binary mode, i.e., always work with the file in its actual EOL format.
I think that is what users expect, don't they?

2. This bug report has showed that the modeline flag corresponding to
the EOL format isn't always correct. So I think this should be fixed
if possible, and if not, this limitation should be documented at
least.


Thanks for your time, Michael.

--
Dani Moncayo

bug#7383: 24.0.50; end-of-line style on remote files

[Michael Albinus]

Dani Moncayo <dmoncayo@gmail.com> writes:

> So now I wonder two things:
> 1. In what cases do we want to have text-mode ftp transfers when
> working with remote files?
>    Because, IMO TRT would be to always use binary mode, i.e., always
> work with the file
>    in its actual EOL format. I think that is what users expect, don't they?

See the code of `ange-ftp-write-region'. Binary transfer is explicitely
disabled for Emacsen on MS Windows machines, if not said otherwise by
`ange-ftp-binary-file-name-regexp':

	       (binary (or (ange-ftp-binary-file filename)
			   (and (not (memq system-type
					   '(ms-dos windows-nt)))
				(memq (ange-ftp-host-type host user)
				      '(unix dumb-unix)))))

Similar (but not identical) checks are in `ange-ftp-insert-file-contents'
and `ange-ftp-copy-file-internal'.

And there are comments about it, like "Binary file transfers between
machines of different architectures can be a risky business.".

> 2. This bug report has showed that the modeline flag corresponding to the EOL
>    format isn't always correct. So I think this should be fixed if possible,
>    and if not, this limitation should be documented at least.

Emacs can show only, what it knows. It sends a given file to the FTP
client. It cannot know, whether that external program changes EOL
conversion.

Likely, we shall add a note to section 8.3 of the "GNU Emacs FAQ for MS
Windows", see <http://www.gnu.org/software/emacs/windows/Network-access.html#Network-access>

> Thanks for your time, Michael.

Best regards, Michael.

bug#7383: 24.0.50; end-of-line style on remote files

[Stefan Monnier]

>> So now I wonder two things:
>> 1. In what cases do we want to have text-mode ftp transfers when
>> working with remote files?
>> Because, IMO TRT would be to always use binary mode, i.e., always
>> work with the file
>> in its actual EOL format. I think that is what users expect, don't they?

> See the code of `ange-ftp-write-region'. Binary transfer is explicitely
> disabled for Emacsen on MS Windows machines, if not said otherwise by
> `ange-ftp-binary-file-name-regexp':

> 	       (binary (or (ange-ftp-binary-file filename)
> 			   (and (not (memq system-type
> 					   '(ms-dos windows-nt)))
> 				(memq (ange-ftp-host-type host user)
> 				      '(unix dumb-unix)))))

> Similar (but not identical) checks are in `ange-ftp-insert-file-contents'
> and `ange-ftp-copy-file-internal'.

I think all this non-binary business in ange-ftp is inherited from
pre-Mule issues.  So I'd be tempted to always use binary now.

> And there are comments about it, like "Binary file transfers between
> machines of different architectures can be a risky business.".

I can't think of any case where a non-binary FTP transfer will correctly
convert to/from the internal file-format, and where Emacs can't do the
same or better.


        Stefan

bug#7383: 24.0.50; end-of-line style on remote files

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think all this non-binary business in ange-ftp is inherited from
> pre-Mule issues.  So I'd be tempted to always use binary now.
>
>> And there are comments about it, like "Binary file transfers between
>> machines of different architectures can be a risky business.".
>
> I can't think of any case where a non-binary FTP transfer will correctly
> convert to/from the internal file-format, and where Emacs can't do the
> same or better.

I don't oppose, but given that I do not use MS Windows, I cannot judge
whether this change will harm, or not.

OK, let's change to binary transfer. Shall we remove the checks for
different systems in the code, or shall we default
`ange-ftp-binary-file-name-regexp' to ""?

>         Stefan

Best regards, Michael.

bug#7383: 24.0.50; end-of-line style on remote files

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 04 May 2011 12:14:09 -0300
> Cc: 7383@debbugs.gnu.org
> 
> > See the code of `ange-ftp-write-region'. Binary transfer is explicitely
> > disabled for Emacsen on MS Windows machines, if not said otherwise by
> > `ange-ftp-binary-file-name-regexp':
> 
> > 	       (binary (or (ange-ftp-binary-file filename)
> > 			   (and (not (memq system-type
> > 					   '(ms-dos windows-nt)))
> > 				(memq (ange-ftp-host-type host user)
> > 				      '(unix dumb-unix)))))
> 
> > Similar (but not identical) checks are in `ange-ftp-insert-file-contents'
> > and `ange-ftp-copy-file-internal'.
> 
> I think all this non-binary business in ange-ftp is inherited from
> pre-Mule issues.  So I'd be tempted to always use binary now.

Yes, definitely.  The change was made (by one Stefan Monnier) in
2001.  I see no reasons to keep it.  Let's remove that and see if
someone comes back crying.

bug#7383: 24.0.50; end-of-line style on remote files

[Stefan Monnier]

>> > See the code of `ange-ftp-write-region'. Binary transfer is explicitely
>> > disabled for Emacsen on MS Windows machines, if not said otherwise by
>> > `ange-ftp-binary-file-name-regexp':
>> 
>> > 	       (binary (or (ange-ftp-binary-file filename)
>> > 			   (and (not (memq system-type
>> > 					   '(ms-dos windows-nt)))
>> > 				(memq (ange-ftp-host-type host user)
>> > 				      '(unix dumb-unix)))))
>> 
>> > Similar (but not identical) checks are in `ange-ftp-insert-file-contents'
>> > and `ange-ftp-copy-file-internal'.
>> 
>> I think all this non-binary business in ange-ftp is inherited from
>> pre-Mule issues.  So I'd be tempted to always use binary now.

> Yes, definitely.  The change was made (by one Stefan Monnier) in
> 2001.  I see no reasons to keep it.  Let's remove that and see if
> someone comes back crying.

Just to be clear: I added the (memq system-type '(ms-dos windows-nt))
test, whose purpose was to apply the "same" test locally as remotely.
And I think we should remove both local and remote tests, i.e. bind
`binary' to t regardless of any system-type or ange-ftp-host-type or
ange-ftp-binary-file.


        Stefan

bug#7383: 24.0.50; end-of-line style on remote files

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: michael.albinus@gmx.de,  7383@debbugs.gnu.org
> Date: Wed, 04 May 2011 15:47:05 -0300
> 
> And I think we should remove both local and remote tests, i.e. bind
> `binary' to t regardless of any system-type or ange-ftp-host-type or
> ange-ftp-binary-file.

Yes.

bug#7383: 24.0.50; end-of-line style on remote files

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: michael.albinus@gmx.de,  7383@debbugs.gnu.org
>> Date: Wed, 04 May 2011 15:47:05 -0300
>> 
>> And I think we should remove both local and remote tests, i.e. bind
>> `binary' to t regardless of any system-type or ange-ftp-host-type or
>> ange-ftp-binary-file.
>
> Yes.

I've fixed it in the tunk.

Best regards, Michael.

bug#7383: 24.0.50; end-of-line style on remote files

[Glenn Morris]

Michael Albinus wrote:

> I've fixed it in the tunk.

So can this be closed?

bug#7383: 24.0.50; end-of-line style on remote files

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

> Michael Albinus wrote:
>
>> I've fixed it in the tunk.
>
> So can this be closed?

Yes. I wanted to wait for the confirmation of Dani, but OTOH he said
already that setting of `ange-ftp-binary-file-name-regexp' to "" works
for him.

So I close it.

Best regards, Michael.

bug#7383: 24.0.50; end-of-line style on remote files

[Dani Moncayo]

On Fri, May 6, 2011 at 20:00, Michael Albinus <michael.albinus@gmx.de> wrote:
> Glenn Morris <rgm@gnu.org> writes:
>
>> Michael Albinus wrote:
>>
>>> I've fixed it in the tunk.
>>
>> So can this be closed?
>
> Yes. I wanted to wait for the confirmation of Dani, but OTOH he said
> already that setting of `ange-ftp-binary-file-name-regexp' to "" works
> for him.
>

I'm sorry, I didn't say anything because I've not yet tested the last
changes made by Michael (I was waiting for the next pre-compiled
windows version by Sean).

But I guess Michael is right, because the current behavior must be
like the one I've already tested (setting
ange-ftp-binary-file-name-regexp to "").

Thanks a lot.

-- 
Dani Moncayo