EOL conversion of files in .tar archives

EOL conversion of files in .tar archives

[Eli Zaretskii]

Lately I noticed that, in the Windows port of Emacs, visiting files in
.tar archives shows them as if they had DOS EOLs, even though the
files have Unix EOLs.  (This is not due to some recent change: it
looks like Emacs behaved like that since before the pretest.)

Is this intentional, or should I debug it?  If the former, can someone
explain the rationale?

Re: EOL conversion of files in .tar archives

[Kenichi Handa]

In article <u1whb5m6p.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> Lately I noticed that, in the Windows port of Emacs, visiting files in
> .tar archives shows them as if they had DOS EOLs, even though the
> files have Unix EOLs.  (This is not due to some recent change: it
> looks like Emacs behaved like that since before the pretest.)

> Is this intentional, or should I debug it?  If the former, can someone
> explain the rationale?

I think it is a bug of find-buffer-file-type-coding-system
(dos-w32.el).  If TARGET is (FILENAME . BUFFER), it should
assume that the file contents is available.  Could you
verify that the attached patch is correct?

---
Kenichi Handa
handa@m17n.org

*** dos-w32.el	24 Apr 2007 15:08:02 +0900	1.45
--- dos-w32.el	21 May 2007 13:43:47 +0900	
***************
*** 129,135 ****
    (let ((op (nth 0 command))
  	(target)
  	(binary nil) (text nil)
! 	(undecided nil) (undecided-unix nil))
      (cond ((eq op 'insert-file-contents)
  	   (setq target (nth 1 command))
  	   ;; If TARGET is a cons cell, it has the form (FILENAME . BUFFER),
--- 129,135 ----
    (let ((op (nth 0 command))
  	(target)
  	(binary nil) (text nil)
! 	(undecided nil) (undecided-unix nil) (buffer nil))
      (cond ((eq op 'insert-file-contents)
  	   (setq target (nth 1 command))
  	   ;; If TARGET is a cons cell, it has the form (FILENAME . BUFFER),
***************
*** 138,144 ****
  	   ;; arguments is used, e.g., in arc-mode.el.)  This function
  	   ;; doesn't care about the contents, it only looks at the file's
  	   ;; name, which is the CAR of the cons cell.
! 	   (if (consp target) (setq target (car target)))
  	   ;; First check for a file name that indicates
  	   ;; it is truly binary.
  	   (setq binary (find-buffer-file-type target))
--- 138,144 ----
  	   ;; arguments is used, e.g., in arc-mode.el.)  This function
  	   ;; doesn't care about the contents, it only looks at the file's
  	   ;; name, which is the CAR of the cons cell.
! 	   (if (consp target) (setq buffer (cdr target) target (car target)))
  	   ;; First check for a file name that indicates
  	   ;; it is truly binary.
  	   (setq binary (find-buffer-file-type target))
***************
*** 147,153 ****
  		 ((find-buffer-file-type-match target)
  		  (setq text t))
  		 ;; For any other existing file, decide based on contents.
! 		 ((file-exists-p target)
  		  (setq undecided t))
  		 ;; Next check for a non-DOS file system.
  		 ((untranslated-file-p target)
--- 147,153 ----
  		 ((find-buffer-file-type-match target)
  		  (setq text t))
  		 ;; For any other existing file, decide based on contents.
! 		 ((or buffer (file-exists-p target))
  		  (setq undecided t))
  		 ;; Next check for a non-DOS file system.
  		 ((untranslated-file-p target)

Re: EOL conversion of files in .tar archives

[David Kastrup]

Kenichi Handa <handa@m17n.org> writes:

> In article <u1whb5m6p.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
>
>> Lately I noticed that, in the Windows port of Emacs, visiting files in
>> .tar archives shows them as if they had DOS EOLs, even though the
>> files have Unix EOLs.  (This is not due to some recent change: it
>> looks like Emacs behaved like that since before the pretest.)
>
>> Is this intentional, or should I debug it?  If the former, can someone
>> explain the rationale?
>
> I think it is a bug of find-buffer-file-type-coding-system
> (dos-w32.el).  If TARGET is (FILENAME . BUFFER), it should
> assume that the file contents is available.  Could you
> verify that the attached patch is correct?

Apropos: take the tar.gz file in
<URL:http://www.codemonkey.org.uk/projects/git-snapshots/git/git-2007-05-20.tar.gz>
(it will be available only a week or so).

Move cursor down to .gitignore and press RET.  Instead you'll get
Makefile, the file one line lower.  In general, on every line you'll
get the file one line lower.

It would appear that this is packed with some version of tar that GNU
tar can perfectly well unpack, but with output confusing tar-mode.  My
tar version is tar (GNU tar) 1.16.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> From: David Kastrup <dak@gnu.org>
> Date: Mon, 21 May 2007 07:15:50 +0200
> 
> Apropos: take the tar.gz file in
> <URL:http://www.codemonkey.org.uk/projects/git-snapshots/git/git-2007-05-20.tar.gz>
> (it will be available only a week or so).
> 
> Move cursor down to .gitignore and press RET.  Instead you'll get
> Makefile, the file one line lower.  In general, on every line you'll
> get the file one line lower.
> 
> It would appear that this is packed with some version of tar that GNU
> tar can perfectly well unpack, but with output confusing tar-mode.

I rather think that the pax_global_header entry (the 1st one) is what
confuses tar-mode, because it shows after it the funny comment=XXXXXX
entry that isn't in the tarball.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Mon, 21 May 2007 13:48:20 +0900
> 
> I think it is a bug of find-buffer-file-type-coding-system
> (dos-w32.el).  If TARGET is (FILENAME . BUFFER), it should
> assume that the file contents is available.  Could you
> verify that the attached patch is correct?

I will try when I have time, but in the meantime, I'd like to ask that
Emacs 22.1 not be released without resolving this issue.

Re: EOL conversion of files in .tar archives

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> I think it is a bug of find-buffer-file-type-coding-system
>> (dos-w32.el).  If TARGET is (FILENAME . BUFFER), it should
>> assume that the file contents is available.  Could you
>> verify that the attached patch is correct?
>
> I will try when I have time, but in the meantime, I'd like to ask that
> Emacs 22.1 not be released without resolving this issue.

Why? This behavior has been present since before the pretest, and no
one complained; this suggests it is not urgent.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> Cc: Kenichi Handa <handa@m17n.org>,  emacs-devel@gnu.org
> From: Chong Yidong <cyd@stupidchicken.com>
> Date: Mon, 21 May 2007 18:58:03 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I think it is a bug of find-buffer-file-type-coding-system
> >> (dos-w32.el).  If TARGET is (FILENAME . BUFFER), it should
> >> assume that the file contents is available.  Could you
> >> verify that the attached patch is correct?
> >
> > I will try when I have time, but in the meantime, I'd like to ask that
> > Emacs 22.1 not be released without resolving this issue.
> 
> Why? This behavior has been present since before the pretest, and no
> one complained; this suggests it is not urgent.

It is not urgent, but it's a nasty problem, because preserving the EOL
format is one of fundamental Emacs features.  The way it works now,
Emacs violates the user trust, because changing EOL format will cause
subtle problems with other tools.

In any case, this problem is not going to delay the release, as I'll
probably work on it today, and Handa-san already suggested what seems
like the right solution.  I just didn't want the release to go out a
few minutes before the change is committed, that's all.

Anyway, I said "I'd like to ask"; if others think it's a bad idea to
fix this now, so be it.

Re: EOL conversion of files in .tar archives

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> In any case, this problem is not going to delay the release, as I'll
> probably work on it today, and Handa-san already suggested what
> seems like the right solution.  I just didn't want the release to go
> out a few minutes before the change is committed, that's all.
>
> Anyway, I said "I'd like to ask"; if others think it's a bad idea to
> fix this now, so be it.

Well, the release could happen any year now, so fixes should go into
the trunk exclusively, leaving it to the release manager's discretion
whether he wants to take the plunge and risk to move the material into
the release branch.

Putting fixes in the trunk is nothing people should shy from now that
we have forked the release branch.  But for the release branch, it
does not matter whether "others think it's a bad idea" at this stage.
This clearly is Richard's call to make.

-- 
David Kastrup

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> Cc: Chong Yidong <cyd@stupidchicken.com>, emacs-devel@gnu.org, handa@m17n.org
> From: David Kastrup <dak@gnu.org>
> Date: Tue, 22 May 2007 08:53:24 +0200
> 
> for the release branch, it does not matter whether "others think
> it's a bad idea" at this stage.

It matters to me.

Thanks for providing your opinion on this.

Re: EOL conversion of files in .tar archives

[Richard Stallman]

    Apropos: take the tar.gz file in
    <URL:http://www.codemonkey.org.uk/projects/git-snapshots/git/git-2007-05-20.tar.gz>
    (it will be available only a week or so).

    Move cursor down to .gitignore and press RET.  Instead you'll get
    Makefile, the file one line lower.  In general, on every line you'll
    get the file one line lower.

    It would appear that this is packed with some version of tar that GNU
    tar can perfectly well unpack, but with output confusing tar-mode.  My
    tar version is tar (GNU tar) 1.16.

Would someone please debug this and fix it in the trunk?
Please ack when it is done.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Mon, 21 May 2007 13:48:20 +0900
> 
> In article <u1whb5m6p.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Lately I noticed that, in the Windows port of Emacs, visiting files in
> > .tar archives shows them as if they had DOS EOLs, even though the
> > files have Unix EOLs.  (This is not due to some recent change: it
> > looks like Emacs behaved like that since before the pretest.)
> 
> > Is this intentional, or should I debug it?  If the former, can someone
> > explain the rationale?
> 
> I think it is a bug of find-buffer-file-type-coding-system
> (dos-w32.el).  If TARGET is (FILENAME . BUFFER), it should
> assume that the file contents is available.  Could you
> verify that the attached patch is correct?

I'm uneasy about patching find-buffer-file-type-coding-system at this
stage, since it is so widely used.  For example,
decode-coding-inserted-region also invokes
find-operation-coding-system with second argument a cons cell, and it
is not clear to me that your change will DTRT in that case.

I'd like to suggest an alternative change that is local to
tar-mode.el, see below.  It mimics what arc-mode.el does.

This solves the problem with the EOL decoding, but reveals a very
strange phenomenon: dos-w32.el and dos-w32.elc behave differently.
The latter is preloaded into Emacs on Windows, and if I visit a file
inside a tarball (after applying the patch below), plain text files
with Unix EOLs get latin-1-unix buffer-file-coding-system, instead of
the expected undecided-unix.  But if I manually load dos-w32.el and
visit the same file again, its buffer-file-coding-system is now
undecided-unix, as expected!  Is this some problem with the byte
compiler?


Index: lisp/tar-mode.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/tar-mode.el,v
retrieving revision 1.108
diff -u -r1.108 tar-mode.el
--- lisp/tar-mode.el	21 Jan 2007 03:53:10 -0000	1.108
+++ lisp/tar-mode.el	22 May 2007 11:18:19 -0000
@@ -681,6 +681,12 @@
   (goto-char (posn-point (event-end event)))
   (tar-extract))
 
+(defun tar-file-name-handler (op &rest args)
+  "Helper function for `tar-extract'."
+  (or (eq op 'file-exists-p)
+      (let ((file-name-handler-alist nil))
+	(apply op args))))
+
 (defun tar-extract (&optional other-window-p)
   "In Tar mode, extract this entry of the tar file into its own buffer."
   (interactive)
@@ -735,9 +741,17 @@
 				  (save-excursion
 				    (funcall set-auto-coding-function
 					     name (- (point-max) (point)))))
-			     (car (find-operation-coding-system
-				   'insert-file-contents
-				   (cons name (current-buffer)) t))))
+			     ;; The following binding causes
+			     ;; find-buffer-file-type-coding-system
+			     ;; (defined on dos-w32.el) to act as if
+			     ;; the file being extracted existed, so
+			     ;; that the file's contents' encoding is
+			     ;; auto-detected.
+			     (let ((file-name-handler-alist
+				    '(("" . tar-file-name-handler))))
+			       (car (find-operation-coding-system
+				     'insert-file-contents
+				     (cons name (current-buffer)) t)))))
 			(multibyte enable-multibyte-characters)
 			(detected (detect-coding-region
 				   (point-min)

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> Date: Tue, 22 May 2007 14:19:14 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> I'd like to suggest an alternative change that is local to
> tar-mode.el, see below.

Sorry, that patch was incomplete, and so did an imperfect job.  Please
use the one below.

Index: lisp/tar-mode.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/tar-mode.el,v
retrieving revision 1.108
diff -u -r1.108 tar-mode.el
--- lisp/tar-mode.el	21 Jan 2007 03:53:10 -0000	1.108
+++ lisp/tar-mode.el	22 May 2007 12:19:49 -0000
@@ -681,6 +681,12 @@
   (goto-char (posn-point (event-end event)))
   (tar-extract))
 
+(defun tar-file-name-handler (op &rest args)
+  "Helper function for `tar-extract'."
+  (or (eq op 'file-exists-p)
+      (let ((file-name-handler-alist nil))
+	(apply op args))))
+
 (defun tar-extract (&optional other-window-p)
   "In Tar mode, extract this entry of the tar file into its own buffer."
   (interactive)
@@ -735,9 +741,17 @@
 				  (save-excursion
 				    (funcall set-auto-coding-function
 					     name (- (point-max) (point)))))
-			     (car (find-operation-coding-system
-				   'insert-file-contents
-				   (cons name (current-buffer)) t))))
+			     ;; The following binding causes
+			     ;; find-buffer-file-type-coding-system
+			     ;; (defined on dos-w32.el) to act as if
+			     ;; the file being extracted existed, so
+			     ;; that the file's contents' encoding is
+			     ;; auto-detected.
+			     (let ((file-name-handler-alist
+				    '(("" . tar-file-name-handler))))
+			       (car (find-operation-coding-system
+				     'insert-file-contents
+				     (cons name (current-buffer)) t)))))
 			(multibyte enable-multibyte-characters)
 			(detected (detect-coding-region
 				   (point-min)
@@ -758,7 +772,7 @@
 			      (coding-system-change-text-conversion
 			       coding 'raw-text)))
 		    (decode-coding-region (point-min) (point-max) coding)
-		    (set-buffer-file-coding-system coding))
+		    (set-buffer-file-coding-system last-coding-system-used t))
 		  ;; Set the default-directory to the dir of the
 		  ;; superior buffer.
 		  (setq default-directory

Re: EOL conversion of files in .tar archives

[Richard Stallman]

    I will try when I have time, but in the meantime, I'd like to ask that
    Emacs 22.1 not be released without resolving this issue.

I put this into the list of bugs that must be fixed.
Please try to work on it ASAP!

Re: EOL conversion of files in .tar archives

[Richard Stallman]

    Why? This behavior has been present since before the pretest, and no
    one complained; this suggests it is not urgent.

Just because the pretesters didn't see a bug during 6 months
doesn't mean it isn't a serious bug.

Re: EOL conversion of files in .tar archives

[Richard Stallman]

    In any case, this problem is not going to delay the release, as I'll
    probably work on it today, and Handa-san already suggested what seems
    like the right solution.  I just didn't want the release to go out a
    few minutes before the change is committed, that's all.

    Anyway, I said "I'd like to ask"; if others think it's a bad idea to
    fix this now, so be it.

Please do fix it!

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: handa@m17n.org, emacs-devel@gnu.org
> Date: Tue, 22 May 2007 10:52:22 -0400
> 
>     I will try when I have time, but in the meantime, I'd like to ask that
>     Emacs 22.1 not be released without resolving this issue.
> 
> I put this into the list of bugs that must be fixed.
> Please try to work on it ASAP!

Done, and a suggested patch posted.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Tue, 22 May 2007 11:49:35 -0400
> Cc: cyd@stupidchicken.com, handa@m17n.org, emacs-devel@gnu.org
> 
>     In any case, this problem is not going to delay the release, as I'll
>     probably work on it today, and Handa-san already suggested what seems
>     like the right solution.  I just didn't want the release to go out a
>     few minutes before the change is committed, that's all.
> 
>     Anyway, I said "I'd like to ask"; if others think it's a bad idea to
>     fix this now, so be it.
> 
> Please do fix it!

I will, as soon as I hear Handa-san's opinion on the alternative patch
I suggested.

Re: EOL conversion of files in .tar archives

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> > Date: Tue, 22 May 2007 14:19:14 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: emacs-devel@gnu.org
> > 
> > I'd like to suggest an alternative change that is local to
> > tar-mode.el, see below.
> 
> Sorry, that patch was incomplete, and so did an imperfect job.  Please
> use the one below.

Maybe we should condition this change to win32, so any bug
inadvertently introduced here won't affect other operating systems.
 
> +			     (let ((file-name-handler-alist
> +				    '(("" . tar-file-name-handler))))
> +			       (car (find-operation-coding-system
> +				     'insert-file-contents
> +				     (cons name (current-buffer)) t)))))

Re: EOL conversion of files in .tar archives

[Lennart Borgman (gmail)]

Chong Yidong wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>>> Date: Tue, 22 May 2007 14:19:14 +0300
>>> From: Eli Zaretskii <eliz@gnu.org>
>>> Cc: emacs-devel@gnu.org
>>>
>>> I'd like to suggest an alternative change that is local to
>>> tar-mode.el, see below.
>> Sorry, that patch was incomplete, and so did an imperfect job.  Please
>> use the one below.
> 
> Maybe we should condition this change to win32, so any bug
> inadvertently introduced here won't affect other operating systems.


Should Emacs not be able to handle this on other systems too?

Re: EOL conversion of files in .tar archives

[Chong Yidong]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

> Chong Yidong wrote:
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >>> Date: Tue, 22 May 2007 14:19:14 +0300
> >>> From: Eli Zaretskii <eliz@gnu.org>
> >>> Cc: emacs-devel@gnu.org
> >>>
> >>> I'd like to suggest an alternative change that is local to
> >>> tar-mode.el, see below.
> >> Sorry, that patch was incomplete, and so did an imperfect job.  Please
> >> use the one below.
> > Maybe we should condition this change to win32, so any bug
> > inadvertently introduced here won't affect other operating systems.
> 
> Should Emacs not be able to handle this on other systems too?

My understanding is that the bug does not affect other systems.

Re: EOL conversion of files in .tar archives

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     Why? This behavior has been present since before the pretest, and no
>     one complained; this suggests it is not urgent.
>
> Just because the pretesters didn't see a bug during 6 months
> doesn't mean it isn't a serious bug.

Quite so.  However, Emacs is _huge_ and growing.  If we won't release
it as long as there is any serious bug remaining in any subsystem of
it, we might as well strike "release" off the things we are ever going
to consider.

With a system as large and complex as Emacs, there is no chance we'll
_ever_ see a release if we don't cut off the attempts of fixing things
at one point of time.  We _have_ to raise the threshold of severity
for bugs we even _attempt_ to fix for the release in order to have the
amount of fixes causing potential havoc (and thus requiring an
extension of the release schedule with additional pretests) shrink.

It will be a huge coincidence (with no really important correlation to
the actual amount of severe bugs remaining) if there will be no
serious bug being _reported_ in any of the many subsystems of Emacs
for an extended period of time.

And that means, serious bugs remaining or not, that pretty much every
point of time is as good for a release as the next one.  In fact, I
would say that we are currently drifting away from the optimal time
for a release since the attention span for last-minute fixes is pretty
much over.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: EOL conversion of files in .tar archives

[Richard Stallman]

    I'd like to suggest an alternative change that is local to
    tar-mode.el, see below.  It mimics what arc-mode.el does.

Ok with me (especially since this is already tested in arc-mode).

    The latter is preloaded into Emacs on Windows, and if I visit a file
    inside a tarball (after applying the patch below), plain text files
    with Unix EOLs get latin-1-unix buffer-file-coding-system, instead of
    the expected undecided-unix.  But if I manually load dos-w32.el and
    visit the same file again, its buffer-file-coding-system is now
    undecided-unix, as expected!

What happens if you load the file and then use `byte-compile' to
compile that function?  Does that fail?

If so, what happens if you add some debugging setqs in the code of
that function, and then use `byte-compile' to compile that function?
Does that fail?  You could debug it in detail that way.

It could also be useful to use M-x disassemble to verify that the
bytecode correctly fits the Lisp code.

Re: EOL conversion of files in .tar archives

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> This solves the problem with the EOL decoding, but reveals a very
> strange phenomenon: dos-w32.el and dos-w32.elc behave differently.
> The latter is preloaded into Emacs on Windows, and if I visit a file
> inside a tarball (after applying the patch below), plain text files
> with Unix EOLs get latin-1-unix buffer-file-coding-system, instead of
> the expected undecided-unix.  But if I manually load dos-w32.el and
> visit the same file again, its buffer-file-coding-system is now
> undecided-unix, as expected!  Is this some problem with the byte
> compiler?

It is probably the result of using last-coding-system-used.  ISTR that
in a similar situation, the coding system used during byte-compilation
replaced last-coding-system-used to cause a similar problem.

Re: EOL conversion of files in .tar archives

[Kenichi Handa]

In article <u1wh9kswd.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> I'd like to suggest an alternative change that is local to
> tar-mode.el, see below.  It mimics what arc-mode.el does.

It seems that the alternative change works well too
especially because arc-mode is doing the same thing.  But...

> +(defun tar-file-name-handler (op &rest args)
> +  "Helper function for `tar-extract'."
> +  (or (eq op 'file-exists-p)
> +      (let ((file-name-handler-alist nil))
> +	(apply op args))))

Is it ok to check only file-exists-p?  How about
file-readable-p?

> Sorry, that patch was incomplete, and so did an imperfect job.  Please
> use the one below.

Does this new version have the following problem too?

> This solves the problem with the EOL decoding, but reveals a very
> strange phenomenon: dos-w32.el and dos-w32.elc behave differently.
> The latter is preloaded into Emacs on Windows, and if I visit a file
> inside a tarball (after applying the patch below), plain text files
> with Unix EOLs get latin-1-unix buffer-file-coding-system, instead of
> the expected undecided-unix.  But if I manually load dos-w32.el and
> visit the same file again, its buffer-file-coding-system is now
> undecided-unix, as expected!  Is this some problem with the byte
> compiler?

---
Kenichi Handa
handa@m17n.org

Re: EOL conversion of files in .tar archives

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:
>
> This solves the problem with the EOL decoding, but reveals a very
> strange phenomenon: dos-w32.el and dos-w32.elc behave differently.
> ...
> -	    (set-buffer-file-coding-system coding))
> +	    (set-buffer-file-coding-system last-coding-system-used t))

What's the rationale for this change?  The purpose of the old code was
quite transparent; changing it to use last-coding-system-used is not
obvious, and deserves at least a comment.  (And I think the use of
last-coding-system-used leads to the "strange phenomenon" mentioned
above.)

All this makes me more dubious about adding this code now.  We should
either make this entire change conditional on win32 (hedious as the
resulting code might be) or simply postphone it for Emacs 22.2.

Re: EOL conversion of files in .tar archives

[Kenichi Handa]

In article <87abvwrzwz.fsf@stupidchicken.com>, Chong Yidong <cyd@stupidchicken.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
> >
> > This solves the problem with the EOL decoding, but reveals a very
> > strange phenomenon: dos-w32.el and dos-w32.elc behave differently.
> > ...
> > -	    (set-buffer-file-coding-system coding))
> > +	    (set-buffer-file-coding-system last-coding-system-used t))

> What's the rationale for this change?  The purpose of the old code was
> quite transparent; changing it to use last-coding-system-used is not
> obvious, and deserves at least a comment.

I think that change is to fix a bug of original code that
has not been unveiled in non-Windows system.  We have this
code now:

		    (if coding
			(or (numberp (coding-system-eol-type coding))
			    (vectorp (coding-system-eol-type detected))
			    (setq coding (coding-system-change-eol-conversion
					  coding
					  (coding-system-eol-type detected))))
->		      (setq coding
->			    (find-new-buffer-file-coding-system detected)))

In non-Windows system, usually the last two lines are
executed, thus DETECTED is reflected in CODING by
find-new-buffer-file-coding-system.  So,

	(set-buffer-file-coding-system coding)

is ok.  But, if that part is not executed and CODING is
something like undecided-dos, even if decode-coding itself
detects a coding system, the detected one is not reflected
in buffer-file-coding-system.

For instance, even on GNU/Linux, if a tar file contains
a Latin-1 file "temp.dos", and we have this setting,

(push '("\\.dos\\'" . undecided-dos) file-coding-system-alist)

Buffer-file-coding-system of "temp.dos" will be
undecided-dos, instead of latin-1-*.

> (And I think the use of last-coding-system-used leads to
> the "strange phenomenon" mentioned above.)

I think it is very unlikely that it is so.  But...

> All this makes me more dubious about adding this code now.  We should
> either make this entire change conditional on win32 (hedious as the
> resulting code might be) or simply postphone it for Emacs 22.2.

if we are going to include that change in Emacs 22.1, it
seems that making the change conditional is a good idea.

---
Kenichi Handa
handa@m17n.org

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> Date: Tue, 22 May 2007 23:18:03 +0200
> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
> CC: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org,  handa@m17n.org
> 
> > Maybe we should condition this change to win32, so any bug
> > inadvertently introduced here won't affect other operating systems.
> 
> Should Emacs not be able to handle this on other systems too?

Other systems don't need this because they don't set up
file-coding-system-alist like dos-w32.el does.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: handa@m17n.org, emacs-devel@gnu.org
> Date: Tue, 22 May 2007 19:09:49 -0400
> 
>     The latter is preloaded into Emacs on Windows, and if I visit a file
>     inside a tarball (after applying the patch below), plain text files
>     with Unix EOLs get latin-1-unix buffer-file-coding-system, instead of
>     the expected undecided-unix.  But if I manually load dos-w32.el and
>     visit the same file again, its buffer-file-coding-system is now
>     undecided-unix, as expected!
> 
> What happens if you load the file and then use `byte-compile' to
> compile that function?  Does that fail?
> 
> If so, what happens if you add some debugging setqs in the code of
> that function, and then use `byte-compile' to compile that function?
> Does that fail?  You could debug it in detail that way.
> 
> It could also be useful to use M-x disassemble to verify that the
> bytecode correctly fits the Lisp code.

I think the problems I described were due to the fact that dos-w32.el
doesn't expect to be loaded twice.  For example, this line:

  (modify-coding-system-alist 'file "" 'find-buffer-file-type-coding-system)

if executed the second time, would add
find-buffer-file-type-coding-system to the alist _twice_, right?

In any case, the original patch was incomplete, and the last patch I
posted does the job without any problems.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Wed, 23 May 2007 10:22:06 +0900
> 
> > +(defun tar-file-name-handler (op &rest args)
> > +  "Helper function for `tar-extract'."
> > +  (or (eq op 'file-exists-p)
> > +      (let ((file-name-handler-alist nil))
> > +	(apply op args))))
> 
> Is it ok to check only file-exists-p?  How about
> file-readable-p?

It only catches file-exists-p because that's what
find-buffer-file-type-coding-system tries.

> > Sorry, that patch was incomplete, and so did an imperfect job.  Please
> > use the one below.
> 
> Does this new version have the following problem too?

I didn't try loading dos-w32.el again with the last patch because it
produces the correct results with the preloaded dos-w32.el.  In any
case, that seems to be a different problem (if it is indeed a
problem), see my other message in this thread.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> Cc: handa@m17n.org, emacs-devel@gnu.org
> From: Chong Yidong <cyd@stupidchicken.com>
> Date: Tue, 22 May 2007 23:18:04 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >
> > This solves the problem with the EOL decoding, but reveals a very
> > strange phenomenon: dos-w32.el and dos-w32.elc behave differently.
> > ...
> > -	    (set-buffer-file-coding-system coding))
> > +	    (set-buffer-file-coding-system last-coding-system-used t))
> 
> What's the rationale for this change?

2 reasons: First, without the 3rd arg, set-buffer-file-coding-system
does not force the coding-system of the buffer to `coding', it rather
_merges_ them (see the code for the details).  Second, `coding' could
be `undecided', in which case decode-coding-region would set
last-coding-system-used to the actual encoding it detected and used to
decode the file's contents.  Thus, using last-coding-system-used is
more accurate (and in fact I believe it's TRT on all platforms, not
only on Windows).

> (And I think the use of last-coding-system-used leads to the
> "strange phenomenon" mentioned above.)

No, the ``strange phenomenon'' was observed in the previous patch,
that didn't have this change.

> All this makes me more dubious about adding this code now.  We should
> either make this entire change conditional on win32 (hedious as the
> resulting code might be) or simply postphone it for Emacs 22.2.

I'm okay with conditioning on w32 on the release branch.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> Date: Wed, 23 May 2007 10:15:55 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org, handa@m17n.org
> 
> I'm okay with conditioning on w32 on the release branch.

How's the following for the branch?


Index: lisp/tar-mode.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/tar-mode.el,v
retrieving revision 1.108
diff -u -r1.108 tar-mode.el
--- lisp/tar-mode.el	21 Jan 2007 03:53:10 -0000	1.108
+++ lisp/tar-mode.el	23 May 2007 12:32:19 -0000
@@ -681,6 +681,12 @@
   (goto-char (posn-point (event-end event)))
   (tar-extract))
 
+(defun tar-file-name-handler (op &rest args)
+  "Helper function for `tar-extract'."
+  (or (eq op 'file-exists-p)
+      (let ((file-name-handler-alist nil))
+	(apply op args))))
+
 (defun tar-extract (&optional other-window-p)
   "In Tar mode, extract this entry of the tar file into its own buffer."
   (interactive)
@@ -735,9 +741,19 @@
 				  (save-excursion
 				    (funcall set-auto-coding-function
 					     name (- (point-max) (point)))))
-			     (car (find-operation-coding-system
-				   'insert-file-contents
-				   (cons name (current-buffer)) t))))
+			     ;; The following binding causes
+			     ;; find-buffer-file-type-coding-system
+			     ;; (defined on dos-w32.el) to act as if
+			     ;; the file being extracted existed, so
+			     ;; that the file's contents' encoding and
+			     ;; EOL format are auto-detected.
+			     (let ((file-name-handler-alist
+				    (if (featurep 'dos-w32)
+					'(("" . tar-file-name-handler))
+				      file-name-handler-alist)))
+			       (car (find-operation-coding-system
+				     'insert-file-contents
+				     (cons name (current-buffer)) t)))))
 			(multibyte enable-multibyte-characters)
 			(detected (detect-coding-region
 				   (point-min)
@@ -758,7 +774,9 @@
 			      (coding-system-change-text-conversion
 			       coding 'raw-text)))
 		    (decode-coding-region (point-min) (point-max) coding)
-		    (set-buffer-file-coding-system coding))
+		    ;; Force buffer-file-coding-system to what
+		    ;; decode-coding-region actually used.
+		    (set-buffer-file-coding-system last-coding-system-used t))
 		  ;; Set the default-directory to the dir of the
 		  ;; superior buffer.
 		  (setq default-directory

Re: EOL conversion of files in .tar archives

[Kenichi Handa]

In article <uhcq3k9bm.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > Date: Wed, 23 May 2007 10:15:55 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: emacs-devel@gnu.org, handa@m17n.org
> > 
> > I'm okay with conditioning on w32 on the release branch.

> How's the following for the branch?

I'm ok with it.

---
Kenichi Handa
handa@m17n.org

Re: EOL conversion of files in .tar archives

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> I'm okay with conditioning on w32 on the release branch.
>
> How's the following for the branch?

Please install it ASAP.  Thanks.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, handa@m17n.org
> From: Chong Yidong <cyd@stupidchicken.com>
> Date: Wed, 23 May 2007 09:31:48 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I'm okay with conditioning on w32 on the release branch.
> >
> > How's the following for the branch?
> 
> Please install it ASAP.  Thanks.

Done.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: cyd@stupidchicken.com, emacs-devel@gnu.org
> Date: Wed, 23 May 2007 22:01:50 +0900
> 
> > How's the following for the branch?
> 
> I'm ok with it.

Thanks, I committed it to the branch.  On the trunk the change is not
conditioned on w32.

Re: EOL conversion of files in .tar archives

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Kenichi Handa <handa@m17n.org>
>> CC: cyd@stupidchicken.com, emacs-devel@gnu.org
>> Date: Wed, 23 May 2007 22:01:50 +0900
>> 
>> > How's the following for the branch?
>> 
>> I'm ok with it.
>
> Thanks, I committed it to the branch.  On the trunk the change is not
> conditioned on w32.

If there is no reason to suspect it causes problems on non-w32
systems, this seems appropriate.

We should confine swinging black cats around and other forms of
superstition to the release branches.  Accumulating them in the trunk
is possibly bad luck once one forgets why they might have been there.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Wed, 23 May 2007 20:02:11 +0200
> Cc: cyd@stupidchicken.com, emacs-devel@gnu.org, Kenichi Handa <handa@m17n.org>
> 
> > On the trunk the change is not conditioned on w32.
> 
> If there is no reason to suspect it causes problems on non-w32
> systems, this seems appropriate.

There's no reason to suspect that, as an almost identical technique
works in arc-mode.el on all platforms.

> We should confine swinging black cats around and other forms of
> superstition to the release branches.

That's what I did, didn't I?  Or are you saying that you approve of
what I did?

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: handa@m17n.org, eliz@gnu.org, emacs-devel@gnu.org
> Date: Tue, 22 May 2007 04:30:45 -0400
> 
>     Apropos: take the tar.gz file in
>     <URL:http://www.codemonkey.org.uk/projects/git-snapshots/git/git-2007-05-20.tar.gz>
>     (it will be available only a week or so).
> 
>     Move cursor down to .gitignore and press RET.  Instead you'll get
>     Makefile, the file one line lower.  In general, on every line you'll
>     get the file one line lower.
> 
>     It would appear that this is packed with some version of tar that GNU
>     tar can perfectly well unpack, but with output confusing tar-mode.  My
>     tar version is tar (GNU tar) 1.16.
> 
> Would someone please debug this and fix it in the trunk?
> Please ack when it is done.

Fixed on the trunk with the patch below.  Additional info about the
tarball that causes this can be found here:

  http://lkml.org/lkml/2005/6/18/5

Note that the fix I checked in is not a perfect one: the extended pax
header is still shown as a special entry (unlike GNU Tar, which
doesn't show it at all).  But at least the bug reported by David is
gone, and tar-mode now says correctly that this is a special entry,
not a real file, and does not allow to visit it, as it does with
directories and other special entries.

I used the uppercase `H' to flag this special entry in the tarball
summary, but if someone has a better suggestion, I won't object to
changing the letter.


2007-05-23  Eli Zaretskii  <eliz@gnu.org>

	* tar-mode.el (tar-header-block-summarize, tar-summarize-buffer)
	(tar-get-descriptor): Handle type 55, an extended pax header.

Index: lisp/tar-mode.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/tar-mode.el,v
retrieving revision 1.109
diff -u -r1.109 tar-mode.el
--- lisp/tar-mode.el	23 May 2007 12:38:35 -0000	1.109
+++ lisp/tar-mode.el	23 May 2007 18:29:24 -0000
@@ -363,6 +363,7 @@
 		  ((eq type 29) ?M)	; multivolume continuation
 		  ((eq type 35) ?S)	; sparse
 		  ((eq type 38) ?V)	; volume header
+		  ((eq type 55) ?H)	; extended pax header
 		  (t ?\s)
 		  )
 	    (tar-grind-file-mode mode)
@@ -421,7 +422,7 @@
                                   (buffer-substring pos (+ pos 512)))))))
         (setq pos (+ pos 512))
         (progress-reporter-update progress-reporter pos)
-        (if (eq (tar-header-link-type tokens) 20)
+        (if (memq (tar-header-link-type tokens) '(20 55))
             ;; Foo.  There's an extra empty block after these.
             (setq pos (+ pos 512)))
         (let ((size (tar-header-size tokens)))
@@ -657,13 +658,14 @@
 	 (size (tar-header-size tokens))
 	 (link-p (tar-header-link-type tokens)))
     (if link-p
-	(error "This is a %s, not a real file"
-	       (cond ((eq link-p 5) "directory")
-		     ((eq link-p 20) "tar directory header")
-		     ((eq link-p 28) "next has longname")
-		     ((eq link-p 29) "multivolume-continuation")
-		     ((eq link-p 35) "sparse entry")
-		     ((eq link-p 38) "volume header")
+	(error "This is a%s, not a real file"
+	       (cond ((eq link-p 5) " directory")
+		     ((eq link-p 20) " tar directory header")
+		     ((eq link-p 28) " next has longname")
+		     ((eq link-p 29) " multivolume-continuation")
+		     ((eq link-p 35) " sparse entry")
+		     ((eq link-p 38) " volume header")
+		     ((eq link-p 55) "n extended pax header")
 		     (t "link"))))
     (if (zerop size) (error "This is a zero-length file"))
     descriptor))

Re: EOL conversion of files in .tar archives

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Wed, 23 May 2007 20:02:11 +0200
>> Cc: cyd@stupidchicken.com, emacs-devel@gnu.org, Kenichi Handa <handa@m17n.org>
>> 
>> > On the trunk the change is not conditioned on w32.
>> 
>> If there is no reason to suspect it causes problems on non-w32
>> systems, this seems appropriate.
>
> There's no reason to suspect that, as an almost identical technique
> works in arc-mode.el on all platforms.
>
>> We should confine swinging black cats around and other forms of
>> superstition to the release branches.
>
> That's what I did, didn't I?  Or are you saying that you approve of
> what I did?

Sorry to startle, but it is the latter (actually, the former too since
your usage of "Or" does not make much sense).  That's why I wrote
"appropriate" above.

Insert the obvious "Who are you and what have you down with David?"
joke here.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: EOL conversion of files in .tar archives

[Chong Yidong]

Chong Yidong <cyd@stupidchicken.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> This solves the problem with the EOL decoding, but reveals a very
>> strange phenomenon: dos-w32.el and dos-w32.elc behave differently.
>> The latter is preloaded into Emacs on Windows, and if I visit a file
>> inside a tarball (after applying the patch below), plain text files
>> with Unix EOLs get latin-1-unix buffer-file-coding-system, instead of
>> the expected undecided-unix.  But if I manually load dos-w32.el and
>> visit the same file again, its buffer-file-coding-system is now
>> undecided-unix, as expected!  Is this some problem with the byte
>> compiler?
>
> It is probably the result of using last-coding-system-used.  ISTR that
> in a similar situation, the coding system used during byte-compilation
> replaced last-coding-system-used to cause a similar problem.

Please ignore this; this was an old message that my mailer screwed up.

Sorry for the noise.

Re: EOL conversion of files in .tar archives

[Richard Stallman]

    I think the problems I described were due to the fact that dos-w32.el
    doesn't expect to be loaded twice.  For example, this line:

      (modify-coding-system-alist 'file "" 'find-buffer-file-type-coding-system)

    if executed the second time, would add
    find-buffer-file-type-coding-system to the alist _twice_, right?

Could someone fix those problems in the trunk, just for cleanliness'
sake and to avoid future red herrings?

Please install the patch for Emacs 22, conditionalized only for
Windows just for caution's sake.  The only other thing we need for the
release is the PROBLEMS entry that was suggested yesterday.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> Cc: Kenichi Handa <handa@m17n.org>,  emacs-devel@gnu.org
> From: Chong Yidong <cyd@stupidchicken.com>
> Date: Tue, 22 May 2007 20:59:44 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This solves the problem with the EOL decoding, but reveals a very
> > strange phenomenon: dos-w32.el and dos-w32.elc behave differently.
> > The latter is preloaded into Emacs on Windows, and if I visit a file
> > inside a tarball (after applying the patch below), plain text files
> > with Unix EOLs get latin-1-unix buffer-file-coding-system, instead of
> > the expected undecided-unix.  But if I manually load dos-w32.el and
> > visit the same file again, its buffer-file-coding-system is now
> > undecided-unix, as expected!  Is this some problem with the byte
> > compiler?
> 
> It is probably the result of using last-coding-system-used.

No, it cannot be that: the version of tar-mode.el with which I saw
this problem did not yet include the change which uses
last-coding-system-used.

Re: EOL conversion of files in .tar archives

[Kenichi Handa]

In article <ud50rjvgt.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, I committed it to the branch.

Thank yo.

> On the trunk the change is not conditioned on w32.

For the trunk, how about trying the fix of my version
(i.e. the change to find-buffer-file-type-coding-system)?

By the way, for emacs-unicode-2, I have a plan of new
functions insert-region-contents and write-region-to-buffer
that do the similar thing as insert-file-contents and
write-region but gets/puts the contents from/to a region of
a specified buffer.  They encapsulte various weird tasks for
decoding and encoding, and can easility be used both by
tar-mode and arc-mode.

---
Kenichi Handa
handa@m17n.org

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: cyd@stupidchicken.com, emacs-devel@gnu.org
> Date: Thu, 24 May 2007 10:01:43 +0900
> 
> > On the trunk the change is not conditioned on w32.
> 
> For the trunk, how about trying the fix of my version
> (i.e. the change to find-buffer-file-type-coding-system)?

I'm uneasy with that change, because there are other places that use a
cons cell as the second argument to find-operation-coding-system, and
I'm not sure what you propose is good for them.  For example, mule.el
defines a utility function decode-coding-inserted-region that also
uses a cons cell.  (This utility function is currently used in
jka-compr.el, pcvs.el and url-handlers.el.)  Both it and its users are
sufficiently complex that I couldn't easily decide whether the change
you suggest will DTRT for them.  Can you help me understand that?

> By the way, for emacs-unicode-2, I have a plan of new
> functions insert-region-contents and write-region-to-buffer
> that do the similar thing as insert-file-contents and
> write-region but gets/puts the contents from/to a region of
> a specified buffer.  They encapsulte various weird tasks for
> decoding and encoding, and can easility be used both by
> tar-mode and arc-mode.

That's good to hear.  I hope that insert-file-contents and
write-region will be implemented on top of these new primitives, or at
least that they all will use the same subroutines to encode/decode
text and set the buffer's file coding system, because currently
tar-mode and arc-mode mimic those parts of insert-file-contents (in
two subtly different ways).

Re: EOL conversion of files in .tar archives

[Kenichi Handa]

In article <u3b1mk6gd.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > For the trunk, how about trying the fix of my version
> > (i.e. the change to find-buffer-file-type-coding-system)?

> I'm uneasy with that change, because there are other places that use a
> cons cell as the second argument to find-operation-coding-system, and
> I'm not sure what you propose is good for them.  For example, mule.el
> defines a utility function decode-coding-inserted-region that also
> uses a cons cell.  (This utility function is currently used in
> jka-compr.el, pcvs.el and url-handlers.el.)  Both it and its users are
> sufficiently complex that I couldn't easily decide whether the change
> you suggest will DTRT for them.  Can you help me understand that?

The docstring of find-operation-coding-system has this
paragraph.

  If operation is `insert-file-contents', the argument corresponding to
  TARGET may be a cons (FILENAME . BUFFER).  In that case, FILENAME is a
  file name to look up, and BUFFER is a buffer that contains the file's
  contents (not yet decoded).  If `file-coding-system-alist' specifies a
  function to call for FILENAME, that function should examine the
  contents of BUFFER instead of reading the file.

So, find-buffer-file-type-coding-system should handle
(FILENAME . BUFFER) as described above.  That function has
this code:

		 ;; For any other existing file, decide based on contents.
		 ((file-exists-p target)
		  (setq undecided t))

If file-exists-p is used to check whether or not there
exists contents that can decide EOL type, isn't it TRT to
return undecided in (FILENAME . BUFFER) case.

> > By the way, for emacs-unicode-2, I have a plan of new
> > functions insert-region-contents and write-region-to-buffer
> > that do the similar thing as insert-file-contents and
> > write-region but gets/puts the contents from/to a region of
> > a specified buffer.  They encapsulte various weird tasks for
> > decoding and encoding, and can easility be used both by
> > tar-mode and arc-mode.

> That's good to hear.  I hope that insert-file-contents and
> write-region will be implemented on top of these new primitives, or at
> least that they all will use the same subroutines to encode/decode
> text and set the buffer's file coding system, because currently
> tar-mode and arc-mode mimic those parts of insert-file-contents (in
> two subtly different ways).

I have not yet decided how to implement them.  As soon as I
find a good way, I'll write the plan here to hear more
opinions.

---
Kenichi Handa
handa@m17n.org

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> CC: emacs-devel@gnu.org
> Date: Thu, 24 May 2007 17:15:23 +0900
> 
> > I'm uneasy with that change, because there are other places that use a
> > cons cell as the second argument to find-operation-coding-system, and
> > I'm not sure what you propose is good for them.  For example, mule.el
> > defines a utility function decode-coding-inserted-region that also
> > uses a cons cell.  (This utility function is currently used in
> > jka-compr.el, pcvs.el and url-handlers.el.)  Both it and its users are
> > sufficiently complex that I couldn't easily decide whether the change
> > you suggest will DTRT for them.  Can you help me understand that?
> 
> The docstring of find-operation-coding-system has this
> paragraph.
> 
>   If operation is `insert-file-contents', the argument corresponding to
>   TARGET may be a cons (FILENAME . BUFFER).  In that case, FILENAME is a
>   file name to look up, and BUFFER is a buffer that contains the file's
>   contents (not yet decoded).  If `file-coding-system-alist' specifies a
>   function to call for FILENAME, that function should examine the
>   contents of BUFFER instead of reading the file.
> 
> So, find-buffer-file-type-coding-system should handle
> (FILENAME . BUFFER) as described above.

Well, but it doesn't: it does NOT examine the contents of BUFFER at
all; it relies on that to be done when it returns undecided.  And that
is the reason I asked the question about decode-coding-inserted-region:
I'm not sure it will look at the region's contents to find the encoding,
and neither am I sure what should be done in that context, since we
are inserting only a portion of the buffer.

So I'd still be happy to know whether your suggestion is not going to
break decode-coding-inserted-region and its users.

> That function has this code:
> 
> 		 ;; For any other existing file, decide based on contents.
> 		 ((file-exists-p target)
> 		  (setq undecided t))
> 
> If file-exists-p is used to check whether or not there
> exists contents that can decide EOL type, isn't it TRT to
> return undecided in (FILENAME . BUFFER) case.

It's probably so, but I'm still not sure.  Also, please note that
returning undecided will cause the text encoding to be detected, not
only the EOL type.

Re: EOL conversion of files in .tar archives

[Kenichi Handa]

In article <uy7jeiptz.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> >   If operation is `insert-file-contents', the argument corresponding to
> >   TARGET may be a cons (FILENAME . BUFFER).  In that case, FILENAME is a
> >   file name to look up, and BUFFER is a buffer that contains the file's
> >   contents (not yet decoded).  If `file-coding-system-alist' specifies a
> >   function to call for FILENAME, that function should examine the
> >   contents of BUFFER instead of reading the file.
> > 
> > So, find-buffer-file-type-coding-system should handle
> > (FILENAME . BUFFER) as described above.

> Well, but it doesn't: it does NOT examine the contents of BUFFER at
> all; it relies on that to be done when it returns undecided.

Ah, for that the docstring should be changed something like
this.

If `file-coding-system-alist' specifies a function to call
for FILENAME, and that function is going to descide the
coding system by examining the contents for decoding, it
should examine the contents of BUFFER instead of reading the
file.

Ummm, a little bit wordy, isn't it?  Could someone make it
more concise?

> And that
> is the reason I asked the question about decode-coding-inserted-region:
> I'm not sure it will look at the region's contents to find the encoding,
> and neither am I sure what should be done in that context, since we
> are inserting only a portion of the buffer.

> So I'd still be happy to know whether your suggestion is not going to
> break decode-coding-inserted-region and its users.

decode-coding-inserted-region itself doesn't detect a coding
system, but internal function decode-coding-region does if
`undecided' is given.

So, the meaning of find-operation-coding-system returning a
not-fully-specified coding system (e.g. undecided, latin-1,
undecided-dos) is to ask decoding routine to detect the
undecided part by the default detection algorithm.

> > That function has this code:
> > 
> > 		 ;; For any other existing file, decide based on contents.
> > 		 ((file-exists-p target)
> > 		  (setq undecided t))
> > 
> > If file-exists-p is used to check whether or not there
> > exists contents that can decide EOL type, isn't it TRT to
> > return undecided in (FILENAME . BUFFER) case.

> It's probably so, but I'm still not sure.  Also, please note that
> returning undecided will cause the text encoding to be detected, not
> only the EOL type.

Is it the problem?  If the file doesn't contain EOL code,
the resulting buffer-file-coding-system doesn't specify EOL
code.  Then, if a user edits it and adds a newline, on
writing, that newline is encoded by the system's detault EOL
code.  On Windows, that is CR LF.

---
Kenichi Handa
handa@m17n.org

Re: EOL conversion of files in .tar archives

[Richard Stallman]

Is there a problem here that we need to fix for Emacs 22?
I cannot tell whether you are talking about a real problem,
or just something that isn't elegant.

Re: EOL conversion of files in .tar archives

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, emacs-devel@gnu.org
> Date: Sun, 27 May 2007 19:21:06 -0400
> 
> Is there a problem here that we need to fix for Emacs 22?

No.  Emacs 22 (i.e. the release branch) already has a fix for the
original problem.

> I cannot tell whether you are talking about a real problem,
> or just something that isn't elegant.

We were talking about how to fix this on the trunk.  The fix I checked
in on the trunk was a slightly less safe version of the one committed
to the branch.  Handa-san suggested a different, more general way to
fix this on the trunk, and we've been discussing his suggestion.

Re: EOL conversion of files in .tar archives

[Richard Stallman]

    > I cannot tell whether you are talking about a real problem,
    > or just something that isn't elegant.

    We were talking about how to fix this on the trunk.  The fix I checked
    in on the trunk was a slightly less safe version of the one committed
    to the branch.  Handa-san suggested a different, more general way to
    fix this on the trunk, and we've been discussing his suggestion.

Ok, thanks.

Please make sure that you install some sort of fix for this in the
trunk, one way or another.