[PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[David Glasser]

[-- Attachment #1: Type: text/plain, Size: 3268 bytes --]

I noticed this morning that when opening a tarball containing a file
with a long filename in tar-mode, the entry for that file didn't list
the full filename; it only listed some suffix of the filename.  I did
some digging and found that newish tar standards (among others)
support a "prefix" field in the tar header block to overcome tar
format limitations; however, tar-mode doesn't support it.

You should be able to test this by opening the attached tarball.  The
correct output should look something like:

 drwxr-x--- glasser/eng           0 foo1
 drwxr-x--- glasser/eng           0 foo1/foo2
 drwxr-x--- glasser/eng           0 foo1/foo2/foo3
 drwxr-x--- glasser/eng           0 foo1/foo2/foo3/foo4
 drwxr-x--- glasser/eng           0 foo1/foo2/foo3/foo4/foo5
 drwxr-x--- glasser/eng           0 foo1/foo2/foo3/foo4/foo5/foo6
 drwxr-x--- glasser/eng           0 foo1/foo2/foo3/foo4/foo5/foo6/foo7
 drwxr-x--- glasser/eng           0 foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8
 drwxr-x--- glasser/eng           0 foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12/foo13
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12/foo13/foo14
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12/foo13/foo14/foo15
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12/foo13/foo14/foo15/foo16
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12/foo13/foo14/foo15/foo16/foo17
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12/foo13/foo14/foo15/foo16/foo17/foo18
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12/foo13/foo14/foo15/foo16/foo17/foo18/foo19
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12/foo13/foo14/foo15/foo16/foo17/foo18/foo19/foo20
 drwxr-x--- glasser/eng           0
foo1/foo2/foo3/foo4/foo5/foo6/foo7/foo8/foo9/foo10/foo11/foo12/foo13/foo14/foo15/foo16/foo17/foo18/foo19/foo20/foo21

Without my patch at all, the bottom few look like they aren't inside foo1/foo2/.

With all of the patch except for the change in the initial
initialization of name-end, the results look correct, except the
penultimate line ends in "foo2" instead of "foo20"; that is because
the name field here fill the entire field with no nulls, but the code
previously assumed there would be at least one null.  (It's possible
that similar adjustments are required for the initial values of
link-end, gname-end, and uname-end.)

Let me know if this is a significant enough change that papers are necessary.

Also, please note that until an hour ago I had no idea what the TAR
format looked like!

--dave

-- 
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/

[-- Attachment #2: tar-prefix.patch.txt --]
[-- Type: text/plain, Size: 3686 bytes --]

Index: tar-mode.el
===================================================================
*** tar-mode.el	(revision 200)
--- tar-mode.el	(working copy)
*************** This information is useful, but it takes
*** 197,203 ****
  (defconst tar-gname-offset (+ tar-uname-offset 32))
  (defconst tar-dmaj-offset (+ tar-gname-offset 32))
  (defconst tar-dmin-offset (+ tar-dmaj-offset 8))
! (defconst tar-end-offset (+ tar-dmin-offset 8))
  
  (defun tar-header-block-tokenize (string)
    "Return a `tar-header' structure.
--- 197,204 ----
  (defconst tar-gname-offset (+ tar-uname-offset 32))
  (defconst tar-dmaj-offset (+ tar-gname-offset 32))
  (defconst tar-dmin-offset (+ tar-dmaj-offset 8))
! (defconst tar-prefix-offset (+ tar-dmin-offset 8))
! (defconst tar-end-offset (+ tar-prefix-offset 155))
  
  (defun tar-header-block-tokenize (string)
    "Return a `tar-header' structure.
*************** write-date, checksum, link-type, and lin
*** 207,219 ****
  	(;(some 'plusp string)		 ; <-- oops, massive cycle hog!
  	 (or (not (= 0 (aref string 0))) ; This will do.
  	     (not (= 0 (aref string 101))))
! 	 (let* ((name-end (1- tar-mode-offset))
  		(link-end (1- tar-magic-offset))
  		(uname-end (1- tar-gname-offset))
  		(gname-end (1- tar-dmaj-offset))
  		(link-p (aref string tar-linkp-offset))
  		(magic-str (substring string tar-magic-offset (1- tar-uname-offset)))
! 		(uname-valid-p (or (string= "ustar  " magic-str) (string= "GNUtar " magic-str)))
  		name linkname
  		(nulsexp   "[^\000]*\000"))
  	   (when (string-match nulsexp string tar-name-offset)
--- 208,221 ----
  	(;(some 'plusp string)		 ; <-- oops, massive cycle hog!
  	 (or (not (= 0 (aref string 0))) ; This will do.
  	     (not (= 0 (aref string 101))))
! 	 (let* ((name-end tar-mode-offset)
  		(link-end (1- tar-magic-offset))
  		(uname-end (1- tar-gname-offset))
  		(gname-end (1- tar-dmaj-offset))
  		(link-p (aref string tar-linkp-offset))
  		(magic-str (substring string tar-magic-offset (1- tar-uname-offset)))
! 		(uname-valid-p (or (string= "ustar  " magic-str) (string= "GNUtar " magic-str)
!                                    (string= "ustar\0000" magic-str)))
  		name linkname
  		(nulsexp   "[^\000]*\000"))
  	   (when (string-match nulsexp string tar-name-offset)
*************** write-date, checksum, link-type, and lin
*** 229,234 ****
--- 231,242 ----
  			    nil
  			  (- link-p ?0)))
  	   (setq linkname (substring string tar-link-offset link-end))
+            (when (and uname-valid-p
+                       (string-match nulsexp string tar-prefix-offset)
+                       (> (match-end 0) (1+ tar-prefix-offset)))
+              (setq name (concat (substring string tar-prefix-offset
+                                            (1- (match-end 0)))
+                                 "/" name)))
  	   (if default-enable-multibyte-characters
  	       (setq name
  		     (decode-coding-string name
*************** write-date, checksum, link-type, and lin
*** 255,261 ****
  	     (and uname-valid-p (substring string tar-uname-offset uname-end))
  	     (and uname-valid-p (substring string tar-gname-offset gname-end))
  	     (tar-parse-octal-integer string tar-dmaj-offset tar-dmin-offset)
! 	     (tar-parse-octal-integer string tar-dmin-offset tar-end-offset)
  	     )))
  	(t 'empty-tar-block)))
  
--- 263,269 ----
  	     (and uname-valid-p (substring string tar-uname-offset uname-end))
  	     (and uname-valid-p (substring string tar-gname-offset gname-end))
  	     (tar-parse-octal-integer string tar-dmaj-offset tar-dmin-offset)
! 	     (tar-parse-octal-integer string tar-dmin-offset tar-prefix-offset)
  	     )))
  	(t 'empty-tar-block)))
  

[-- Attachment #3: test.tar.gz --]
[-- Type: application/x-gzip, Size: 437 bytes --]

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[Juri Linkov]

> With all of the patch except for the change in the initial
> initialization of name-end, the results look correct, except the
> penultimate line ends in "foo2" instead of "foo20"; that is because
> the name field here fill the entire field with no nulls, but the code
> previously assumed there would be at least one null.  (It's possible
> that similar adjustments are required for the initial values of
> link-end, gname-end, and uname-end.)

Does your patch also handle @LongLink currently unsupported by tar-mode.el?

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[David Glasser]

On Sun, Apr 13, 2008 at 3:23 PM, Juri Linkov <juri@jurta.org> wrote:
> > With all of the patch except for the change in the initial
>  > initialization of name-end, the results look correct, except the
>  > penultimate line ends in "foo2" instead of "foo20"; that is because
>  > the name field here fill the entire field with no nulls, but the code
>  > previously assumed there would be at least one null.  (It's possible
>  > that similar adjustments are required for the initial values of
>  > link-end, gname-end, and uname-end.)
>
>  Does your patch also handle @LongLink currently unsupported by tar-mode.el?

Is it actually unsupported?  When I tried to make a tarball to test
this issue with GNU tar, I saw stuff about "@LongLink" when I opened
it in tar-mode.  (Try making a tarball from the one I sent using GNU
tar; you'll see what I mean; I made my test tarball with pax, since
that was what was used to make the one I opened when I first noticed
this issue.)  I don't know if it was correct or not, though.

--dave

-- 
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

>> > With all of the patch except for the change in the initial
>>  > initialization of name-end, the results look correct, except the
>>  > penultimate line ends in "foo2" instead of "foo20"; that is because
>>  > the name field here fill the entire field with no nulls, but the code
>>  > previously assumed there would be at least one null.  (It's possible
>>  > that similar adjustments are required for the initial values of
>>  > link-end, gname-end, and uname-end.)
>>
>>  Does your patch also handle @LongLink currently unsupported by tar-mode.el?
>
> Is it actually unsupported?  When I tried to make a tarball to test
> this issue with GNU tar, I saw stuff about "@LongLink" when I opened
> it in tar-mode.  (Try making a tarball from the one I sent using GNU
> tar; you'll see what I mean; I made my test tarball with pax, since
> that was what was used to make the one I opened when I first noticed
> this issue.)  I don't know if it was correct or not, though.

Please see a test tarball attached below.  It is correctly created with
GNU tar, but opening it in Emacs displays the following message:

    Warning: premature EOF parsing tar file

Also its file listing is incomplete, file names are truncated, and content
of some files is inserted into the file listing in the tar-mode buffer.

I thought your patch will fix these problems.


[-- Attachment #2: test.tgz --]
[-- Type: application/x-gtar, Size: 108633 bytes --]

[-- Attachment #3: Type: text/plain, Size: 45 bytes --]


-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[David Glasser]

On Tue, Apr 15, 2008 at 3:35 PM, Juri Linkov <juri@jurta.org> wrote:
> >> > With all of the patch except for the change in the initial
>  >>  > initialization of name-end, the results look correct, except the
>  >>  > penultimate line ends in "foo2" instead of "foo20"; that is because
>  >>  > the name field here fill the entire field with no nulls, but the code
>  >>  > previously assumed there would be at least one null.  (It's possible
>  >>  > that similar adjustments are required for the initial values of
>  >>  > link-end, gname-end, and uname-end.)
>  >>
>  >>  Does your patch also handle @LongLink currently unsupported by tar-mode.el?
>  >
>  > Is it actually unsupported?  When I tried to make a tarball to test
>  > this issue with GNU tar, I saw stuff about "@LongLink" when I opened
>  > it in tar-mode.  (Try making a tarball from the one I sent using GNU
>  > tar; you'll see what I mean; I made my test tarball with pax, since
>  > that was what was used to make the one I opened when I first noticed
>  > this issue.)  I don't know if it was correct or not, though.
>
>  Please see a test tarball attached below.  It is correctly created with
>  GNU tar, but opening it in Emacs displays the following message:
>
>     Warning: premature EOF parsing tar file
>
>  Also its file listing is incomplete, file names are truncated, and content
>  of some files is inserted into the file listing in the tar-mode buffer.
>
>  I thought your patch will fix these problems.

Ugh, that's pretty gross.  Mine fixes issues with a different way of
encoding long file names, as far as I understand.  However, there is
*some* long-link support in tar-mode.el, just not whatever's up with
that file.

--dave


-- 
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[Juri Linkov]

>> >> > With all of the patch except for the change in the initial
>>  >>  > initialization of name-end, the results look correct, except the
>>  >>  > penultimate line ends in "foo2" instead of "foo20"; that is because
>>  >>  > the name field here fill the entire field with no nulls, but the code
>>  >>  > previously assumed there would be at least one null.  (It's possible
>>  >>  > that similar adjustments are required for the initial values of
>>  >>  > link-end, gname-end, and uname-end.)
>>  >>
>>  >>  Does your patch also handle @LongLink currently unsupported by tar-mode.el?
>>  >
>>  > Is it actually unsupported?  When I tried to make a tarball to test
>>  > this issue with GNU tar, I saw stuff about "@LongLink" when I opened
>>  > it in tar-mode.  (Try making a tarball from the one I sent using GNU
>>  > tar; you'll see what I mean; I made my test tarball with pax, since
>>  > that was what was used to make the one I opened when I first noticed
>>  > this issue.)  I don't know if it was correct or not, though.
>>
>>  Please see a test tarball attached below.  It is correctly created with
>>  GNU tar, but opening it in Emacs displays the following message:
>>
>>     Warning: premature EOF parsing tar file
>>
>>  Also its file listing is incomplete, file names are truncated, and content
>>  of some files is inserted into the file listing in the tar-mode buffer.
>>
>>  I thought your patch will fix these problems.
>
> Ugh, that's pretty gross.  Mine fixes issues with a different way of
> encoding long file names, as far as I understand.  However, there is
> *some* long-link support in tar-mode.el, just not whatever's up with
> that file.

It seems tar-mode.el has some limitations on the file name lengths.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[David Glasser]

On Wed, Apr 16, 2008 at 4:17 PM, Juri Linkov <juri@jurta.org> wrote:
>
> >> >> > With all of the patch except for the change in the initial
>  >>  >>  > initialization of name-end, the results look correct, except the
>  >>  >>  > penultimate line ends in "foo2" instead of "foo20"; that is because
>  >>  >>  > the name field here fill the entire field with no nulls, but the code
>  >>  >>  > previously assumed there would be at least one null.  (It's possible
>  >>  >>  > that similar adjustments are required for the initial values of
>  >>  >>  > link-end, gname-end, and uname-end.)
>  >>  >>
>  >>  >>  Does your patch also handle @LongLink currently unsupported by tar-mode.el?
>  >>  >
>  >>  > Is it actually unsupported?  When I tried to make a tarball to test
>  >>  > this issue with GNU tar, I saw stuff about "@LongLink" when I opened
>  >>  > it in tar-mode.  (Try making a tarball from the one I sent using GNU
>  >>  > tar; you'll see what I mean; I made my test tarball with pax, since
>  >>  > that was what was used to make the one I opened when I first noticed
>  >>  > this issue.)  I don't know if it was correct or not, though.
>  >>
>  >>  Please see a test tarball attached below.  It is correctly created with
>  >>  GNU tar, but opening it in Emacs displays the following message:
>  >>
>  >>     Warning: premature EOF parsing tar file
>  >>
>  >>  Also its file listing is incomplete, file names are truncated, and content
>  >>  of some files is inserted into the file listing in the tar-mode buffer.
>  >>
>  >>  I thought your patch will fix these problems.
>  >
>  > Ugh, that's pretty gross.  Mine fixes issues with a different way of
>  > encoding long file names, as far as I understand.  However, there is
>  > *some* long-link support in tar-mode.el, just not whatever's up with
>  > that file.
>
>  It seems tar-mode.el has some limitations on the file name lengths.

Apparently so.  I think my patch improve it, though, at least for one
version of tar file.  Is there something I need to do to increase the
chances of it being applied?

--dave


-- 
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[Juri Linkov]

>>  It seems tar-mode.el has some limitations on the file name lengths.
>
> Apparently so.  I think my patch improve it, though, at least for one
> version of tar file.  Is there something I need to do to increase the
> chances of it being applied?

Let our maintainers decide if your patch is tiny enough to install
without papers.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[Chong Yidong]

Juri Linkov <juri@jurta.org> writes:

>>>  It seems tar-mode.el has some limitations on the file name lengths.
>>
>> Apparently so.  I think my patch improve it, though, at least for one
>> version of tar file.  Is there something I need to do to increase the
>> chances of it being applied?
>
> Let our maintainers decide if your patch is tiny enough to install
> without papers.

The patch is small enough to install without papers.

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[Juri Linkov]

>>>>  It seems tar-mode.el has some limitations on the file name lengths.
>>>
>>> Apparently so.  I think my patch improve it, though, at least for one
>>> version of tar file.  Is there something I need to do to increase the
>>> chances of it being applied?
>>
>> Let our maintainers decide if your patch is tiny enough to install
>> without papers.
>
> The patch is small enough to install without papers.

David, could you send a ChangeLog entry that describes changes of your patch?

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: [PATCH] support filename prefixes and 100-character filenames in tar-mode.el

[David Glasser]

On Tue, Apr 22, 2008 at 1:51 PM, Juri Linkov <juri@jurta.org> wrote:
>
> >>>>  It seems tar-mode.el has some limitations on the file name lengths.
>  >>>
>  >>> Apparently so.  I think my patch improve it, though, at least for one
>  >>> version of tar file.  Is there something I need to do to increase the
>  >>> chances of it being applied?
>  >>
>  >> Let our maintainers decide if your patch is tiny enough to install
>  >> without papers.
>  >
>  > The patch is small enough to install without papers.
>
>  David, could you send a ChangeLog entry that describes changes of your patch?

     2008-04-22  David Glasser  <glasser@davidglasser.net>

             * tar-mode.el (tar-prefix-offset): New constant.
             (tar-header-block-tokenize): Support paths with long names
              which use the "ustar" standard.

--dave

-- 
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/