bug#29235: 26.0.50; tar.xz broken

bug#29235: 26.0.50; tar.xz broken

[Devon Sean McCullough]

Emacs fails to open tar.xz files

	Error while executing "xz -c -q -d < /tmp/bar.tar.xz"
	/bin/sh: xz: command not found

Note that MacOSX lacks xz but has tar
which automagically decompresses xz.

		Peace
			--Devon

P.S. Transcript follows:

$ type xz
bash: type: xz: not found
$ type tar
tar is /usr/bin/tar
$ tar -h
tar(bsdtar): manipulate archive files
First option must be a mode specifier:
  -c Create  -r Add/Replace  -t List  -u Update  -x Extract
Common Options:
  -b #  Use # 512-byte records per I/O block
  -f <filename>  Location of archive
  -v    Verbose
  -w    Interactive
Create: tar -c [options] [<file> | <dir> | @<archive> | -C <dir> ]
  <file>, <dir>  add these items to archive
  -z, -j, -J, --lzma  Compress archive with gzip/bzip2/xz/lzma
  --format {ustar|pax|cpio|shar}  Select archive format
  --exclude <pattern>  Skip files that match pattern
  -C <dir>  Change to <dir> before processing remaining files
  @<archive>  Add entries from <archive> to output
List: tar -t [options] [<patterns>]
  <patterns>  If specified, list only entries that match
Extract: tar -x [options] [<patterns>]
  <patterns>  If specified, extract only entries that match
  -k    Keep (don't overwrite) existing files
  -m    Don't restore modification times
  -O    Write entries to stdout, don't restore to disk
  -p    Restore permissions (including ACLs, owner, file flags)
bsdtar 2.8.3 - libarchive 2.8.3

$ cd /tmp
$ echo foo > bar
$ tar -cJ bar > bar.tar.xz
$ tar -tv < bar.tar.xz
-rw-r--r--  0 devon  wheel       4 Nov  9 14:43 bar
$ tar -xO bar < bar.tar.xz
foo

$ cat > test.el <<.
(print emacs-version)
(dolist (b (buffer-list))
  (with-current-buffer b
    (print b)
    (print (buffer-string))))
.
$ Emacs --batch -Q -q --eval '(setq debug-on-error t)' /tmp/bar.tar.xz --load test.el
Warning: arch-dependent data dir '/Users/build/workspace/Emacs-Multi-Build/label/mavericks/emacs-source/nextstep/Emacs.app/Contents/MacOS/libexec/': No such file or directory
XZ uncompressing bar.tar.xz...
Parsing tar file...
Parsing tar file...done

"26.0.50"

#<buffer *scratch*>

""

#<buffer  *Minibuf-0*>

""

#<buffer *Messages*>

"Warning: arch-dependent data dir \342\200\231/Users/build/workspace/Emacs-Multi-Build/label/mavericks/emacs-source/nextstep/Emacs.app/Contents/MacOS/libexec/\342\200\231: No such file or directory

XZ uncompressing bar.tar.xz...
Parsing tar file...done
"

#<buffer  *code-conversion-work*>

"/private/tmp/test.el"

#<buffer bar.tar.xz>

"
"

#<buffer  *jka-compr-error*>

"Error while executing \"xz -c -q -d < /tmp/bar.tar.xz\"

/bin/sh: xz: command not found
"

#<buffer  *tar-data bar.tar.xz*>

""

#<buffer  *load*>

"(print emacs-version)
(dolist (b (buffer-list))
  (with-current-buffer b
    (print b)
    (print (buffer-string))))
"

In GNU Emacs 26.0.50 (build 1, x86_64-apple-darwin13.4.0, NS appkit-1265.21 Version 10.9.5 (Build 13F1911))
 of 2017-06-07 built on builder10-9.local
Windowing system distributor 'Apple', version 10.3.1404
Recent messages:
Checking 57 files in /Applications/Emacs-2017f07w.app/Contents/Resources/lisp/eshell...
Checking 70 files in /Applications/Emacs-2017f07w.app/Contents/Resources/lisp/erc...
Checking 34 files in /Applications/Emacs-2017f07w.app/Contents/Resources/lisp/emulation...
Checking 174 files in /Applications/Emacs-2017f07w.app/Contents/Resources/lisp/emacs-lisp...
Checking 24 files in /Applications/Emacs-2017f07w.app/Contents/Resources/lisp/cedet...
Checking 57 files in /Applications/Emacs-2017f07w.app/Contents/Resources/lisp/calendar...
Checking 87 files in /Applications/Emacs-2017f07w.app/Contents/Resources/lisp/calc...
Checking 103 files in /Applications/Emacs-2017f07w.app/Contents/Resources/lisp/obsolete...
Checking for load-path shadows...done
Undo! [2 times]
user-error: No further undo information
Configured using:
 'configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp' --with-modules'

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Shell

Minor modes in effect:
  erc-list-mode: t
  erc-menu-mode: t
  erc-autojoin-mode: t
  erc-ring-mode: t
  erc-networks-mode: t
  erc-pcomplete-mode: t
  erc-track-mode: t
  erc-match-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-netsplit-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rfc822 mml mml-sec epa epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils pulse eieio-opt
speedbar sb-image ezimage dframe find-func erc-list erc-menu erc-join
erc-ring erc-networks erc-pcomplete erc-track erc-match erc-button
wid-edit erc-fill erc-stamp erc-netsplit erc-goodies erc erc-backend
erc-compat pcmpl-unix pcmpl-gnu tar-mode jka-compr view dired-aux
tramp-cache ange-ftp tramp-ftp tramp tramp-compat tramp-loaddefs
trampver shell pcomplete parse-time format-spec dired dired-loaddefs
cl-indent network-stream puny nsm subr-x auth-source password-cache
starttls tls gnutls debug ibuf-ext ibuffer ibuffer-loaddefs seq lpr
mule-diag misearch multi-isearch cl-print help-fns radix-tree
slime-fancy slime-trace-dialog slime-fontifying-fu slime-package-fu
slime-references slime-compiler-notes-tree slime-scratch
slime-presentations advice bridge slime-macrostep macrostep
slime-mdot-fu slime-enclosing-context slime-fuzzy slime-fancy-trace
slime-fancy-inspector slime-c-p-c slime-editing-commands slime-autodoc
slime-repl elp slime-parse slime derived cl-extra help-mode edmacro
kmacro lisp-mnt gud apropos compile etags xref cl-seq project eieio
byte-opt bytecomp byte-compile cconv eieio-core eieio-loaddefs arc-mode
archive-mode noutline outline easy-mmode easymenu pp comint ansi-color
ring hyperspec thingatpt browse-url slime-autoloads cl-macs cl gv
cl-loaddefs cl-lib time-date tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/ns-win ns-win ucs-normalize
mule-util term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote kqueue cocoa ns multi-tty make-network-process emacs)

Memory information:
((conses 16 385031 15149)
 (symbols 48 32323 0)
 (miscs 40 177 721)
 (strings 32 58176 9683)
 (string-bytes 1 1721858)
 (vectors 16 47494)
 (vector-slots 8 840851 12047)
 (floats 8 99 452)
 (intervals 56 4661 418)
 (buffers 976 31))

bug#29235: 26.0.50; tar.xz broken

[Eli Zaretskii]

> From: Devon Sean McCullough <Emacs-Hacker2017@jovi.net>
> Date: Thu, 9 Nov 2017 16:41:34 -0500
> 
> Emacs fails to open tar.xz files
> 
> 	Error while executing "xz -c -q -d < /tmp/bar.tar.xz"
> 	/bin/sh: xz: command not found
> 
> Note that MacOSX lacks xz but has tar
> which automagically decompresses xz.

Having tar or bsdtar does not help in this case, because what Emacs
needs is the uncompressed tar file (which it will then display in
tar-mode).  By contrast, bsdtar will unpack the archive completely,
writing the files to the disk.  AFAIK, tar and bsdtar don't have an
option to just uncompress the archive, or did I miss something?

So yes, you really need top install xz for this feature to work.

bug#29235: 26.0.50; tar.xz broken

[Andreas Schwab]

On Nov 09 2017, Devon Sean McCullough <Emacs-Hacker2017@jovi.net> wrote:

> Note that MacOSX lacks xz

Not even under a different name?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#29235: 26.0.50; tar.xz broken

[Devon Sean McCullough]

> On Nov 10, 2017, at 4:37 AM, Andreas Schwab <schwab@linux-m68k.org> wrote:
> 
> On Nov 09 2017, Devon Sean McCullough <Emacs-Hacker2017@jovi.net> wrote:
> 
>> Note that MacOSX lacks xz
> 
> Not even under a different name?

Turns out xz is redundant because gunzip, gzcat and zcat all work.

		Peace
			—Devon

P.S. Transcript follows:
$ for C in cat gzcat zcat gunzip; do echo -en "$C\t"; "$C" < bar.tar.xz | hexdump -C | head -1; done
cat	00000000  fd 37 7a 58 5a 00 00 04  e6 d6 b4 46 02 00 21 01  |.7zXZ......F..!.|
gzcat	00000000  62 61 72 00 00 00 00 00  00 00 00 00 00 00 00 00  |bar.............|
zcat	00000000  62 61 72 00 00 00 00 00  00 00 00 00 00 00 00 00  |bar.............|
gunzip	00000000  62 61 72 00 00 00 00 00  00 00 00 00 00 00 00 00  |bar.............|

P.P.S. Users need to list and retrieve archived files.
MacOSX bsdtar -tv and -xO provide these capabilities.
I’ve never heard of anyone saving archives from Emacs
so dired and find-file satisfy 99% of mundane user needs.
Other requirements are artifacts of a brittle implementation
which, fortunately in this particular case, is sufficient after all.

bug#29235: Builtin bsdtar decompression for .tar.xz not used

[Lars Ingebrigtsen]

Devon Sean McCullough <Emacs-Hacker2017@jovi.net> writes:

>>> Note that MacOSX lacks xz
>> 
>> Not even under a different name?
>
> Turns out xz is redundant because gunzip, gzcat and zcat all work.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

So Macos doesn't have a separate binary for xz, but makes gunzip handle
that format, too?

So would the following change make sense, or do we need more parameters
in the compressing case to make Macos gzip make an .xz archive?

diff --git a/lisp/jka-cmpr-hook.el b/lisp/jka-cmpr-hook.el
index 6933a7c1d0..5a884c7ade 100644
--- a/lisp/jka-cmpr-hook.el
+++ b/lisp/jka-cmpr-hook.el
@@ -203,7 +203,7 @@ jka-compr-compression-info-list
   ;; can-append strip-extension-flag file-magic-bytes
   ;; uncompress-function]
   (mapcar 'purecopy
-  '(["\\.Z\\'"
+  `(["\\.Z\\'"
      "compressing"    "compress"     ("-c")
      ;; gzip is more common than uncompress. It can only read, not write.
      "uncompressing"  "gzip"   ("-c" "-q" "-d")
@@ -238,10 +238,15 @@ jka-compr-compression-info-list
      "LZMA compressing"   "lzma"         ("-c" "-q" "-z")
      "LZMA uncompressing" "lzma"         ("-c" "-q" "-d")
      t t ""]
-    ["\\.xz\\'"
-     "XZ compressing"     "xz"           ("-c" "-q")
-     "XZ uncompressing"   "xz"           ("-c" "-q" "-d")
-     t t "\3757zXZ\0"]
+    ,(if (featurep 'ns)
+         ["\\.xz\\'"
+          "XZ compressing"     "gzip"           ("-c" "-q")
+          "XZ uncompressing"   "gzip"           ("-c" "-q" "-d")
+          t t "\3757zXZ\0"]
+       ["\\.xz\\'"
+        "XZ compressing"     "xz"           ("-c" "-q")
+        "XZ uncompressing"   "xz"           ("-c" "-q" "-d")
+        t t "\3757zXZ\0"])
     ["\\.txz\\'"
      "XZ compressing"     "xz"           ("-c" "-q")
      "XZ uncompressing"   "xz"           ("-c" "-q" "-d")


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29235: Builtin bsdtar decompression for .tar.xz not used

[Andreas Schwab]

On Sep 03 2021, Lars Ingebrigtsen wrote:

> +    ,(if (featurep 'ns)
> +         ["\\.xz\\'"
> +          "XZ compressing"     "gzip"           ("-c" "-q")

Don't you need to tell gzip to produce XZ format instead of GZIP?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#29235: Builtin bsdtar decompression for .tar.xz not used

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Sep 03 2021, Lars Ingebrigtsen wrote:
>
>> +    ,(if (featurep 'ns)
>> +         ["\\.xz\\'"
>> +          "XZ compressing"     "gzip"           ("-c" "-q")
>
> Don't you need to tell gzip to produce XZ format instead of GZIP?

I don't know -- that's why I asked:

> So would the following change make sense, or do we need more parameters
> in the compressing case to make Macos gzip make an .xz archive?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29235: Builtin bsdtar decompression for .tar.xz not used

[Robert Pluim]

>>>>> On Fri, 03 Sep 2021 12:45:52 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Andreas Schwab <schwab@linux-m68k.org> writes:
    >> On Sep 03 2021, Lars Ingebrigtsen wrote:
    >> 
    >>> +    ,(if (featurep 'ns)
    >>> +         ["\\.xz\\'"
    >>> +          "XZ compressing"     "gzip"           ("-c" "-q")
    >> 
    >> Don't you need to tell gzip to produce XZ format instead of GZIP?

    Lars> I don't know -- that's why I asked:

    >> So would the following change make sense, or do we need more parameters
    >> in the compressing case to make Macos gzip make an .xz archive?

From 'man gzip' on a mac:

    This version of gzip is also capable of decompressing files compressed
    using compress(1), bzip2(1), or xz(1).

So I think people need to install xz if they want to create xz format
archives.

Robert
-- 

bug#29235: Builtin bsdtar decompression for .tar.xz not used

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

>>From 'man gzip' on a mac:
>
>     This version of gzip is also capable of decompressing files compressed
>     using compress(1), bzip2(1), or xz(1).
>
> So I think people need to install xz if they want to create xz format
> archives.

Thanks.  So I've now pushed a version of the patch that only uses gzip
when decompressing .xz files.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no