unofficial mirror of bug-gnu-emacs@gnu.org 
 help / color / mirror / code / Atom feed
* bug#33919: [PATCH 0/5] Add Zstandard compression for installation and utilities
@ 2018-12-30  6:46 Alex
  2018-12-30 10:06 ` Michael Albinus
  2018-12-30 15:21 ` Eli Zaretskii
  0 siblings, 2 replies; 9+ messages in thread
From: Alex @ 2018-12-30  6:46 UTC (permalink / raw)
  To: 33919

[-- Attachment #1: Type: text/plain, Size: 2217 bytes --]


I figured that it would be nice to support other compression programs
for installed files, and Zstandard[1] seems like a good alternative
for gzip in this area due to its fast decompression and wide range of
options.

This patch series depends on the patch I posted for bug#33880[2].

I have a few questions/comments:

1) I tested about everything except the Tramp part. One possible issue
is that zstd prints a progress/summary line to stdout without the -q
flag when compressing a single file; does Tramp care about that here?

2) Is there a convenient method to avoid the duplication of setting up
the COMPR* (and perhaps other) variables in the doc/*/Makefile.in
files?  What about making a common file that other files load at the
beginning?

3) Hopefully using eval is not an issue here.  I believe the security
should be the same, since I double quoted the filename strings (which
I confirmed could be used to execute code without the extra quotes),
and the old GZIP_PROG could also be exploited.

[1] https://facebook.github.io/zstd/
[2]
https://lists.gnu.org/archive/html/bug-gnu-emacs/2018-12/msg00926.html

Alexander Gramiak (5):
  Add Zstandard compression support for etags
  Add Zstandard compression support for dired
  Add Zstandard compression support for Tramp
  * make-dist: Add option for zstd compression
  Add Zstandard compression option for make install

 INSTALL                   | 34 +++++++++++++++++++----
 Makefile.in               | 57 ++++++++++++++++++++++++++-------------
 admin/admin.el            |  2 +-
 configure.ac              | 36 ++++++++++++++++++-------
 doc/emacs/Makefile.in     | 25 ++++++++++++-----
 doc/lispintro/Makefile.in | 25 ++++++++++++-----
 doc/lispref/Makefile.in   | 25 ++++++++++++-----
 doc/man/etags.1           |  2 +-
 doc/misc/Makefile.in      | 25 ++++++++++++-----
 doc/misc/tramp.texi       |  9 ++++---
 etc/NEWS                  | 30 +++++++++++++++++++++
 lib-src/etags.c           |  3 ++-
 lisp/dired-aux.el         | 21 ++++++++++-----
 lisp/jka-cmpr-hook.el     |  4 +--
 lisp/net/tramp-archive.el |  8 +++---
 lisp/net/tramp-sh.el      |  1 +
 make-dist                 |  8 +++++-
 17 files changed, 239 insertions(+), 76 deletions(-)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-Zstandard-compression-support-for-etags.patch --]
[-- Type: text/x-patch, Size: 2354 bytes --]

From e6d4e51cc73274189e9d12062698679f79829e71 Mon Sep 17 00:00:00 2001
From: Alexander Gramiak <agrambot@gmail.com>
Date: Sat, 29 Dec 2018 11:54:23 -0600
Subject: [PATCH 1/5] Add Zstandard compression support for etags

* lib-src/etags.c: (compressors): Add zstd support.
(print_language_names): Report zstd support.
* doc/man/etags.1: Update doc.
---
 doc/man/etags.1 | 2 +-
 etc/NEWS        | 3 +++
 lib-src/etags.c | 3 ++-
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/doc/man/etags.1 b/doc/man/etags.1
index 558b249f31..42c6fcd6da 100644
--- a/doc/man/etags.1
+++ b/doc/man/etags.1
@@ -64,7 +64,7 @@ Files specified with absolute file names will be recorded
 with absolute file names.  Files generated from a source file\-\-like
 a C file generated from a source Cweb file\-\-will be recorded with
 the name of the source file.
-Compressed files are supported using gzip, bzip2, and xz.
+Compressed files are supported using gzip, bzip2, xz, and zstd.
 The programs recognize the language used in an input file based on its
 file name and contents.  The \fB\-\-language\fP switch can be used to force
 parsing of the file names following the switch according to the given
diff --git a/etc/NEWS b/etc/NEWS
index 586e7065d1..2bacb66d4a 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -373,6 +373,9 @@ The mode is automatically enabled in files that start with the
 use the new 'multifile-initialize' and 'multifile-continue' functions
 instead.
 
++++
+*** etags is now able to read Zstandard-compressed files.
+
 ** bibtex
 
 ---
diff --git a/lib-src/etags.c b/lib-src/etags.c
index 102d867b38..61f5ff7a08 100644
--- a/lib-src/etags.c
+++ b/lib-src/etags.c
@@ -527,6 +527,7 @@ static compressor compressors[] =
   { "GZ", "gzip -d -c"},
   { "bz2", "bzip2 -d -c" },
   { "xz", "xz -d -c" },
+  { "zst", "zstd -d -c -q" },
   { NULL }
 };
 
@@ -869,7 +870,7 @@ followed by the name of an interpreter.  If no such sequence is found,\n\
 Fortran is tried first; if no tags are found, C is tried next.\n\
 When parsing any C file, a \"class\" or \"template\" keyword\n\
 switches to C++.");
-  puts ("Compressed files are supported using gzip, bzip2, and xz.\n\
+  puts ("Compressed files are supported using gzip, bzip2, xz, and zstd.\n\
 \n\
 For detailed help on a given language use, for example,\n\
 etags --help --lang=ada.");
-- 
2.20.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Add-Zstandard-compression-support-for-dired.patch --]
[-- Type: text/x-patch, Size: 3055 bytes --]

From 4e60a56756958af4d8af095fab0dad661628ffcb Mon Sep 17 00:00:00 2001
From: Alexander Gramiak <agrambot@gmail.com>
Date: Sat, 29 Dec 2018 11:54:24 -0600
Subject: [PATCH 2/5] Add Zstandard compression support for dired

* lisp/dired-aux.el (dired-compress-file-suffixes): Add suffixes for
zstd compressed tarballs and other files.
(dired-compress-file): Treat options found in
`dired-compress-file-suffixes' as arguments instead of part of the
program name.
---
 etc/NEWS          |  4 ++++
 lisp/dired-aux.el | 21 +++++++++++++++------
 2 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 2bacb66d4a..af99e29e40 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -388,6 +388,10 @@ remapped to these, respectively.
 +++
 *** New command 'dired-create-empty-file'.
 
+---
+*** Zstandard compression is now supported for 'dired-do-compress' and
+'dired-do-compress-to'.
+
 ** Change Logs and VC
 
 *** Recording ChangeLog entries doesn't require an actual file.
diff --git a/lisp/dired-aux.el b/lisp/dired-aux.el
index 9cd7998216..909d6f6666 100644
--- a/lisp/dired-aux.el
+++ b/lisp/dired-aux.el
@@ -996,6 +996,9 @@ dired-compress-file-suffixes
     ("\\.bz2\\'" "" "bunzip2")
     ("\\.xz\\'" "" "unxz")
     ("\\.zip\\'" "" "unzip -o -d %o %i")
+    ("\\.tar\\.zst\\'" "" "unzstd -c %i | tar -xf -")
+    ("\\.tzst\\'" "" "unzstd -c %i | tar -xf -")
+    ("\\.zst\\'" "" "unzstd --rm")
     ("\\.7z\\'" "" "7z x -aoa -o%o %i")
     ;; This item controls naming for compression.
     ("\\.tar\\'" ".tgz" nil)
@@ -1020,6 +1023,7 @@ dired-compress-files-alist
   '(("\\.tar\\.gz\\'" . "tar -cf - %i | gzip -c9 > %o")
     ("\\.tar\\.bz2\\'" . "tar -cf - %i | bzip2 -c9 > %o")
     ("\\.tar\\.xz\\'" . "tar -cf - %i | xz -c9 > %o")
+    ("\\.tar\\.zst\\'" . "tar -cf - %i | zstd -19 -o %o")
     ("\\.zip\\'" . "zip %o -r --filesync %i"))
   "Control the compression shell command for `dired-do-compress-to'.
 
@@ -1103,12 +1107,17 @@ dired-compress-file
                     nil t)
                    nil t)))
              ;; We found an uncompression rule.
-             (when (not
-                    (dired-check-process
-                     (concat "Uncompressing " file)
-                     command
-                     file))
-               newname)))
+             (let ((match (string-match " " command))
+                   (msg (concat "Uncompressing " file)))
+               (unless (if match
+                           (dired-check-process msg
+                                                (substring command 0 match)
+                                                (substring command (1+ match))
+                                                file)
+                         (dired-check-process msg
+                                              command
+                                              file))
+                 newname))))
           (t
            ;; We don't recognize the file as compressed, so compress it.
            ;; Try gzip; if we don't have that, use compress.
-- 
2.20.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-Add-Zstandard-compression-support-for-Tramp.patch --]
[-- Type: text/x-patch, Size: 4643 bytes --]

From 7f8c658d5423947cc9b88d137cec5a2e2e53afd1 Mon Sep 17 00:00:00 2001
From: Alexander Gramiak <agrambot@gmail.com>
Date: Sat, 29 Dec 2018 11:54:24 -0600
Subject: [PATCH 3/5] Add Zstandard compression support for Tramp

* lisp/net/tramp-archive.el (tramp-archive-suffixes)
(tramp-archive-compression-suffixes): Add zstd compression suffixes.
* lisp/net/tramp-sh.el (tramp-inline-compress-commands): Add
zstd (de)compression commands.
---
 doc/misc/tramp.texi       | 9 ++++++---
 etc/NEWS                  | 4 ++++
 lisp/net/tramp-archive.el | 8 ++++----
 lisp/net/tramp-sh.el      | 1 +
 4 files changed, 15 insertions(+), 7 deletions(-)

diff --git a/doc/misc/tramp.texi b/doc/misc/tramp.texi
index c9f1e75d8e..d491d04918 100644
--- a/doc/misc/tramp.texi
+++ b/doc/misc/tramp.texi
@@ -3332,18 +3332,21 @@ Archive file names
 @cindex @file{shar} file archive suffix
 @cindex file archive suffix @file{shar}
 
-@item @samp{.tar}, @samp{.tbz}, @samp{.tgz}, @samp{.tlz}, @samp{.txz} ---
+@item @samp{.tar}, @samp{.tbz}, @samp{.tgz}, @samp{.tlz}, @samp{.txz},
+@samp{.tzst} ---
 (Compressed) tape archives
 @cindex @file{tar} file archive suffix
 @cindex @file{tbz} file archive suffix
 @cindex @file{tgz} file archive suffix
 @cindex @file{tlz} file archive suffix
 @cindex @file{txz} file archive suffix
+@cindex @file{tzst} file archive suffix
 @cindex file archive suffix @file{tar}
 @cindex file archive suffix @file{tbz}
 @cindex file archive suffix @file{tgz}
 @cindex file archive suffix @file{tlz}
 @cindex file archive suffix @file{txz}
+@cindex file archive suffix @file{tzst}
 
 @item @samp{.warc} ---
 Web archives
@@ -3378,8 +3381,8 @@ Archive file names
 compression suffix.  Valid compression suffixes are listed in the
 constant @code{tramp-archive-compression-suffixes}.  They are
 @samp{.bz2}, @samp{.gz}, @samp{.lrz}, @samp{.lz}, @samp{.lz4},
-@samp{.lzma}, @samp{.lzo}, @samp{.uu}, @samp{.xz} and @samp{.Z}.  A
-valid archive file name would be
+@samp{.lzma}, @samp{.lzo}, @samp{.uu}, @samp{.xz}, @samp{.Z}, and
+@samp{.zst}.  A valid archive file name would be
 @file{/path/to/dir/file.tar.gz/dir/file}.  Even several suffixes in a
 row are possible, like @file{/path/to/dir/file.tar.gz.uu/dir/file}.
 
diff --git a/etc/NEWS b/etc/NEWS
index af99e29e40..e65823413b 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -927,6 +927,10 @@ ad-hoc multi-hop file names must match the previous hop.
 timeout, after which the underlying session is disabled.  This is for
 security reasons.
 
++++
+*** Zstandard compression is now supported when using libarchive 3.3
+or newer.
+
 ** Rcirc
 
 ---
diff --git a/lisp/net/tramp-archive.el b/lisp/net/tramp-archive.el
index 03b1eed970..910982978d 100644
--- a/lisp/net/tramp-archive.el
+++ b/lisp/net/tramp-archive.el
@@ -77,8 +77,8 @@
 ;; File archives could also be compressed, identified by an additional
 ;; compression suffix.  Valid compression suffixes are listed in the
 ;; constant `tramp-archive-compression-suffixes'.  They are ".bz2",
-;; ".gz", ".lrz", ".lz", ".lz4", ".lzma", ".lzo", ".uu", ".xz" and
-;; ".Z".  A valid archive file name would be
+;; ".gz", ".lrz", ".lz", ".lz4", ".lzma", ".lzo", ".uu", ".xz",".Z",
+;; and .zst.  A valid archive file name would be
 ;; "/path/to/dir/file.tar.gz/dir/file".  Even several suffixes in a
 ;; row are possible, like "/path/to/dir/file.tar.gz.uu/dir/file".
 
@@ -149,7 +149,7 @@ tramp-archive-suffixes
     "rar" ;; RAR archives.
     "rpm" ;; Red Hat packages.
     "shar" ;; Shell archives.  Not in libarchive testsuite.
-    "tar" "tbz" "tgz" "tlz" "txz" ;; (Compressed) tape archives.
+    "tar" "tbz" "tgz" "tlz" "txz" ".tzst" ;; (Compressed) tape archives.
     "warc" ;; Web archives.
     "xar" ;; macOS XAR archives.  Not in libarchive testsuite.
     "xpi" ;; XPInstall Mozilla addons.  Not in libarchive testsuite.
@@ -164,7 +164,7 @@ tramp-archive-suffixes
 
 ;;;###autoload
 (defconst tramp-archive-compression-suffixes
-  '("bz2" "gz" "lrz" "lz" "lz4" "lzma" "lzo" "uu" "xz" "Z")
+  '("bz2" "gz" "lrz" "lz" "lz4" "lzma" "lzo" "uu" "xz" "Z" "zst")
   "List of suffixes which indicate a compressed file.
 It must be supported by libarchive(3).")
 
diff --git a/lisp/net/tramp-sh.el b/lisp/net/tramp-sh.el
index 991a210263..332cd356bf 100644
--- a/lisp/net/tramp-sh.el
+++ b/lisp/net/tramp-sh.el
@@ -4542,6 +4542,7 @@ tramp-inline-compress-commands
     ("env GZIP= gzip" "env GZIP= gzip -d")
     ("bzip2" "bzip2 -d")
     ("xz" "xz -d")
+    ("zstd --rm" "zstd -d --rm")
     ("compress" "compress -d"))
   "List of compress and decompress commands for inline transfer.
 Each item is a list that looks like this:
-- 
2.20.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: 0004-make-dist-Add-option-for-zstd-compression.patch --]
[-- Type: text/x-patch, Size: 1367 bytes --]

From 1a76a5c680ab539c992d06f9d10166781fdef0d2 Mon Sep 17 00:00:00 2001
From: Alexander Gramiak <agrambot@gmail.com>
Date: Sat, 29 Dec 2018 11:54:24 -0600
Subject: [PATCH 4/5] * make-dist: Add option for zstd compression

---
 make-dist | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/make-dist b/make-dist
index 926665b8f0..761477a83b 100755
--- a/make-dist
+++ b/make-dist
@@ -99,6 +99,10 @@ verbose=
     "--xz")
       default_gzip="xz"
     ;;
+    ## Same with zstd.
+    "--zstd")
+      default_gzip="zstd"
+    ;;
     "--no-compress")
       default_gzip="cat"
     ;;
@@ -125,6 +129,7 @@ verbose=
       echo "  --bzip2	use bzip2 instead of gzip"
       echo "  --clean-up	delete staging directories when done"
       echo "  --xz		use xz instead of gzip"
+      echo "  --zstd		use zstd instead of gzip"
       echo "  --no-compress	don't compress"
       echo "  --newer=TIME	don't include files older than TIME"
       echo "  --no-check	don't check for bad file names etc."
@@ -684,7 +689,8 @@ files=
   fi
   case "${default_gzip}" in
     bzip2) gzip_extension=.bz2 ;;
-    xz)  gzip_extension=.xz ;;
+    xz)    gzip_extension=.xz ;;
+    zstd)  gzip_extension=.zst; default_gzip="zstd -19q --rm" ;;
     gzip)  gzip_extension=.gz ; default_gzip="gzip --best --no-name";;
        *)  gzip_extension= ;;
   esac
-- 
2.20.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: 0005-Add-Zstandard-compression-option-for-make-install.patch --]
[-- Type: text/x-patch, Size: 18753 bytes --]

From d5a8d41cd30e78bb614a8324acc1e7220368185a Mon Sep 17 00:00:00 2001
From: Alexander Gramiak <agrambot@gmail.com>
Date: Sat, 29 Dec 2018 11:54:24 -0600
Subject: [PATCH 5/5] Add Zstandard compression option for make install

* lisp/jka-cmpr-hook.el (jka-compr-load-suffixes): Add .zst by
default.

* configure.ac (--with-compress-install): Add `zstd' option and
deprecate `GZIP_PROG' in favour of the more general `COMPR_PROG'.

* Makefile.in: Add and use Make variables `COMPR_PROG', `COMPR_ARGS',
`COMPR_TYPE', `COMPR_EXT', `COMPR_INFO_PROG', `COMPR_INFO_ARGS',
`COMPR_INFO_EXT'.  Deprecate, but still support `GZIP_PROG'.

* doc/emacs/Makefile.in:
doc/lispintro/Makefile.in:
doc/lispref/Makefile.in:
doc/misc/Makefile.in: Same as Makefile.in, but without the
`COMPR_INFO*' Make variables.

* admin/admin.el (make-manuals-dist-output-variables): Use
`COMPR_PROG' over `GZIP_PROG'.

* INSTALL: Document new `COMPR_{PROG,TYPE,ARGS,EXT}' Make variables.
---
 INSTALL                   | 34 +++++++++++++++++++----
 Makefile.in               | 57 ++++++++++++++++++++++++++-------------
 admin/admin.el            |  2 +-
 configure.ac              | 36 ++++++++++++++++++-------
 doc/emacs/Makefile.in     | 25 ++++++++++++-----
 doc/lispintro/Makefile.in | 25 ++++++++++++-----
 doc/lispref/Makefile.in   | 25 ++++++++++++-----
 doc/misc/Makefile.in      | 25 ++++++++++++-----
 etc/NEWS                  | 19 +++++++++++++
 lisp/jka-cmpr-hook.el     |  4 +--
 10 files changed, 192 insertions(+), 60 deletions(-)

diff --git a/INSTALL b/INSTALL
index 0c56fff6d4..ae9047d1ea 100644
--- a/INSTALL
+++ b/INSTALL
@@ -101,8 +101,9 @@ sections if you need to.
 		make distclean
 
   Note that the install automatically saves space by compressing
-  (provided you have the 'gzip' program) those installed Lisp source (.el)
-  files that have corresponding .elc versions, as well as the Info files.
+  (provided you have 'gzip' or another compression program) those
+  installed Lisp source (.el) files that have corresponding .elc
+  versions, as well as the Info files.
 
 
 ADDITIONAL DISTRIBUTION FILES
@@ -641,9 +642,32 @@ GNU software; the following variables are specific to Emacs.
 	see), is '/usr/local/libexec/emacs/VERSION/CONFIGURATION-NAME'
 	(where VERSION and CONFIGURATION-NAME are as described above).
 
-'GZIP_PROG' is the name of the executable that compresses installed info,
-	manual, and .el files.  It defaults to gzip.  Setting it to
-	the empty string suppresses compression.
+'COMPR_PROG' is the name of the executable that compresses various
+	installed files, such as info, manual, and .el files.  It
+	defaults to gzip.  Setting it to the empty string suppresses
+	compression.
+
+'GZIP_PROG' is a deprecated variable that behaves like 'COMPR_PROG',
+	but also forces 'COMPR_TYPE' to gzip if 'COMPR_TYPE' is
+	unspecified.
+
+'COMPR_TYPE' is the type of compression used for the above, which for
+	supported values (currently 'gzip' and 'zstd') determines the
+	default values of 'COMPR_ARGS' and 'COMPR_EXT'.  Its default
+	value is based on the configure option
+	'--with-compress-install'; if that option is specified, then
+	'COMPR_TYPE' is set to the specified value of that option,
+	otherwise 'gzip'.
+
+'COMPR_ARGS' is the list of arguments passed to 'COMPR_PROG', which is
+	set according to 'COMPR_TYPE'.  The default arguments for
+	supported 'COMPR_TYPE' set the slowest (best) compression
+	level that is enabled by default, and signify that the
+	original files are deleted after compression.
+
+'COMPR_EXT' is the extension used by 'COMPR_PROG', which is set
+	according to 'COMPR_TYPE'.  It defaults to the canonical
+	extension of supported 'COMPR_TYPE'.
 
 Remember that you must specify any variable values you need each time
 you run 'make' in the top directory.  If you run 'make' once to build
diff --git a/Makefile.in b/Makefile.in
index bbb028a74a..5020abc494 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -295,8 +295,33 @@ MKDIR_P =
 # Create a link to a file in the same directory as the target.
 LN_S_FILEONLY = @LN_S_FILEONLY@
 
-# We use gzip to compress installed .el and some .txt files.
-GZIP_PROG = @GZIP_PROG@
+# We compress many installed .el and other files.
+# GZIP_PROG is used here for backwards-compatibility.
+ifdef GZIP_PROG
+  COMPR_PROG = ${GZIP_PROG}
+  COMPR_TYPE = gzip
+else
+  COMPR_PROG = @COMPR_PROG@
+  COMPR_TYPE = @COMPR_TYPE@
+endif
+
+ifeq (${COMPR_TYPE}, gzip)
+  COMPR_ARGS = "-9n"
+  COMPR_EXT  = ".gz"
+
+  # Info/man don't support zstd at the time of writing, so provide an
+  # alternative for them that defaults to gzip.
+  COMPR_INFO_PROG = ${COMPR_PROG}
+  COMPR_INFO_ARGS = ${COMPR_ARGS}
+  COMPR_INFO_EXT  = ${COMPR_EXT}
+else ifeq (${COMPR_TYPE}, zstd)
+  COMPR_ARGS = "-19q --rm"
+  COMPR_EXT  = ".zst"
+
+  COMPR_INFO_PROG = @COMPR_INFO_PROG@
+  COMPR_INFO_ARGS = "-9n"
+  COMPR_INFO_EXT  = ".gz"
+endif
 
 # ============================= Targets ==============================
 
@@ -609,16 +634,16 @@ install-arch-indep:
 	  ${write_subdir}
 	subdir="$(DESTDIR)${datadir}/emacs/site-lisp" ; \
 	  ${write_subdir} || true
-	[ -z "${GZIP_PROG}" ] || { \
+	[ -z "${COMPR_PROG}" ] || { \
 	  echo "Compressing *.el etc. ..." && \
 	  cd "$(DESTDIR)${lispdir}" && \
 	  for f in `find . -name "*.elc" -print | sed 's/.elc$$/.el/'`; do \
-	    ${GZIP_PROG} -9n "$$f"; \
+	    eval ${COMPR_PROG} ${COMPR_ARGS} "\"$$f\""; \
 	  done; \
 	  cd "$(DESTDIR)${etcdir}" && \
 	  for f in `find . -maxdepth 1 -name "*NEWS*" -not -name NEWS \
 	-not -name ORG-NEWS` `find refcards -name "*.ps"` "publicsuffix.txt"; do \
-	    ${GZIP_PROG} -9n "$$f"; \
+	    eval ${COMPR_PROG} ${COMPR_ARGS} "\"$$f\""; \
 	  done; \
 	}
 	-chmod -R a+r "$(DESTDIR)${datadir}/emacs/${version}" ${COPYDESTS}
@@ -668,9 +693,9 @@ install-info:
 	      for f in `ls $$elt $$elt-[1-9] $$elt-[1-9][0-9] 2>/dev/null`; do \
 	       (cd "$${thisdir}"; \
 	        ${INSTALL_DATA} ${srcdir}/info/$$f "$(DESTDIR)${infodir}/$$f"); \
-	        [ -n "${GZIP_PROG}" ] || continue ; \
-	        rm -f "$(DESTDIR)${infodir}/$$f.gz"; \
-	        ${GZIP_PROG} -9n "$(DESTDIR)${infodir}/$$f"; \
+	        [ -n "${COMPR_INFO_PROG}" ] || continue ; \
+	        rm -f "$(DESTDIR)${infodir}/$${f}${COMPR_INFO_EXT}"; \
+	        eval ${COMPR_INFO_PROG} ${COMPR_INFO_ARGS} "\"$(DESTDIR)${infodir}/$$f\""; \
 	      done; \
 	     (cd "$${thisdir}"; \
 	      ${INSTALL_INFO} --info-dir="$(DESTDIR)${infodir}" "$(DESTDIR)${infodir}/$$elt"); \
@@ -692,9 +717,9 @@ install-man:
 	  dest=`echo "$${page}" | sed -e 's/\.1$$//' -e '$(TRANSFORM)'`.1; \
 	  (cd "$${thisdir}"; \
 	   ${INSTALL_DATA} ${mansrcdir}/$${page} "$(DESTDIR)${man1dir}/$${dest}"); \
-	  [ -n "${GZIP_PROG}" ] || continue ; \
-	  rm -f "$(DESTDIR)${man1dir}/$${dest}.gz"; \
-	  ${GZIP_PROG} -9n "$(DESTDIR)${man1dir}/$${dest}" || true; \
+	  [ -n "${COMPR_INFO_PROG}" ] || continue ; \
+	  rm -f "$(DESTDIR)${man1dir}/$${dest}${COMPR_INFO_EXT}"; \
+	  eval ${COMPR_INFO_PROG} ${COMPR_INFO_ARGS} "\"$(DESTDIR)${man1dir}/$${dest}\"" || true; \
 	done
 
 ## Install those items from etc/ that need to end up elsewhere.
@@ -774,16 +799,12 @@ uninstall:
 	   for elt in ${INFO_NONMISC} $${info_misc}; do \
 	     (cd "$${thisdir}"; \
 	      $(INSTALL_INFO) --remove --info-dir="$(DESTDIR)${infodir}" "$(DESTDIR)${infodir}/$$elt"); \
-	     if [ -n "${GZIP_PROG}" ]; then \
-	        ext=.gz; else ext=; fi; \
-	     rm -f $$elt$$ext $$elt-[1-9]$$ext $$elt-[1-9][0-9]$$ext; \
+	     rm -f $$elt${COMPR_INFO_EXT} $$elt-[1-9]${COMPR_INFO_EXT} $$elt-[1-9][0-9]${COMPR_INFO_EXT}; \
 	   done; \
 	 fi)
-	(if [ -n "${GZIP_PROG}" ]; then \
-	    ext=.gz; else ext=; fi; \
-	 if cd ${mansrcdir}; then \
+	(if cd ${mansrcdir}; then \
 	   for page in *.1; do \
-	     rm -f "$(DESTDIR)${man1dir}"/`echo "$${page}" | sed -e 's/\.1$$//' -e '$(TRANSFORM)'`.1$$ext; done; \
+	     rm -f "$(DESTDIR)${man1dir}"/`echo "$${page}" | sed -e 's/\.1$$//' -e '$(TRANSFORM)'`.1${COMPR_INFO_EXT}; done; \
 	 fi)
 	rm -f "$(DESTDIR)${bindir}/$(EMACS)" "$(DESTDIR)${bindir}/$(EMACSFULL)"
 	(if cd "$(DESTDIR)${icondir}"; then \
diff --git a/admin/admin.el b/admin/admin.el
index 3fc50afe9f..61792bea89 100644
--- a/admin/admin.el
+++ b/admin/admin.el
@@ -671,7 +671,7 @@ make-manuals-dist-output-variables
     ("@PACKAGE_TARNAME@" . "emacs")
     ("@docdir@" . "${datarootdir}/doc/${PACKAGE_TARNAME}")
     ("@\\(dvi\\|html\\|pdf\\|ps\\)dir@" . "${docdir}")
-    ("@GZIP_PROG@" . "gzip")
+    ("@COMPR_PROG@" . "gzip")
     ("@INSTALL@" . "install -c")
     ("@INSTALL_DATA@" . "${INSTALL} -m 644")
     ("@configure_input@" . "")
diff --git a/configure.ac b/configure.ac
index 8b34c3b658..d0061d9865 100644
--- a/configure.ac
+++ b/configure.ac
@@ -415,11 +415,32 @@ AC_DEFUN
 ## Makefile.in needs the cache file name.
 AC_SUBST(cache_file)
 
-## This is an option because I do not know if all info/man support
-## compressed files, nor how to test if they do so.
-OPTION_DEFAULT_ON([compress-install],
-  [don't compress some files (.el, .info, etc.) when installing.  Equivalent to:
-make GZIP_PROG= install])
+AC_ARG_WITH([compress-install],dnl
+[AS_HELP_STRING([--with-compress-install=PROGRAM],
+                [compression program used to compress some files
+                 (.el, .info, etc.) when installing (PROGRAM one of: yes, gzip,
+                 zstd, no; default 'yes'='gzip').  Note that Info/man files are
+                 still compressed with gzip when 'zstd' is selected.])],
+[ case "${withval}" in
+    yes|no|gzip|zstd) ;;
+    *) AC_MSG_ERROR(['--with-compress-install=$withval' is invalid;
+this option's value should be 'yes', 'no', 'gzip', or 'zstd'.])
+    ;;
+  esac
+],
+[with_compress_install=$with_features])
+
+if test "${with_compress_install}" != no; then
+  if test "${with_compress_install}" = yes; then
+    with_compress_install=gzip;
+  fi
+  COMPR_TYPE=$with_compress_install
+  AC_PATH_PROG(COMPR_PROG, $with_compress_install)
+
+  # Support separate compression for Info/man files for compatibility
+  AC_PATH_PROG(COMPR_INFO_PROG, gzip)
+fi
+AC_SUBST(COMPR_TYPE)
 
 AC_ARG_WITH(gameuser,dnl
 [AS_HELP_STRING([--with-gameuser=USER_OR_GROUP],
@@ -1209,11 +1230,6 @@ AC_DEFUN
 
 AC_PATH_PROG(INSTALL_INFO, install-info, :,
   $PATH$PATH_SEPARATOR/usr/sbin$PATH_SEPARATOR/sbin)
-dnl Don't use GZIP, which is used by gzip for additional parameters.
-AC_PATH_PROG(GZIP_PROG, gzip)
-
-test $with_compress_install != yes && test -n "$GZIP_PROG" && \
-   GZIP_PROG=" # $GZIP_PROG # (disabled by configure --without-compress-install)"
 
 AC_CACHE_CHECK([for 'find' args to delete a file],
   [emacs_cv_find_delete],
diff --git a/doc/emacs/Makefile.in b/doc/emacs/Makefile.in
index 54e173f8d6..81825c9f82 100644
--- a/doc/emacs/Makefile.in
+++ b/doc/emacs/Makefile.in
@@ -51,7 +51,21 @@ psdir =
 
 MKDIR_P = @MKDIR_P@
 
-GZIP_PROG = @GZIP_PROG@
+ifdef GZIP_PROG
+  COMPR_PROG = ${GZIP_PROG}
+  COMPR_TYPE = gzip
+else
+  COMPR_PROG = @COMPR_PROG@
+  COMPR_TYPE = @COMPR_TYPE@
+endif
+
+ifeq (${COMPR_TYPE}, gzip)
+  COMPR_ARGS = "-9n"
+  COMPR_EXT  = ".gz"
+else ifeq (${COMPR_TYPE}, zstd)
+  COMPR_ARGS = "-19q --rm"
+  COMPR_EXT  = ".zst"
+endif
 
 HTML_OPTS = --no-split --html
 
@@ -241,9 +255,9 @@ install-ps:
 	umask 022; $(MKDIR_P) "$(DESTDIR)$(psdir)"
 	for file in $(PS_TARGETS); do \
 	  $(INSTALL_DATA) $${file} "$(DESTDIR)$(psdir)"; \
-	  [ -n "${GZIP_PROG}" ] || continue; \
-	  rm -f "$(DESTDIR)$(psdir)/$${file}.gz"; \
-	  ${GZIP_PROG} -9n "$(DESTDIR)$(psdir)/$${file}"; \
+	  [ -n "${COMPR_PROG}" ] || continue; \
+	  rm -f "$(DESTDIR)$(psdir)/$${file}${COMPR_EXT}"; \
+	  eval ${COMPR_PROG} ${COMPR_ARGS} "\"$(DESTDIR)$(psdir)/$${file}\""; \
 	done
 
 ## Top-level Makefile installs the info pages.
@@ -261,9 +275,8 @@ uninstall-html:
 	  rm -f "$(DESTDIR)$(htmldir)/$${file}"; \
 	done
 uninstall-ps:
-	ext= ; [ -n "${GZIP_PROG}" ] && ext=.gz; \
 	for file in $(PS_TARGETS); do \
-	  rm -f "$(DESTDIR)$(psdir)/$${file}$${ext}"; \
+	  rm -f "$(DESTDIR)$(psdir)/$${file}${COMPR_EXT}"; \
 	done
 uninstall-pdf:
 	for file in $(PDF_TARGETS); do \
diff --git a/doc/lispintro/Makefile.in b/doc/lispintro/Makefile.in
index e2a1229d5c..989163b7c6 100644
--- a/doc/lispintro/Makefile.in
+++ b/doc/lispintro/Makefile.in
@@ -39,7 +39,21 @@ psdir =
 
 MKDIR_P = @MKDIR_P@
 
-GZIP_PROG = @GZIP_PROG@
+ifdef GZIP_PROG
+  COMPR_PROG = ${GZIP_PROG}
+  COMPR_TYPE = gzip
+else
+  COMPR_PROG = @COMPR_PROG@
+  COMPR_TYPE = @COMPR_TYPE@
+endif
+
+ifeq (${COMPR_TYPE}, gzip)
+  COMPR_ARGS = "-9n"
+  COMPR_EXT  = ".gz"
+else ifeq (${COMPR_TYPE}, zstd)
+  COMPR_ARGS = "-19q --rm"
+  COMPR_EXT  = ".zst"
+endif
 
 HTML_OPTS = --no-split --html
 
@@ -140,9 +154,9 @@ install-ps:
 	umask 022; $(MKDIR_P) "$(DESTDIR)$(psdir)"
 	for file in $(PS_TARGETS); do \
 	  $(INSTALL_DATA) $${file} "$(DESTDIR)$(psdir)"; \
-	  [ -n "${GZIP_PROG}" ] || continue; \
-	  rm -f "$(DESTDIR)$(psdir)/$${file}.gz"; \
-	  ${GZIP_PROG} -9n "$(DESTDIR)$(psdir)/$${file}"; \
+	  [ -n "${COMPR_PROG}" ] || continue; \
+	  rm -f "$(DESTDIR)$(psdir)/$${file}${COMPR_EXT}"; \
+	  eval ${COMPR_PROG} ${COMPR_ARGS} "\"$(DESTDIR)$(psdir)/$${file}\""; \
 	done
 
 ## Top-level Makefile installs the info pages.
@@ -160,9 +174,8 @@ uninstall-html:
 	  rm -f "$(DESTDIR)$(htmldir)/$${file}"; \
 	done
 uninstall-ps:
-	ext= ; [ -n "${GZIP_PROG}" ] && ext=.gz; \
 	for file in $(PS_TARGETS); do \
-	  rm -f "$(DESTDIR)$(psdir)/$${file}$${ext}"; \
+	  rm -f "$(DESTDIR)$(psdir)/$${file}${COMPR_EXT}"; \
 	done
 uninstall-pdf:
 	for file in $(PDF_TARGETS); do \
diff --git a/doc/lispref/Makefile.in b/doc/lispref/Makefile.in
index 221f4f97f5..352cbab258 100644
--- a/doc/lispref/Makefile.in
+++ b/doc/lispref/Makefile.in
@@ -43,7 +43,21 @@ psdir =
 
 MKDIR_P = @MKDIR_P@
 
-GZIP_PROG = @GZIP_PROG@
+ifdef GZIP_PROG
+  COMPR_PROG = ${GZIP_PROG}
+  COMPR_TYPE = gzip
+else
+  COMPR_PROG = @COMPR_PROG@
+  COMPR_TYPE = @COMPR_TYPE@
+endif
+
+ifeq (${COMPR_TYPE}, gzip)
+  COMPR_ARGS = "-9n"
+  COMPR_EXT  = ".gz"
+else ifeq (${COMPR_TYPE}, zstd)
+  COMPR_ARGS = "-19q --rm"
+  COMPR_EXT  = ".zst"
+endif
 
 HTML_OPTS = --no-split --html
 
@@ -201,9 +215,9 @@ install-ps:
 	umask 022; $(MKDIR_P) "$(DESTDIR)$(psdir)"
 	for file in $(PS_TARGETS); do \
 	  $(INSTALL_DATA) $${file} "$(DESTDIR)$(psdir)"; \
-	  [ -n "${GZIP_PROG}" ] || continue; \
-	  rm -f "$(DESTDIR)$(psdir)/$${file}.gz"; \
-	  ${GZIP_PROG} -9n "$(DESTDIR)$(psdir)/$${file}"; \
+	  [ -n "${COMPR_PROG}" ] || continue; \
+	  rm -f "$(DESTDIR)$(psdir)/$${file}${COMPR_EXT}"; \
+	  eval ${COMPR_PROG} ${COMPR_ARGS} "\"$(DESTDIR)$(psdir)/$${file}\""; \
 	done
 
 ## Top-level Makefile installs the info pages.
@@ -221,9 +235,8 @@ uninstall-html:
 	  rm -f "$(DESTDIR)$(htmldir)/$${file}"; \
 	done
 uninstall-ps:
-	ext= ; [ -n "${GZIP_PROG}" ] && ext=.gz; \
 	for file in $(PS_TARGETS); do \
-	  rm -f "$(DESTDIR)$(psdir)/$${file}$${ext}"; \
+	  rm -f "$(DESTDIR)$(psdir)/$${file}${COMPR_EXT}"; \
 	done
 uninstall-pdf:
 	for file in $(PDF_TARGETS); do \
diff --git a/doc/misc/Makefile.in b/doc/misc/Makefile.in
index fd07ea4ca1..380efb8761 100644
--- a/doc/misc/Makefile.in
+++ b/doc/misc/Makefile.in
@@ -44,7 +44,21 @@ psdir =
 
 MKDIR_P = @MKDIR_P@
 
-GZIP_PROG = @GZIP_PROG@
+ifdef GZIP_PROG
+  COMPR_PROG = ${GZIP_PROG}
+  COMPR_TYPE = gzip
+else
+  COMPR_PROG = @COMPR_PROG@
+  COMPR_TYPE = @COMPR_TYPE@
+endif
+
+ifeq (${COMPR_TYPE}, gzip)
+  COMPR_ARGS = "-9n"
+  COMPR_EXT  = ".gz"
+else ifeq (${COMPR_TYPE}, zstd)
+  COMPR_ARGS = "-19q --rm"
+  COMPR_EXT  = ".zst"
+endif
 
 HTML_OPTS = --no-split --html
 
@@ -262,9 +276,9 @@ install-ps:
 	umask 022; $(MKDIR_P) "$(DESTDIR)$(psdir)"
 	for file in $(PS_TARGETS); do \
 	  $(INSTALL_DATA) $${file} "$(DESTDIR)$(psdir)"; \
-	  [ -n "${GZIP_PROG}" ] || continue; \
-	  rm -f "$(DESTDIR)$(psdir)/$${file}.gz"; \
-	  ${GZIP_PROG} -9n "$(DESTDIR)$(psdir)/$${file}"; \
+	  [ -n "${COMPR_PROG}" ] || continue; \
+	  rm -f "$(DESTDIR)$(psdir)/$${file}${COMPR_EXT}"; \
+	  eval ${COMPR_PROG} ${COMPR_ARGS} "\"$(DESTDIR)$(psdir)/$${file}\""; \
 	done
 
 ## Top-level Makefile installs the info pages.
@@ -283,9 +297,8 @@ uninstall-html:
 	  rm -f "$(DESTDIR)$(htmldir)/$${file}"; \
 	done
 uninstall-ps:
-	ext= ; [ -n "${GZIP_PROG}" ] && ext=.gz; \
 	for file in $(PS_TARGETS); do \
-	  rm -f "$(DESTDIR)$(psdir)/$${file}$${ext}"; \
+	  rm -f "$(DESTDIR)$(psdir)/$${file}${COMPR_EXT}"; \
 	done
 uninstall-pdf:
 	for file in $(PDF_TARGETS); do \
diff --git a/etc/NEWS b/etc/NEWS
index e65823413b..633ddd9f62 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -40,6 +40,20 @@ counterparts from json.el.
 ** NEWS files for past Emacs versions and packages are now compressed
 by default.
 
++++
+** Emacs now supports Zstandard compression for most compressed
+installation files using '--with-compress-install=zstd'.  Info/man
+files are still compressed with 'gzip' due to lack of support in
+'info' and 'man'.
+
++++
+** The make variable GZIP_PROG is now a deprecated alias for the new
+variable COMPR_PROG.
+
+** New make variables COMPR_TYPE, COMPR_ARGS, and COMPR_EXT allow for
+fine-tuning the compression type, arguments, and extension used when
+(un)installing compressed files.
+
 ** The etags program now uses the C library's regular expression matcher
 when possible, and a compatible regex substitute otherwise.  This will
 let developers maintain Emacs's own regex code without having to also
@@ -313,6 +327,11 @@ the node "(emacs) Directory Variables" of the user manual.
 \f
 * Changes in Specialized Modes and Packages in Emacs 27.1
 
+** Auto Compression mode
+*** Zstandard files are now recognized as valid compressed
+representative files by default.  See 'load-file-rep-suffixes' for
+details.
+
 ** map.el
 *** Now also understands plists.
 *** Now defined via generic functions that can be extended via 'cl-defmethod'.
diff --git a/lisp/jka-cmpr-hook.el b/lisp/jka-cmpr-hook.el
index d800b60513..355a72df26 100644
--- a/lisp/jka-cmpr-hook.el
+++ b/lisp/jka-cmpr-hook.el
@@ -155,7 +155,7 @@ jka-compr-install
   (setq auto-mode-alist
 	(append auto-mode-alist jka-compr-mode-alist-additions))
 
-  ;; Make sure that (load "foo") will find /bla/foo.el.gz.
+  ;; Make sure that (load "foo") will find, e.g., /bla/foo.el.gz.
   (setq load-file-rep-suffixes
 	(append load-file-rep-suffixes jka-compr-load-suffixes nil)))
 
@@ -331,7 +331,7 @@ jka-compr-mode-alist-additions
   :set 'jka-compr-set
   :group 'jka-compr)
 
-(defcustom jka-compr-load-suffixes (purecopy '(".gz"))
+(defcustom jka-compr-load-suffixes (purecopy '(".gz" ".zst"))
   "List of compression related suffixes to try when loading files.
 Enabling Auto Compression mode appends this list to `load-file-rep-suffixes',
 which see.  Disabling Auto Compression mode removes all suffixes
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-06-24 16:27 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-12-30  6:46 bug#33919: [PATCH 0/5] Add Zstandard compression for installation and utilities Alex
2018-12-30 10:06 ` Michael Albinus
2019-01-02 13:04   ` Michael Albinus
2018-12-30 15:21 ` Eli Zaretskii
2019-06-23 18:32   ` Lars Ingebrigtsen
2019-06-23 18:49     ` Michael Albinus
2019-06-24 14:07       ` Lars Ingebrigtsen
2019-06-24 14:44     ` Eli Zaretskii
2019-06-24 16:27       ` Lars Ingebrigtsen

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).