bug#21637: Subject: 24.5; dired-do-compress

bug#21637: Subject: 24.5; dired-do-compress

[Devon Sean McCullough]

dired-do-compress does foo.zip --> foo.zip.gz, should do foo.zip --> foo instead.

		Peace
			—Devon

P.S. To replicate, type capital-Z after starting Emacs like this:
bash$ Open -a Emacs -n --args -Q --eval '(shell-command "zip <<< bar > /tmp/foo.zip")' --eval '(dired "/tmp/foo.zip")'
bash$ type zip
zip is /usr/bin/zip

In GNU Emacs 24.5.1 (x86_64-apple-darwin13.4.0, NS apple-appkit-1265.21)
 of 2015-04-10 on builder10-9.porkrind.org
Windowing system distributor `Apple', version 10.3.1265
Configured using:
 `configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp''

Important settings:
  locale-coding-system: utf-8-unix

Major mode: Dired by name

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
  adding: - (stored 0%)
ls does not support --dired; see `dired-use-ls-dired' for more details.
Compress or uncompress foo.zip? (y or n) y
Compressing /tmp/foo.zip...done
Compress or uncompress: 1 file.
foo.zip.gz

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util help-fns mail-prsvr mail-utils dired-aux dired time-date tooltip
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel ns-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
cocoa ns multi-tty emacs)

Memory information:
((conses 16 77286 4373)
 (symbols 48 17640 0)
 (miscs 40 42 130)
 (strings 32 11369 3914)
 (string-bytes 1 297483)
 (vectors 16 9448)
 (vector-slots 8 384612 15166)
 (floats 8 53 270)
 (intervals 56 185 0)
 (buffers 960 13))

bash$ zip --help
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
Zip 3.0 (July 5th 2008). Usage:
zip [-options] [-b path] [-t mmddyyyy] [-n suffixes] [zipfile list] [-xi list]
  The default action is to add or replace zipfile entries from list, which
  can include the special name - to compress standard input.
  If zipfile and list are omitted, zip compresses stdin to stdout.
  -f   freshen: only changed files  -u   update: only changed or new files
  -d   delete entries in zipfile    -m   move into zipfile (delete OS files)
  -r   recurse into directories     -j   junk (don't record) directory names
  -0   store only                   -l   convert LF to CR LF (-ll CR LF to LF)
  -1   compress faster              -9   compress better
  -q   quiet operation              -v   verbose operation/print version info
  -c   add one-line comments        -z   add zipfile comment
  -@   read names from stdin        -o   make zipfile as old as latest entry
  -x   exclude the following names  -i   include only the following names
  -F   fix zipfile (-FF try harder) -D   do not add directory entries
  -A   adjust self-extracting exe   -J   junk zipfile prefix (unzipsfx)
  -T   test zipfile integrity       -X   eXclude eXtra file attributes
  -y   store symbolic links as the link instead of the referenced file
  -e   encrypt                      -n   don't compress these suffixes
  -h2  show more help
  
bash$ zip -h2

Extended Help for Zip

See the Zip Manual for more detailed help


Zip stores files in zip archives.  The default action is to add or replace
zipfile entries.

Basic command line:
  zip options archive_name file file ...

Some examples:
  Add file.txt to z.zip (create z if needed):      zip z file.txt
  Zip all files in current dir:                    zip z *
  Zip files in current dir and subdirs also:       zip -r z .

Basic modes:
 External modes (selects files from file system):
        add      - add new files/update existing files in archive (default)
  -u    update   - add new files/update existing files only if later date
  -f    freshen  - update existing files only (no files added)
  -FS   filesync - update if date or size changed, delete if no OS match
 Internal modes (selects entries in archive):
  -d    delete   - delete files from archive (see below)
  -U    copy     - select files in archive to copy (use with --out)

Basic options:
  -r        recurse into directories (see Recursion below)
  -m        after archive created, delete original files (move into archive)
  -j        junk directory names (store just file names)
  -q        quiet operation
  -v        verbose operation (just "zip -v" shows version information)
  -c        prompt for one-line comment for each entry
  -z        prompt for comment for archive (end with just "." line or EOF)
  -@        read names to zip from stdin (one path per line)
  -o        make zipfile as old as latest entry


Syntax:
  The full command line syntax is:

    zip [-shortopts ...] [--longopt ...] [zipfile [path path ...]] [-xi list]

  Any number of short option and long option arguments are allowed
  (within limits) as well as any number of path arguments for files
  to zip up.  If zipfile exists, the archive is read in.  If zipfile
  is "-", stream to stdout.  If any path is "-", zip stdin.

Options and Values:
  For short options that take values, use -ovalue or -o value or -o=value
  For long option values, use either --longoption=value or --longoption value
  For example:
    zip -ds 10 --temp-dir=path zipfile path1 path2 --exclude pattern pattern
  Avoid -ovalue (no space between) to avoid confusion
  In particular, be aware of 2-character options.  For example:
    -d -s is (delete, split size) while -ds is (dot size)
  Usually better to break short options across multiple arguments by function
    zip -r -dbdcds 10m -lilalf logfile archive input_directory -ll

  All args after just "--" arg are read verbatim as paths and not options.
    zip zipfile path path ... -- verbatimpath verbatimpath ...
  Use -nw to also disable wildcards, so paths are read literally:
    zip zipfile -nw -- "-leadingdashpath" "a[path].c" "path*withwildcard"
  You may still have to escape or quote arguments to avoid shell expansion

Wildcards:
  Internally zip supports the following wildcards:
    ?       (or %% or #, depending on OS) matches any single character
    *       matches any number of characters, including zero
    [list]  matches char in list (regex), can do range [ac-f], all but [!bf]
  If port supports [], must escape [ as [[] or use -nw to turn off wildcards
  For shells that expand wildcards, escape (\* or "*") so zip can recurse
    zip zipfile -r . -i "*.h"

  Normally * crosses dir bounds in path, e.g. 'a*b' can match 'ac/db'.  If
   -ws option used, * does not cross dir bounds but ** does

  For DOS and Windows, [list] is now disabled unless the new option
  -RE       enable [list] (regular expression) matching
  is used to avoid problems with file paths containing "[" and "]":
    zip files_ending_with_number -RE foo[0-9].c

Include and Exclude:
  -i pattern pattern ...   include files that match a pattern
  -x pattern pattern ...   exclude files that match a pattern
  Patterns are paths with optional wildcards and match paths as stored in
  archive.  Exclude and include lists end at next option, @, or end of line.
    zip -x pattern pattern @ zipfile path path ...

Case matching:
  On most OS the case of patterns must match the case in the archive, unless
  the -ic option is used.
  -ic       ignore case of archive entries
  This option not available on case-sensitive file systems.  On others, case
  ignored when matching files on file system but matching against archive
  entries remains case sensitive for modes -f (freshen), -U (archive copy),
  and -d (delete) because archive paths are always case sensitive.  With
  -ic, all matching ignores case, but it's then possible multiple archive
  entries that differ only in case will match.

End Of Line Translation (text files only):
  -l        change CR or LF (depending on OS) line end to CR LF (Unix->Win)
  -ll       change CR LF to CR or LF (depending on OS) line end (Win->Unix)
  If first buffer read from file contains binary the translation is skipped

Recursion:
  -r        recurse paths, include files in subdirs:  zip -r a path path ...
  -R        recurse current dir and match patterns:   zip -R a ptn ptn ...
  Use -i and -x with either to include or exclude paths
  Path root in archive starts at current dir, so if /a/b/c/file and
   current dir is /a/b, 'zip -r archive .' puts c/file in archive

Date filtering:
  -t date   exclude before (include files modified on this date and later)
  -tt date  include before (include files modified before date)
  Can use both at same time to set a date range
  Dates are mmddyyyy or yyyy-mm-dd

Deletion, File Sync:
  -d        delete files
  Delete archive entries matching internal archive paths in list
    zip archive -d pattern pattern ...
  Can use -t and -tt to select files in archive, but NOT -x or -i, so
    zip archive -d "*" -t 2005-12-27
  deletes all files from archive.zip with date of 27 Dec 2005 and later
  Note the * (escape as "*" on Unix) to select all files in archive

  -FS       file sync
  Similar to update, but files updated if date or size of entry does not
  match file on OS.  Also deletes entry from archive if no matching file
  on OS.
    zip archive_to_update -FS -r dir_used_before
  Result generally same as creating new archive, but unchanged entries
  are copied instead of being read and compressed so can be faster.
      WARNING:  -FS deletes entries so make backup copy of archive first

Compression:
  -0        store files (no compression)
  -1 to -9  compress fastest to compress best (default is 6)
  -Z cm     set compression method to cm:
              store   - store without compression, same as option -0
              deflate - original zip deflate, same as -1 to -9 (default)
            if bzip2 is enabled:
              bzip2 - use bzip2 compression (need modern unzip)

Encryption:
  -e        use standard (weak) PKZip 2.0 encryption, prompt for password
  -P pswd   use standard encryption, password is pswd

Splits (archives created as a set of split files):
  -s ssize  create split archive with splits of size ssize, where ssize nm
              n number and m multiplier (kmgt, default m), 100k -> 100 kB
  -sp       pause after each split closed to allow changing disks
      WARNING:  Archives created with -sp use data descriptors and should
                work with most unzips but may not work with some
  -sb       ring bell when pause
  -sv       be verbose about creating splits
      Split archives CANNOT be updated, but see --out and Copy Mode below

Using --out (output to new archive):
  --out oa  output to new archive oa
  Instead of updating input archive, create new output archive oa.
  Result is same as without --out but in new archive.  Input archive
  unchanged.
      WARNING:  --out ALWAYS overwrites any existing output file
  For example, to create new_archive like old_archive but add newfile1
  and newfile2:
    zip old_archive newfile1 newfile2 --out new_archive
  Cannot update split archive, so use --out to out new archive:
    zip in_split_archive newfile1 newfile2 --out out_split_archive
  If input is split, output will default to same split size
  Use -s=0 or -s- to turn off splitting to convert split to single file:
    zip in_split_archive -s 0 --out out_single_file_archive
      WARNING:  If overwriting old split archive but need less splits,
                old splits not overwritten are not needed but remain

Copy Mode (copying from archive to archive):
  -U        (also --copy) select entries in archive to copy (reverse delete)
  Copy Mode copies entries from old to new archive with --out and is used by
  zip when either no input files on command line or -U (--copy) used.
    zip inarchive --copy pattern pattern ... --out outarchive
  To copy only files matching *.c into new archive, excluding foo.c:
    zip old_archive --copy "*.c" --out new_archive -x foo.c
  If no input files and --out, copy all entries in old archive:
    zip old_archive --out new_archive

Streaming and FIFOs:
  prog1 | zip -ll z -      zip output of prog1 to zipfile z, converting CR LF
  zip - -R "*.c" | prog2   zip *.c files in current dir and stream to prog2 
  prog1 | zip | prog2      zip in pipe with no in or out acts like zip - -
  If Zip is Zip64 enabled, streaming stdin creates Zip64 archives by default
   that need PKZip 4.5 unzipper like UnZip 6.0
  WARNING:  Some archives created with streaming use data descriptors and
            should work with most unzips but may not work with some
  Can use -fz- to turn off Zip64 if input not large (< 4 GB):
    prog_with_small_output | zip archive -fz-

  Zip now can read Unix FIFO (named pipes).  Off by default to prevent zip
  from stopping unexpectedly on unfed pipe, use -FI to enable:
    zip -FI archive fifo

Dots, counts:
  -db       display running count of bytes processed and bytes to go
              (uncompressed size, except delete and copy show stored size)
  -dc       display running count of entries done and entries to go
  -dd       display dots every 10 MB (or dot size) while processing files
  -dg       display dots globally for archive instead of for each file
    zip -qdgds 10m   will turn off most output except dots every 10 MB
  -ds siz   each dot is siz processed where siz is nm as splits (0 no dots)
  -du       display original uncompressed size for each entry as added
  -dv       display volume (disk) number in format in_disk>out_disk
  Dot size is approximate, especially for dot sizes less than 1 MB
  Dot options don't apply to Scanning files dots (dot/2sec) (-q turns off)

Logging:
  -lf path  open file at path as logfile (overwrite existing file)
  -la       append to existing logfile
  -li       include info messages (default just warnings and errors)

Testing archives:
  -T        test completed temp archive with unzip before updating archive
  -TT cmd   use command cmd instead of 'unzip -tqq' to test archive
             On Unix, to use unzip in current directory, could use:
               zip archive file1 file2 -T -TT "./unzip -tqq"
             The return code is checked for success (0 on Unix)

Fixing archives:
  -F        attempt to fix a mostly intact archive (try this first)
  -FF       try to salvage what can (may get more but less reliable)
  Fix options copy entries from potentially bad archive to new archive.
  -F tries to read archive normally and copy only intact entries, while
  -FF tries to salvage what can and may result in incomplete entries.
  Must use --out option to specify output archive:
    zip -F bad.zip --out fixed.zip
  Use -v (verbose) with -FF to see details:
    zip reallybad.zip -FF -v --out fixed.zip
  Currently neither option fixes bad entries, as from text mode ftp get.

Difference mode:
  -DF       (also --dif) only include files that have changed or are
             new as compared to the input archive
  Difference mode can be used to create incremental backups.  For example:
    zip --dif full_backup.zip -r somedir --out diff.zip
  will store all new files, as well as any files in full_backup.zip where
  either file time or size have changed from that in full_backup.zip,
  in new diff.zip.  Output archive not excluded automatically if exists,
  so either use -x to exclude it or put outside what is being zipped.

DOS Archive bit (Windows only):
  -AS       include only files with the DOS Archive bit set
  -AC       after archive created, clear archive bit of included files
      WARNING: Once the archive bits are cleared they are cleared
               Use -T to test the archive before the bits are cleared
               Can also use -sf to save file list before zipping files

Show files:
  -sf       show files to operate on and exit (-sf- logfile only)
  -su       as -sf but show escaped UTF-8 Unicode names also if exist
  -sU       as -sf but show escaped UTF-8 Unicode names instead
  Any character not in the current locale is escaped as #Uxxxx, where x
  is hex digit, if 16-bit code is sufficient, or #Lxxxxxx if 24-bits
  are needed.  If add -UN=e, Zip escapes all non-ASCII characters.

Unicode:
  If compiled with Unicode support, Zip stores UTF-8 path of entries.
  This is backward compatible.  Unicode paths allow better conversion
  of entry names between different character sets.

  New Unicode extra field includes checksum to verify Unicode path
  goes with standard path for that entry (as utilities like ZipNote
  can rename entries).  If these do not match, use below options to
  set what Zip does:
      -UN=Quit     - if mismatch, exit with error
      -UN=Warn     - if mismatch, warn, ignore UTF-8 (default)
      -UN=Ignore   - if mismatch, quietly ignore UTF-8
      -UN=No       - ignore any UTF-8 paths, use standard paths for all
  An exception to -UN=N are entries with new UTF-8 bit set (instead
  of using extra fields).  These are always handled as Unicode.

  Normally Zip escapes all chars outside current char set, but leaves
  as is supported chars, which may not be OK in path names.  -UN=Escape
  escapes any character not ASCII:
    zip -sU -UN=e archive
  Can use either normal path or escaped Unicode path on command line
  to match files in archive.

  Zip now stores UTF-8 in entry path and comment fields on systems
  where UTF-8 char set is default, such as most modern Unix, and
  and on other systems in new extra fields with escaped versions in
  entry path and comment fields for backward compatibility.
  Option -UN=UTF8 will force storing UTF-8 in entry path and comment
  fields:
      -UN=UTF8     - store UTF-8 in entry path and comment fields
  This option can be useful for multi-byte char sets on Windows where
  escaped paths and comments can be too long to be valid as the UTF-8
  versions tend to be shorter.

  Only UTF-8 comments on UTF-8 native systems supported.  UTF-8 comments
  for other systems planned in next release.

Self extractor:
  -A        Adjust offsets - a self extractor is created by prepending
             the extractor executable to archive, but internal offsets
             are then off.  Use -A to fix offsets.
  -J        Junk sfx - removes prepended extractor executable from
             self extractor, leaving a plain zip archive.

More option highlights (see manual for additional options and details):
  -b dir    when creating or updating archive, create the temp archive in
             dir, which allows using seekable temp file when writing to a
             write once CD, such archives compatible with more unzips
             (could require additional file copy if on another device)
  -MM       input patterns must match at least one file and matched files
             must be readable or exit with OPEN error and abort archive
             (without -MM, both are warnings only, and if unreadable files
             are skipped OPEN error (18) returned after archive created)
  -nw       no wildcards (wildcards are like any other character)
  -sc       show command line arguments as processed and exit
  -sd       show debugging as Zip does each step
  -so       show all available options on this system
  -X        default=strip old extra fields, -X- keep old, -X strip most
  -ws       wildcards don't span directory boundaries in paths

bash$ 

bug#21637: Subject: 24.5; dired-do-compress

[Glenn Morris]

Devon Sean McCullough wrote:

> dired-do-compress does foo.zip --> foo.zip.gz, should do foo.zip -->
> foo instead.

"Uncompressing" a zip file may (and normally does?) result in multiple
output files, which is beyond the scope of dired-do-compress,
in the same way that it does not extract .tar files.

bug#21637: Subject: 24.5; dired-do-compress

[Devon Sean McCullough]

Ask yourself, what would a reasonable user expect?
If “beyond the scope” means “we don't fix problems” well,
yes, times have changed since I was a paid Emacs maintainer.
My answer would’ve been more of the form, hmm… what's the right thing?

When foo.zip contains only names of the form foo/* perhaps ask

	Uncompress foo.zip into directory foo?  (y or n) 

of course if foo already exists, get confirmation before writing to it
and prompt for an alternative directory name if necessary.

Given one of those archives that would spam the containing directory
(I’ve read ancient religious debates on the topic but no matter)
simply derive a directory name from the archive name.

In the case of a one-file archive, just unpack it
making sure to mention the unpacked name.

		Peace
			—Devon

P.S. Obviously foo.tar would be treated likewise.

On Oct 6, 2015, at 5:57 PM, Glenn Morris <rgm@gnu.org> wrote:

> Devon Sean McCullough wrote:
>> dired-do-compress does foo.zip --> foo.zip.gz, should do foo.zip -->
>> foo instead.
> "Uncompressing" a zip file may (and normally does?) result in multiple
> output files, which is beyond the scope of dired-do-compress,
> in the same way that it does not extract .tar files.

bug#21637: Subject: 24.5; dired-do-compress

[Ken Brown]

On 10/7/2015 4:22 PM, Devon Sean McCullough wrote:
> P.S. Obviously foo.tar would be treated likewise.

That's not at all obvious.  I would expect foo.tar to be compressed to 
foo.tar.gz.  I think that's a much more common use case.

Ken

bug#21637: Subject: 24.5; dired-do-compress

[Devon Sean McCullough]

True, I conflated/confused archiving/compressing, somehow inevitable with *.zip but not with *.tgz and so on
yet many if not most users naturally consider the combination as a single concept.

		Peace
			—Devon

P.S. Even if dired-do-archive existed to serve this major use case,
the difference between foo.zip and foo.gz is utterly obscure
and to create foo.zip.gz is insanely perverse.

P.P.S. (dired "/sudo::”) works, can (dired “/zip::”) be far?
Then dired-do-copy would work.  Might be easier for
shift-C to work in Zip-Archive mode as in dired.

On Oct 7, 2015, at 9:56 PM, Ken Brown <kbrown@cornell.edu> wrote:

> On 10/7/2015 4:22 PM, Devon Sean McCullough wrote:
>> P.S. Obviously foo.tar would be treated likewise.
> 
> That's not at all obvious.  I would expect foo.tar to be compressed to foo.tar.gz.  I think that's a much more common use case.
> 
> Ken
> 

bug#21637: dired-do-compress and zip files

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> Devon Sean McCullough wrote:
>
>> dired-do-compress does foo.zip --> foo.zip.gz, should do foo.zip -->
>> foo instead.
>
> "Uncompressing" a zip file may (and normally does?) result in multiple
> output files, which is beyond the scope of dired-do-compress,
> in the same way that it does not extract .tar files.

So I think everything here works as designed, and I'm therefore closing
this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no