bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

[awrhygty]

With python 3.10, ZIP archive can be created with:
  python -m zipfile -c ARCHIVE.zip subfile

If the subfile name contains non-ASCII characters, they are encoded with
utf-8 in anyway. Such subfile names are decoded with local language
encoding(cp932 for Japanese Windows OS) in archive-mode.

For example, archive 一.txt in test.zip with python:
  python -m zipfile -c test.zip 一.txt
The subfile name is shown as 荳\200.txt in archive-mode buffer and the
entry can not be extracted. 


In GNU Emacs 29.1 (build 2, x86_64-w64-mingw32) of 2023-08-02 built on
 AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Pro (v10.0.2009.19045.3324)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation=aot
 --without-compress-install --with-tree-sitter CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

Important settings:
  value of $LANG: JPN
  locale-coding-system: cp932

Major mode: Dired

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(wdired qp files-x shell pcomplete comint ansi-osc ansi-color ring
dired-aux image-mode exif arc-mode archive-mode pp shadow emacsbug
help-mode gnutls network-stream nsm mailalias smtpmail textsec
uni-scripts url url-proxy url-privacy url-expand url-methods url-history
url-cookie generate-lisp-file url-domsuf url-util url-parse auth-source
eieio eieio-core cl-macs json map byte-opt gv bytecomp byte-compile
url-vars idna-mapping ucs-normalize uni-confusable textsec-check sort
cl-seq misearch multi-isearch mail-extr message sendmail mailcap
yank-media puny dired dired-loaddefs rfc822 mml mml-sec password-cache
epa derived epg rfc6068 epg-config gnus-util text-property-search
time-date subr-x mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils
gmm-utils mailheader cl-loaddefs cl-lib term/bobcat japan-util rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 239724 23153)
 (symbols 48 8782 7)
 (strings 32 40794 3846)
 (string-bytes 1 887485)
 (vectors 16 45221)
 (vector-slots 8 1276135 125856)
 (floats 8 105 354)
 (intervals 56 1746 0)
 (buffers 984 17))

bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

[Eli Zaretskii]

> From: awrhygty@outlook.com
> Date: Tue, 15 Aug 2023 13:31:16 +0900
> 
> 
> With python 3.10, ZIP archive can be created with:
>   python -m zipfile -c ARCHIVE.zip subfile
> 
> If the subfile name contains non-ASCII characters, they are encoded with
> utf-8 in anyway. Such subfile names are decoded with local language
> encoding(cp932 for Japanese Windows OS) in archive-mode.
> 
> For example, archive 一.txt in test.zip with python:
>   python -m zipfile -c test.zip 一.txt
> The subfile name is shown as 荳\200.txt in archive-mode buffer and the
> entry can not be extracted. 

Is there any way of distinguishing these Python-created ZIP archives
from ZIP archives created by other Windows programs?

Emacs by default assumes that file names in a ZIP archive created by a
Windows program are encoded in the console codepage, and it enforces
using that encoding for file names when the "creator" of the ZIP
archive indicates the archive was created by Windows programs such as
InfoZip's zip.exe and the File Explorer.  In my testing, zip archives
created by Python as above record the "creator" as number 0 (zero),
which is identical to what InfoZip does.  So, unless someone explains
how to distinguish these zip archives from those created by InfoZip, I
don't see how can Emacs know whether to use the InfoZip heuristics or
the Python heuristics.  Without the InfoZip/File Explorer heuristics
we have in arc-mode.el today, Emacs on Windows would be completely
unable to support non-ASCII file names in ZIP archives.

bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

[awrhygty]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: awrhygty@outlook.com
>> Date: Tue, 15 Aug 2023 13:31:16 +0900
>> 
>> 
>> With python 3.10, ZIP archive can be created with:
>>   python -m zipfile -c ARCHIVE.zip subfile
>> 
>> If the subfile name contains non-ASCII characters, they are encoded with
>> utf-8 in anyway. Such subfile names are decoded with local language
>> encoding(cp932 for Japanese Windows OS) in archive-mode.
>> 
>> For example, archive 一.txt in test.zip with python:
>>   python -m zipfile -c test.zip 一.txt
>> The subfile name is shown as 荳\200.txt in archive-mode buffer and the
>> entry can not be extracted. 
>
> Is there any way of distinguishing these Python-created ZIP archives
> from ZIP archives created by other Windows programs?
>
> Emacs by default assumes that file names in a ZIP archive created by a
> Windows program are encoded in the console codepage, and it enforces
> using that encoding for file names when the "creator" of the ZIP
> archive indicates the archive was created by Windows programs such as
> InfoZip's zip.exe and the File Explorer.  In my testing, zip archives
> created by Python as above record the "creator" as number 0 (zero),
> which is identical to what InfoZip does.  So, unless someone explains
> how to distinguish these zip archives from those created by InfoZip, I
> don't see how can Emacs know whether to use the InfoZip heuristics or
> the Python heuristics.  Without the InfoZip/File Explorer heuristics
> we have in arc-mode.el today, Emacs on Windows would be completely
> unable to support non-ASCII file names in ZIP archives.

There is a bit flag indicating that the subfile name is encoded with
utf-8. Bytes 6-7 in local file header or bytes 8-9 in central directory
header are general purpose bit flag. And bit 11 of the flag represents
file encoding flag(1 for utf-8 encoding).

I guess unzip.exe does not support utf-8 encoded subfile name.
Writing batch file with utf-8 encoding:
  c:\Emacs\emacs-29.1\bin\unzip.exe test.zip 一.txt
and run with chcp 932, 荳\200.txt is extracted.
With chcp 65001, extraction failed.

Writing batch file with cp932 encoding:(same as above)
  c:\Emacs\emacs-29.1\bin\unzip.exe test.zip 一.txt
and run with chcp 65001, 荳\200.txt is extracted.
With chcp 932, extraction failed.
This is not an ideal behavior, but extraction to STDOUT may work.

To the contrary, 7z.exe extracts 一.txt correctly.
If batch file is encoded with utf-8, it works with chcp 65001.
If batch file is encoded with cp932, it works with chcp 932.

bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

[Eli Zaretskii]

> From: awrhygty@outlook.com
> Cc: 65305@debbugs.gnu.org
> Date: Tue, 15 Aug 2023 22:53:01 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Is there any way of distinguishing these Python-created ZIP archives
> > from ZIP archives created by other Windows programs?
> >
> > Emacs by default assumes that file names in a ZIP archive created by a
> > Windows program are encoded in the console codepage, and it enforces
> > using that encoding for file names when the "creator" of the ZIP
> > archive indicates the archive was created by Windows programs such as
> > InfoZip's zip.exe and the File Explorer.  In my testing, zip archives
> > created by Python as above record the "creator" as number 0 (zero),
> > which is identical to what InfoZip does.  So, unless someone explains
> > how to distinguish these zip archives from those created by InfoZip, I
> > don't see how can Emacs know whether to use the InfoZip heuristics or
> > the Python heuristics.  Without the InfoZip/File Explorer heuristics
> > we have in arc-mode.el today, Emacs on Windows would be completely
> > unable to support non-ASCII file names in ZIP archives.
> 
> There is a bit flag indicating that the subfile name is encoded with
> utf-8. Bytes 6-7 in local file header or bytes 8-9 in central directory
> header are general purpose bit flag. And bit 11 of the flag represents
> file encoding flag(1 for utf-8 encoding).

Thanks, please try the patch below.  If it gives good results, I will
install it.

> I guess unzip.exe does not support utf-8 encoded subfile name.
> Writing batch file with utf-8 encoding:
>   c:\Emacs\emacs-29.1\bin\unzip.exe test.zip 一.txt
> and run with chcp 932, 荳\200.txt is extracted.
> With chcp 65001, extraction failed.
> 
> Writing batch file with cp932 encoding:(same as above)
>   c:\Emacs\emacs-29.1\bin\unzip.exe test.zip 一.txt
> and run with chcp 65001, 荳\200.txt is extracted.
> With chcp 932, extraction failed.
> This is not an ideal behavior, but extraction to STDOUT may work.
> 
> To the contrary, 7z.exe extracts 一.txt correctly.
> If batch file is encoded with utf-8, it works with chcp 65001.
> If batch file is encoded with cp932, it works with chcp 932.

Like I said: support for UTF-8 encoded file names on Windows is
sporadic and incomplete.  It will remain so until Windows file-related
APIs support UTF-8 encoded file names.

diff --git a/lisp/arc-mode.el b/lisp/arc-mode.el
index 5e696c0..05a71fb 100644
--- a/lisp/arc-mode.el
+++ b/lisp/arc-mode.el
@@ -1990,6 +1990,7 @@ archive-zip-summarize
     (setq p (+ p (point-min)))
     (while (string= "PK\001\002" (buffer-substring p (+ p 4)))
       (let* ((creator (get-byte (+ p 5)))
+             (gpflags (archive-l-e (+ p 8) 2))
 	     ;; (method  (archive-l-e (+ p 10) 2))
              (modtime (archive-l-e (+ p 12) 2))
              (moddate (archive-l-e (+ p 14) 2))
@@ -2001,7 +2002,12 @@ archive-zip-summarize
              (efnname (let ((str (buffer-substring (+ p 46) (+ p 46 fnlen))))
 			(decode-coding-string
 			 str
-                         (or (if (and w32-fname-encoding
+                         ;; Bit 11 of general purpose bit flags (bytes
+                         ;; 8-9) of Central Directory: 1 means UTF-8
+                         ;; encoded file names.
+                         (or (if (/= 0 (logand gpflags #x0800))
+                                 'utf-8-unix)
+                             (if (and w32-fname-encoding
                                       (memq creator
                                             ;; This should be just 10 and
                                             ;; 14, but InfoZip uses 0 and

bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

[awrhygty]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: awrhygty@outlook.com
>> Cc: 65305@debbugs.gnu.org
>> Date: Tue, 15 Aug 2023 22:53:01 +0900
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Is there any way of distinguishing these Python-created ZIP archives
>> > from ZIP archives created by other Windows programs?
>> >
>> > Emacs by default assumes that file names in a ZIP archive created by a
>> > Windows program are encoded in the console codepage, and it enforces
>> > using that encoding for file names when the "creator" of the ZIP
>> > archive indicates the archive was created by Windows programs such as
>> > InfoZip's zip.exe and the File Explorer.  In my testing, zip archives
>> > created by Python as above record the "creator" as number 0 (zero),
>> > which is identical to what InfoZip does.  So, unless someone explains
>> > how to distinguish these zip archives from those created by InfoZip, I
>> > don't see how can Emacs know whether to use the InfoZip heuristics or
>> > the Python heuristics.  Without the InfoZip/File Explorer heuristics
>> > we have in arc-mode.el today, Emacs on Windows would be completely
>> > unable to support non-ASCII file names in ZIP archives.
>> 
>> There is a bit flag indicating that the subfile name is encoded with
>> utf-8. Bytes 6-7 in local file header or bytes 8-9 in central directory
>> header are general purpose bit flag. And bit 11 of the flag represents
>> file encoding flag(1 for utf-8 encoding).
>
> Thanks, please try the patch below.  If it gives good results, I will
> install it.
>
>> I guess unzip.exe does not support utf-8 encoded subfile name.
>> Writing batch file with utf-8 encoding:
>>   c:\Emacs\emacs-29.1\bin\unzip.exe test.zip 一.txt
>> and run with chcp 932, 荳\200.txt is extracted.
>> With chcp 65001, extraction failed.
>> 
>> Writing batch file with cp932 encoding:(same as above)
>>   c:\Emacs\emacs-29.1\bin\unzip.exe test.zip 一.txt
>> and run with chcp 65001, 荳\200.txt is extracted.
>> With chcp 932, extraction failed.
>> This is not an ideal behavior, but extraction to STDOUT may work.
>> 
>> To the contrary, 7z.exe extracts 一.txt correctly.
>> If batch file is encoded with utf-8, it works with chcp 65001.
>> If batch file is encoded with cp932, it works with chcp 932.
>
> Like I said: support for UTF-8 encoded file names on Windows is
> sporadic and incomplete.  It will remain so until Windows file-related
> APIs support UTF-8 encoded file names.
>
> diff --git a/lisp/arc-mode.el b/lisp/arc-mode.el
> index 5e696c0..05a71fb 100644
> --- a/lisp/arc-mode.el
> +++ b/lisp/arc-mode.el
> @@ -1990,6 +1990,7 @@ archive-zip-summarize
>      (setq p (+ p (point-min)))
>      (while (string= "PK\001\002" (buffer-substring p (+ p 4)))
>        (let* ((creator (get-byte (+ p 5)))
> +             (gpflags (archive-l-e (+ p 8) 2))
>  	     ;; (method  (archive-l-e (+ p 10) 2))
>               (modtime (archive-l-e (+ p 12) 2))
>               (moddate (archive-l-e (+ p 14) 2))
> @@ -2001,7 +2002,12 @@ archive-zip-summarize
>               (efnname (let ((str (buffer-substring (+ p 46) (+ p 46 fnlen))))
>  			(decode-coding-string
>  			 str
> -                         (or (if (and w32-fname-encoding
> +                         ;; Bit 11 of general purpose bit flags (bytes
> +                         ;; 8-9) of Central Directory: 1 means UTF-8
> +                         ;; encoded file names.
> +                         (or (if (/= 0 (logand gpflags #x0800))
> +                                 'utf-8-unix)
> +                             (if (and w32-fname-encoding
>                                        (memq creator
>                                              ;; This should be just 10 and
>                                              ;; 14, but InfoZip uses 0 and

The patch works to list entries, and the contents can be extracted with
7z.exe. unzip.exe does not work well.

I tried the settings below, but rewriting entries does not work.
(archive-zip-* variables' values are default if archive-7z-program is set
and zip.exe/unzip.exe are non-existent)

(setq archive-7z-program "c:/Program Files/7-Zip/7z.exe"
      archive-zip-extract '("c:/Program Files/7-Zip/7z.exe" "x" "-so")
      archive-zip-expunge '("c:/Program Files/7-Zip/7z.exe" "d")
      archive-zip-update  '("c:/Program Files/7-Zip/7z.exe" "u")
      archive-zip-update-case archive-zip-update)

It is because update command needs "-si" option followed by an entry
name. It should be one argument like (format "-si%s" name). 

bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

[Eli Zaretskii]

> From: awrhygty@outlook.com
> Cc: 65305@debbugs.gnu.org
> Date: Wed, 16 Aug 2023 12:47:14 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> There is a bit flag indicating that the subfile name is encoded with
> >> utf-8. Bytes 6-7 in local file header or bytes 8-9 in central directory
> >> header are general purpose bit flag. And bit 11 of the flag represents
> >> file encoding flag(1 for utf-8 encoding).
> >
> > Thanks, please try the patch below.  If it gives good results, I will
> > install it.
> 
> The patch works to list entries, and the contents can be extracted with
> 7z.exe. unzip.exe does not work well.

Thanks, I installed the patch on the emacs-29 branch.  I'm not
surprised that unzip.exe cannot extract such files, and I think it
works for you with 7z.exe by sheer luck: Windows transparently
converts non-ASCII characters to the system codepage when it invokes
programs via the "narrow" APIs, so it could mangle the UTF-8 encoded
file name into something unrecognizable.

> I tried the settings below, but rewriting entries does not work.
> (archive-zip-* variables' values are default if archive-7z-program is set
> and zip.exe/unzip.exe are non-existent)
> 
> (setq archive-7z-program "c:/Program Files/7-Zip/7z.exe"
>       archive-zip-extract '("c:/Program Files/7-Zip/7z.exe" "x" "-so")
>       archive-zip-expunge '("c:/Program Files/7-Zip/7z.exe" "d")
>       archive-zip-update  '("c:/Program Files/7-Zip/7z.exe" "u")
>       archive-zip-update-case archive-zip-update)
> 
> It is because update command needs "-si" option followed by an entry
> name. It should be one argument like (format "-si%s" name). 

Sorry, I don't understand: is this the same problem, or is this an
additional problem?  For example, does rewriting entries work with
ASCII file names?

If this is a separate problem, I prefer that you submit a separate bug
report with all the pertinent details.

Thanks.

bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

[awrhygty]

[-- Attachment #1: Type: text/plain, Size: 2383 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: awrhygty@outlook.com
>> Cc: 65305@debbugs.gnu.org
>> Date: Wed, 16 Aug 2023 12:47:14 +0900
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> There is a bit flag indicating that the subfile name is encoded with
>> >> utf-8. Bytes 6-7 in local file header or bytes 8-9 in central directory
>> >> header are general purpose bit flag. And bit 11 of the flag represents
>> >> file encoding flag(1 for utf-8 encoding).
>> >
>> > Thanks, please try the patch below.  If it gives good results, I will
>> > install it.
>> 
>> The patch works to list entries, and the contents can be extracted with
>> 7z.exe. unzip.exe does not work well.
>
> Thanks, I installed the patch on the emacs-29 branch.  I'm not
> surprised that unzip.exe cannot extract such files, and I think it
> works for you with 7z.exe by sheer luck: Windows transparently
> converts non-ASCII characters to the system codepage when it invokes
> programs via the "narrow" APIs, so it could mangle the UTF-8 encoded
> file name into something unrecognizable.
>
>> I tried the settings below, but rewriting entries does not work.
>> (archive-zip-* variables' values are default if archive-7z-program is set
>> and zip.exe/unzip.exe are non-existent)
>> 
>> (setq archive-7z-program "c:/Program Files/7-Zip/7z.exe"
>>       archive-zip-extract '("c:/Program Files/7-Zip/7z.exe" "x" "-so")
>>       archive-zip-expunge '("c:/Program Files/7-Zip/7z.exe" "d")
>>       archive-zip-update  '("c:/Program Files/7-Zip/7z.exe" "u")
>>       archive-zip-update-case archive-zip-update)
>> 
>> It is because update command needs "-si" option followed by an entry
>> name. It should be one argument like (format "-si%s" name). 
>
> Sorry, I don't understand: is this the same problem, or is this an
> additional problem?  For example, does rewriting entries work with
> ASCII file names?
>
> If this is a separate problem, I prefer that you submit a separate bug
> report with all the pertinent details.
>
> Thanks.

Sorry, I have mistaken something.
7z.exe works for rewriting with the settings above.
(not only ascii subfiles but also cp932 encodable subfiles)

I imagine if there is a special program which interprets ascii arguments
into multilingual strings.
Any subfile in any archive file will be treated correctly within emacs.
So I implemented an instant program using base64.


[-- Attachment #2: archive-utf.el --]
[-- Type: application/emacs-lisp, Size: 2940 bytes --]

bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

[Eli Zaretskii]

> From: awrhygty@outlook.com
> Cc: 65305@debbugs.gnu.org
> Date: Thu, 17 Aug 2023 22:56:54 +0900
> 
> Sorry, I have mistaken something.
> 7z.exe works for rewriting with the settings above.
> (not only ascii subfiles but also cp932 encodable subfiles)
> 
> I imagine if there is a special program which interprets ascii arguments
> into multilingual strings.
> Any subfile in any archive file will be treated correctly within emacs.
> So I implemented an instant program using base64.

I guess it's a special feature of 7-zip that it supports
base64-encoded file names?  Since 7-zip is not really Free Software,
I'm reluctant to go to such lengths on its behalf, especially since
it's only for MS-Windows.

bug#65305: 29.1; archive-mode can not handle subfile names encoded with utf-8

[Eli Zaretskii]

> Cc: 65305@debbugs.gnu.org
> Date: Thu, 17 Aug 2023 17:19:19 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: awrhygty@outlook.com
> > Cc: 65305@debbugs.gnu.org
> > Date: Thu, 17 Aug 2023 22:56:54 +0900
> > 
> > Sorry, I have mistaken something.
> > 7z.exe works for rewriting with the settings above.
> > (not only ascii subfiles but also cp932 encodable subfiles)
> > 
> > I imagine if there is a special program which interprets ascii arguments
> > into multilingual strings.
> > Any subfile in any archive file will be treated correctly within emacs.
> > So I implemented an instant program using base64.
> 
> I guess it's a special feature of 7-zip that it supports
> base64-encoded file names?  Since 7-zip is not really Free Software,
> I'm reluctant to go to such lengths on its behalf, especially since
> it's only for MS-Windows.

No further comments, so I'm now closing this bug.