bug#71500: 28.2; Can't create/open .gpg file name contains non-ascii encoding in windows

bug#71500: 28.2; Can't create/open .gpg file name contains non-ascii encoding in windows

[Tatsuya Kumazawa]

When I save .gpg file contains non-ASCII(eg: e:/あ.gpg).

I got message below
(It happens anytime when user name contains non-ASCII
(it must to change epg-gpg-home-directory to ASCII only path))

----
Error while encrypting with "e:/msys2/usr/bin/gpg.exe":

gpg: can't create 'c:/Users/(my name with CP932 encoded by
UTF-8)/AppData/Local/Temp/epg-output9UGAFQ': No such file or directory
gpg: symmetric encryption of '[stdin]' failed: No such file or directory
----


When I open e:/あ.gpg, I got

----
Error while decrypting with "e:/msys2/usr/bin/gpg.exe":

gpg: can't open 'e:/邵ｺ繝ｻgpg': No such file or directory
gpg: decrypt_message failed: No such file or directory
----

It seems UTF-8 encode by CP932 binary sequence.

share\emacs\28.2\lisp\epg.el

673:                                                     :coding 'raw-text

to

673:                                                     :coding
'(raw-text . cp932)

(CP932 is japanese charactor code of windows)
seemsly worked in my environment.



It might be good from cp932 to system process encoding for multilanguage?

(Sorry, I don't know emacs well)


In GNU Emacs 28.2 (build 2, x86_64-w64-mingw32)
 of 2022-09-14 built on AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.22631
System Description: Microsoft Windows 10 Pro (v10.0.2009.22631.3672)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation
 --without-compress-install CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

Important settings:
  value of $LANG: JPN
  locale-coding-system: cp932

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(mailalias mailclient browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util url-parse
url-vars mailcap qp pp shadow sort mail-extr emacsbug message rmc puny
dired dired-loaddefs rfc822 mml mml-sec epa derived epg cl-extra
help-mode rfc6068 epg-config gnus-util rmail rmail-loaddefs auth-source
cl-seq eieio eieio-core cl-macs eieio-loaddefs password-cache json map
text-property-search time-date subr-x seq byte-opt gv bytecomp
byte-compile cconv mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils japan-util iso-transl
tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 94210 14660)
 (symbols 48 8115 2)
 (strings 32 24243 3223)
 (string-bytes 1 785123)
 (vectors 16 22517)
 (vector-slots 8 1119837 217090)
 (floats 8 31 74)
 (intervals 56 1138 23)
 (buffers 992 15))

bug#71500: 28.2; Can't create/open .gpg file name contains non-ascii encoding in windows

[Eli Zaretskii]

> From: Tatsuya Kumazawa <billy.hotwind@gmail.com>
> Date: Tue, 11 Jun 2024 22:35:23 +0900
> 
> When I save .gpg file contains non-ASCII(eg: e:/あ.gpg).
> 
> I got message below
> (It happens anytime when user name contains non-ASCII
> (it must to change epg-gpg-home-directory to ASCII only path))
> 
> ----
> Error while encrypting with "e:/msys2/usr/bin/gpg.exe":
> 
> gpg: can't create 'c:/Users/(my name with CP932 encoded by
> UTF-8)/AppData/Local/Temp/epg-output9UGAFQ': No such file or directory
> gpg: symmetric encryption of '[stdin]' failed: No such file or directory
> ----
> 
> 
> When I open e:/あ.gpg, I got
> 
> ----
> Error while decrypting with "e:/msys2/usr/bin/gpg.exe":
> 
> gpg: can't open 'e:/邵ｺ繝ｻgpg': No such file or directory
> gpg: decrypt_message failed: No such file or directory
> ----
> 
> It seems UTF-8 encode by CP932 binary sequence.
> 
> share\emacs\28.2\lisp\epg.el
> 
> 673:                                                     :coding 'raw-text
> 
> to
> 
> 673:                                                     :coding
> '(raw-text . cp932)
> 
> (CP932 is japanese charactor code of windows)
> seemsly worked in my environment.
> 
> 
> 
> It might be good from cp932 to system process encoding for multilanguage?

There were some encoding problems in epg that were fixed recently in
the development sources.  Can try Emacs 30 from the master branch of
the Emacs Git repository and see if the problem still exists there?

In any case, on MS-Windows Emacs can only support GPG-encoding of file
names that can be encoded in your system codepage, due to limitations
of how Emacs on Windows invokes sub-processes.  So this will only work
of cp932 is your system codepage.

bug#71500: 28.2; Can't create/open .gpg file name contains non-ascii encoding in windows

[Eli Zaretskii]

Ping!  Did you have a chance to try Emacs 30 (now on the emacs-30
branch of the Emacs Git repository)?

> Cc: 71500@debbugs.gnu.org
> Date: Wed, 12 Jun 2024 10:52:23 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Tatsuya Kumazawa <billy.hotwind@gmail.com>
> > Date: Tue, 11 Jun 2024 22:35:23 +0900
> > 
> > When I save .gpg file contains non-ASCII(eg: e:/あ.gpg).
> > 
> > I got message below
> > (It happens anytime when user name contains non-ASCII
> > (it must to change epg-gpg-home-directory to ASCII only path))
> > 
> > ----
> > Error while encrypting with "e:/msys2/usr/bin/gpg.exe":
> > 
> > gpg: can't create 'c:/Users/(my name with CP932 encoded by
> > UTF-8)/AppData/Local/Temp/epg-output9UGAFQ': No such file or directory
> > gpg: symmetric encryption of '[stdin]' failed: No such file or directory
> > ----
> > 
> > 
> > When I open e:/あ.gpg, I got
> > 
> > ----
> > Error while decrypting with "e:/msys2/usr/bin/gpg.exe":
> > 
> > gpg: can't open 'e:/邵ｺ繝ｻgpg': No such file or directory
> > gpg: decrypt_message failed: No such file or directory
> > ----
> > 
> > It seems UTF-8 encode by CP932 binary sequence.
> > 
> > share\emacs\28.2\lisp\epg.el
> > 
> > 673:                                                     :coding 'raw-text
> > 
> > to
> > 
> > 673:                                                     :coding
> > '(raw-text . cp932)
> > 
> > (CP932 is japanese charactor code of windows)
> > seemsly worked in my environment.
> > 
> > 
> > 
> > It might be good from cp932 to system process encoding for multilanguage?
> 
> There were some encoding problems in epg that were fixed recently in
> the development sources.  Can try Emacs 30 from the master branch of
> the Emacs Git repository and see if the problem still exists there?
> 
> In any case, on MS-Windows Emacs can only support GPG-encoding of file
> names that can be encoded in your system codepage, due to limitations
> of how Emacs on Windows invokes sub-processes.  So this will only work
> of cp932 is your system codepage.
> 
> 
> 
> 

bug#71500: 28.2; Can't create/open .gpg file name contains non-ascii encoding in windows

[Kazuhiro Ito]

> Ping!  Did you have a chance to try Emacs 30 (now on the emacs-30
> branch of the Emacs Git repository)?

Unfortunately, the issue is not resolved.

As Tatsuya wrote, epg--start function calls gpg2 program with setting
:coding property to raw-text, thus each argument should be encoded in
proper coding system by a caller.  The problem would occur on any OS
whose file name coding system was not utf-8.

For example, in case of epg-decrypt-file, following change is required
if we need to set :coding property to raw-text for ENCODING.

diff --git a/lisp/epg.el b/lisp/epg.el
index 7bec91f616d..59870761565 100644
--- a/lisp/epg.el
+++ b/lisp/epg.el
@@ -1468,7 +1468,10 @@ epg-decrypt-file
       (progn
 	(setf (epg-context-output-file context)
               (or plain (make-temp-file "epg-output")))
-	(epg-start-decrypt context (epg-make-data-from-file cipher))
+	(epg-start-decrypt context
+                           (epg-make-data-from-file
+                            (encode-coding-string
+                             cipher file-name-coding-system)))
 	(epg-wait-for-completion context)
 	(epg--check-error-for-decrypt context)
 	(unless plain

-- 
Kazuhiro Ito


> > Cc: 71500@debbugs.gnu.org
> > Date: Wed, 12 Jun 2024 10:52:23 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > 
> > > From: Tatsuya Kumazawa <billy.hotwind@gmail.com>
> > > Date: Tue, 11 Jun 2024 22:35:23 +0900
> > > 
> > > When I save .gpg file contains non-ASCII(eg: e:/あ.gpg).
> > > 
> > > I got message below
> > > (It happens anytime when user name contains non-ASCII
> > > (it must to change epg-gpg-home-directory to ASCII only path))
> > > 
> > > ----
> > > Error while encrypting with "e:/msys2/usr/bin/gpg.exe":
> > > 
> > > gpg: can't create 'c:/Users/(my name with CP932 encoded by
> > > UTF-8)/AppData/Local/Temp/epg-output9UGAFQ': No such file or directory
> > > gpg: symmetric encryption of '[stdin]' failed: No such file or directory
> > > ----
> > > 
> > > 
> > > When I open e:/あ.gpg, I got
> > > 
> > > ----
> > > Error while decrypting with "e:/msys2/usr/bin/gpg.exe":
> > > 
> > > gpg: can't open 'e:/邵ｺ繝ｻgpg': No such file or directory
> > > gpg: decrypt_message failed: No such file or directory
> > > ----
> > > 
> > > It seems UTF-8 encode by CP932 binary sequence.
> > > 
> > > share\emacs\28.2\lisp\epg.el
> > > 
> > > 673:                                                     :coding 'raw-text
> > > 
> > > to
> > > 
> > > 673:                                                     :coding
> > > '(raw-text . cp932)
> > > 
> > > (CP932 is japanese charactor code of windows)
> > > seemsly worked in my environment.
> > > 
> > > 
> > > 
> > > It might be good from cp932 to system process encoding for multilanguage?
> > 
> > There were some encoding problems in epg that were fixed recently in
> > the development sources.  Can try Emacs 30 from the master branch of
> > the Emacs Git repository and see if the problem still exists there?

bug#71500: 28.2; Can't create/open .gpg file name contains non-ascii encoding in windows

[Eli Zaretskii]

> Date: Fri, 28 Jun 2024 20:30:08 +0900
> From: Kazuhiro Ito <kzhr@d1.dion.ne.jp>
> Cc: billy.hotwind@gmail.com, eliz@gnu.org
> 
> > Ping!  Did you have a chance to try Emacs 30 (now on the emacs-30
> > branch of the Emacs Git repository)?
> 
> Unfortunately, the issue is not resolved.
> 
> As Tatsuya wrote, epg--start function calls gpg2 program with setting
> :coding property to raw-text, thus each argument should be encoded in
> proper coding system by a caller.  The problem would occur on any OS
> whose file name coding system was not utf-8.
> 
> For example, in case of epg-decrypt-file, following change is required
> if we need to set :coding property to raw-text for ENCODING.
> 
> diff --git a/lisp/epg.el b/lisp/epg.el
> index 7bec91f616d..59870761565 100644
> --- a/lisp/epg.el
> +++ b/lisp/epg.el
> @@ -1468,7 +1468,10 @@ epg-decrypt-file
>        (progn
>  	(setf (epg-context-output-file context)
>                (or plain (make-temp-file "epg-output")))
> -	(epg-start-decrypt context (epg-make-data-from-file cipher))
> +	(epg-start-decrypt context
> +                           (epg-make-data-from-file
> +                            (encode-coding-string
> +                             cipher file-name-coding-system)))
>  	(epg-wait-for-completion context)
>  	(epg--check-error-for-decrypt context)
>  	(unless plain

Thanks, but this doesn't look right to me.  The problem is likely in
the fact that epg--start sets the encoding/decoding for the epg
process to raw-text in a way that is too broad: it affects not only
how we encode and decode text sent to and received from GnuPG, but
also how its command-line arguments are encoded.  And that's not what
we want.

So I think the right fix is in epg--start: instead of setting :coding
as part of the make-process call, it should call
set-process-coding-system _after_ the make-process call, so that
raw-text will affect only the text we send and receive from gpg.

Can you try such a change and see if it fixes the problem?

bug#71500: 28.2; Can't create/open .gpg file name contains non-ascii encoding in windows

[Kazuhiro Ito]

[-- Attachment #1: Type: text/plain, Size: 2069 bytes --]

> > As Tatsuya wrote, epg--start function calls gpg2 program with setting
> > :coding property to raw-text, thus each argument should be encoded in
> > proper coding system by a caller.  The problem would occur on any OS
> > whose file name coding system was not utf-8.
> > 
> > For example, in case of epg-decrypt-file, following change is required
> > if we need to set :coding property to raw-text for ENCODING.
> > 
> > diff --git a/lisp/epg.el b/lisp/epg.el
> > index 7bec91f616d..59870761565 100644
> > --- a/lisp/epg.el
> > +++ b/lisp/epg.el
> > @@ -1468,7 +1468,10 @@ epg-decrypt-file
> >        (progn
> >  	(setf (epg-context-output-file context)
> >                (or plain (make-temp-file "epg-output")))
> > -	(epg-start-decrypt context (epg-make-data-from-file cipher))
> > +	(epg-start-decrypt context
> > +                           (epg-make-data-from-file
> > +                            (encode-coding-string
> > +                             cipher file-name-coding-system)))
> >  	(epg-wait-for-completion context)
> >  	(epg--check-error-for-decrypt context)
> >  	(unless plain
> 
> Thanks, but this doesn't look right to me.  The problem is likely in
> the fact that epg--start sets the encoding/decoding for the epg
> process to raw-text in a way that is too broad: it affects not only
> how we encode and decode text sent to and received from GnuPG, but
> also how its command-line arguments are encoded.  And that's not what
> we want.
> 
> So I think the right fix is in epg--start: instead of setting :coding
> as part of the make-process call, it should call
> set-process-coding-system _after_ the make-process call, so that
> raw-text will affect only the text we send and receive from gpg.
> 
> Can you try such a change and see if it fixes the problem?

I tested attached patch and confirmed the issue was resolved.  I'm not
sure that :coding setting for decoding in make-process can be removed.
I mean, I don't know whether Emacs can receive output from process
between make-process and set-process-coding-system.

-- 
Kazuhiro Ito

[-- Attachment #2: 0001-Fix-non-ASCII-filename-operatiion-on-EasyPG-bug-7150.patch --]
[-- Type: message/rfc822, Size: 882 bytes --]

From: Kazuhiro Ito <kzhr@d1.dion.ne.jp>
Subject: [PATCH] Fix non-ASCII filename operatiion on EasyPG (bug#71500)
Date: Sat, 29 Jun 2024 20:56:02 +0900

* lisp/epg.el (epg--start): Don't suppress to encode arguments for
gpg2 program.
---
 lisp/epg.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/epg.el b/lisp/epg.el
index 7bec91f616d..cf59f03c724 100644
--- a/lisp/epg.el
+++ b/lisp/epg.el
@@ -676,10 +676,11 @@ epg--start
 				    :command (cons (epg-context-program context)
 						   args)
 				    :connection-type 'pipe
-				    :coding 'raw-text
+				    :coding '(raw-text . nil)
 				    :filter #'epg--process-filter
 				    :stderr error-process
 				    :noquery t))))
+    (set-process-coding-system process 'raw-text 'raw-text)
     (setf (epg-context-process context) process)))
 
 (defun epg--process-filter (process input)
-- 
2.45.1

bug#71500: 28.2; Can't create/open .gpg file name contains non-ascii encoding in windows

[Eli Zaretskii]

> Date: Sat, 29 Jun 2024 21:39:38 +0900
> From: Kazuhiro Ito <kzhr@d1.dion.ne.jp>
> Cc: 71500@debbugs.gnu.org,
> 	billy.hotwind@gmail.com
> 
> > > As Tatsuya wrote, epg--start function calls gpg2 program with setting
> > > :coding property to raw-text, thus each argument should be encoded in
> > > proper coding system by a caller.  The problem would occur on any OS
> > > whose file name coding system was not utf-8.
> > > 
> > > For example, in case of epg-decrypt-file, following change is required
> > > if we need to set :coding property to raw-text for ENCODING.
> > > 
> > > diff --git a/lisp/epg.el b/lisp/epg.el
> > > index 7bec91f616d..59870761565 100644
> > > --- a/lisp/epg.el
> > > +++ b/lisp/epg.el
> > > @@ -1468,7 +1468,10 @@ epg-decrypt-file
> > >        (progn
> > >  	(setf (epg-context-output-file context)
> > >                (or plain (make-temp-file "epg-output")))
> > > -	(epg-start-decrypt context (epg-make-data-from-file cipher))
> > > +	(epg-start-decrypt context
> > > +                           (epg-make-data-from-file
> > > +                            (encode-coding-string
> > > +                             cipher file-name-coding-system)))
> > >  	(epg-wait-for-completion context)
> > >  	(epg--check-error-for-decrypt context)
> > >  	(unless plain
> > 
> > Thanks, but this doesn't look right to me.  The problem is likely in
> > the fact that epg--start sets the encoding/decoding for the epg
> > process to raw-text in a way that is too broad: it affects not only
> > how we encode and decode text sent to and received from GnuPG, but
> > also how its command-line arguments are encoded.  And that's not what
> > we want.
> > 
> > So I think the right fix is in epg--start: instead of setting :coding
> > as part of the make-process call, it should call
> > set-process-coding-system _after_ the make-process call, so that
> > raw-text will affect only the text we send and receive from gpg.
> > 
> > Can you try such a change and see if it fixes the problem?
> 
> I tested attached patch and confirmed the issue was resolved.

Thanks, installed on the emacs-30 release branch, and closing the bug.

> I'm not sure that :coding setting for decoding in make-process can
> be removed.  I mean, I don't know whether Emacs can receive output
> from process between make-process and set-process-coding-system.

Emacs is a single-threaded program, so until epg--start returns, no
output can be received and decoded by Emacs from the gpg sub-process.
So this is safe.