Passing unicode filenames to start-process on Windows?

Passing unicode filenames to start-process on Windows?

[Klaus-Dieter Bauer]

[-- Attachment #1: Type: text/plain, Size: 3251 bytes --]

Hello!

Is there a reliable way to pass unicode file names as
arguments through `start-process'?

I realized two limitations:


1. Using `prefer-coding-system' with anything other than
   `locale-default-encoding', e.g.

       (prefer-coding-system 'utf-8),

   causes a file name "Ö.txt" to be misdecoded as by
   subprocesses -- notably including "emacs.exe", but also
   all other executables I tried (both Windows builtins like
   where.exe and third party executables like ffmpeg.exe or
   GnuWin32 utilities).

   In my case (German locale, 'utf-8 preferred coding
   system) it is mis-decoded as "Ö.txt", i.e. emacs encodes
   the process argument as 'utf-8 but the subprocess decodes
   it as 'latin-1 (in my case).

   While this can be fixed by an explicit encoding

       (start-process ...
         (encode-coding-string filename locale-coding-system))

   such code will probably not be used in most projects, as
   the issue occurs only on Windows, dependent on the user
   configuration (-> hard-to-find bug?). I have added some
   elisp for demonstration at the end of the mail.


2. When a file-name contains characters that cannot be
   encoded in the locale's encoding, e.g. Japanese
   characters in a German locale, I cannot find any way to
   pass the file name through the `start-process' interface;
   Unlike for characters, that are supported by the locale,
   it fails even in a clean "emacs -Q" session.

   Curiously the file name can still be used in cmd.exe,
   though entering it may require TAB-completion (even
   though the active codepage shouldn't support them).


- Klaus


---------------- EXAMPLE CODE --------------------

;; Setup: Create a file "unifilebug/Ö.txt" with
;; some arbitrary text. Make sure it is the only file in
;; "unifilebug".
;;
;; Note that for this issue it doesn't matter what coding system
;; is chosen for file names (Unix only; On Windows the coding
;; system for file names is fixed anyway.)


;; Set the preferred coding system.
(prefer-coding-system 'utf-8)


;; Try opening it in an emacs subprocess.
;;
;; On Windows this breaks
;; if `prefer-coding-system' was called with anything other than
;; `locale-coding-system', here 'utf-8.
;;
;; On Unix (tested with cygwin), it works fine; Presumably because
;; the file name is decoded (in `directory-files') and encoded (in
;; `start-process') with the same preferred coding system.
(let ((file-name (car (directory-files "unifilebug" t "txt$"))))
  (start-process "" nil "emacs" "-Q" file-name))


;; It can be fixed by explicitly encoding file-names. This
;; thankfully works both in the W32 and the Cygwin version of
;; emacs.
(let ((file-name (car (directory-files "unifilebug" t "txt$"))))
  (start-process "" nil "emacs" "-Q"
    (encode-coding-string file-name locale-coding-system)))


;; Now we create a file called "ufb2/こんにちは世界.txt"
;; Even in a emacs-session without prefer-coding-system it will
;; fail, decoding the file-name as "ufb2/ .txt".
(let ((file-name (car (directory-files "ufb2" t "txt$"))))
  (start-process "" nil "emacs" "-Q" file-name))


--------------------------------------------------

[-- Attachment #2: Type: text/html, Size: 10843 bytes --]

Re: Passing unicode filenames to start-process on Windows?

[Eli Zaretskii]

> From: Klaus-Dieter Bauer <bauer.klaus.dieter@gmail.com>
> Date: Wed, 6 Jan 2016 16:20:29 +0100
> 
> Is there a reliable way to pass unicode file names as
> arguments through `start-process'?

No, not at the moment, not in the native Windows build of Emacs.
Arguments to subprocesses are forced to be encoded in the current
system codepage.  This commentary in w32.c tells a few more details:

   . Running subprocesses in non-ASCII directories and with non-ASCII
     file arguments is limited to the current codepage (even though
     Emacs is perfectly capable of finding an executable program file
     in a directory whose name cannot be encoded in the current
     codepage).  This is because the command-line arguments are
     encoded _before_ they get to the w32-specific level, and the
     encoding is not known in advance (it doesn't have to be the
     current ANSI codepage), so w32proc.c functions cannot re-encode
     them in UTF-16.  This should be fixed, but will also require
     changes in cmdproxy.  The current limitation is not terribly bad
     anyway, since very few, if any, Windows console programs that are
     likely to be invoked by Emacs support UTF-16 encoded command
     lines.

   . For similar reasons, server.el and emacsclient are also limited
     to the current ANSI codepage for now.

   . Emacs itself can only handle command-line arguments encoded in
     the current codepage.

The main reason for this being a low-priority problem is that the
absolute majority of console programs Emacs might invoke don't support
UTF-16 encoded command-line arguments anyway, so the efforts to enable
this would yield very little gains.  However, patches to do that will
be welcome.  (Note that, as the comment above says, the changes will
also need to touch cmdproxy, since we invoke all the programs through
it.)

> I realized two limitations:
> 
> 1. Using `prefer-coding-system' with anything other than
> `locale-default-encoding', e.g. 
> (prefer-coding-system 'utf-8), 
> causes a file name "Ö.txt" to be misdecoded as by
> subprocesses -- notably including "emacs.exe", but also
> all other executables I tried (both Windows builtins like
> where.exe and third party executables like ffmpeg.exe or
> GnuWin32 utilities). 
> In my case (German locale, 'utf-8 preferred coding
> system) it is mis-decoded as "Ö.txt", i.e. emacs encodes
> the process argument as 'utf-8 but the subprocess decodes
> it as 'latin-1 (in my case).
> While this can be fixed by an explicit encoding 
> (start-process ... 
> (encode-coding-string filename locale-coding-system))
> such code will probably not be used in most projects, as
> the issue occurs only on Windows, dependent on the user
> configuration (-> hard-to-find bug?). I have added some
> elisp for demonstration at the end of the mail.
> 
> 2. When a file-name contains characters that cannot be
> encoded in the locale's encoding, e.g. Japanese
> characters in a German locale, I cannot find any way to
> pass the file name through the `start-process' interface; 
> Unlike for characters, that are supported by the locale, 
> it fails even in a clean "emacs -Q" session. 
> Curiously the file name can still be used in cmd.exe,
> though entering it may require TAB-completion (even
> though the active codepage shouldn't support them).

Does the program which you invoke support UTF-16 encoded command-line
arguments?  It would need to either use '_wmain' instead of 'main', or
access the command-line arguments via GetCommandLineW or such likes,
and process them as wchar_t strings.

If the program doesn't have these capabilities, it won't help that
Emacs passes it UTF-16 encoded arguments, because Windows will attempt
to convert them to strings encoded in the current codepage, and will
replace any un-encodable characters with question marks or blanks.

> ;; Set the preferred coding system. 
> (prefer-coding-system 'utf-8)

You cannot use UTF-8 to encode command-line arguments on Windows, not
in general, even if the program you invoke does understand UTF-8
strings as its command-line arguments.  (I can explain if you want.)

> ;; On Unix (tested with cygwin), it works fine; Presumably because
> ;; the file name is decoded (in `directory-files') and encoded (in
> ;; `start-process') with the same preferred coding system.

It works with Cygwin because Cygwin does support UTF-8 for passing
strings to subprograms.  That support lives inside the Cygwin DLL,
which replaces most of the Windows runtime with Posix-compatible
APIs.  The native Windows build of Emacs doesn't have that luxury.

Re: Passing unicode filenames to start-process on Windows?

[Klaus-Dieter Bauer]

[-- Attachment #1: Type: text/plain, Size: 5862 bytes --]

​
2016-01-06 17:13 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:

> > From: Klaus-Dieter Bauer <bauer.klaus.dieter@gmail.com>
> > Date: Wed, 6 Jan 2016 16:20:29 +0100
> >
> > Is there a reliable way to pass unicode file names as
> > arguments through `start-process'?
>
> No, not at the moment, not in the native Windows build of Emacs.
> Arguments to subprocesses are forced to be encoded in the current
> system codepage.  This commentary in w32.c tells a few more details:
>
>    . Running subprocesses in non-ASCII directories and with non-ASCII
>      file arguments is limited to the current codepage (even though
>      Emacs is perfectly capable of finding an executable program file
>      in a directory whose name cannot be encoded in the current
>      codepage).  This is because the command-line arguments are
>      encoded _before_ they get to the w32-specific level, and the
>      encoding is not known in advance (it doesn't have to be the
>      current ANSI codepage), so w32proc.c functions cannot re-encode
>      them in UTF-16.  This should be fixed, but will also require
>      changes in cmdproxy.  The current limitation is not terribly bad
>      anyway, since very few, if any, Windows console programs that are
>      likely to be invoked by Emacs support UTF-16 encoded command
>      lines.
>
>    . For similar reasons, server.el and emacsclient are also limited
>      to the current ANSI codepage for now.
>
>    . Emacs itself can only handle command-line arguments encoded in
>      the current codepage.
>
> The main reason for this being a low-priority problem is that the
> absolute majority of console programs Emacs might invoke don't support
> UTF-16 encoded command-line arguments anyway, so the efforts to enable
> this would yield very little gains.  However, patches to do that will
> be welcome.  (Note that, as the comment above says, the changes will
> also need to touch cmdproxy, since we invoke all the programs through
> it.)
>
> > I realized two limitations:
> >
> > 1. Using `prefer-coding-system' with anything other than
> > `locale-default-encoding', e.g.
> > (prefer-coding-system 'utf-8),
> > causes a file name "Ö.txt" to be misdecoded as by
> > subprocesses -- notably including "emacs.exe", but also
> > all other executables I tried (both Windows builtins like
> > where.exe and third party executables like ffmpeg.exe or
> > GnuWin32 utilities).
> > In my case (German locale, 'utf-8 preferred coding
> > system) it is mis-decoded as "Ö.txt", i.e. emacs encodes
> > the process argument as 'utf-8 but the subprocess decodes
> > it as 'latin-1 (in my case).
> > While this can be fixed by an explicit encoding
> > (start-process ...
> > (encode-coding-string filename locale-coding-system))
> > such code will probably not be used in most projects, as
> > the issue occurs only on Windows, dependent on the user
> > configuration (-> hard-to-find bug?). I have added some
> > elisp for demonstration at the end of the mail.
> >
> > 2. When a file-name contains characters that cannot be
> > encoded in the locale's encoding, e.g. Japanese
> > characters in a German locale, I cannot find any way to
> > pass the file name through the `start-process' interface;
> > Unlike for characters, that are supported by the locale,
> > it fails even in a clean "emacs -Q" session.
> > Curiously the file name can still be used in cmd.exe,
> > though entering it may require TAB-completion (even
> > though the active codepage shouldn't support them).
>
> Does the program which you invoke support UTF-16 encoded command-line
> arguments?  It would need to either use '_wmain' instead of 'main', or
> access the command-line arguments via GetCommandLineW or such likes,
> and process them as wchar_t strings.
>
> If the program doesn't have these capabilities, it won't help that
> Emacs passes it UTF-16 encoded arguments, because Windows will attempt
> to convert them to strings encoded in the current codepage, and will
> replace any un-encodable characters with question marks or blanks.
>
> > ;; Set the preferred coding system.
> > (prefer-coding-system 'utf-8)
>
> You cannot use UTF-8 to encode command-line arguments on Windows, not
> in general, even if the program you invoke does understand UTF-8
> strings as its command-line arguments.  (I can explain if you want.)
>
> > ;; On Unix (tested with cygwin), it works fine; Presumably because
> > ;; the file name is decoded (in `directory-files') and encoded (in
> > ;; `start-process') with the same preferred coding system.
>
> It works with Cygwin because Cygwin does support UTF-8 for passing
> strings to subprograms.  That support lives inside the Cygwin DLL,
> which replaces most of the Windows runtime with Posix-compatible
> APIs.  The native Windows build of Emacs doesn't have that luxury.
>


I checked again and found that indeed some of the utilities I tested before
(specifically the GnuWin32 tools) can't handle japanese characters when
called from cmd.exe;

ffmpeg on the other hand supports unicode file names in cmd.exe, but I
agree that this is quite a niche usage.

I thought up some workarounds, but they all run into limitations:


   - w32-short-file-name: Doesn't work, because in modern Windows systems
   8.3 file names may not be generated, so it may just return the unchanged
   filename.
   - rename-file: Allows working with a name via a temporary supported file
   name. Sadly there is no way to guarantee that such renaming is undone
   afterwards.
   - copy-file (to a temporary directory): Would work for the current
   application, but unviable when larger amounts of data are involved.

Would you happen to know any other possible workaround?

thanks for the explanations,
- Klaus

[-- Attachment #2: Type: text/html, Size: 8023 bytes --]

Re: Passing unicode filenames to start-process on Windows?

[Davis Herring]

> Would you happen to know any other possible workaround?

You could use a temporary symbolic link, but those may still require
root on Windows...

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Passing unicode filenames to start-process on Windows?

[Eli Zaretskii]

> Date: Wed, 06 Jan 2016 16:05:46 -0700
> From: Davis Herring <herring@lanl.gov>
> 
> > Would you happen to know any other possible workaround?
> 
> You could use a temporary symbolic link, but those may still require
> root on Windows...

Yes, creating symlinks requires elevation on Windows.

Hard links are free from this problem, though.  But I think using
hard links will cause complications: e.g., deleting a file will not
actually delete the original.

Re: Passing unicode filenames to start-process on Windows?

[Eli Zaretskii]

> From: Klaus-Dieter Bauer <bauer.klaus.dieter@gmail.com>
> Date: Wed, 6 Jan 2016 22:19:39 +0100
> Cc: emacs-devel@gnu.org
> 
> I thought up some workarounds, but they all run into limitations:
> 
> * w32-short-file-name: Doesn't work, because in modern Windows systems 8.3 file
>   names may not be generated, so it may just return the unchanged filename.
> * rename-file: Allows working with a name via a temporary supported file name.
>   Sadly there is no way to guarantee that such renaming is undone afterwards.
> * copy-file (to a temporary directory): Would work for the current application,
>   but unviable when larger amounts of data are involved. 
> 
> Would you happen to know any other possible workaround?

The only one that would work reliably is to pass arguments via a file
or a pipe.  (Some program support "response files" as a replacement
for command-line arguments, or can read the arguments from stdin.)

Do you really have programs that can support text outside of the
current system codepage?  If you don't, then passing arguments with
such strings is the least of your problems: once you do get these
strings into the program, the program won't be able to do anything
useful with them: all the library functions that receive C strings
will misbehave, you won't be able to open files with such names, etc.

IOW, I'm not sure I understand your use case in enough detail to
provide useful advice.  Perhaps describe what you want to do and the
program you want to invoke from Emacs in more detail.

Re: Passing unicode filenames to start-process on Windows?

[Klaus-Dieter Bauer]

[-- Attachment #1: Type: text/plain, Size: 2574 bytes --]

2016-01-07 17:00 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:

> > From: Klaus-Dieter Bauer <bauer.klaus.dieter@gmail.com>
> > Date: Wed, 6 Jan 2016 22:19:39 +0100
> > Cc: emacs-devel@gnu.org
> >
> > I thought up some workarounds, but they all run into limitations:
> >
> > * w32-short-file-name: Doesn't work, because in modern Windows systems
> 8.3 file
> >   names may not be generated, so it may just return the unchanged
> filename.
> > * rename-file: Allows working with a name via a temporary supported file
> name.
> >   Sadly there is no way to guarantee that such renaming is undone
> afterwards.
> > * copy-file (to a temporary directory): Would work for the current
> application,
> >   but unviable when larger amounts of data are involved.
> >
> > Would you happen to know any other possible workaround?
>
> The only one that would work reliably is to pass arguments via a file
> or a pipe.  (Some program support "response files" as a replacement
> for command-line arguments, or can read the arguments from stdin.)
>
> Do you really have programs that can support text outside of the
> current system codepage?  If you don't, then passing arguments with
> such strings is the least of your problems: once you do get these
> strings into the program, the program won't be able to do anything
> useful with them: all the library functions that receive C strings
> will misbehave, you won't be able to open files with such names, etc.
>
> IOW, I'm not sure I understand your use case in enough detail to
> provide useful advice.  Perhaps describe what you want to do and the
> program you want to invoke from Emacs in more detail.
>

​I have two usecases where I run into the issue:

- I want at some point to write an incremental backup utility
  that uses md5sum to identify renamed files. Since precompiled
  Windows binaries are 32bit, only the first 512MB of any given
  file are accessible to elisp however, so I wanted to use
  GnuWin32's md5sum.exe (but it turns out that it doesn't
  support unicode filenames anyway).

- I want to verify a convention where filenames should mirror
  the metadata in my music library. Here I intended to write
  an elisp tool (for easy interactive processing im Emacs)
  and tried to use ffmpeg (which does support unicode filenames
  in cmd.exe).

I checked and both tools allow reading the input data from
a pipe (`type UNICODE.mp3 | ffmpeg -i - ...` or `md5sum`
respectively), so that workaround is applicable to all my usecases.

Thanks for the help!
- Klaus

[-- Attachment #2: Type: text/html, Size: 4959 bytes --]

Re: Passing unicode filenames to start-process on Windows?

[Eli Zaretskii]

> From: Klaus-Dieter Bauer <bauer.klaus.dieter@gmail.com>
> Date: Fri, 8 Jan 2016 00:31:38 +0100
> Cc: emacs-devel@gnu.org
> 
> - I want at some point to write an incremental backup utility 
> that uses md5sum to identify renamed files. Since precompiled
> Windows binaries are 32bit, only the first 512MB of any given
> file are accessible to elisp however, so I wanted to use 
> GnuWin32's md5sum.exe (but it turns out that it doesn't 
> support unicode filenames anyway). 

Emacs 25 can be built --with-wide-int, in which case the 512MB limit
goes up to almost 2GB.  I believe the precomiled binaries of the next
version will use this configure-time option.  So maybe this is still a
relevant alternative for you.

> - I want to verify a convention where filenames should mirror 
> the metadata in my music library. Here I intended to write 
> an elisp tool (for easy interactive processing im Emacs) 
> and tried to use ffmpeg (which does support unicode filenames
> in cmd.exe). 

You could have Emacs write a batch file that invokes ffmpeg with those
Unicode file names (encoded in UTF-16, of course, not UTF-8!), and
then run the batch file as the sub-process.  Will that work for you?

> I checked and both tools allow reading the input data from 
> a pipe (`type UNICODE.mp3 | ffmpeg -i - ...` or `md5sum` 
> respectively), so that workaround is applicable to all my usecases. 

Yes, that's another possibility.

Re: Passing unicode filenames to start-process on Windows?

[Klaus-Dieter Bauer]

[-- Attachment #1: Type: text/plain, Size: 1743 bytes --]

2016-01-08 10:17 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:

> > From: Klaus-Dieter Bauer <bauer.klaus.dieter@gmail.com>
> > Date: Fri, 8 Jan 2016 00:31:38 +0100
> > Cc: emacs-devel@gnu.org
> >
> > - I want at some point to write an incremental backup utility
> > that uses md5sum to identify renamed files. Since precompiled
> > Windows binaries are 32bit, only the first 512MB of any given
> > file are accessible to elisp however, so I wanted to use
> > GnuWin32's md5sum.exe (but it turns out that it doesn't
> > support unicode filenames anyway).
>
> Emacs 25 can be built --with-wide-int, in which case the 512MB limit
> goes up to almost 2GB.  I believe the precomiled binaries of the next
> version will use this configure-time option.  So maybe this is still a
> relevant alternative for you.
>
> > - I want to verify a convention where filenames should mirror
> > the metadata in my music library. Here I intended to write
> > an elisp tool (for easy interactive processing im Emacs)
> > and tried to use ffmpeg (which does support unicode filenames
> > in cmd.exe).
>
> You could have Emacs write a batch file that invokes ffmpeg with those
> Unicode file names (encoded in UTF-16, of course, not UTF-8!), and
> then run the batch file as the sub-process.  Will that work for you?
>

​It will, though I'll probably stick with the pipe version
where it's possible. It is the more portable solution (in
case that I use the script on a linux system).
​


>
> > I checked and both tools allow reading the input data from
> > a pipe (`type UNICODE.mp3 | ffmpeg -i - ...` or `md5sum`
> > respectively), so that workaround is applicable to all my usecases.
>
> Yes, that's another possibility.
>

[-- Attachment #2: Type: text/html, Size: 2926 bytes --]