Codepages and shell function on w32

Codepages and shell function on w32

[Lennart Borgman (gmail)]

I do not know how to solve the problem with the code pages and the shell 
function on w32, but played a bit with it. There are actually functions 
to get the code pages and I tested this at the end of the shell function:

      (shell-mode)
      (when (eq system-type 'windows-nt)
        (let ((shell-file-name prog)
              (proc (get-buffer-process (current-buffer)))
              (fullprog (executable-find prog))
              (cp-out (read (format "cp%s" 
(w32-get-console-output-codepage))))
              (cp-in  (read (format "cp%s" (w32-get-console-codepage)))))
          (cond
           ((w32-shell-dos-semantics)
            (set-process-coding-system proc cp-out cp-in))
           ((string-match "/msys/" fullprog)
            (message "think it is MSYS...")
            ;; MSYS: fix-me, does not work
            (set-process-coding-system proc 'windows-1252 'windows-1252))
           ((string-match "/cygwin/" fullprog)
            (message "think it is Cygwin...")
            ;; Cygwin: fix-me, does not work
            (set-process-coding-system proc 'windows-1252 'windows-1252))
           (t
            (message "unrecognized shell program: %s" fullprog))
           )))))
  buffer)

It seems to work for cmdproxy, but I do not know how the code should 
look. I do not know what code pages to use for MSYS and Cygwin, the code 
above is just a test.

Re: Codepages and shell function on w32

[Eli Zaretskii]

> Date: Sun, 31 Dec 2006 14:22:09 +0100
> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
> 
>               (cp-out (read (format "cp%s" (w32-get-console-output-codepage))))
>               (cp-in  (read (format "cp%s" (w32-get-console-codepage)))))
>           (cond
>            ((w32-shell-dos-semantics)
>             (set-process-coding-system proc cp-out cp-in))

This is wrong: w32-fns.el sets this as follows:

  (setq default-process-coding-system
	(if default-enable-multibyte-characters
	    '(undecided-dos . undecided-unix)
	  '(raw-text-dos . raw-text-unix)))

The output encoding uses *-unix for a good reason.

>             ;; MSYS: fix-me, does not work
>             (set-process-coding-system proc 'windows-1252 'windows-1252))
>            ((string-match "/cygwin/" fullprog)
>             (message "think it is Cygwin...")
>             ;; Cygwin: fix-me, does not work
>             (set-process-coding-system proc 'windows-1252 'windows-1252))

Does Cygwin at all support anything beyond the C locale?  If not,
that's the reason why windows-1252 doesn't work for you.

Re: Codepages and shell function on w32

[Lennart Borgman (gmail)]

Eli Zaretskii wrote:
>> Date: Sun, 31 Dec 2006 14:22:09 +0100
>> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
>>
>>               (cp-out (read (format "cp%s" (w32-get-console-output-codepage))))
>>               (cp-in  (read (format "cp%s" (w32-get-console-codepage)))))
>>           (cond
>>            ((w32-shell-dos-semantics)
>>             (set-process-coding-system proc cp-out cp-in))
>>     
>
> This is wrong: w32-fns.el sets this as follows:
>
>   (setq default-process-coding-system
> 	(if default-enable-multibyte-characters
> 	    '(undecided-dos . undecided-unix)
> 	  '(raw-text-dos . raw-text-unix)))
>
> The output encoding uses *-unix for a good reason.
>   

Can you please explain a bit more? How should it look? In the case 
cmdproxy case (the case above) it works for me with this code using 
Swedish characters in file names for example.

>   
>>             ;; MSYS: fix-me, does not work
>>             (set-process-coding-system proc 'windows-1252 'windows-1252))
>>            ((string-match "/cygwin/" fullprog)
>>             (message "think it is Cygwin...")
>>             ;; Cygwin: fix-me, does not work
>>             (set-process-coding-system proc 'windows-1252 'windows-1252))
>>     
>
> Does Cygwin at all support anything beyond the C locale?  If not,
> that's the reason why windows-1252 doesn't work for you.
>   

I have no idea. I just sent this code in the hope that someone knew and 
could correct it. It would be nice if we got it correct in the release. 
What do I write to test with the C locale?

Re: Codepages and shell function on w32

[Eli Zaretskii]

> Date: Mon, 01 Jan 2007 04:05:23 +0100
> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
> CC:  emacs-devel@gnu.org
> 
> Eli Zaretskii wrote:
> >> Date: Sun, 31 Dec 2006 14:22:09 +0100
> >> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
> >>
> >>               (cp-out (read (format "cp%s" (w32-get-console-output-codepage))))
> >>               (cp-in  (read (format "cp%s" (w32-get-console-codepage)))))
> >>           (cond
> >>            ((w32-shell-dos-semantics)
> >>             (set-process-coding-system proc cp-out cp-in))
> >>     
> >
> > This is wrong: w32-fns.el sets this as follows:
> >
> >   (setq default-process-coding-system
> > 	(if default-enable-multibyte-characters
> > 	    '(undecided-dos . undecided-unix)
> > 	  '(raw-text-dos . raw-text-unix)))
> >
> > The output encoding uses *-unix for a good reason.
> 
> Can you please explain a bit more? How should it look?

I meant the EOL conversion that you've left unspecified (i.e. open to
Emacs guesswork on input and subtle black magic on output).  You need
to use coding-system-change-eol-conversion to force the car of the
cons cell use -dos, and the cdr to use -unix.  (I hope this answers
your question because, frankly, I'm not quite sure what you were
asking.)

> In the case 
> cmdproxy case (the case above) it works for me with this code using 
> Swedish characters in file names for example.

``Works for me'' is not a replacement for good engineering, because
your testing can never find all problems.  We use these specific EOL
conversions for input and output because of the reason explained in
this comment in w32-fns.el:

  ;; Most programs on Windows will accept Unix line endings on input
  ;; (and some programs ported from Unix require it) but most will
  ;; produce DOS line endings on output.

> > Does Cygwin at all support anything beyond the C locale?  If not,
> > that's the reason why windows-1252 doesn't work for you.
> 
> I have no idea.

It's very easy to verify, if you have Cygwin programs installed, no?

> What do I write to test with the C locale?

I'd suggest the other way around: try to get a Cygwin program to work
correctly in a non-C locale.  For example, `sort' should use the
correct sorting order with non-ASCII (Swedish) characters, `ls' should
display file names with correct Swedish characters, etc.

Re: Codepages and shell function on w32

[Jason Rumney]

Lennart Borgman (gmail) wrote:
> I have no idea. I just sent this code in the hope that someone knew 
> and could correct it. It would be nice if we got it correct in the 
> release.
There is no absolute "correct", as a user may run any programs from the 
command-line, not just native Windows console programs, and their input 
files could be in any encoding. In any case, hardcoding windows-1252 is 
clearly wrong in non-European locales. Why not just set the locale for 
dos-semantic shells and leave it at that?

Re: Codepages and shell function on w32

[Eli Zaretskii]

> Date: Mon, 01 Jan 2007 06:08:08 +0000
> From: Jason Rumney <jasonr@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> There is no absolute "correct", as a user may run any programs from the 
> command-line, not just native Windows console programs, and their input 
> files could be in any encoding.

But the same problem exists on Unix, doesn't it?  On a GNU/Linux
system I'm typing this, default-process-coding-system is set to
`(undecided . iso-latin-1)'.  How about if we follow the same logic,
with the exception that we want to force EOL conversions to specific
values as w32-fns.el does now?  That is, use undecided-dos for input
and CP-unix for output, where CP is whatever
w32-get-console-output-codepage returns.

The other problem is that `undecided' might use the wrong defaults on
w32, because console programs and GUI programs use different
encodings...

> In any case, hardcoding windows-1252 is clearly wrong in
> non-European locales.

100% agreement.  The days of Europe-centric code should be long gone.

> Why not just set the locale for dos-semantic shells and leave it at
> that?

If we find no better alternative that is safe, let's do at least this.

Re: Codepages and shell function on w32

[Lennart Borgman (gmail)]

Eli Zaretskii wrote:
>> Date: Mon, 01 Jan 2007 04:05:23 +0100
>> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
>> CC:  emacs-devel@gnu.org
>>
>> Eli Zaretskii wrote:
>>     
>>>> Date: Sun, 31 Dec 2006 14:22:09 +0100
>>>> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
>>>>
>>>>               (cp-out (read (format "cp%s" (w32-get-console-output-codepage))))
>>>>               (cp-in  (read (format "cp%s" (w32-get-console-codepage)))))
>>>>           (cond
>>>>            ((w32-shell-dos-semantics)
>>>>             (set-process-coding-system proc cp-out cp-in))
>>>>     
>>>>         
>>> This is wrong: w32-fns.el sets this as follows:
>>>
>>>   (setq default-process-coding-system
>>> 	(if default-enable-multibyte-characters
>>> 	    '(undecided-dos . undecided-unix)
>>> 	  '(raw-text-dos . raw-text-unix)))
>>>
>>> The output encoding uses *-unix for a good reason.
>>>       
>> Can you please explain a bit more? How should it look?
>>     
>
> I meant the EOL conversion that you've left unspecified (i.e. open to
> Emacs guesswork on input and subtle black magic on output).  You need
> to use coding-system-change-eol-conversion to force the car of the
> cons cell use -dos, and the cdr to use -unix.
>   

Thanks. Should the cmdproxy case look something like this then:

        (let ((shell-file-name prog)
              (proc (get-buffer-process (current-buffer)))
              (fullprog (executable-find prog))
              (cp-out
               (coding-system-change-eol-conversion
                (read (format "cp%s" (w32-get-console-output-codepage)))))
              (cp-in  (read (format "cp%s" (w32-get-console-codepage)))))
          (cond
           ((w32-shell-dos-semantics)
            (set-process-coding-system proc cp-out cp-in))

Should it be in defun shell? Is there a better way then (read (format 
...)) to convert from integer to symbol here?

> It's very easy to verify, if you have Cygwin programs installed, no?
>   

Yes, both Cygwin and MSYS for testing.

>   
>> What do I write to test with the C locale?
>>     
>
> I'd suggest the other way around: try to get a Cygwin program to work
> correctly in a non-C locale.  For example, `sort' should use the
> correct sorting order with non-ASCII (Swedish) characters, `ls' should
> display file names with correct Swedish characters, etc.
>   

Both MSYS and Cygwin have troubles with Swedish characters in file names.

Re: Codepages and shell function on w32

[Lennart Borgman (gmail)]

Eli Zaretskii wrote:
>> Date: Mon, 01 Jan 2007 06:08:08 +0000
>> From: Jason Rumney <jasonr@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>>
>> There is no absolute "correct", as a user may run any programs from the 
>> command-line, not just native Windows console programs, and their input 
>> files could be in any encoding.
>>     
>
> But the same problem exists on Unix, doesn't it?  On a GNU/Linux
> system I'm typing this, default-process-coding-system is set to
> `(undecided . iso-latin-1)'.  How about if we follow the same logic,
> with the exception that we want to force EOL conversions to specific
> values as w32-fns.el does now?  That is, use undecided-dos for input
> and CP-unix for output, where CP is whatever
> w32-get-console-output-codepage returns.
>   

Oh, it is a bit easy to mix in and out here. This is what I use for 
testing now:

      (shell-mode)
      (when (eq system-type 'windows-nt)
        (let* ((shell-file-name prog)
               (proc (get-buffer-process (current-buffer)))
               (fullprog (executable-find prog))
               (cp-out (read (format "cp%s" 
(w32-get-console-output-codepage))))
               (cp-out-dos (coding-system-change-eol-conversion cp-out 
'dos))
               (cp-out-unix (coding-system-change-eol-conversion cp-out 
'unix))
               (cp-in  (read (format "cp%s" (w32-get-console-codepage))))
               (cp-in-dos (coding-system-change-eol-conversion cp-in 'dos))
               (cp-in-unix (coding-system-change-eol-conversion cp-in 
'unix)))
          (cond
           ((w32-shell-dos-semantics)
            (set-process-coding-system proc cp-out-dos cp-in-unix))
           ((string-match "/msys/" fullprog)
            (message "think it is MSYS...")
            (set-process-coding-system proc cp-out-dos 'undecided-unix))
           ((string-match "/cygwin/" fullprog)
            (message "think it is Cygwin...")
            (set-process-coding-system proc cp-out-dos 'undecided-unix))
           (t
            (message "unrecognized shell program: %s" fullprog))
           )))))
  buffer)

Is this what you meant Eli?

Nothing especially weird happens with this version. It works OK for 
cmdproxy. Cygwin and MSYS still can't handle Swedish characters. I guess 
there is nothing to do about that.

Re: Codepages and shell function on w32

[Eli Zaretskii]

> Date: Mon, 01 Jan 2007 15:20:47 +0100
> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
> CC: Jason Rumney <jasonr@gnu.org>,  emacs-devel@gnu.org
> 
>       (shell-mode)
>       (when (eq system-type 'windows-nt)
>         (let* ((shell-file-name prog)
>                (proc (get-buffer-process (current-buffer)))
>                (fullprog (executable-find prog))
>                (cp-out (read (format "cp%s" (w32-get-console-output-codepage))))
>                (cp-out-dos (coding-system-change-eol-conversion cp-out 'dos))
>                (cp-out-unix (coding-system-change-eol-conversion cp-out 'unix))
>                (cp-in  (read (format "cp%s" (w32-get-console-codepage))))
>                (cp-in-dos (coding-system-change-eol-conversion cp-in 'dos))
>                (cp-in-unix (coding-system-change-eol-conversion cp-in 'unix)))
>           (cond
>            ((w32-shell-dos-semantics)
>             (set-process-coding-system proc cp-out-dos cp-in-unix))
>            ((string-match "/msys/" fullprog)
>             (message "think it is MSYS...")
>             (set-process-coding-system proc cp-out-dos 'undecided-unix))
>            ((string-match "/cygwin/" fullprog)
>             (message "think it is Cygwin...")
>             (set-process-coding-system proc cp-out-dos 'undecided-unix))
>            (t
>             (message "unrecognized shell program: %s" fullprog))
>            )))))
>   buffer)
> 
> Is this what you meant Eli?

Something like that, but I doubt that using cp-out with MSYS and
Cygwin is a good idea, since they don't support that.

Re: Codepages and shell function on w32

[Lennart Borgman (gmail)]

Eli Zaretskii wrote:
>> Date: Mon, 01 Jan 2007 15:20:47 +0100
>> From: "Lennart Borgman (gmail)" <lennart.borgman@gmail.com>
>> CC: Jason Rumney <jasonr@gnu.org>,  emacs-devel@gnu.org
>>
>>       (shell-mode)
>>       (when (eq system-type 'windows-nt)
>>         (let* ((shell-file-name prog)
>>                (proc (get-buffer-process (current-buffer)))
>>                (fullprog (executable-find prog))
>>                (cp-out (read (format "cp%s" (w32-get-console-output-codepage))))
>>                (cp-out-dos (coding-system-change-eol-conversion cp-out 'dos))
>>                (cp-out-unix (coding-system-change-eol-conversion cp-out 'unix))
>>                (cp-in  (read (format "cp%s" (w32-get-console-codepage))))
>>                (cp-in-dos (coding-system-change-eol-conversion cp-in 'dos))
>>                (cp-in-unix (coding-system-change-eol-conversion cp-in 'unix)))
>>           (cond
>>            ((w32-shell-dos-semantics)
>>             (set-process-coding-system proc cp-out-dos cp-in-unix))
>>            ((string-match "/msys/" fullprog)
>>             (message "think it is MSYS...")
>>             (set-process-coding-system proc cp-out-dos 'undecided-unix))
>>            ((string-match "/cygwin/" fullprog)
>>             (message "think it is Cygwin...")
>>             (set-process-coding-system proc cp-out-dos 'undecided-unix))
>>            (t
>>             (message "unrecognized shell program: %s" fullprog))
>>            )))))
>>   buffer)
>>
>> Is this what you meant Eli?
>>     
>
> Something like that, but I doubt that using cp-out with MSYS and
> Cygwin is a good idea, since they don't support that.
>   

The only reason to use cp-out-dos for MSYS and Cygwin is that this makes 
the output correct for w32 console programs they may run. However I do 
not know if it creates some problems since I do not know the char tables 
very well. Maybe it is better to just leave the coding system for MSYS 
and Cygwin as before?.

Do we agree about the cmdproxy part? How about installing that?

Re: Codepages and shell function on w32

[Benjamin Riefenstahl]

Hi Lennart,

"Lennart Borgman (gmail)" writes:
> I do not know what code pages to use for MSYS and Cygwin, the code
> above is just a test.

For Cygwin, see
<http://cygwin.com/cygwin-ug-net/using-cygwinenv.html>,
CYGWIN=codepage:ansi vs. CYGWIN=codepage:oem.

benny

Re: Codepages and shell function on w32

[Lennart Borgman (gmail)]

Benjamin Riefenstahl wrote:
> Hi Lennart,
>
> "Lennart Borgman (gmail)" writes:
>   
>> I do not know what code pages to use for MSYS and Cygwin, the code
>> above is just a test.
>>     
>
> For Cygwin, see
> <http://cygwin.com/cygwin-ug-net/using-cygwinenv.html>,
> CYGWIN=codepage:ansi vs. CYGWIN=codepage:oem.
>
> benny
>   

Thanks Benny,

I read from there that codepage ansi is the default for Cygwin. But what 
black magic do we need to use to specify what Cygwin calls codepage ansi 
or oem in Emacs?

Re: Codepages and shell function on w32

[Benjamin Riefenstahl]

Hi Lennart,


Sorry for the late reply. 

"Lennart Borgman (gmail)" writes:
> I read from there that codepage ansi is the default for Cygwin. But
> what black magic do we need to use to specify what Cygwin calls
> codepage ansi or oem in Emacs?

Cygwin just uses the MS Windows naming conventions here.  "ANSI" is
the codepage that the graphics subsystem uses and "OEM" is the default
codepage for console I/O, like CMD.EXE uses it, e.g.

Emacs should have the coding system for the "ANSI" codepage in the
variable locale-coding-system, I think.


benny

Re: Codepages and shell function on w32

[Lennart Borgman (gmail)]

Benjamin Riefenstahl wrote:
> Hi Lennart,
> 
> 
> Sorry for the late reply. 
> 
> "Lennart Borgman (gmail)" writes:
>> I read from there that codepage ansi is the default for Cygwin. But
>> what black magic do we need to use to specify what Cygwin calls
>> codepage ansi or oem in Emacs?
> 
> Cygwin just uses the MS Windows naming conventions here.  "ANSI" is
> the codepage that the graphics subsystem uses and "OEM" is the default
> codepage for console I/O, like CMD.EXE uses it, e.g.
> 
> Emacs should have the coding system for the "ANSI" codepage in the
> variable locale-coding-system, I think.


Thanks Benjamin. I tested this with both Cygwin and MSYS and it seems to 
work. So now here is how the end of defun shell could look like at the 
moment:

       (shell-mode)
       (when (eq system-type 'windows-nt)
         (let* ((shell-file-name prog)
                (proc (get-buffer-process (current-buffer)))
                (fullprog (executable-find prog))
                (cp-out (read (format "cp%s" 
(w32-get-console-output-codepage))))
                (cp-out-dos (coding-system-change-eol-conversion cp-out 
'dos))
                (cp-out-unix (coding-system-change-eol-conversion cp-out 
'unix))
                (cp-in  (read (format "cp%s" (w32-get-console-codepage))))
                (cp-in-dos (coding-system-change-eol-conversion cp-in 'dos))
                (cp-in-unix (coding-system-change-eol-conversion cp-in 
'unix))
                (cp-locale-dos (coding-system-change-eol-conversion 
locale-coding-system 'dos))
                (cp-locale-unix (coding-system-change-eol-conversion 
locale-coding-system 'unix)))
           (cond
            ((w32-shell-dos-semantics)
             (set-process-coding-system proc cp-out-dos cp-in-unix))
            ((string-match "/msys/" fullprog)
             (message "think it is MSYS...")
             (set-process-coding-system proc cp-locale-dos cp-locale-unix))
            ((string-match "/cygwin/" fullprog)
             (message "think it is Cygwin...")
             (set-process-coding-system proc cp-locale-dos cp-locale-unix))
            (t
             (message "unrecognized shell program: %s" fullprog))
            )))))
   buffer)


Maybe we could install something like this and get on with the quesion 
about file name completion which as we earlier saw also is broken on w32?

I would however want to add one thing to the above: The possibility to 
explicitly set the coding system for the interactive shell.

And maybe the tests for Cygwin and MSYS should be enhanced, but let us 
take some steps forward please!