On Windows(R) Emacs «shell» mode doesn't prints an output.

On Windows(R) Emacs «shell» mode doesn't prints an output.

[Hi-Angel]

When I am working on Windows, I am using Emacs «shell» mode. The
problem is that it sometimes doesn't prints an applications output
(more precisely prints only after the app died).
The problem is present exclusively in Emacs, e.g. if I am run an
application in Windows® native terminal, it is works fine.

I thought that the problem is in encoding, and set «shell»'s buffer
with «C-x <RET> f» to UTF-16, which, as I know, uses MS, but it didn't
fixed the problem.

Below is a simple example of an application that won't print an output in Emacs:

#include <cstdio>
#include <cstring>

int main(){
    puts("Hello, enter something");
    char buf[256];
    memset(buf, 0, sizeof(buf));
    fgets(buf, sizeof(buf)-1, stdin);
    puts(buf);
    return 0;
}

This works fine in both GNU/Linux, and the terminal of Windows®, but
not in Emacs shell mode on Windowsor.

Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Eli Zaretskii]

> Date: Thu, 25 Sep 2014 00:07:09 +0400
> From: Hi-Angel <hiangel999@gmail.com>
> 
> When I am working on Windows, I am using Emacs «shell» mode. The
> problem is that it sometimes doesn't prints an applications output
> (more precisely prints only after the app died).
> The problem is present exclusively in Emacs, e.g. if I am run an
> application in Windows® native terminal, it is works fine.

Emacs on Windows redirects the shell's standard handles to a pipe, and
pipes on Windows are fully-buffered (as opposed to line-buffering for
the console device).  So if your program doesn't flush its standard
output frequently enough, you may see this problem.

> I thought that the problem is in encoding, and set «shell»'s buffer
> with «C-x <RET> f» to UTF-16, which, as I know, uses MS, but it didn't
> fixed the problem.

Console applications on Windows use the current OEM codepage, not
UTF-16.

Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Yuri Khan]

On Thu, Sep 25, 2014 at 2:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>> I thought that the problem is in encoding, and set «shell»'s buffer
>> with «C-x <RET> f» to UTF-16, which, as I know, uses MS, but it didn't
>> fixed the problem.
>
> Console applications on Windows use the current OEM codepage, not
> UTF-16.

This is correct in general, although *some* Windows console
applications can be persuaded to output in UTF-16. (The only example I
know of is cmd.exe /U.)

Also, there is an unofficial unsupported ugly hack to use “chcp 65001”
which makes applications think that the current OEM codepage is UTF-8.
(It is unsupported because that there is some code that assumes at
most two bytes for each character, while UTF-8 may produce up to
four.)

Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Hi-Angel]

Dear Pascal J. Bourguignon, no need for an abuses, don't be a
barbarian. I am always heard and was sure that the buffers flushes on
a «newline». That's weird to me that it is wrong, but after your words
I googled, and found that this is indeed a truth. Thx anyway for the
answer, I surely wouldn't found it soon by myself.


>> When I am working on Windows, I am using Emacs «shell» mode. The
>> problem is that it sometimes doesn't prints an applications output
>> (more precisely prints only after the app died).
>> The problem is present exclusively in Emacs, e.g. if I am run an
>> application in Windows® native terminal, it is works fine.
>>
>> I thought that the problem is in encoding, and set «shell»'s buffer
>> with «C-x <RET> f» to UTF-16, which, as I know, uses MS, but it didn't
>> fixed the problem.
>
>1- this is not an emacs problem.
>
>2- this is a basic programming question, that dates back since the dawn
   >of computing.
>
>3- what the fuck are you taught in school?
>
>
>> Below is a simple example of an application that won't print an output in Emacs:
>>
>> #include <cstdio>
>> #include <cstring>
>>
>> int main(){
>>     puts("Hello, enter something");
>>     char buf[256];
>>     memset(buf, 0, sizeof(buf));
>>     fgets(buf, sizeof(buf)-1, stdin);
>>     puts(buf);
>>     return 0;
>> }
>>
>> This works fine in both GNU/Linux, and the terminal of Windows®,
>
>By pure chance.
>
>> but not in Emacs shell mode on Windowsor.
>
>
>YOU FORGOT TO FLUSH YOUR BUFFERS!

Re: Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Eli Zaretskii]

> Date: Thu, 25 Sep 2014 15:25:48 +0700
> From: Yuri Khan <yuri.v.khan@gmail.com>
> Cc: "help-gnu-emacs@gnu.org" <help-gnu-emacs@gnu.org>
> 
> On Thu, Sep 25, 2014 at 2:43 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >> I thought that the problem is in encoding, and set «shell»'s buffer
> >> with «C-x <RET> f» to UTF-16, which, as I know, uses MS, but it didn't
> >> fixed the problem.
> >
> > Console applications on Windows use the current OEM codepage, not
> > UTF-16.
> 
> This is correct in general, although *some* Windows console
> applications can be persuaded to output in UTF-16. (The only example I
> know of is cmd.exe /U.)

cmd.exe is not a "normal" console application.  And AFAIK, the /U
switch only affects output to files and pipes, not to the console.

> Also, there is an unofficial unsupported ugly hack to use “chcp 65001”
> which makes applications think that the current OEM codepage is UTF-8.
> (It is unsupported because that there is some code that assumes at
> most two bytes for each character, while UTF-8 may produce up to
> four.)

First, UTF-8 is still not UTF-16.

And second, there's nothing unsupported about codepage 65001, the only
problem with switching the console to that codepage is that the fonts
available for it in the console cover only a very small portion of the
BMP, so you don't really get a Unicode-capable console.

Re: Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Yuri Khan]

On Thu, Sep 25, 2014 at 3:55 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> cmd.exe is not a "normal" console application.  And AFAIK, the /U
> switch only affects output to files and pipes, not to the console.

True.

> And second, there's nothing unsupported about codepage 65001,

OK, my knowledge of Windows console is stuck in the XP era. At that
time, once you switched to 65001, you couldn’t run any batch files or
.cmd scripts[1]. Apparently it was fixed for Windows 7.

[1]: http://consolesoft.com/p/cmd-xp-65001-fix/index.html

And there was this comment thread[2] in the blog of Michael Kaplan,
the guy who was Microsoft’s i18n expert, stating that the multibyte
string API functions are only supported for (up to double)-byte
encodings.

[2]: http://www.siao2.com/2006/07/04/656051.aspx#comment-7971

> the only
> problem with switching the console to that codepage is that the fonts
> available for it in the console cover only a very small portion of the
> BMP, so you don't really get a Unicode-capable console.

I was about to retort that setting a TrueType font with sufficient
coverage (finding which can be left as an excercise to the reader)
gets you at least Western, Cyrillic, Greek and CJK and possibly some
more, but what do I know, compared to a user of a right-to-left
script?

Re: Re: Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Eli Zaretskii]

> Date: Thu, 25 Sep 2014 17:18:22 +0700
> From: Yuri Khan <yuri.v.khan@gmail.com>
> Cc: "help-gnu-emacs@gnu.org" <help-gnu-emacs@gnu.org>
> 
> And there was this comment thread[2] in the blog of Michael Kaplan,
> the guy who was Microsoft’s i18n expert, stating that the multibyte
> string API functions are only supported for (up to double)-byte
> encodings.
> 
> [2]: http://www.siao2.com/2006/07/04/656051.aspx#comment-7971

That's true, but you can work around that with wide (a.k.a. "Unicode")
APIs, such as WriteConsoleW.  Not easy, but its doable; you can find
the recipes and code on the Internet.

> > the only
> > problem with switching the console to that codepage is that the fonts
> > available for it in the console cover only a very small portion of the
> > BMP, so you don't really get a Unicode-capable console.
> 
> I was about to retort that setting a TrueType font with sufficient
> coverage (finding which can be left as an excercise to the reader)
> gets you at least Western, Cyrillic, Greek and CJK and possibly some
> more, but what do I know, compared to a user of a right-to-left
> script?

I think Western, Greek, and Cyrillic are covered, but CJK aren't, see
http://www.microsoft.com/typography/fonts/font.aspx?FMID=1262.  So its
coverage is mostly limited to European scripts.

Re: Re: Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Yuri Khan]

On Thu, Sep 25, 2014 at 6:19 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>> [2]: http://www.siao2.com/2006/07/04/656051.aspx#comment-7971
>
> That's true, but you can work around that with wide (a.k.a. "Unicode")
> APIs, such as WriteConsoleW.  Not easy, but its doable; you can find
> the recipes and code on the Internet.

Sure but in that case chcp doesn’t affect you at all and is therefore
unnecessary.

> I think Western, Greek, and Cyrillic are covered, but CJK aren't, see
> http://www.microsoft.com/typography/fonts/font.aspx?FMID=1262.  So its
> coverage is mostly limited to European scripts.

Lucida Console is not the only font which is usable in the Windows
console. See, for example, MS Gothic:

http://www.microsoft.com/typography/fonts/font.aspx?FMID=2145

Re: Re: Re: Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Eli Zaretskii]

> Date: Thu, 25 Sep 2014 19:16:07 +0700
> From: Yuri Khan <yuri.v.khan@gmail.com>
> Cc: "help-gnu-emacs@gnu.org" <help-gnu-emacs@gnu.org>
> 
> On Thu, Sep 25, 2014 at 6:19 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >> [2]: http://www.siao2.com/2006/07/04/656051.aspx#comment-7971
> >
> > That's true, but you can work around that with wide (a.k.a. "Unicode")
> > APIs, such as WriteConsoleW.  Not easy, but its doable; you can find
> > the recipes and code on the Internet.
> 
> Sure but in that case chcp doesn’t affect you at all and is therefore
> unnecessary.

I don't think so.  Windows will always convert the stuff you write to
the console to the codepage that is in effect on that console, because
cmd.exe is not a Unicode app.  So if you want to mix characters from
different scripts, codepage 65001 is still required.

That said, I never tried these techniques myself, so I can easily be
wrong.

> > I think Western, Greek, and Cyrillic are covered, but CJK aren't, see
> > http://www.microsoft.com/typography/fonts/font.aspx?FMID=1262.  So its
> > coverage is mostly limited to European scripts.
> 
> Lucida Console is not the only font which is usable in the Windows
> console. See, for example, MS Gothic:
> 
> http://www.microsoft.com/typography/fonts/font.aspx?FMID=2145

I don't see this font in the list of fonts you can select in the
Property sheets of the cmd prompt on Windows XP and on Windows 7.  On
XP, I can only select Lucida Console; on Windows 7 I can also select
Consolas (whose coverage is slightly larger, and includes Thai, but
still does not include CJK.

Re: Re: Re: Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Yuri Khan]

On Thu, Sep 25, 2014 at 8:31 PM, Eli Zaretskii <eliz@gnu.org> wrote:

>> > That's true, but you can work around that with wide (a.k.a. "Unicode")
>> > APIs, such as WriteConsoleW.  Not easy, but its doable; you can find
>> > the recipes and code on the Internet.
>>
>> Sure but in that case chcp doesn’t affect you at all and is therefore
>> unnecessary.
>
> I don't think so.  Windows will always convert the stuff you write to
> the console to the codepage that is in effect on that console, because
> cmd.exe is not a Unicode app.  So if you want to mix characters from
> different scripts, codepage 65001 is still required.

No. The console uses Unicode internally, as do …W functions. If your
program does too, all is well.

However, many programs are written against …A functions or against the
C runtime library, which also uses …A functions. In this case, the
console converts Unicode user input into the OEM code page expected by
the application, and converts the application output from the OEM code
page into Unicode.

>> Lucida Console is not the only font which is usable in the Windows
>> console. See, for example, MS Gothic:
>>
>> http://www.microsoft.com/typography/fonts/font.aspx?FMID=2145
>
> I don't see this font in the list of fonts you can select in the
> Property sheets of the cmd prompt on Windows XP and on Windows 7.  On
> XP, I can only select Lucida Console; on Windows 7 I can also select
> Consolas (whose coverage is slightly larger, and includes Thai, but
> still does not include CJK.

http://blogs.msdn.com/b/oldnewthing/archive/2007/05/16/2659903.aspx
and the MSDN article linked from there. Before I fully migrated to
Linux, I used Andale Mono and later Liberation Mono in the Windows
console to great effect (just because I like tall and narrow rather
than fat and squarish console fonts, not because of coverage).

Re: On Windows(R) Emacs «shell» mode doesn't prints an output.

[Eli Zaretskii]

> Date: Thu, 25 Sep 2014 22:44:51 +0700
> From: Yuri Khan <yuri.v.khan@gmail.com>
> Cc: "help-gnu-emacs@gnu.org" <help-gnu-emacs@gnu.org>
> 
> >> Lucida Console is not the only font which is usable in the Windows
> >> console. See, for example, MS Gothic:
> >>
> >> http://www.microsoft.com/typography/fonts/font.aspx?FMID=2145
> >
> > I don't see this font in the list of fonts you can select in the
> > Property sheets of the cmd prompt on Windows XP and on Windows 7.  On
> > XP, I can only select Lucida Console; on Windows 7 I can also select
> > Consolas (whose coverage is slightly larger, and includes Thai, but
> > still does not include CJK.
> 
> http://blogs.msdn.com/b/oldnewthing/archive/2007/05/16/2659903.aspx
> and the MSDN article linked from there.

You cannot expect your users to do such things, many of them won't.