Latin1 language environment breaks Cygwin shell buffer

Latin1 language environment breaks Cygwin shell buffer

[Eric Hanchrow]

In GNU Emacs 21.2.1 (i386-mingw-nt5.0.2195)
 of 2002-03-22 on ALPHA
configured using `configure --with-gcc (2.95)'
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  locale-coding-system: iso-latin-1
  default-enable-multibyte-characters: t

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

On a Cygwin system (excruciatingly painful details below), type
`runemacs' at the shell, and then evaluate these four forms in order
(adjust the name of the "bash" program as needed for your system):

(set-language-environment "Latin-1")

(setq shell-file-name		"c:/cygwin/bin/bash")
(setq explicit-shell-file-name	"c:/cygwin/bin/bash")

(shell)

You will, not surprisingly, see a "*shell*" buffer appear.  Now switch
to that buffer with C-x o, and hit the Enter key -- you'll see 

	$ 
	: command not found

(You'll also see a lot of cruft from escape sequences in the Bash
prompt; you may ignore those)

Now do (set-language-environment "English"), kill the shell buffer, do
M-x shell, and again hit enter -- this time you won't see the `command
not found' message.

I expected my choice of language environment to have no effect on the
functioning of the shell.

I think that the shell is "seeing" a carriage-return at the end of the
input line, and complaining about it; I think we see nothing before
the colon in the error message because the carriage-return has caused
Emacs to move the cursor to the beginning of the line and erase to the
end of the line.

Here's an experiment that I think confirms my carriage-return theory:

	$ od -c ; echo
	hey you
	buz
	0000000   h   e   y       y   o   u  \r  \n   b   u   z  \r  \n
	0000016
	: command not found

I typed "hey you" and "buz" on separate lines, and the `od' program
indeed "saw" carriage-returns at the ends of those lines.
\f
output of `cygcheck -sv':

Cygwin Win95/NT Configuration Diagnostics
Current System Time: Sat Mar 30 08:16:31 2002

Windows 2000 Professional Ver 5.0 Build 2195 Service Pack 2

Path:	C:\cygwin\usr\local\bin
	C:\cygwin\bin
	C:\cygwin\bin
	c:\WINNT\system32
	c:\WINNT
	c:\WINNT\System32\Wbem
	C:\cygwin\usr\X11R6\bin
	C:\cygwin\usr\games

SysDir: C:\WINNT\System32
WinDir: C:\WINNT

HOME = `C:\cygwin\home\Administrator'
MAKE_MODE = `unix'
PWD = `/usr/local'
USER = `Administrator'

ALLUSERSPROFILE = `C:\Documents and Settings\All Users'
APPDATA = `C:\Documents and Settings\administrator\Application Data'
COMMONPROGRAMFILES = `C:\Program Files\Common Files'
COMPUTERNAME = `RUSTY'
COMSPEC = `C:\WINNT\system32\cmd.exe'
CVS_RSH = `ssh'
HOMEDRIVE = `C:'
HOMEPATH = `\'
LOGONSERVER = `\\RUSTY'
MANPATH = `:/usr/ssl/man'
NUMBER_OF_PROCESSORS = `1'
OLDPWD = `/home/Administrator'
OS2LIBPATH = `C:\WINNT\system32\os2\dll;'
OS = `Windows_NT'
PATHEXT = `.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH'
PROCESSOR_ARCHITECTURE = `x86'
PROCESSOR_IDENTIFIER = `x86 Family 6 Model 6 Stepping 5, GenuineIntel'
PROCESSOR_LEVEL = `6'
PROCESSOR_REVISION = `0605'
PROGRAMFILES = `C:\Program Files'
PROMPT = `$P$G'
PS1 = `\t [\u@\h \W]\$ '
SHLVL = `1'
SYSTEMDRIVE = `C:'
SYSTEMROOT = `C:\WINNT'
TEMP = `c:\DOCUME~1\ADMINI~1\LOCALS~1\Temp'
TERM = `cygwin'
TMP = `c:\DOCUME~1\ADMINI~1\LOCALS~1\Temp'
USERDOMAIN = `RUSTY'
USERNAME = `administrator'
USERPROFILE = `C:\Documents and Settings\administrator'
WINDIR = `C:\WINNT'
_ = `/usr/bin/cygcheck.exe'

Use `-r' to scan registry

c:  hd  NTFS    4104Mb  32% CP CS UN PA FC     
d:  cd           N/A    N/A                    
e:  net NTFS    5793Mb  62% CP CS    PA        offby1

.              /cygdrive  user    binmode,noumount
C:/cygwin      /          system  binmode
C:/cygwin/bin  /usr/bin   system  binmode
C:/cygwin/lib  /usr/lib   system  binmode

Found: C:\cygwin\bin\bash.exe
Found: C:\cygwin\bin\cat.exe
Not Found: cpp (good!)
Found: C:\cygwin\bin\find.exe
Not Found: gcc
Not Found: gdb
Not Found: ld
Found: C:\cygwin\bin\ls.exe
Not Found: make
Found: C:\cygwin\bin\sh.exe

   56k 2000/12/03 C:\cygwin\bin\cygbz21.0.dll - os=4.0 img=1.0 sys=4.0
                  "cygbz21.0.dll" v0.0 ts=2000/11/20 15:53
  621k 2002/01/16 C:\cygwin\bin\cygcrypto.dll - os=4.0 img=1.0 sys=4.0
                  "cygcrypto.dll" v0.0 ts=2002/1/16 1:54
   45k 2001/04/25 C:\cygwin\bin\cygform5.dll - os=4.0 img=1.0 sys=4.0
                  "cygform5.dll" v0.0 ts=2001/4/24 22:28
   35k 2002/01/09 C:\cygwin\bin\cygform6.dll - os=4.0 img=1.0 sys=4.0
                  "cygform6.dll" v0.0 ts=2002/1/8 22:03
   19k 2002/02/20 C:\cygwin\bin\cyggdbm.dll - os=4.0 img=1.0 sys=4.0
                  "cyggdbm.dll" v0.0 ts=2002/2/19 19:05
   17k 2001/06/28 C:\cygwin\bin\cyghistory4.dll - os=4.0 img=1.0 sys=4.0
                  "cyghistory4.dll" v0.0 ts=2001/1/6 20:34
   20k 2002/01/13 C:\cygwin\bin\cyghistory5.dll - os=4.0 img=1.0 sys=4.0
                  "cyghistory5.dll" v0.0 ts=2002/1/12 17:27
   22k 2001/12/13 C:\cygwin\bin\cygintl-1.dll - os=4.0 img=1.0 sys=4.0
                  "cygintl-1.dll" v0.0 ts=2001/12/13 1:28
   26k 2001/04/25 C:\cygwin\bin\cygmenu5.dll - os=4.0 img=1.0 sys=4.0
                  "cygmenu5.dll" v0.0 ts=2001/4/24 22:27
   20k 2002/01/09 C:\cygwin\bin\cygmenu6.dll - os=4.0 img=1.0 sys=4.0
                  "cygmenu6.dll" v0.0 ts=2002/1/8 22:03
  156k 2001/04/25 C:\cygwin\bin\cygncurses++5.dll - os=4.0 img=1.0 sys=4.0
                  "cygncurses++5.dll" v0.0 ts=2001/4/24 22:29
  175k 2002/01/09 C:\cygwin\bin\cygncurses++6.dll - os=4.0 img=1.0 sys=4.0
                  "cygncurses++6.dll" v0.0 ts=2002/1/8 22:03
  226k 2001/04/25 C:\cygwin\bin\cygncurses5.dll - os=4.0 img=1.0 sys=4.0
                  "cygncurses5.dll" v0.0 ts=2001/4/24 22:17
  202k 2002/01/09 C:\cygwin\bin\cygncurses6.dll - os=4.0 img=1.0 sys=4.0
                  "cygncurses6.dll" v0.0 ts=2002/1/8 22:03
   15k 2001/04/25 C:\cygwin\bin\cygpanel5.dll - os=4.0 img=1.0 sys=4.0
                  "cygpanel5.dll" v0.0 ts=2001/4/24 22:27
   12k 2002/01/09 C:\cygwin\bin\cygpanel6.dll - os=4.0 img=1.0 sys=4.0
                  "cygpanel6.dll" v0.0 ts=2002/1/8 22:03
   40k 2001/11/21 C:\cygwin\bin\cygpcre.dll - os=4.0 img=1.0 sys=4.0
                  "cygpcre.dll" v0.0 ts=2001/11/21 14:15
   39k 2001/11/21 C:\cygwin\bin\cygpcreposix.dll - os=4.0 img=1.0 sys=4.0
                  "cygpcreposix.dll" v0.0 ts=2001/11/21 14:15
  108k 2001/06/28 C:\cygwin\bin\cygreadline4.dll - os=4.0 img=1.0 sys=4.0
                  "cygreadline4.dll" v0.0 ts=2001/1/6 20:34
  121k 2002/01/13 C:\cygwin\bin\cygreadline5.dll - os=4.0 img=1.0 sys=4.0
                  "cygreadline5.dll" v0.0 ts=2002/1/12 17:27
  156k 2002/01/16 C:\cygwin\bin\cygssl.dll - os=4.0 img=1.0 sys=4.0
                  "cygssl.dll" v0.0 ts=2002/1/16 1:54
   50k 2002/03/12 C:\cygwin\bin\cygz.dll - os=4.0 img=1.0 sys=4.0
                  "cygz.dll" v0.0 ts=2002/3/11 20:38
  751k 2002/02/25 C:\cygwin\bin\cygwin1.dll - os=4.0 img=1.0 sys=4.0
                  "cygwin1.dll" v0.0 ts=2002/2/25 8:14
    Cygwin DLL version info:
        DLL version: 1.3.10
        DLL epoch: 19
        DLL bad signal mask: 19005
        DLL old termios: 5
        DLL malloc env: 28
        API major: 0
        API minor: 51
        Shared data: 3
        DLL identifier: cygwin1
        Mount registry: 2
        Cygnus registry name: Cygnus Solutions
        Cygwin registry name: Cygwin
        Program options name: Program Options
        Cygwin mount registry name: mounts v2
        Cygdrive flags: cygdrive flags
        Cygdrive prefix: cygdrive prefix
        Cygdrive default prefix: 
        Build date: Mon Feb 25 11:14:34 EST 2002
        Shared id: cygwin1S3


Cygwin Package Information
Last downloaded files to: C:\Documents and Settings\administrator\Desktop
Last downloaded files from: ftp://mirrors.rcn.net/mirrors/sources.redhat.com/cygwin

Package             Version             
ash                 20020131-1          
bash                2.05a-3             
bzip2               1.0.1-6             
crypt               1.0-1               
cvs                 1.11.0-1            
cygwin              1.3.10-1            
diff                0.0                 
fileutils           4.1-1               
findutils           4.1                 
gawk                3.0.4-1             
gdbm                1.8.0-4             
grep                2.5g                
gzip                1.3.2-1             
libintl1            0.10.40-1           
libncurses5         5.2-1               
libncurses6         5.2-8               
libreadline4        4.1-2               
libreadline5        4.2a-1              
login               1.4-3               
ncurses             5.2-8               
openssh             3.1p1-1             
openssl             0.9.6c-3            
pcre                3.7-1               
readline            4.2a-1              
sed                 3.02-1              
sh-utils            2.0-2               
tar                 1.13.19-1           
termcap             20010825-1          
terminfo            5.2-1               
textutils           2.0.21-1            
unzip               5.41-1              
which               1.5-1               
zlib                1.1.4-1             

Use -h to see help about each section

-- 
PGP Fingerprint: 3E7B A3F3 96CA 8958 ACC5  C8BD 6337 0041 C01C 5276

Re: Latin1 language environment breaks Cygwin shell buffer

[Eli Zaretskii]

> From: Eric Hanchrow <offby1@blarg.net>
> Date: 30 Mar 2002 08:32:39 -0800
> 
> On a Cygwin system (excruciatingly painful details below), type
> `runemacs' at the shell, and then evaluate these four forms in order
> (adjust the name of the "bash" program as needed for your system):

I don't have Cygwin installed, so I can only speculate about this.
Therefore, please take what's below with a grain of salt.

> (set-language-environment "Latin-1")
> 
> (setq shell-file-name		"c:/cygwin/bin/bash")
> (setq explicit-shell-file-name	"c:/cygwin/bin/bash")

You should probably use an explicit .exe extension in setting these
variables.  (Actually, I don't recommend setting these variables at
all in Emacs, but instead set up PATH and SHELL variables _outside_
Emacs so that the right shell is found.  But this is a different
issue, probably unrelated to your problems.)

> (shell)
> 
> You will, not surprisingly, see a "*shell*" buffer appear.  Now switch
> to that buffer with C-x o, and hit the Enter key -- you'll see 
> 
> 	$ 
> 	: command not found
> 
> (You'll also see a lot of cruft from escape sequences in the Bash
> prompt; you may ignore those)

I rather think that this ``cruft'' should not be ignored, but instead
closely scrutinized.  I suspect that, in a Latin-1 environment, Emacs
finds in that cruft something that utterly confuses the shell mode.

How about if you disable all the prompt rigamarole in your Bash, and
see if that helps?

Also, what about your settings of process-coding-system?  What is it
in both language environments you used, and does it help to use
set-buffer-process-coding-system in the shell buffer so that Emacs
decodes Bash output with undecided-dos?

> I expected my choice of language environment to have no effect on the
> functioning of the shell.

The language environment sets the defaults for decoding subprocess
I/O, so there are valid reasons you should assume it does in fact have
effect on the interaction between Bash and the shell buffer.

> I think that the shell is "seeing" a carriage-return at the end of the
> input line, and complaining about it

It is (or, rather, should be) normal for the Cygwin Bash to see a
CR-LF pair at the end of each line.  It shouldn't complain about it,
but should rather DTRT: remove the CR and act on the rest as a normal
text line.

You might wish to experiment with mounting your filesystems in binary
(instead of text) mode.

Re: Latin1 language environment breaks Cygwin shell buffer

[Eli Zaretskii]

[Please keep the mailing list on the CC.]

On 30 Mar 2002, Eric Hanchrow wrote:

> 
>     > (setq shell-file-name		"c:/cygwin/bin/bash")
>     > (setq explicit-shell-file-name	"c:/cygwin/bin/bash")
> 
>     ... Actually, I don't recommend setting these variables at all in
>     Emacs, but instead set up PATH and SHELL variables _outside_ Emacs
>     so that the right shell is found.
> 
> I wanted to do that, since it seems cleaner than setting Emacs
> variables, but the `runemacs' program appears to unconditionally set
> the SHELL variable to cmdproxy.exe.  (And I prefer `runemacs' to plain
> `emacs' since the former doesn't open a console window, which is
> annoying.)

AFAIK, `emacs' does not open a console window for quite some time.  When 
did you last try that, and how did you invoke Emacs when you did?

>     Also, what about your settings of process-coding-system?  What is
>     it in both language environments you used
> 
> Well, I didn't set it explicitly, as you can see, and I don't know how
> to find out what coding system is in effect.  Can you tell me which
> command to type to find out?

I believe the doc string of the `shell' command explains this.

>     does it help to use set-buffer-process-coding-system in the shell
>     buffer so that Emacs decodes Bash output with undecided-dos?
> 
> No, that doesn't help.

Then it's probably not something related to CR characters.

>     You might wish to experiment with mounting your filesystems in binary
>     (instead of text) mode.
> 
> I'm afraid they already are mounted in binary mode.

Then how come `od' shows \r characters in the output of `echo'?  
Something strange is going on here, and you probably need a Cygwin guru 
to help you out.  I suggest to ask on the Cygwin mailing list.

Re: Latin1 language environment breaks Cygwin shell buffer

[Jason Rumney]

> > I prefer `runemacs' to plain `emacs' since the former doesn't open
> > a console window, which is annoying.)

> AFAIK, `emacs' does not open a console window for quite some time.  When 
> did you last try that, and how did you invoke Emacs when you did?

emacs.exe run directly from the graphical interface will open a
console window on MS Windows. runemacs.exe exists solely to get around
this limitation (these days; it used to perform more environment
initialisation, but that got moved into emacs.exe quite some time ago).


-- 
Jason Rumney

Re: Latin1 language environment breaks Cygwin shell buffer

[Harald.Maier.BW]

>>>>> "Eric" == Eric Hanchrow <offby1@blarg.net> writes:

    Eric> (set-language-environment "Latin-1")

    Eric> (setq shell-file-name "c:/cygwin/bin/bash") (setq
    Eric> explicit-shell-file-name "c:/cygwin/bin/bash")

    Eric> (shell)

    Eric> You will, not surprisingly, see a "*shell*" buffer appear.
    Eric> Now switch to that buffer with C-x o, and hit the Enter key
    Eric> -- you'll see

    Eric> 	$ : command not found

I had a similar problem on a German system. My workaround was to set
the LANG variable in the control panel / system dialog to "C". Then
all worked fine.

Harald

Re: Latin1 language environment breaks Cygwin shell buffer

[Harald.Maier.BW]

>>>>> "Eric" == Eric Hanchrow <offby1@blarg.net> writes:
>>>>> "Harald" == Harald Maier BW <Harald.Maier.BW@t-online.de> writes:
      Eric> $ : command not found

    Harald> ... set the LANG variable in the control panel / system
    Harald> dialog to "C".

  Eric> Interesting; it works for me too.  How did you think of that
  Eric> workaround?

First, it's an easy workaround. Second, I am not sure how much people
have problem with this issue. But as far as I know I have the problem
always with German Windows systems so other people may have problems
too. If I interpret your cygcheck correctly then you are using a
English version, so the problem seems to be a more common problem. I
reported this for some time but Eli did not believe my findings.
Personally I would add a entry to the etc/PROBLEMS file or to the
Emacs FAQ for Windows about that issue.

Harald

Re: Latin1 language environment breaks Cygwin shell buffer

[Jason Rumney]

Harald.Maier.BW@t-online.de writes:

> >>>>> "Eric" == Eric Hanchrow <offby1@blarg.net> writes:
> >>>>> "Harald" == Harald Maier BW <Harald.Maier.BW@t-online.de> writes:
>       Eric> $ : command not found
> 
>     Harald> ... set the LANG variable in the control panel / system
>     Harald> dialog to "C".

Interesting, I'm confused as to why (set-language-environment "Latin-1") 
produces different results for default-process-coding-system
depending on whether the previous language environment was "Default"
or "German".

Another workaround could be to remove the (set-language-environment "Latin-1")
from your .emacs, and just change whatever it is you don't like about
the "German" language environment (they don't differ by much).

I see why this happens now.  The Windows specific startup code tries
to DTRT for default-process-coding-system, depending on which shell
you are using.  But set-language-environment overrides this. I think
an exception needs to be made within set-language-environment to not
mess with default-process-coding-system on Windows, as the problem of
different shells is too complex to deal with there. Alternatively, we
could use the same logic as at startup, but if that did not DTRT and
the user overrode it, it would only cause the same problems we are
seeing now.


-- 
Jason Rumney

Re: Latin1 language environment breaks Cygwin shell buffer

[Eli Zaretskii]

> From: Harald.Maier.BW@t-online.de
> Date: 06 Apr 2002 13:25:54 +0100
> 
> I reported this for some time but Eli did not believe my findings.

Did I actually say I didn't believe you?  IIRC, this issue wasn't
solved because no one, including Jason, could reproduce the problem on
their machine.  We need to reproduce the problem to be able to debug
it, or depend on you to debug it on your machine.  Surely, you
understand this, right?

Re: Latin1 language environment breaks Cygwin shell buffer

[Eli Zaretskii]

> From: Jason Rumney <jasonr@gnu.org>
> Date: 06 Apr 2002 17:35:00 +0100
> 
> I see why this happens now.  The Windows specific startup code tries
> to DTRT for default-process-coding-system, depending on which shell
> you are using.  But set-language-environment overrides this. I think
> an exception needs to be made within set-language-environment to not
> mess with default-process-coding-system on Windows, as the problem of
> different shells is too complex to deal with there. Alternatively, we
> could use the same logic as at startup, but if that did not DTRT and
> the user overrode it, it would only cause the same problems we are
> seeing now.

I suspect that the problem is with the EOL conversion in process I/O:
where w32-fns.el carefully sets up the EOL conversions as apropriate
for both input and output, the language environment leaves the EOL
conversion undecided.  Perhaps set-language-environment should inherit
the EOL conversion from the previous setting, at least on Windows?

Re: Latin1 language environment breaks Cygwin shell buffer

[Jason Rumney]

"Eli Zaretskii" <eliz@is.elta.co.il> writes:

> I suspect that the problem is with the EOL conversion in process I/O:
> where w32-fns.el carefully sets up the EOL conversions as apropriate
> for both input and output, the language environment leaves the EOL
> conversion undecided.

It only leaves the EOL conversion undecided when moving from
"Default" to "Latin-1", which is the case that works. In the case
of moving from "German" to "Latin-1", it sets both to iso-latin-1-dos.
I can't see why the two cases should be any different.

> Perhaps set-language-environment should inherit
> the EOL conversion from the previous setting, at least on Windows?

That might work, but since I can't see where the "-dos" is coming from
in the first place, I don't know where to make this change.

-- 
Jason Rumney

Re: Latin1 language environment breaks Cygwin shell buffer

[Eli Zaretskii]

> From: Jason Rumney <jasonr@gnu.org>
> Date: 06 Apr 2002 18:43:03 +0100
> 
> "Eli Zaretskii" <eliz@is.elta.co.il> writes:
> 
> > I suspect that the problem is with the EOL conversion in process I/O:
> > where w32-fns.el carefully sets up the EOL conversions as apropriate
> > for both input and output, the language environment leaves the EOL
> > conversion undecided.
> 
> It only leaves the EOL conversion undecided when moving from
> "Default" to "Latin-1", which is the case that works. In the case
> of moving from "German" to "Latin-1", it sets both to iso-latin-1-dos.

I see in w32-fns that it sometimes sets the encode part of
default-process-coding-system to *-unix.  Wasn't the problem in the
OP's case the DOS encoding of command lines sent to the shell?

Re: Latin1 language environment breaks Cygwin shell buffer

[Jason Rumney]

"Eli Zaretskii" <eliz@is.elta.co.il> writes:

> > From: Jason Rumney <jasonr@gnu.org>
> > Date: 06 Apr 2002 18:43:03 +0100
> > 
> > "Eli Zaretskii" <eliz@is.elta.co.il> writes:
> > 
> > > I suspect that the problem is with the EOL conversion in process I/O:
> > > where w32-fns.el carefully sets up the EOL conversions as apropriate
> > > for both input and output, the language environment leaves the EOL
> > > conversion undecided.
> > 
> > It only leaves the EOL conversion undecided when moving from
> > "Default" to "Latin-1", which is the case that works. In the case
> > of moving from "German" to "Latin-1", it sets both to iso-latin-1-dos.
> 
> I see in w32-fns that it sometimes sets the encode part of
> default-process-coding-system to *-unix.  Wasn't the problem in the
> OP's case the DOS encoding of command lines sent to the shell?

Yes, that is the code that sets it initially at startup. But my own
testing shows that leaving the encode eol as undecided actually works
(I'm not sure why, I'd expect it default to -dos on Windows), and the
problemetic case is caused by something, somewhere, explicitly
setting it to -dos.

-- 
Jason Rumney

Re: Latin1 language environment breaks Cygwin shell buffer

[Eric Hanchrow]

I haven't understood most of this thread, but that's OK; I'd be happy
to test changes for you if you'd like.
 
-- 
PGP Fingerprint: 3E7B A3F3 96CA 8958 ACC5  C8BD 6337 0041 C01C 5276