Help - Tcl/Expect Parsing Strangeness In Emacs Shell

Help - Tcl/Expect Parsing Strangeness In Emacs Shell

[Tennis Smith]

Hi,

Apologies for the wide distribution. This is in one of those grey areas
between tools so I'm not sure where it fits.

Basically, in certain circumstances, Tcl/Expect doesn't seem to work
correctly while in a tcsh/bash shell under Emacs.  Application output that
Expect is "expecting" will not be detected.

I ran into this while trying to parse the output of a telnet session to a MS
Windows machine.  Initially, I thought it was something wrong with the
Windows output (or with your humble programmer ;-)). During debugging, I've
winnowed out the various Windows and networking components . Now I can
recreate it at will on Solaris and Linux with no telnet/Windows involvement.

By contrast, everything works correctly on rxvt, xterm and Emacs version
20.7.1 and earlier.

The only environment that consistently has this occur is Emacs versions
21.2.95.2 and later. In addition, this only happens parsing strings with
"tabs" (0x09) frequently used.

How to observe/recreate:

Attached, is tar file with two tiny scripts that illustrate the problem.

After uncompress/untar, run the "test.exp" script. The "test.exp" script
spawns a task that will print a reproduction of a command output. When
everything is working correctly, you'll see two copies of the command
output, one from stdout and one printed from expect itself.

If run under an xterm, expect will be able to detect output and put it into
the "accum" variable (which is then printed). But, if run in a shell under
Emacs (tcsh or bash in my case), expect will not detect the data and will
put only blank lines into the "accum" variable.

Has anyone else seen this?

TIA,
-Tennis


begin 666 exptest.tar.Z
M'YV0+EZ4P0.'3IDY=%X`6,BPH<.'$"-*G$BQ8D48,&C N%&C!@",-&K Z/@1
M(\89,4J:7(D10 P8+V_<H"&CA@T;'V/$N!'#(PB/%H,*'4JTZ,,Z",/(`0$"
M@$$W;M+,,0JQC)LS.JEJW<JUJ]>O8+<&'%CP8$(R8>B$.6/5!9TQ;,+*W0H3
M(T>/($629(E2Y<H8,V[ <%E7)DV;.&%JK D`Q."YD",?3;JTZ=.H4[U:Q9I2
MLN?/H$-['A'BA9@T;E[,0:-@! @\L&/+GDV[MNW;N'/G5C"G#!T0E TNW0."
MQ)X8.E:(( %$1!\%3)FN3F.&#O3HTZM?9SJPS!@09+W_;F&F. P0+5H4/YY\
M>?/GSUWKGD^_OOW:"A3 J4-G#@@1:< QQAMNF)'&&5PHP(4;"UZ!&AEOW.&?
M#":!D 04( Q!H(%GU"%'6FD0F."""Q9!!QIER.&&;R"$@5992\4@@PT@..';
M'6_(L88.(S+H1@(:0B5>B&ZT, <<WE&7QG=$.#$%"%/488:!>##E`@@Z@-!C
M`A>"$ 099,AQT!Q7E@F"F6B>J::96<+@@IMP;AFE&"O^UD08<ZRQYIYI]HFE
M8V\&"L.61)1A1AAUL/';$6F5<4<8>? I:9I9]ECBB2G6V>*+PH&PDPPUWICC
MCI;^&.2*8]!!I)%(CJ$DDTY"*2655OZY99=?ACFFG[Q*FN6G+L200[ WX.""
M##3)60>=+-Z9YZ304@I"334<VY&UU0ZJX(^%'IKHHHT^&FFO?OYZ@PS!#KN3
ML3&(H(!H\,8K[[ST:C46008A]$*^=+A 5KVBU;717AGIA==*?;&$T4XTX/12
M3#/5=%-.,] D0V./`:PQ1$BI59E35F'VU699;6SRR2@+19IIJ*G&FGSWQ2SS
MS+#QQF)P*8) G''(*<><<]L!AP9UUD4G--%!=_==>*FB5QX)YZ6W7L_N`1W?
M:S1GK35N^8W0FH5VE $"'7*D40898P_M'[\@U.$&&3F_M5J+;X,@!IYH7"G$
M&R>"@$888:<M]AQAM%'&UV+.\:T+K>G'GW\B^"#YY)(+4<0123@!@A5%2#%%
M$D]HGKD13_!@^NFH)VBTEFY0,003FW?^>>@\NO$6&R"$+<<<1(*P!6IFO#$V
M7+KS3F 7) ;->A%80%'$$%3$[CGH3M2^A=+@$20>>G8@SZ#R"PH1Q!1(;)ZB
M\6Y8C_W=<Z=7/)'>JV[T@JZ3;_[N1-;N._9RHX%>"^\[7O)6MR#*3:X(3B""
M]&8G.B>0#G40A"")W. NWL A#'=P@]WPEA^R?*$W=1-!0-"B%K:XP2UP65 %
M>_.;,(QA#'5H@\Z>HP"F_68/^3%:"\3TGQ[B<'7889$+82C#Y>QAB#'L`PEL
M^(4W\ <%8IA5BE+@+B!RAR!?&)#M4%.'PQF-AD93E>&<>$/E18<Z.@O!%A!2
MMJN @ V;Z=L2M9>J)CXQBE.:8A?ZH#,S`I&%+7IA#)DB`C&>#01D!,&CTJ J
M-P9O*20,0Q6MR,<RL*$W?;3BZ@")Q"(:IY-]F* 14!,&W!F2C"KT(PB>\T4%
MT' __?E/)W5 @DZZ*V6XS*4N=\G+7OKRE\ ,IC"'2<QB&O.8R$RF,I?)S&8Z
M\YG0C*8TITG-:EKSFMC,IC:WR<UN>O.;X RG.,=)SG*:\YSH3*<ZU\G.=KKS
<G?",ISSG2<]ZVO.>^,RG/O?)SW[Z\Y\`#:@U`0``
`
end

Help - Tcl/Expect Parsing Strangeness In Emacs Shell

[Tennis Smith]

Hi,

Apologies for the wide distribution. This is in one of those grey areas
between tools so I'm not sure where it fits. I tried posting this once
before
with a "tar" file containing my test scripts, but  the mailer barfed back at
me.

The problem is that in certain circumstances, Tcl/Expect doesn't work
correctly
while in an Emacs shell.  Application output that Expect is "expecting"
will not be detected.

I ran into this while trying to parse the output of a telnet session to an
MS
Windows machine.  Initially, I thought it was something wrong with the
Windows output (or with your humble programmer ;-)). But after much
debugging, I've narrowed it down to two salient facts:

1 - Everything works correctly on rxvt, xterm and Emacs version
20.7.1 and earlier.

2 - The only environment that consistently has this occur is Emacs versions
21.2.95.2 and later.

Steps To Recreate:

Below are two tiny scripts.

 The "test.exp" script spawns a task. The "datagen.tcl" script is
called in the spawned task and simply writes to the screen. The "test.exp"
then reads output from "datagen.tcl" Run the "test.exp" script with the
"datagen.tcl" script in the same directory.

When everything is working _correctly_, you'll see two copies of  command
output from "datagen.tcl", one from stdout and one printed from  expect
itself.

If run under an xterm, expect will be able to detect output and put it into
the "accum" variable (which is then printed). But, if run in a shell under
Emacs, expect will not detect the data and will put only blank lines into
the "accum" variable.

Has anyone else seen this or know how to fix it?

TIA,
-Tennis

#!/bin/sh
#
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x
set starter { ${1:+"$@"}
    shift
    shift
    exec expect -f $0 -- ${1:+"$@"}
}
#
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x
#
# Ive tried this test under tcsh and bash. Both have the same
# result.
#
puts ">>>>>>>>BEGIN VERSION INFO<<<<<<<<<<\
     \nTCL VERSION:\ntcl version [info tclversion]\n\
     \nEXPECT VERSION:\n[exec expect -v]\n\
     \nBASH Version:\n[exec bash --version]\n\
     \nTCSH Version:\n [exec tcsh --version]\n\
     \n>>>>>>>END VERSION INFO<<<<<<<<<<<<<<\n\n"
spawn bash
exp_send "./datagen.tcl\n"
set accum {}
expect {
    -re " " {
        set accum "${accum}$expect_out(buffer)"
        exp_continue
    }
    timeout {
        if {![string length $expect_out(buffer)]} {
            set accum  "timed out waiting for data"
        } else {
            set accum "${accum}\n\nFinal timeout\n"
        }
    }
}
puts "accum:$accum"

# End of test.exp


#!/bin/sh
#
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x
set starter { ${1:+"$@"}
    shift
    shift
    exec expect -f $0 -- ${1:+"$@"}
}
#
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x
puts "ipconfig\
\n\nWindows 2000 IP Configuration\
\n\nEthernet adapter 126 Network:\
\n\n    Connection-specific DNS Suffix  . : \
\n      IP Address. . . . . . . . . . . . : 0.0.0.0\
\n      Subnet Mask . . . . . . . . . . . : 0.0.0.0\
\n      Default Gateway . . . . . . . . . : \
\n\nEthernet adapter 172 Network:\
\n\n    Connection-specific DNS Suffix  . : \
\n      IP Address. . . . . . . . . . . . : 172.19.178.242\
\n      Subnet Mask . . . . . . . . . . . : 255.255.255.0\
\n      Default Gateway . . . . . . . . . : 172.19.178.1"
# End of datagen.tcl

--
Remove "-remove-to-reply" to respond to my  email address directly.

Re: Help - Tcl/Expect Parsing Strangeness In Emacs Shell

[Don Libes]

Emacs plays around with tty settings in a way that can break
simplistic Expect scripts.  Have you read the Expect man page - it
talks about this and how to avoid it.  The Expect book has a section
that describes the same thing albeit in a different way.

Don

"Tennis Smith" <tennis_smith@yahoo-remove-to-reply.com> writes:

> Hi,
> 
> Apologies for the wide distribution. This is in one of those grey areas
> between tools so I'm not sure where it fits. I tried posting this once
> before
> with a "tar" file containing my test scripts, but  the mailer barfed back at
> me.
> 
> The problem is that in certain circumstances, Tcl/Expect doesn't work
> correctly
> while in an Emacs shell.  Application output that Expect is "expecting"
> will not be detected.
> 
> I ran into this while trying to parse the output of a telnet session to an
> MS
> Windows machine.  Initially, I thought it was something wrong with the
> Windows output (or with your humble programmer ;-)). But after much
> debugging, I've narrowed it down to two salient facts:
> 
> 1 - Everything works correctly on rxvt, xterm and Emacs version
> 20.7.1 and earlier.
> 
> 2 - The only environment that consistently has this occur is Emacs versions
> 21.2.95.2 and later.
> 
> Steps To Recreate:
> 
> Below are two tiny scripts.
> 
>  The "test.exp" script spawns a task. The "datagen.tcl" script is
> called in the spawned task and simply writes to the screen. The "test.exp"
> then reads output from "datagen.tcl" Run the "test.exp" script with the
> "datagen.tcl" script in the same directory.
> 
> When everything is working _correctly_, you'll see two copies of  command
> output from "datagen.tcl", one from stdout and one printed from  expect
> itself.
> 
> If run under an xterm, expect will be able to detect output and put it into
> the "accum" variable (which is then printed). But, if run in a shell under
> Emacs, expect will not detect the data and will put only blank lines into
> the "accum" variable.
> 
> Has anyone else seen this or know how to fix it?
> 
> TIA,
> -Tennis
> 
> #!/bin/sh
> #
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> x
> set starter { ${1:+"$@"}
>     shift
>     shift
>     exec expect -f $0 -- ${1:+"$@"}
> }
> #
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> x
> #
> # Ive tried this test under tcsh and bash. Both have the same
> # result.
> #
> puts ">>>>>>>>BEGIN VERSION INFO<<<<<<<<<<\
>      \nTCL VERSION:\ntcl version [info tclversion]\n\
>      \nEXPECT VERSION:\n[exec expect -v]\n\
>      \nBASH Version:\n[exec bash --version]\n\
>      \nTCSH Version:\n [exec tcsh --version]\n\
>      \n>>>>>>>END VERSION INFO<<<<<<<<<<<<<<\n\n"
> spawn bash
> exp_send "./datagen.tcl\n"
> set accum {}
> expect {
>     -re " " {
>         set accum "${accum}$expect_out(buffer)"
>         exp_continue
>     }
>     timeout {
>         if {![string length $expect_out(buffer)]} {
>             set accum  "timed out waiting for data"
>         } else {
>             set accum "${accum}\n\nFinal timeout\n"
>         }
>     }
> }
> puts "accum:$accum"
> 
> # End of test.exp
> 
> 
> #!/bin/sh
> #
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> x
> set starter { ${1:+"$@"}
>     shift
>     shift
>     exec expect -f $0 -- ${1:+"$@"}
> }
> #
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> x
> puts "ipconfig\
> \n\nWindows 2000 IP Configuration\
> \n\nEthernet adapter 126 Network:\
> \n\n    Connection-specific DNS Suffix  . : \
> \n      IP Address. . . . . . . . . . . . : 0.0.0.0\
> \n      Subnet Mask . . . . . . . . . . . : 0.0.0.0\
> \n      Default Gateway . . . . . . . . . : \
> \n\nEthernet adapter 172 Network:\
> \n\n    Connection-specific DNS Suffix  . : \
> \n      IP Address. . . . . . . . . . . . : 172.19.178.242\
> \n      Subnet Mask . . . . . . . . . . . : 255.255.255.0\
> \n      Default Gateway . . . . . . . . . : 172.19.178.1"
> # End of datagen.tcl
> 
> --
> Remove "-remove-to-reply" to respond to my  email address directly.

Re: Help - Tcl/Expect Parsing Strangeness In Emacs Shell

[Tennis Smith]

Hi Don,

Thanks for the reply.   The man page pointer helps.

I'm thinking now that shell handling in Emacs is the culprit.  As I said
below, this used to work fine in Emacs 20. It doesn't work in Emacs 21.

Emacs has lost functionality that used to be there.  I don't think its
Expect's problem.

-Tennis

"Don Libes" <libes@nist.gov> wrote in message
news:s6aisbhu96c.fsf@peace.mel.nist.gov...
> Emacs plays around with tty settings in a way that can break
> simplistic Expect scripts.  Have you read the Expect man page - it
> talks about this and how to avoid it.  The Expect book has a section
> that describes the same thing albeit in a different way.
>
> Don
>
> "Tennis Smith" <tennis_smith@yahoo-remove-to-reply.com> writes:
>
> > Hi,
> >
> > Apologies for the wide distribution. This is in one of those grey areas
> > between tools so I'm not sure where it fits. I tried posting this once
> > before
> > with a "tar" file containing my test scripts, but  the mailer barfed
back at
> > me.
> >
> > The problem is that in certain circumstances, Tcl/Expect doesn't work
> > correctly
> > while in an Emacs shell.  Application output that Expect is "expecting"
> > will not be detected.
> >
> > I ran into this while trying to parse the output of a telnet session to
an
> > MS
> > Windows machine.  Initially, I thought it was something wrong with the
> > Windows output (or with your humble programmer ;-)). But after much
> > debugging, I've narrowed it down to two salient facts:
> >
> > 1 - Everything works correctly on rxvt, xterm and Emacs version
> > 20.7.1 and earlier.
> >
> > 2 - The only environment that consistently has this occur is Emacs
versions
> > 21.2.95.2 and later.
> >
> > Steps To Recreate:
> >
> > Below are two tiny scripts.
> >
> >  The "test.exp" script spawns a task. The "datagen.tcl" script is
> > called in the spawned task and simply writes to the screen. The
"test.exp"
> > then reads output from "datagen.tcl" Run the "test.exp" script with the
> > "datagen.tcl" script in the same directory.
> >
> > When everything is working _correctly_, you'll see two copies of
command
> > output from "datagen.tcl", one from stdout and one printed from  expect
> > itself.
> >
> > If run under an xterm, expect will be able to detect output and put it
into
> > the "accum" variable (which is then printed). But, if run in a shell
under
> > Emacs, expect will not detect the data and will put only blank lines
into
> > the "accum" variable.
> >
> > Has anyone else seen this or know how to fix it?
> >
> > TIA,
> > -Tennis
> >
> > #!/bin/sh
> > #
> >
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> > x
> > set starter { ${1:+"$@"}
> >     shift
> >     shift
> >     exec expect -f $0 -- ${1:+"$@"}
> > }
> > #
> >
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> > x
> > #
> > # Ive tried this test under tcsh and bash. Both have the same
> > # result.
> > #
> > puts ">>>>>>>>BEGIN VERSION INFO<<<<<<<<<<\
> >      \nTCL VERSION:\ntcl version [info tclversion]\n\
> >      \nEXPECT VERSION:\n[exec expect -v]\n\
> >      \nBASH Version:\n[exec bash --version]\n\
> >      \nTCSH Version:\n [exec tcsh --version]\n\
> >      \n>>>>>>>END VERSION INFO<<<<<<<<<<<<<<\n\n"
> > spawn bash
> > exp_send "./datagen.tcl\n"
> > set accum {}
> > expect {
> >     -re " " {
> >         set accum "${accum}$expect_out(buffer)"
> >         exp_continue
> >     }
> >     timeout {
> >         if {![string length $expect_out(buffer)]} {
> >             set accum  "timed out waiting for data"
> >         } else {
> >             set accum "${accum}\n\nFinal timeout\n"
> >         }
> >     }
> > }
> > puts "accum:$accum"
> >
> > # End of test.exp
> >
> >
> > #!/bin/sh
> > #
> >
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> > x
> > set starter { ${1:+"$@"}
> >     shift
> >     shift
> >     exec expect -f $0 -- ${1:+"$@"}
> > }
> > #
> >
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> > x
> > puts "ipconfig\
> > \n\nWindows 2000 IP Configuration\
> > \n\nEthernet adapter 126 Network:\
> > \n\n    Connection-specific DNS Suffix  . : \
> > \n      IP Address. . . . . . . . . . . . : 0.0.0.0\
> > \n      Subnet Mask . . . . . . . . . . . : 0.0.0.0\
> > \n      Default Gateway . . . . . . . . . : \
> > \n\nEthernet adapter 172 Network:\
> > \n\n    Connection-specific DNS Suffix  . : \
> > \n      IP Address. . . . . . . . . . . . : 172.19.178.242\
> > \n      Subnet Mask . . . . . . . . . . . : 255.255.255.0\
> > \n      Default Gateway . . . . . . . . . : 172.19.178.1"
> > # End of datagen.tcl
> >
> > --
> > Remove "-remove-to-reply" to respond to my  email address directly.