bug#28500: 25.3; shell mode starts echoing command

bug#28500: 25.3; shell mode starts echoing command

[Barry A. Warsaw]

* M-x shell RET
* Go to the shell and type `ls RET`
* See the output of the `ls` but *not* the actual `ls` command
* Enter an interpreter; two choices which reproduce the problem include `sqlite3` and `python3`
  Type `sqlite3` at the shell prompt
* Type .exit at the sqlite3 prompt (notice that the `.exit` was echo'd
  before the shell prompt came back; the bug is now active
* Type `ls RET` at the shell prompt.  Notice that the `ls RET` was
  echoed before the ls output was printed.

Note that using `python3` or even `python` at the prompt will also
exhibit the bug.  (To exit Python, just type ctrl-D).  The only way I've
found to "fix" the problem is to ctrl-D to exit the shell, then `M-x
shell RET` to restart it in the same buffer.



In GNU Emacs 25.3.1 (x86_64-apple-darwin16.7.0, NS appkit-1504.83 Version 10.12.6 (Build 16G29))
 of 2017-09-14 built on resist.local
Windowing system distributor 'Apple', version 10.3.1504
Configured using:
 'configure --disable-dependency-tracking --disable-silent-rules
 --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs/25.3/share/info/emacs
 --prefix=/usr/local/Cellar/emacs/25.3 --without-x --with-xml2
 --without-dbus --without-gnutls --without-imagemagick --with-ns
 --disable-ns-self-contained'

Configured features:
JPEG NOTIFY ACL LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Shell

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
The mark is not set now, so there is no region
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode easymenu
cl-loaddefs pcase cl-lib mail-prsvr mail-utils shell pcomplete comint
ansi-color ring time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel ns-win ucs-normalize
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
kqueue cocoa ns multi-tty make-network-process emacs)

Memory information:
((conses 16 200070 5259)
 (symbols 48 19931 0)
 (miscs 40 59 194)
 (strings 32 16829 5243)
 (string-bytes 1 489691)
 (vectors 16 33562)
 (vector-slots 8 655967 4691)
 (floats 8 158 281)
 (intervals 56 252 16)
 (buffers 976 20))

bug#28500: 25.3; shell mode starts echoing command

[Glenn Morris]

Barry A. Warsaw wrote:

> * M-x shell RET
> * Go to the shell and type `ls RET`
> * See the output of the `ls` but *not* the actual `ls` command
> * Enter an interpreter; two choices which reproduce the problem include `sqlite3` and `python3`
>   Type `sqlite3` at the shell prompt
> * Type .exit at the sqlite3 prompt (notice that the `.exit` was echo'd
>   before the shell prompt came back; the bug is now active
> * Type `ls RET` at the shell prompt.  Notice that the `ls RET` was
>   echoed before the ls output was printed.

FWIW, I cannot reproduce this on RHEL7.

bug#28500: 25.3; shell mode starts echoing command

[Nick Helm]

barry@python.org (Barry A. Warsaw) writes:

> * M-x shell RET
> * Go to the shell and type `ls RET`
> * See the output of the `ls` but *not* the actual `ls` command
> * Enter an interpreter; two choices which reproduce the problem include `sqlite3` and `python3`
>   Type `sqlite3` at the shell prompt
> * Type .exit at the sqlite3 prompt (notice that the `.exit` was echo'd
>   before the shell prompt came back; the bug is now active
> * Type `ls RET` at the shell prompt.  Notice that the `ls RET` was
>   echoed before the ls output was printed.

I can reproduce this with Emacs 25.3 on macOS 10.12.6.

bug#28500: 25.3; shell mode starts echoing command

[Mike Kupfer]

I see the same issue after using openssl to encrypt or decrypt a file.
Just never got around to filing a bug report...

-----8<-----8<-----
alto$ openssl aes-256-cbc -in foo.txt -out foo.text.enc
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
alto$ ls
ls
deb9-upgrade.log  foo.text.enc	ieee-cs-election-receipt.pdf
enchant.dummy*	  foo.txt	net-neutrality-comments.pdf
alto$ echo $SHELL
echo $SHELL
/bin/bash
alto$ 
----->8----->8-----

I see this on Debian 8 and on Solaris, with various versions of Emacs.

mike

bug#28500: 25.3; shell mode starts echoing command

[Barry Warsaw]

[-- Attachment #1: Type: text/plain, Size: 1251 bytes --]

On Sep 18, 2017, at 21:05, Mike Kupfer <mkupfer@alum.berkeley.edu> wrote:
> 
> I see the same issue after using openssl to encrypt or decrypt a file.
> Just never got around to filing a bug report…

Interesting.  I’ve only noticed it in 25.2 and 25.3 on macOS (installed from brew).  I can’t reproduce it in 25.2 on Ubuntu 17.10.

> alto$ openssl aes-256-cbc -in foo.txt -out foo.text.enc
> enter aes-256-cbc encryption password:
> Verifying - enter aes-256-cbc encryption password:
> alto$ ls
> ls
> deb9-upgrade.log  foo.text.enc	ieee-cs-election-receipt.pdf
> enchant.dummy*	  foo.txt	net-neutrality-comments.pdf
> alto$ echo $SHELL
> echo $SHELL
> /bin/bash
> alto$

This is reproducible for me too on macOS.

I can also reproduce it by running the Mailman 3 shell from git checkout.  That made me think it could be readline related, but I built a Mailman 3 without readline support and it made no difference.  I haven’t figured out what the common denominator is, but I edebug’d it down to comint-send-string and then process-send-string.  There the trail ended with C code and I haven’t gdb’d into that.  It seems like some state in process-send-string gets confused and starts echoing the input.

-Barry


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28500: 25.3; shell mode starts echoing command

[Andreas Schwab]

On Sep 18 2017, Mike Kupfer <mkupfer@alum.berkeley.edu> wrote:

> I see the same issue after using openssl to encrypt or decrypt a file.
> Just never got around to filing a bug report...
>
> -----8<-----8<-----
> alto$ openssl aes-256-cbc -in foo.txt -out foo.text.enc
> enter aes-256-cbc encryption password:
> Verifying - enter aes-256-cbc encryption password:
> alto$ ls
> ls
> deb9-upgrade.log  foo.text.enc	ieee-cs-election-receipt.pdf
> enchant.dummy*	  foo.txt	net-neutrality-comments.pdf
> alto$ echo $SHELL
> echo $SHELL
> /bin/bash
> alto$ 
> ----->8----->8-----
>
> I see this on Debian 8 and on Solaris, with various versions of Emacs.

This is a bug in openssl, it doesn't properly restore the tty settings
after reading the password.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#28500: 25.3; shell mode starts echoing command

[Barry Warsaw]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]

On Sep 19, 2017, at 03:34, Andreas Schwab <schwab@suse.de> wrote:
> 
> On Sep 18 2017, Mike Kupfer <mkupfer@alum.berkeley.edu> wrote:
> 
>> I see the same issue after using openssl to encrypt or decrypt a file.
>> Just never got around to filing a bug report...
>> 
>> -----8<-----8<-----
>> alto$ openssl aes-256-cbc -in foo.txt -out foo.text.enc
>> enter aes-256-cbc encryption password:
>> Verifying - enter aes-256-cbc encryption password:
>> alto$ ls
>> ls
>> deb9-upgrade.log  foo.text.enc	ieee-cs-election-receipt.pdf
>> enchant.dummy*	  foo.txt	net-neutrality-comments.pdf
>> alto$ echo $SHELL
>> echo $SHELL
>> /bin/bash
>> alto$
>> ----->8----->8-----
>> 
>> I see this on Debian 8 and on Solaris, with various versions of Emacs.
> 
> This is a bug in openssl, it doesn't properly restore the tty settings
> after reading the password.

Andreas, can you provide any additional details?  What exactly is not getting restored?  Are the other two examples I posted (python3 and sqlite3) related?  Why would it occur on (for me) macOS but not Ubuntu?


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28500: 25.3; shell mode starts echoing command

[Andreas Schwab]

On Sep 19 2017, Barry Warsaw <barry@python.org> wrote:

> On Sep 19, 2017, at 03:34, Andreas Schwab <schwab@suse.de> wrote:
>> 
>> On Sep 18 2017, Mike Kupfer <mkupfer@alum.berkeley.edu> wrote:
>> 
>>> I see the same issue after using openssl to encrypt or decrypt a file.
>>> Just never got around to filing a bug report...
>>> 
>>> -----8<-----8<-----
>>> alto$ openssl aes-256-cbc -in foo.txt -out foo.text.enc
>>> enter aes-256-cbc encryption password:
>>> Verifying - enter aes-256-cbc encryption password:
>>> alto$ ls
>>> ls
>>> deb9-upgrade.log  foo.text.enc	ieee-cs-election-receipt.pdf
>>> enchant.dummy*	  foo.txt	net-neutrality-comments.pdf
>>> alto$ echo $SHELL
>>> echo $SHELL
>>> /bin/bash
>>> alto$
>>> ----->8----->8-----
>>> 
>>> I see this on Debian 8 and on Solaris, with various versions of Emacs.
>> 
>> This is a bug in openssl, it doesn't properly restore the tty settings
>> after reading the password.
>
> Andreas, can you provide any additional details?  What exactly is not getting restored?  Are the other two examples I posted (python3 and sqlite3) related?  Why would it occur on (for me) macOS but not Ubuntu?

$ stty; openssl aes-256-cbc -in foo.txt -out foo.text.enc; stty

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#28500: 25.3; shell mode starts echoing command

[Barry Warsaw]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

On Sep 19, 2017, at 09:36, Andreas Schwab <schwab@suse.de> wrote:

> $ stty; openssl aes-256-cbc -in foo.txt -out foo.text.enc; stty

Thanks, that’s exactly the clue I needed.

% stty
speed 9600 baud;
lflags: -echo echoe echoke echoctl
oflags: -oxtabs
cflags: cs8 -parenb
% sqlite3
SQLite version 3.16.0 2016-11-04 19:09:39
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .exit
.exit
% stty
stty
speed 9600 baud;
lflags: echoe echoke echoctl
oflags: -oxtabs
cflags: cs8 -parenb
% stty -echo
stty -echo
|resist=|@resist[~:1020]% ls
Applications/	Library/	Sites/

Notice that -echo gets lost when exiting sqlite3.  Setting -echo explicitly fixes the problem.  So not a bug per se in Emacs, although I wonder if a coming-output-filter-function could be added to do the fix until the offending processes themselves are fixed?


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28500: 25.3; shell mode starts echoing command

[Barry Warsaw]

[-- Attachment #1: Type: text/plain, Size: 1453 bytes --]

On Sep 19, 2017, at 09:36, Andreas Schwab <schwab@suse.de> wrote:
> 
> $ stty; openssl aes-256-cbc -in foo.txt -out foo.text.enc; stty

What is happening is that -echo is getting lost.  `stty -echo` after exiting Python will restore normal behavior in a shell buffer.  I’m not entirely sure why, but this is neither set nor necessary in a Terminal window.

My suspicion now falls on readline from home-brew.  Specifically, if I build Python from git master but *remove* the readline.<blah>.so file so there’s no way that readline can be invoked, then I don’t see the problem.  readline gets imported implicitly if available when entering the interactive prompt, and I’ve proven to myself that if that happens, echo gets reenabled.  It’s the common feature with sqlite3 too, which for me comes from brew.  Further, because openssl comes from Apple, it isn’t linked with brew’s readline and that does *not* exhibit the problem.

And then I found this: https://github.com/Homebrew/brew/issues/651

I will follow up on that issue, but it’s clearly now not an Emacs bug.

FWIW, this little bit of Python in your $PYTHONSTARTUP file will workaround the problem:

import sys
import atexit
import termios

# Reset `stty -echo` on exit.
def no_echo():
    flags = termios.tcgetattr(sys.stdin)
    flags[3] &= ~termios.ECHO
    termios.tcsetattr(sys.stdin, termios.TCSADRAIN, flags)

atexit.register(no_echo)

Cheers!


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28500: 25.3; shell mode starts echoing command

[Eli Zaretskii]

> From: Barry Warsaw <barry@python.org>
> Date: Tue, 19 Sep 2017 20:19:40 -0400
> Cc: 28500@debbugs.gnu.org, Mike Kupfer <mkupfer@alum.berkeley.edu>
> 
> I will follow up on that issue, but it’s clearly now not an Emacs bug.

Thanks for investigating.  Does that mean we can close this bug
report?

bug#28500: 25.3; shell mode starts echoing command

[Barry Warsaw]

[-- Attachment #1: Type: text/plain, Size: 410 bytes --]

On Sep 20, 2017, at 02:12, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Barry Warsaw <barry@python.org>
>> Date: Tue, 19 Sep 2017 20:19:40 -0400
>> Cc: 28500@debbugs.gnu.org, Mike Kupfer <mkupfer@alum.berkeley.edu>
>> 
>> I will follow up on that issue, but it’s clearly now not an Emacs bug.
> 
> Thanks for investigating.  Does that mean we can close this bug
> report?

Yep!  And thanks.


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#28500: 25.3; shell mode starts echoing command

[Eli Zaretskii]

> From: Barry Warsaw <barry@python.org>
> Date: Wed, 20 Sep 2017 09:39:14 -0400
> Cc: Andreas Schwab <schwab@suse.de>,
>  28500@debbugs.gnu.org,
>  mkupfer@alum.berkeley.edu
> 
> > Thanks for investigating.  Does that mean we can close this bug
> > report?
> 
> Yep!  And thanks.

Closing.