bug#43450: 28.0.50; Python shell doesn't echo if shebang

bug#43450: 28.0.50; Python shell doesn't echo if shebang

[Gunnar Horrigmo]

Open a .py file
M-x run-python RET
In .py file, write some statement, say "1+2" and evaluate it with
(f.ex.) python-shell-send-statement. Observe that python shell echoes
the result, "3", and prompts on a new line.

Add a shebang to the top of .py file (f.ex. "#!/usr/bin/pyton3" on my
system). Evaluate the same statement as over. Observe that python shell
doesn't echo the result, but does print a new prompt, but without a
newline. 

You can add and remove the shebang as much you want, and see the echo
behaviour change accordingly. You do _not_ need to save the file.

You can see a replay of the issue here:
https://asciinema.org/a/DOq15ldlyiKI5HKLrKpF7gIeg

I've verified this behaviour using emacs -Q on versions 28.0.50
(including master HEAD as of 2020-09-16) and 27.1.50. And it seems all
the python-shell-send-* functions exhibit the same, not just
python-shell-send-statement. 



In GNU Emacs 28.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version 3.24.13, cairo version 1.16.0)
 of 2020-05-23 built on hot-rats.uio.no
Repository revision: f8581bcf6a1942ebd331cae20e32945a3a86a3d1
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Fedora 31 (Workstation Edition)

Recent messages:
Checking 71 files in /home/horrigmo/src/emacs/emacs/lisp/erc...
Checking 34 files in /home/horrigmo/src/emacs/emacs/lisp/emulation...
Checking 180 files in /home/horrigmo/src/emacs/emacs/lisp/emacs-lisp...
Checking 24 files in /home/horrigmo/src/emacs/emacs/lisp/cedet...
Checking 59 files in /home/horrigmo/src/emacs/emacs/lisp/calendar...
Checking 87 files in /home/horrigmo/src/emacs/emacs/lisp/calc...
Checking 105 files in /home/horrigmo/src/emacs/emacs/lisp/obsolete...
Checking for load-path shadows...done
C-x C-g is undefined [2 times]
Buffer *unsent mail to bug-gnu-emacs@gnu.org* modified; kill anyway? (y or n) y

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS PDUMPER LCMS2
GMP

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: ERC

Minor modes in effect:
  erc-list-mode: t
  erc-menu-mode: t
  erc-ring-mode: t
  erc-networks-mode: t
  erc-pcomplete-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-netsplit-mode: t
  erc-autojoin-mode: t
  erc-track-mode: t
  erc-track-minor-mode: t
  erc-match-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  shell-dirtrack-mode: t
  savehist-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug erc-list erc-menu erc-ring erc-networks erc-pcomplete
erc-button erc-fill erc-stamp erc-netsplit erc-join erc-track erc-match
erc-goodies erc erc-backend erc-compat thingatpt pp erc-loaddefs
mailalias bbdb-message nnir vc-git diff-mode easy-mmode sort gnus-cite
smiley shr-color mm-archive mail-extr gnus-async gnus-bcklg mule-util qp
gnus-ml disp-table gnus-topic nndraft nnmh utf-7 gnutls bbdb-gnus
network-stream nsm gnus-agent gnus-srvr gnus-score score-mode nnvirtual
gnus-msg gnus-art mm-uu mml2015 nntp gnus-cache gnus-sum url url-proxy
url-privacy url-expand url-methods url-history shr url-cookie url-domsuf
url-util svg xml dom gnus-group gnus-undo powerline advice
powerline-separators color cyberpunk-theme powerline-themes cus-edit
cus-start cus-load tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat shell pcomplete comint ansi-color ring ls-lisp ido
nnfolder smtpmail sendmail mm-view mml-smime smime dig mailcap
gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo
parse-time iso8601 gnus-spec gnus-int gnus-range message rmc puny
format-spec rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 gmm-utils mailheader gnus-win gnus nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums text-property-search time-date
mail-utils mm-util mail-prsvr wid-edit server pinentry epa-file epa
derived epg epg-config yatemplate autoinsert cl-extra yasnippet
help-mode edmacro kmacro dired-sidebar face-remap dired-subtree
dired-hacks-utils dash dired dired-loaddefs bbdb-mua bbdb-com crm
mailabbrev bbdb bbdb-site timezone finder-inf info package easymenu
browse-url url-handlers url-parse auth-source cl-seq eieio eieio-core
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt
bytecomp byte-compile cconv make-password cl-macs help-misc markstack cl
doders html-misc stoffe-misc gv savehist avoid cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 441208 78375)
 (symbols 48 30936 1)
 (strings 32 123601 7188)
 (string-bytes 1 3885469)
 (vectors 16 47667)
 (vector-slots 8 1304186 200108)
 (floats 8 378 407)
 (intervals 56 4655 1922)
 (buffers 992 33))

-- 
Gunnar

bug#43450: 28.0.50; Python shell doesn't echo if shebang

[Lars Ingebrigtsen]

Gunnar Horrigmo <horrigmo@runbox.no> writes:

> Open a .py file
> M-x run-python RET
> In .py file, write some statement, say "1+2" and evaluate it with
> (f.ex.) python-shell-send-statement. Observe that python shell echoes
> the result, "3", and prompts on a new line.
>
> Add a shebang to the top of .py file (f.ex. "#!/usr/bin/pyton3" on my
> system). Evaluate the same statement as over. Observe that python shell
> doesn't echo the result, but does print a new prompt, but without a
> newline. 

I can reproduce this in Emacs 28, but it doesn't seem to be triggered by
the shebang:  Adding some blank lines to the top of the buffer also
triggers the error.  Or ... anything:

(python-shell-buffer-substring 2 (point) nil)
=> "# -*- coding: utf-8 -*-

1+2"

(python-shell-buffer-substring 1 (point) nil)
=> "1+2"

         (fillstr (when (not starts-at-point-min-p)
                    (concat
                     (format "# -*- coding: %s -*-\n" encoding)
                     (make-string
                      ;; Subtract 2 because of the coding cookie.
                      (- (line-number-at-pos start) 2) ?\n)))))

This was added by (in a slightly different form):

commit 2dd5163d764f395eb31a2306dba385d123af4aba
Author:     Fabián Ezequiel Gallina <fgallina@gnu.org>
AuthorDate: Sat Dec 27 01:30:21 2014 -0300

    python.el: Handle file encoding for shell.
    
    * lisp/progmodes/python.el (python-rx-constituents): Add coding-cookie.
    (python-shell--save-temp-file): Write file with proper encoding.
    (python-shell-buffer-substring): Add coding cookie for detected
    encoding to generated content.  Fix blank lines when removing
    if-name-main block.
    (python-shell-send-file): Handle file encoding.
    (python-info-encoding-from-cookie)
    (python-info-encoding): New functions.

I've added Fabián to the Cc, perhaps he has some insight here...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#43450: 28.0.50; Python shell doesn't echo if shebang

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I can reproduce this in Emacs 28, but it doesn't seem to be triggered by
> the shebang:  Adding some blank lines to the top of the buffer also
> triggers the error.  Or ... anything:
>
> (python-shell-buffer-substring 2 (point) nil)
> => "# -*- coding: utf-8 -*-
>
> 1+2"

I have to admit that I don't understand the reason for the coding cookie
at all -- we bind coding-system-for-write to utf-8 before writing the
temp file anyway, and on the Python side, we specify the coding system
explicitly there, too.

But in case it does have an effect, I haven't removed it altogether, but
instead made the C-c C-e command avoid outputting it, which fixes the
problem.  C-M-x, probably a more important command, is not affected by
the cookie one way or another (since it's just side-effecting), so I've
left it alone there.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#43450: 28.0.50; Python shell doesn't echo if shebang

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I have to admit that I don't understand the reason for the coding cookie
> at all -- we bind coding-system-for-write to utf-8 before writing the
> temp file anyway, and on the Python side, we specify the coding system
> explicitly there, too.

OK, I think this is basically the point of it all:

(defun python-shell--save-temp-file (string)
  (let* ((temporary-file-directory
          (if (file-remote-p default-directory)
              (concat (file-remote-p default-directory) "/tmp")
            temporary-file-directory))
         (temp-file-name (make-temp-file "py"))
         (coding-system-for-write (python-info-encoding)))
    (with-temp-file temp-file-name
      (insert string)
      (delete-trailing-whitespace))
    temp-file-name))

If the current buffer has a coding cookie, we use that in the temp files
we load, too.  That's logical (er, somewhat, why not just use the
current buffer coding system?), but if there's no coding cookie, we
still add one, which isn't.

Anyway, I think the change I made is unlikely to break anything...
possibly...  unless you have a Python file with a coding system cookie,
and you're eval-ing a non-ASCII string, and it's not all utf-8.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no