bug#49822: master e32c7d2: Change Python eval to send directly instead of using temporary files

[Augusto Stoffel <arstoffel@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

On Fri,  3 Sep 2021 at 19:04, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>>     Change Python eval to send directly instead of using temporary files
>
> FWIW, sending large amounts of text via pty can be troublesome (some
> OSes have been known to insert additional control chars every 256
> bytes, or to do weird things after 4096 bytes leading the send to never
> complete, ...), which is why it's very common for comint modes to send
> regions of text via temp files.
>
>
>         Stefan

Okay, the 4096 bytes limit indeed exists in IPython (but not in the
standard Python interpreters, apparently).

I've attached a patch that reverts to using temporary files for
sufficiently long strings (would this magic 4096 ever require
customization?).  The patch also solves bug#32042, which is mostly
unrelated.

I would like pass text inline as much as possible because the
back-and-forth of temp files is pretty slow over Tramp, which makes the
Python shell rather annoying to use.

As to some OSes inserting additional control characters every 256 bytes,
I've never encountered this, but it seems a rather tricky problem to
work around.  Before this commit, the "plumbing code" sent to the
interpreter could already be above 256 bytes (e.g., if the generated
temp file names are modestly long).  Inserting newline characters at
strategic places would print random prompt strings, and therefore also
introduce complications.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fixes-for-python-shell-send-string-and-python-shell-.patch --]
[-- Type: text/x-patch, Size: 3852 bytes --]

From 6fdbecb66d4409e2ce03c559861138966abb4acd Mon Sep 17 00:00:00 2001
From: Augusto Stoffel <arstoffel@gmail.com>
Date: Sat, 4 Sep 2021 11:16:11 +0200
Subject: [PATCH] Fixes for 'python-shell-send-string' and
 'python-shell-send-file'

* lisp/progmodes/python.el (python-shell-send-string): use a temporary
file for sufficiently long strings.
(python-shell-send-file, python-shell-eval-file-setup-code): Avoid
showing "plumbing code" in the traceback (bug#32042).
---
 lisp/progmodes/python.el | 46 +++++++++++++++++++++++++++-------------
 1 file changed, 31 insertions(+), 15 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 306cc3a542..d8ec032402 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3127,12 +3127,18 @@ python-shell-send-string
 t when called interactively."
   (interactive
    (list (read-string "Python command: ") nil t))
-  (comint-send-string
-   (or process (python-shell-get-process-or-error msg))
-   (format "exec(%s);__PYTHON_EL_eval(%s, %s)\n"
-           (python-shell--encode-string python-shell-eval-setup-code)
-           (python-shell--encode-string string)
-           (python-shell--encode-string (or (buffer-file-name) "<string>")))))
+  (let ((process (or process (python-shell-get-process-or-error msg)))
+        (code (format "exec(%s);__PYTHON_EL_eval(%s, %s)\n"
+                      (python-shell--encode-string python-shell-eval-setup-code)
+                      (python-shell--encode-string string)
+                      (python-shell--encode-string (or (buffer-file-name)
+                                                       "<string>")))))
+    (if (<= (string-bytes code) 4096)
+        (comint-send-string process code)
+      (let* ((temp-file-name (with-current-buffer (process-buffer process)
+                               (python-shell--save-temp-file string)))
+             (file-name (or (buffer-file-name) temp-file-name)))
+        (python-shell-send-file file-name process temp-file-name t)))))
 
 (defvar python-shell-output-filter-in-progress nil)
 (defvar python-shell-output-filter-buffer nil)
@@ -3372,6 +3378,18 @@ python-shell-send-defun
        nil ;; noop
        msg))))
 
+
+(defconst python-shell-eval-file-setup-code
+  "\
+def __PYTHON_EL_eval_file(filename, tempname, encoding, delete):
+    import codecs, os
+    with codecs.open(tempname or filename, encoding=encoding) as file:
+        source = file.read().encode(encoding)
+    if delete and tempname:
+        os.remove(tempname)
+    return __PYTHON_EL_eval(source, filename)"
+  "Code used to evaluate files in inferior Python processes.")
+
 (defun python-shell-send-file (file-name &optional process temp-file-name
                                          delete msg)
   "Send FILE-NAME to inferior Python PROCESS.
@@ -3401,15 +3419,13 @@ python-shell-send-file
     (comint-send-string
      process
      (format
-      (concat
-       "import codecs, os;"
-       "__pyfile = codecs.open('''%s''', encoding='''%s''');"
-       "__code = __pyfile.read().encode('''%s''');"
-       "__pyfile.close();"
-       (when (and delete temp-file-name)
-         (format "os.remove('''%s''');" temp-file-name))
-       "exec(compile(__code, '''%s''', 'exec'));")
-      (or temp-file-name file-name) encoding encoding file-name))))
+      "exec(%s);exec(%s);__PYTHON_EL_eval_file(%s, %s, %s, %s)\n"
+      (python-shell--encode-string python-shell-eval-setup-code)
+      (python-shell--encode-string python-shell-eval-file-setup-code)
+      (python-shell--encode-string file-name)
+      (python-shell--encode-string (or temp-file-name ""))
+      (python-shell--encode-string (symbol-name encoding))
+      (if delete "True" "False")))))
 
 (defun python-shell-switch-to-shell (&optional msg)
   "Switch to inferior Python process buffer.
-- 
2.31.1