bug#50503: [PATCH] Properly encode all strings sent to the Python shell

bug#50503: [PATCH] Properly encode all strings sent to the Python shell

[Augusto Stoffel]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

Tags: patch

Some Python shell functions naively assume that '''<arbitrary text>'''
is valid Python code, which is not true, say, if the arbitrary text
ends with a quote sign.  This patch fixes this.

Some more remarks:

- python.el now depends on Emacs 28 because it uses 'format-prompt' and
  'comint-max-line-length' (luckily, only the latter is my fault).  Is
  this a problem?
- I've deleted some defcustoms which are not usable in my opinion.  The
  intended customizability is not impossible, but can only be achieved
  by advising certain functions.
- There are some long-obsolete functions (since say Emacs 24) that I
  would like to delete.  One example is 'python-send-string' which is
  easy to confuse with the 'python-shell-send-string', which is real
  deal.  What are the rules for this?

In GNU Emacs 28.0.50 (build 6, x86_64-pc-linux-gnu, GTK+ Version 3.24.29, cairo version 1.17.4)
 of 2021-09-09 built on toolbox
Repository revision: 8c023e5ea159c6756c92fd02643983aa449e0da9
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101002
System Description: Fedora 34 (Workstation Edition)


[-- Attachment #2: 0001-Properly-encode-all-strings-sent-to-Python-shell.patch --]
[-- Type: text/patch, Size: 5664 bytes --]

From 032b81954685ccd0c2d0a34cfc79bda23393d8e2 Mon Sep 17 00:00:00 2001
From: Augusto Stoffel <arstoffel@gmail.com>
Date: Fri, 10 Sep 2021 11:06:03 +0200
Subject: [PATCH] Properly encode all strings sent to Python shell

* lisp/progmodes/python.el: Now depends on Emacs 28.
(python-shell-package-enable, python-shell-completion-get-completions,
python-ffap-module-path, python-eldoc--get-doc-at-point): Enconde
Python strings using 'python-shell--encode-string' instead of triple
quotes.
(python-shell-completion-string-code, python-eldoc-string-code,
python-ffap-string-code): Remove defcustoms.
---
 lisp/progmodes/python.el | 66 ++++++++++++++--------------------------
 1 file changed, 22 insertions(+), 44 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e342ce7f56..2898897755 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -5,7 +5,7 @@
 ;; Author: Fabián E. Gallina <fgallina@gnu.org>
 ;; URL: https://github.com/fgallina/python.el
 ;; Version: 0.27.1
-;; Package-Requires: ((emacs "24.2") (cl-lib "1.0"))
+;; Package-Requires: ((emacs "28.1") (cl-lib "1.0"))
 ;; Maintainer: emacs-devel@gnu.org
 ;; Created: Jul 2010
 ;; Keywords: languages
@@ -2571,10 +2571,12 @@ python-shell-package-enable
    (format
     (concat
      "import os.path;import sys;"
-     "sys.path.append(os.path.dirname(os.path.dirname('''%s''')));"
-     "__package__ = '''%s''';"
+     "sys.path.append(os.path.dirname(os.path.dirname(%s)));"
+     "__package__ = %s;"
      "import %s")
-    directory package package)
+    (python-shell--encode-string directory)
+    (python-shell--encode-string package)
+    package)
    (python-shell-get-process)))
 
 (defun python-shell-accept-process-output (process &optional timeout regexp)
@@ -3532,14 +3534,6 @@ python-shell-completion-setup-code
   "25.1"
   "Completion string code must work for (i)pdb.")
 
-(defcustom python-shell-completion-string-code
-  "';'.join(__PYTHON_EL_get_completions('''%s'''))"
-  "Python code used to get a string of completions separated by semicolons.
-The string passed to the function is the current python name or
-the full statement in the case of imports."
-  :type 'string
-  :group 'python)
-
 (defcustom python-shell-completion-native-disabled-interpreters
   ;; PyPy's readline cannot handle some escape sequences yet.  Native
   ;; completion doesn't work on w32 (Bug#28580).
@@ -3834,9 +3828,10 @@ python-shell-completion-get-completions
            (python-util-strip-string
             (python-shell-send-string-no-output
              (format
-              (concat python-shell-completion-setup-code
-                      "\nprint (" python-shell-completion-string-code ")")
-              input) process))))
+              "%s\nprint(';'.join(__PYTHON_EL_get_completions(%s)))"
+              python-shell-completion-setup-code
+              (python-shell--encode-string input))
+             process))))
       (when (> (length completions) 2)
         (split-string completions
                       "^'\\|^\"\\|;\\|'$\\|\"$" t)))))
@@ -4559,28 +4554,16 @@ python-ffap-setup-code
   :type 'string
   :group 'python)
 
-(defcustom python-ffap-string-code
-  "__FFAP_get_module_path('''%s''')"
-  "Python code used to get a string with the path of a module."
-  :type 'string
-  :group 'python)
-
 (defun python-ffap-module-path (module)
   "Function for `ffap-alist' to return path for MODULE."
-  (let ((process (or
-                  (and (derived-mode-p 'inferior-python-mode)
-                       (get-buffer-process (current-buffer)))
-                  (python-shell-get-process))))
-    (if (not process)
-        nil
-      (let ((module-file
-             (python-shell-send-string-no-output
-              (concat
-               python-ffap-setup-code
-               "\nprint (" (format python-ffap-string-code module) ")")
-              process)))
-        (unless (zerop (length module-file))
-          (python-util-strip-string module-file))))))
+  (when-let ((process (python-shell-get-process))
+             (module-file
+              (python-shell-send-string-no-output
+               (format "%s\nprint(__FFAP_get_module_path(%s))"
+                       python-ffap-setup-code
+                       (python-shell--encode-string module)))))
+    (unless (string-empty-p module-file)
+      (python-util-strip-string module-file))))
 
 (defvar ffap-alist)
 
@@ -4671,12 +4654,6 @@ python-eldoc-setup-code
   :type 'string
   :group 'python)
 
-(defcustom python-eldoc-string-code
-  "__PYDOC_get_help('''%s''')"
-  "Python code used to get a string with the documentation of an object."
-  :type 'string
-  :group 'python)
-
 (defun python-eldoc--get-symbol-at-point ()
   "Get the current symbol for eldoc.
 Returns the current symbol handling point within arguments."
@@ -4706,11 +4683,12 @@ python-eldoc--get-doc-at-point
                 ;; enabled.  Bug#18794.
                 (python-util-strip-string
                  (python-shell-send-string-no-output
-                  (concat
+                  (format
+                   "%s\nprint(__PYDOC_get_help(%s))"
                    python-eldoc-setup-code
-                   "\nprint(" (format python-eldoc-string-code input) ")")
+                   (python-shell--encode-string input))
                   process)))))
-        (unless (zerop (length docstring))
+        (unless (string-empty-p docstring)
           docstring)))))
 
 (defvar-local python-eldoc-get-doc t
-- 
2.31.1

bug#50503: [PATCH] Properly encode all strings sent to the Python shell

[Lars Ingebrigtsen]

Augusto Stoffel <arstoffel@gmail.com> writes:

> Some Python shell functions naively assume that '''<arbitrary text>'''
> is valid Python code, which is not true, say, if the arbitrary text
> ends with a quote sign.  This patch fixes this.

Thanks; pushed to Emacs 28.

> Some more remarks:
>
> - python.el now depends on Emacs 28 because it uses 'format-prompt' and
>   'comint-max-line-length' (luckily, only the latter is my fault).  Is
>   this a problem?

Yes -- that was a mistake, so I've now reverted the usage of
format-prompt here.

> - There are some long-obsolete functions (since say Emacs 24) that I
>   would like to delete.  One example is 'python-send-string' which is
>   easy to confuse with the 'python-shell-send-string', which is real
>   deal.  What are the rules for this?

We don't have any specific rules here, but I think we can start removing
Emacs 24.x obsoletions when we get to Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#50503: [PATCH] Properly encode all strings sent to the Python shell

[Augusto Stoffel]

On Fri, 10 Sep 2021 at 13:07, Lars Ingebrigtsen <larsi@gnus.org> wrote:

>> - python.el now depends on Emacs 28 because it uses 'format-prompt' and
>>   'comint-max-line-length' (luckily, only the latter is my fault).  Is
>>   this a problem?
>
> Yes -- that was a mistake, so I've now reverted the usage of
> format-prompt here.

Thanks!  But note that there is at least one Emacs 28 dependency left,
namely `comint-max-line-length'.  There's also a byte-compiler warning
regarding `comint-highlight-input' even on Emacs 27.

bug#50503: [PATCH] Properly encode all strings sent to the Python shell

[Lars Ingebrigtsen]

Augusto Stoffel <arstoffel@gmail.com> writes:

> Thanks!  But note that there is at least one Emacs 28 dependency left,
> namely `comint-max-line-length'.

Perhaps there should be some compat code in that case?

> There's also a byte-compiler warning regarding
> `comint-highlight-input' even on Emacs 27.

Warnings are normal on older Emacs versions -- I mean, it'd be good to
avoid them, but we're mainly worried about actual breakages.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#50503: [PATCH] Properly encode all strings sent to the Python shell

[Augusto Stoffel]

On Sat, 11 Sep 2021 at 14:38, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Augusto Stoffel <arstoffel@gmail.com> writes:
>
>> Thanks!  But note that there is at least one Emacs 28 dependency left,
>> namely `comint-max-line-length'.
>
> Perhaps there should be some compat code in that case?

If it is stylistically acceptable, I think

  (or (bound-and-true-p comint-max-line-length) 1024)

is the simplest solution.  One could also replace the mysterious 1024 by
0, which was more or less the previous situation anyway.

>
>> There's also a byte-compiler warning regarding
>> `comint-highlight-input' even on Emacs 27.
>
> Warnings are normal on older Emacs versions -- I mean, it'd be good to
> avoid them, but we're mainly worried about actual breakages.

All right then.

bug#50503: [PATCH] Properly encode all strings sent to the Python shell

[Lars Ingebrigtsen]

Augusto Stoffel <arstoffel@gmail.com> writes:

> If it is stylistically acceptable, I think
>
>   (or (bound-and-true-p comint-max-line-length) 1024)
>
> is the simplest solution.  One could also replace the mysterious 1024 by
> 0, which was more or less the previous situation anyway.

Sure, that'd be fine.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#50503: [PATCH] Properly encode all strings sent to the Python shell

[Augusto Stoffel]

[-- Attachment #1: Type: text/plain, Size: 418 bytes --]

On Sat, 11 Sep 2021 at 14:45, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Augusto Stoffel <arstoffel@gmail.com> writes:
>
>> If it is stylistically acceptable, I think
>>
>>   (or (bound-and-true-p comint-max-line-length) 1024)
>>
>> is the simplest solution.  One could also replace the mysterious 1024 by
>> 0, which was more or less the previous situation anyway.
>
> Sure, that'd be fine.

I've attached a patch


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Keep-python.el-compatible-with-older-Emacsen.patch --]
[-- Type: text/x-patch, Size: 1218 bytes --]

From d2a6bd70d19f84592235691eb25356bcc5275880 Mon Sep 17 00:00:00 2001
From: Augusto Stoffel <arstoffel@gmail.com>
Date: Sat, 11 Sep 2021 15:29:06 +0200
Subject: [PATCH] Keep python.el compatible with older Emacsen

* progmodes/python.el (python-shell-send-string): Don't assume
comint-max-line-length is defined.
---
 lisp/progmodes/python.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e71a8102df..b17d740036 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3155,7 +3155,9 @@ python-shell-send-string
                       (python-shell--encode-string (or (buffer-file-name)
                                                        "<string>")))))
     (if (or (null (process-tty-name process))
-            (<= (string-bytes code) comint-max-line-length))
+            (<= (string-bytes code)
+                (or (bound-and-true-p comint-max-line-length)
+                    1024))) ;; For Emacs < 28
         (comint-send-string process code)
       (let* ((temp-file-name (with-current-buffer (process-buffer process)
                                (python-shell--save-temp-file string)))
-- 
2.31.1

bug#50503: [PATCH] Properly encode all strings sent to the Python shell

[Lars Ingebrigtsen]

Augusto Stoffel <arstoffel@gmail.com> writes:

> I've attached a patch

Thanks; applied to Emacs 28.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no