From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Augusto Stoffel Newsgroups: gmane.emacs.bugs,gmane.emacs.devel Subject: bug#49822: master e32c7d2: Change Python eval to send directly instead of using temporary files Date: Sat, 04 Sep 2021 11:49:31 +0200 Message-ID: <87r1e4eklw.fsf@gmail.com> References: <20210903122828.16890.65271@vcs0.savannah.gnu.org> <20210903122829.EAAC220B71@vcs0.savannah.gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="40719"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: 49822@debbugs.gnu.org, emacs-devel@gnu.org To: Stefan Monnier Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sat Sep 04 11:50:42 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mMSJt-000AQA-LI for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 04 Sep 2021 11:50:41 +0200 Original-Received: from localhost ([::1]:45772 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mMSJs-0007Gj-4d for geb-bug-gnu-emacs@m.gmane-mx.org; Sat, 04 Sep 2021 05:50:40 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:51218) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mMSJG-0007Fy-QG for bug-gnu-emacs@gnu.org; Sat, 04 Sep 2021 05:50:03 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:35117) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mMSJG-0005wP-FT for bug-gnu-emacs@gnu.org; Sat, 04 Sep 2021 05:50:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mMSJG-00030d-3j for bug-gnu-emacs@gnu.org; Sat, 04 Sep 2021 05:50:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Augusto Stoffel Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 04 Sep 2021 09:50:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49822 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 49822-submit@debbugs.gnu.org id=B49822.163074898311528 (code B ref 49822); Sat, 04 Sep 2021 09:50:02 +0000 Original-Received: (at 49822) by debbugs.gnu.org; 4 Sep 2021 09:49:43 +0000 Original-Received: from localhost ([127.0.0.1]:46660 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMSIw-0002zq-RJ for submit@debbugs.gnu.org; Sat, 04 Sep 2021 05:49:43 -0400 Original-Received: from mail-wr1-f53.google.com ([209.85.221.53]:33673) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMSIu-0002zb-Bv for 49822@debbugs.gnu.org; Sat, 04 Sep 2021 05:49:40 -0400 Original-Received: by mail-wr1-f53.google.com with SMTP id b10so2159016wru.0 for <49822@debbugs.gnu.org>; Sat, 04 Sep 2021 02:49:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=uYx71icbEEV30m8z547je/rE0wdQz2z9amCg22AzY7c=; b=Trd5u+ZtuPw3V2dhB0j+u2/BLIQJKfIAsj932yPkgZbOWsM7Vt22ZVgmsH0EArz0V0 PPn4m0O9jMSpfnN6pm1N7BqRxQcn5za6BA0hzP06qRth6LIu6+iT+6llEjDSeRNgLLDf PZHpdyCCq/wn8kryVtvXgnkX6LlVLJ1YNOYI59o1GkZZgf24H4lExJQFzIuQnBq/yD7c s78w7hvD74mbjikewtqfc6shRcRtIWdNYycHnOqpsjILzC/Met8b6T5me3iwSKnTk7BE nnwRUy7/3vRkjf4NLcU+9kNKjHpCYvNnEVmL6LGvAOzKggwroDdxJOcYfM/j/jNCMWJz RzyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=uYx71icbEEV30m8z547je/rE0wdQz2z9amCg22AzY7c=; b=ZuFmtH8WDbuTyCejDMA15BHqaFyN7YAcomFh7Xljc4QHaO+9p2bLiwMcJHuN9JI3yW LKfKIFn5u/NUhabVT/VGdFrkf1VffVekA6AtRZ2w6OkZ/THa5mLontONErgerCg7UZhS qRNFaloy0lKJkNDgbHYfmCjq8sWgx0eZ7sug5NFPanUtVwmHVv0CpvEwqpiX1h1Vsksh dcr0n9/5SfIidersDgPc27ta0PtMNINpXdDIdjFMUvDLYQSgtDO3bKYqtHcMre7Fwc0C Z1oMn7qzv/QMlA80KEYatW4Lhgmua8Ps6chpvtsRqLzBhijV+ZQp2mo6TRCGaDcshQ/9 GWag== X-Gm-Message-State: AOAM530teL3rl825LEJs1Rvv1W+RVj/VPUMwCdHzNup62r0lnbUHiM/a xEaI6ZVtndGkN914Hv0rghe69nUHLZI= X-Google-Smtp-Source: ABdhPJy0Q871I2edIQHSbv/+sihVnSlTkqHBPYYKTF5zb3jr+4ncrF4KNRO3LmtAb5/6CKlpxHlWVw== X-Received: by 2002:a5d:4951:: with SMTP id r17mr3215977wrs.417.1630748973951; Sat, 04 Sep 2021 02:49:33 -0700 (PDT) Original-Received: from ars3 ([2a02:8109:8ac0:56d0::8aaf]) by smtp.gmail.com with ESMTPSA id e3sm2104183wrv.18.2021.09.04.02.49.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 04 Sep 2021 02:49:32 -0700 (PDT) In-Reply-To: (Stefan Monnier's message of "Fri, 03 Sep 2021 19:04:53 -0400") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:213407 gmane.emacs.devel:273850 Archived-At: --=-=-= Content-Type: text/plain On Fri, 3 Sep 2021 at 19:04, Stefan Monnier wrote: >> Change Python eval to send directly instead of using temporary files > > FWIW, sending large amounts of text via pty can be troublesome (some > OSes have been known to insert additional control chars every 256 > bytes, or to do weird things after 4096 bytes leading the send to never > complete, ...), which is why it's very common for comint modes to send > regions of text via temp files. > > > Stefan Okay, the 4096 bytes limit indeed exists in IPython (but not in the standard Python interpreters, apparently). I've attached a patch that reverts to using temporary files for sufficiently long strings (would this magic 4096 ever require customization?). The patch also solves bug#32042, which is mostly unrelated. I would like pass text inline as much as possible because the back-and-forth of temp files is pretty slow over Tramp, which makes the Python shell rather annoying to use. As to some OSes inserting additional control characters every 256 bytes, I've never encountered this, but it seems a rather tricky problem to work around. Before this commit, the "plumbing code" sent to the interpreter could already be above 256 bytes (e.g., if the generated temp file names are modestly long). Inserting newline characters at strategic places would print random prompt strings, and therefore also introduce complications. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Fixes-for-python-shell-send-string-and-python-shell-.patch >From 6fdbecb66d4409e2ce03c559861138966abb4acd Mon Sep 17 00:00:00 2001 From: Augusto Stoffel Date: Sat, 4 Sep 2021 11:16:11 +0200 Subject: [PATCH] Fixes for 'python-shell-send-string' and 'python-shell-send-file' * lisp/progmodes/python.el (python-shell-send-string): use a temporary file for sufficiently long strings. (python-shell-send-file, python-shell-eval-file-setup-code): Avoid showing "plumbing code" in the traceback (bug#32042). --- lisp/progmodes/python.el | 46 +++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el index 306cc3a542..d8ec032402 100644 --- a/lisp/progmodes/python.el +++ b/lisp/progmodes/python.el @@ -3127,12 +3127,18 @@ python-shell-send-string t when called interactively." (interactive (list (read-string "Python command: ") nil t)) - (comint-send-string - (or process (python-shell-get-process-or-error msg)) - (format "exec(%s);__PYTHON_EL_eval(%s, %s)\n" - (python-shell--encode-string python-shell-eval-setup-code) - (python-shell--encode-string string) - (python-shell--encode-string (or (buffer-file-name) ""))))) + (let ((process (or process (python-shell-get-process-or-error msg))) + (code (format "exec(%s);__PYTHON_EL_eval(%s, %s)\n" + (python-shell--encode-string python-shell-eval-setup-code) + (python-shell--encode-string string) + (python-shell--encode-string (or (buffer-file-name) + ""))))) + (if (<= (string-bytes code) 4096) + (comint-send-string process code) + (let* ((temp-file-name (with-current-buffer (process-buffer process) + (python-shell--save-temp-file string))) + (file-name (or (buffer-file-name) temp-file-name))) + (python-shell-send-file file-name process temp-file-name t))))) (defvar python-shell-output-filter-in-progress nil) (defvar python-shell-output-filter-buffer nil) @@ -3372,6 +3378,18 @@ python-shell-send-defun nil ;; noop msg)))) + +(defconst python-shell-eval-file-setup-code + "\ +def __PYTHON_EL_eval_file(filename, tempname, encoding, delete): + import codecs, os + with codecs.open(tempname or filename, encoding=encoding) as file: + source = file.read().encode(encoding) + if delete and tempname: + os.remove(tempname) + return __PYTHON_EL_eval(source, filename)" + "Code used to evaluate files in inferior Python processes.") + (defun python-shell-send-file (file-name &optional process temp-file-name delete msg) "Send FILE-NAME to inferior Python PROCESS. @@ -3401,15 +3419,13 @@ python-shell-send-file (comint-send-string process (format - (concat - "import codecs, os;" - "__pyfile = codecs.open('''%s''', encoding='''%s''');" - "__code = __pyfile.read().encode('''%s''');" - "__pyfile.close();" - (when (and delete temp-file-name) - (format "os.remove('''%s''');" temp-file-name)) - "exec(compile(__code, '''%s''', 'exec'));") - (or temp-file-name file-name) encoding encoding file-name)))) + "exec(%s);exec(%s);__PYTHON_EL_eval_file(%s, %s, %s, %s)\n" + (python-shell--encode-string python-shell-eval-setup-code) + (python-shell--encode-string python-shell-eval-file-setup-code) + (python-shell--encode-string file-name) + (python-shell--encode-string (or temp-file-name "")) + (python-shell--encode-string (symbol-name encoding)) + (if delete "True" "False"))))) (defun python-shell-switch-to-shell (&optional msg) "Switch to inferior Python process buffer. -- 2.31.1 --=-=-=--