From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Augusto Stoffel Newsgroups: gmane.emacs.devel Subject: Re: bug#49822: master e32c7d2: Change Python eval to send directly instead of using temporary files Date: Wed, 08 Sep 2021 16:05:11 +0200 Message-ID: <87zgsnp3hk.fsf@gmail.com> References: <20210903122828.16890.65271@vcs0.savannah.gnu.org> <20210903122829.EAAC220B71@vcs0.savannah.gnu.org> <87r1e4eklw.fsf@gmail.com> <87v93fnfmm.fsf@igel.home> <87lf4aooge.fsf@gmail.com> <878s0a409g.fsf@gmx.de> <87ilzef65i.fsf@igel.home> <878s0aeyna.fsf@gmx.de> <87r1e1ex7z.fsf@igel.home> <875yvcmgm2.fsf@gmail.com> <877dfr5ww1.fsf@gnus.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="4950"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: Andy Moreton , "Barton, Mark" , emacs-devel@gnu.org, Andreas Schwab , Stefan Monnier , 49822@debbugs.gnu.org, Michael Albinus To: Lars Ingebrigtsen Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Wed Sep 08 16:06:14 2021 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mNyDO-00011P-4W for ged-emacs-devel@m.gmane-mx.org; Wed, 08 Sep 2021 16:06:14 +0200 Original-Received: from localhost ([::1]:53518 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mNyDM-0002RY-MO for ged-emacs-devel@m.gmane-mx.org; Wed, 08 Sep 2021 10:06:12 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:40558) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mNyCV-0001Me-3L for emacs-devel@gnu.org; Wed, 08 Sep 2021 10:05:20 -0400 Original-Received: from mail-wr1-x42b.google.com ([2a00:1450:4864:20::42b]:35603) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mNyCR-0002yI-9x for emacs-devel@gnu.org; Wed, 08 Sep 2021 10:05:18 -0400 Original-Received: by mail-wr1-x42b.google.com with SMTP id i28so3526114wrb.2 for ; Wed, 08 Sep 2021 07:05:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=sYW7XlJZ7MLuf3aqanS/27POetbwdkpqv5vsvdK7huI=; b=DezgbP9CLH7FSCc2RvpiPXy0813pUsndrhiMrbL6mGVhdPCjCs5HubjbPDkRVUBTBH rSEwbaD/zaNVtPGJgfTrQnt7L7SvwdrRA/f2ZSPyOOVupaGkS1z24FVpPVJJaBLylpMf ZUXaOqIL4d/p09Z6WCPSJGksBk21/hGTssI6S+yEPL0YpCcpGZlYU0MCy9Xda2VrwAbf 3JX3rSV/qrxyC8j7eki4Xy8ggpHEw/kBJ3auMANvqgeKSupus2DR+lAbR2vW1hQChjxm Cd4a5LT8cRFiKwT+HvNvHYeDm907Wdo8bbiWZtxX2x7ut5AZCh4pL73NDwofwbZQ/Ebq IovQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=sYW7XlJZ7MLuf3aqanS/27POetbwdkpqv5vsvdK7huI=; b=TYy33HCzmchL7sRgopIstbzA0GelA+yZRFDSwbFh6wDYo9qZwqDUcrN1lkj9j1vDNK 9fW+GZr/Cb59Fn3LLJfKTos5rYZAKAzg1zCvYIGgNJMAYBU47oYSN/EKKnWldISKMYBe PzfYYn6aUV1ZnIfI96E84NETKEDZArlIhjND2M5teWg80Hs+RFW38KOBu7LtiO5FRJtD KyggUYMFAIv3ctVd9gcZy+CX5CdFYhXI4t4pHORMvuyv3Su/Hx723jnx6sFvzNizuyrC nfn5vzCI0jJLpfqboT7/4mkM5SRE/8uBmmC0erD/je0vkD7IsXXedT6mOX0t/aTds6ce U/Xg== X-Gm-Message-State: AOAM533Jm/vebhGW0Hw6YpM0gNthw5+jBSS8GMfiIx7oxEZjEleI28Xn vpn+90hAoJCPy/fzStr+ELk= X-Google-Smtp-Source: ABdhPJwhZQMopoSaLu2iqLVXV8YGXHj11mb7l3CrlmzQS8phCrKTSbDkGR9hHlrmWP5ueiPtzBdjtw== X-Received: by 2002:adf:de8f:: with SMTP id w15mr4409036wrl.277.1631109913518; Wed, 08 Sep 2021 07:05:13 -0700 (PDT) Original-Received: from ars3 ([2a02:8109:8ac0:56d0::b1d]) by smtp.gmail.com with ESMTPSA id i11sm2292249wrb.48.2021.09.08.07.05.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Sep 2021 07:05:12 -0700 (PDT) In-Reply-To: <877dfr5ww1.fsf@gnus.org> (Lars Ingebrigtsen's message of "Wed, 08 Sep 2021 09:50:22 +0200") Received-SPF: pass client-ip=2a00:1450:4864:20::42b; envelope-from=arstoffel@gmail.com; helo=mail-wr1-x42b.google.com X-Spam_score_int: -1 X-Spam_score: -0.2 X-Spam_bar: / X-Spam_report: (-0.2 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:274344 Archived-At: --=-=-= Content-Type: text/plain On Wed, 8 Sep 2021 at 09:50, Lars Ingebrigtsen wrote: > Augusto Stoffel writes: > >> I have also rearranged things a bit so that the setup code is sent to >> the inferior process just once, rather than of on every call to >> `python-shell-send-string'. This way, the smaller line length limit >> doesn't increase too much the use of temp files, which, as I mentioned, >> is slow over ssh. > > This leads to compilation warnings about > python-shell-eval-file-setup-code etc. > > Can you adjust the patch to fix that, and also add the defconst for > 1024, like Eli discussed? > >> PS: I have some more suggestions around the Python shell. Is the ideal >> workflow to keep creating bugs with a small patch to each improvement, >> or do you prefer to review a larger collection of changes bundled >> together? > > Smaller patches are fine by me, and you can submit them with `M-x > submit-emacs-patch'. Hi Lars, I've attached a new patch addressing these 2 points. I've added the max line length defconst to comint.el, since this something general about PTYs. The value reflects my knowledge of the situation (4096 on Linux, possibly 1024 on other OSes). Hopefully other people will fill in more cases eventually. FWIW, when I read the man page for termios, it seems pretty clear that MAX_CANON is the name of this parameter we're discussing here. But if I do 'getconf MAX_CANON /', I get 255, which is definitely not the right value for my system. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Better-treatment-of-line-length-limits-for-the-Pytho.patch >From 3eab275464f4a53bf86153f5f3693e522a240b28 Mon Sep 17 00:00:00 2001 From: Augusto Stoffel Date: Mon, 6 Sep 2021 23:34:48 +0200 Subject: [PATCH] Better treatment of line length limits for the Python inferior * lisp/comint.el (comint-max-line-length): new constant reflecting a safe maximum line size that can be sent to an inferior process. * lisp/progmodes/python.el (python-shell-comint-watch-for-first-prompt-output-filter): Send setup code to the inferior process only once and at this stage. (python-shell-eval-setup-code, python-shell-eval-file-setup-code): Move, unchanged, to an earlier point to avoid byte-compiler warnings. (python-shell-send-string-no-output): Revert changes of e32c7d2a8d (python-shell-send-string): Use 'comint-max-line-length' to decide when to resort to temp files. (python-shell-send-string, python-shell-send-file): Don't send setup code each time. --- lisp/comint.el | 6 +++ lisp/progmodes/python.el | 98 +++++++++++++++++++++------------------- 2 files changed, 57 insertions(+), 47 deletions(-) diff --git a/lisp/comint.el b/lisp/comint.el index e058e6b8cf..c2e528a5d5 100644 --- a/lisp/comint.el +++ b/lisp/comint.el @@ -479,6 +479,12 @@ comint-terminfo-terminal :group 'comint :version "26.1") +(defconst comint-max-line-length + (pcase system-type + ('gnu/linux 4096) + (_ 1024)) + "Maximum line length, in bytes, accepted by the inferior process.") + (defvar comint-mode-map (let ((map (make-sparse-keymap))) ;; Keys: diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el index d8ec032402..4f222b4cf5 100644 --- a/lisp/progmodes/python.el +++ b/lisp/progmodes/python.el @@ -2811,6 +2811,44 @@ python-shell-first-prompt-hook :type 'hook :group 'python) +(defconst python-shell-eval-setup-code + "\ +def __PYTHON_EL_eval(source, filename): + import ast, sys + if sys.version_info[0] == 2: + from __builtin__ import compile, eval, globals + else: + from builtins import compile, eval, globals + sys.stdout.write('\\n') + try: + p, e = ast.parse(source, filename), None + except SyntaxError: + t, v, tb = sys.exc_info() + sys.excepthook(t, v, tb.tb_next) + return + if p.body and isinstance(p.body[-1], ast.Expr): + e = p.body.pop() + try: + g = globals() + exec(compile(p, filename, 'exec'), g, g) + if e: + return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g) + except Exception: + t, v, tb = sys.exc_info() + sys.excepthook(t, v, tb.tb_next)" + "Code used to evaluate statements in inferior Python processes.") + +(defconst python-shell-eval-file-setup-code + "\ +def __PYTHON_EL_eval_file(filename, tempname, encoding, delete): + import codecs, os + with codecs.open(tempname or filename, encoding=encoding) as file: + source = file.read().encode(encoding) + if delete and tempname: + os.remove(tempname) + return __PYTHON_EL_eval(source, filename)" + "Code used to evaluate files in inferior Python processes.") + (defun python-shell-comint-watch-for-first-prompt-output-filter (output) "Run `python-shell-first-prompt-hook' when first prompt is found in OUTPUT." (when (not python-shell--first-prompt-received) @@ -2826,6 +2864,15 @@ python-shell-comint-watch-for-first-prompt-output-filter (setq python-shell--first-prompt-received-output-buffer nil) (setq-local python-shell--first-prompt-received t) (setq python-shell--first-prompt-received-output-buffer nil) + (cl-letf (((symbol-function 'python-shell-send-string) + (lambda (string process) + (comint-send-string + process + (format "exec(%s)\n" (python-shell--encode-string string)))))) + ;; Bootstrap: the normal definition of `python-shell-send-string' + ;; depends on the Python code sent here. + (python-shell-send-string-no-output python-shell-eval-setup-code) + (python-shell-send-string-no-output python-shell-eval-file-setup-code)) (with-current-buffer (current-buffer) (let ((inhibit-quit nil)) (run-hooks 'python-shell-first-prompt-hook)))))) @@ -3081,33 +3128,6 @@ python-shell--save-temp-file (delete-trailing-whitespace)) temp-file-name)) -(defconst python-shell-eval-setup-code - "\ -def __PYTHON_EL_eval(source, filename): - import ast, sys - if sys.version_info[0] == 2: - from __builtin__ import compile, eval, globals - else: - from builtins import compile, eval, globals - sys.stdout.write('\\n') - try: - p, e = ast.parse(source, filename), None - except SyntaxError: - t, v, tb = sys.exc_info() - sys.excepthook(t, v, tb.tb_next) - return - if p.body and isinstance(p.body[-1], ast.Expr): - e = p.body.pop() - try: - g = globals() - exec(compile(p, filename, 'exec'), g, g) - if e: - return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g) - except Exception: - t, v, tb = sys.exc_info() - sys.excepthook(t, v, tb.tb_next)" - "Code used to evaluate statements in inferior Python processes.") - (defalias 'python-shell--encode-string (let ((fun (if (and (fboundp 'json-serialize) (>= emacs-major-version 28)) @@ -3128,12 +3148,11 @@ python-shell-send-string (interactive (list (read-string "Python command: ") nil t)) (let ((process (or process (python-shell-get-process-or-error msg))) - (code (format "exec(%s);__PYTHON_EL_eval(%s, %s)\n" - (python-shell--encode-string python-shell-eval-setup-code) + (code (format "__PYTHON_EL_eval(%s, %s)\n" (python-shell--encode-string string) (python-shell--encode-string (or (buffer-file-name) ""))))) - (if (<= (string-bytes code) 4096) + (if (<= (string-bytes code) comint-max-line-length) (comint-send-string process code) (let* ((temp-file-name (with-current-buffer (process-buffer process) (python-shell--save-temp-file string))) @@ -3180,8 +3199,7 @@ python-shell-send-string-no-output (inhibit-quit t)) (or (with-local-quit - (comint-send-string - process (format "exec(%s)\n" (python-shell--encode-string string))) + (python-shell-send-string string process) (while python-shell-output-filter-in-progress ;; `python-shell-output-filter' takes care of setting ;; `python-shell-output-filter-in-progress' to NIL after it @@ -3378,18 +3396,6 @@ python-shell-send-defun nil ;; noop msg)))) - -(defconst python-shell-eval-file-setup-code - "\ -def __PYTHON_EL_eval_file(filename, tempname, encoding, delete): - import codecs, os - with codecs.open(tempname or filename, encoding=encoding) as file: - source = file.read().encode(encoding) - if delete and tempname: - os.remove(tempname) - return __PYTHON_EL_eval(source, filename)" - "Code used to evaluate files in inferior Python processes.") - (defun python-shell-send-file (file-name &optional process temp-file-name delete msg) "Send FILE-NAME to inferior Python PROCESS. @@ -3419,9 +3425,7 @@ python-shell-send-file (comint-send-string process (format - "exec(%s);exec(%s);__PYTHON_EL_eval_file(%s, %s, %s, %s)\n" - (python-shell--encode-string python-shell-eval-setup-code) - (python-shell--encode-string python-shell-eval-file-setup-code) + "__PYTHON_EL_eval_file(%s, %s, %s, %s)\n" (python-shell--encode-string file-name) (python-shell--encode-string (or temp-file-name "")) (python-shell--encode-string (symbol-name encoding)) -- 2.31.1 --=-=-=--