From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Carlos Pita Newsgroups: gmane.emacs.bugs Subject: bug#32390: 26.1; python.el: cleanup font lock buffer after input is sent Date: Tue, 4 Dec 2018 19:35:09 -0300 Message-ID: References: <87sh3q1075.fsf@gmail.com> <87woshvk7a.fsf@gmail.com> <87va7prkeq.fsf@gmail.com> <877ejzgy2a.fsf@gmail.com> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="00000000000092f301057c39e13c" X-Trace: blaine.gmane.org 1543963465 29901 195.159.176.226 (4 Dec 2018 22:44:25 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 4 Dec 2018 22:44:25 +0000 (UTC) Cc: 32390@debbugs.gnu.org To: Noam Postavsky Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Dec 04 23:44:20 2018 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUJQS-0007cS-JS for geb-bug-gnu-emacs@m.gmane.org; Tue, 04 Dec 2018 23:44:20 +0100 Original-Received: from localhost ([::1]:59242 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gUJSZ-0004SQ-56 for geb-bug-gnu-emacs@m.gmane.org; Tue, 04 Dec 2018 17:46:31 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:58022) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gUJRi-000458-VS for bug-gnu-emacs@gnu.org; Tue, 04 Dec 2018 17:45:42 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gUJIR-0002FM-2P for bug-gnu-emacs@gnu.org; Tue, 04 Dec 2018 17:36:06 -0500 Original-Received: from debbugs.gnu.org ([208.118.235.43]:56865) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gUJIQ-0002EI-Qi for bug-gnu-emacs@gnu.org; Tue, 04 Dec 2018 17:36:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1gUJIQ-0007tk-7B for bug-gnu-emacs@gnu.org; Tue, 04 Dec 2018 17:36:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Carlos Pita Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 04 Dec 2018 22:36:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 32390 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: confirmed Original-Received: via spool by 32390-submit@debbugs.gnu.org id=B32390.154396293030320 (code B ref 32390); Tue, 04 Dec 2018 22:36:02 +0000 Original-Received: (at 32390) by debbugs.gnu.org; 4 Dec 2018 22:35:30 +0000 Original-Received: from localhost ([127.0.0.1]:32890 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUJHu-0007sy-Dz for submit@debbugs.gnu.org; Tue, 04 Dec 2018 17:35:30 -0500 Original-Received: from mail-yw1-f44.google.com ([209.85.161.44]:33679) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gUJHs-0007sl-Mj for 32390@debbugs.gnu.org; Tue, 04 Dec 2018 17:35:29 -0500 Original-Received: by mail-yw1-f44.google.com with SMTP id q11so7710212ywa.0 for <32390@debbugs.gnu.org>; Tue, 04 Dec 2018 14:35:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=0YHTIj0la/m8gdv7DVW5i7+WlO96Fm3pE1V+0/zR3Co=; b=DKwWE2T+wOPAn7es2luW79/zptSKBUSmn4VqvF8pj4e2aOkgWDYhBLjgTCuGUDWHzE ogLsmwP2eC9MhgQVJ1yX+Tq3QRs9HtfiOzE+1E4YZd1meZXzjEUvvTnuVByoQriRyaLh N0yhk4z00iZ/jF1H5xTwYiVj6JPo9iJSztnuLr6sjgmjQ7DF9Z8Nub8h01abztlpEwLJ RLsKAkx0HkKheeOeg+uVDBlqfYlI7re6AJBFXfRbe/3HLyyfrlkPf6tTleHambcIsG2V HWxIwlMXGdoJJpAjwDEuqgsY8GqkYMkPQ9AB+NCOpaJeKZa5l7uxOPDhq9VN6jIQc2nC Aahg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=0YHTIj0la/m8gdv7DVW5i7+WlO96Fm3pE1V+0/zR3Co=; b=FX0PT7JqkJltgugX/DutfU2P0Y/Lg6F2KwrV9hXuFK6TjUuQnHPm9jSIpU5O5Kk26S T6D/G2vbuSRZEvHbuOyfGWpxistGyxHn3Ud9uWiIKxiUSi7ewJr2W7kWfnhC+PgMSWXk Efaml9N9S/9oS244FK/UYRbZxSFCs6DzS3O168ZehP84IDCZ/IK09nzkGoUyEkOQbNjQ EQq2whu+0NNLsQla6dpw1iH7Q8f7QmriV6g5eXH6VzPi13xW3oSC42eZNj7H4ytugBvQ H4Jb47NxJGhaH7fkYk8q0XExmuRJO3PKGR/wHSQ0WIpyzW7BDbscGdwuD3xcTIPmG8Gt iijg== X-Gm-Message-State: AA+aEWaM51aFyVw9d9T//4W+AVrY0OxgfyYfVoWFly2k1/04HaQpYSPj iwgd0zg9oSZG4XKLUzPj2SjkujXEU9oHLwPyUrA= X-Google-Smtp-Source: AFSGD/WxugC0hE3P7b30HNjMgm9nANJnuylEKbwlz/+7IT+M1jJNfsJ8chnA85lBY+1CXVEVjR/AQRb/G3MSPCLB8PA= X-Received: by 2002:a81:d84c:: with SMTP id n12mr21970914ywl.280.1543962923024; Tue, 04 Dec 2018 14:35:23 -0800 (PST) In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:153085 Archived-At: --00000000000092f301057c39e13c Content-Type: text/plain; charset="UTF-8" Hi Noam, I've been carefully studying this issue and below I explain in detail what is really happening and what different cases are being (sometimes wrongly) covered. There is a patch attached, I hope you consider it worth of being applied. The condition for cleaning up of the font lock buffer is: ------------- (if (and (not (string= "" output)) (1) ;; Is end of output and is not just a prompt. (not (member (python-shell-comint-end-of-output-p (ansi-color-filter-apply output)) '(nil (2) 0)))) (3) ------------- First nota that (1) is really irrelevant since (python-shell-comint-end-of-output-p "") returns nil anyway. Now, the problem originating this report was: ------------- In [15]: " File "", line 1 " ^ SyntaxError: EOL while scanning string literal In [16]: string face still here" ------------- This happens because python-shell-font-lock-comint-output-filter-function is called twice, first for the error output and then for the "In [16]: " part (I assume this is because one part is coming from stderr and the other from stdout, but that's just a guess). The first time (2) applies since we're *not* at the end of an input prompt. The second time (3) applies since we're at the end of *just* and input prompt. So in any case the buffer is cleaned up. Now, my first reaction was to ignore the *just* part: what damage could it do to just check if we're at the end of an input prompt disregarding the fact that it could be the only thing there? Well, the problem is with multiline input, of course. Nevertheless the current code is relying in a very weak rule: it considers "just an input prompt" to be a continuation prompt. Another unreliable aspect of the current rule is that sometimes (python-shell-comint-end-of-output-p (ansi-color-filter-apply output)) returns 1 and not 0 for continuation prompts. In short, the rule does a very poor job identifying continuations. So, all in all I had rewritten (in a previous post) the condition above as: ------------- (if (and (python-shell-comint-end-of-output-p (ansi-color-filter-apply output)) (not (string-match "\\.\\.\\.: $" output))) (4) ------------- Where: - Clause (1) is disregarded because it's redundant. - Clause (2) is taken into account. - Clause (3) is disregarded because it's unreliable. - Clause (4) was added to address the multiline input case (continuation prompt). Now, it's a sad fact that python-shell-prompt-input-regexps doesn't distinguish between initial and continuation input prompts, so I explicitly had to put that particular regexp there. I assume this is the main reason while my fix was not yet accepted, isn't it? At this point we have at least two alternatives: a) Consider that an input prompt that includes the pattern "\\.\\.\\." is a continuation prompt. This is a heuristic but I think it's robust enough. I favor this solution and the attached patch implements it. b) Add a new customization option with a list of continuation prompts. I believe this would be too much. So the attached patch implements this new rule: ------------- (if (let ((output (ansi-color-filter-apply output))) (and (python-shell-comint-end-of-output-p output) (not (string-match "\\.\\.\\." output)))) (5) ----------- The difference between (4) and (5) is that (5) relaxes the match to just include three sequential dots (because we already know we have an input prompt at the end of the output!). I've been more careful by matching on the filtered string instead of the raw one also. Please let me know if you prefer option (b) instead. Best regards -- Carlos --00000000000092f301057c39e13c Content-Type: text/x-patch; charset="US-ASCII"; name="font-lock-cleanup-filter.diff" Content-Disposition: attachment; filename="font-lock-cleanup-filter.diff" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_jpabmhoi0 ZGlmZiAtLWdpdCBhL2xpc3AvcHJvZ21vZGVzL3B5dGhvbi5lbCBiL2xpc3AvcHJvZ21vZGVzL3B5 dGhvbi5lbAppbmRleCBjN2JiMmQ5Li4yMThjNmU0IDEwMDY0NAotLS0gYS9saXNwL3Byb2dtb2Rl cy9weXRob24uZWwKKysrIGIvbGlzcC9wcm9nbW9kZXMvcHl0aG9uLmVsCkBAIC0yNTU2LDE0ICsy NTU2LDExIEBAIHB5dGhvbi1zaGVsbC1mb250LWxvY2stY2xlYW51cC1idWZmZXIKIAogKGRlZnVu IHB5dGhvbi1zaGVsbC1mb250LWxvY2stY29taW50LW91dHB1dC1maWx0ZXItZnVuY3Rpb24gKG91 dHB1dCkKICAgIkNsZWFuIHVwIHRoZSBmb250LWxvY2sgYnVmZmVyIGFmdGVyIGFueSBPVVRQVVQu IgotICAoaWYgKGFuZCAobm90IChzdHJpbmc9ICIiIG91dHB1dCkpCi0gICAgICAgICAgIDs7IElz IGVuZCBvZiBvdXRwdXQgYW5kIGlzIG5vdCBqdXN0IGEgcHJvbXB0LgotICAgICAgICAgICAobm90 IChtZW1iZXIKLSAgICAgICAgICAgICAgICAgKHB5dGhvbi1zaGVsbC1jb21pbnQtZW5kLW9mLW91 dHB1dC1wCi0gICAgICAgICAgICAgICAgICAoYW5zaS1jb2xvci1maWx0ZXItYXBwbHkgb3V0cHV0 KSkKLSAgICAgICAgICAgICAgICAgJyhuaWwgMCkpKSkKLSAgICAgIDs7IElmIG91dHB1dCBpcyBv dGhlciB0aGFuIGFuIGlucHV0IHByb21wdCB0aGVuICJyZWFsIiBvdXRwdXQgaGFzCi0gICAgICA7 OyBiZWVuIHJlY2VpdmVkIGFuZCB0aGUgZm9udC1sb2NrIGJ1ZmZlciBtdXN0IGJlIGNsZWFuZWQg dXAuCisgIChpZiAobGV0ICgob3V0cHV0IChhbnNpLWNvbG9yLWZpbHRlci1hcHBseSBvdXRwdXQp KSkKKyAgICAgICAgKGFuZCAocHl0aG9uLXNoZWxsLWNvbWludC1lbmQtb2Ytb3V0cHV0LXAgb3V0 cHV0KQorICAgICAgICAgICAgIChub3QgKHN0cmluZy1tYXRjaCAiXFwuXFwuXFwuIiBvdXRwdXQp KSkpCisgICAgICA7OyBJZiBvdXRwdXQgZW5kcyB3aXRoIGFuIGluaXRpYWwgKG5vdCBjb250aW51 YXRpb24pIGlucHV0IHByb21wdAorICAgICAgOzsgdGhlbiB0aGUgZm9udC1sb2NrIGJ1ZmZlciBt dXN0IGJlIGNsZWFuZWQgdXAuCiAgICAgICAocHl0aG9uLXNoZWxsLWZvbnQtbG9jay1jbGVhbnVw LWJ1ZmZlcikKICAgICA7OyBPdGhlcndpc2UganVzdCBhZGQgYSBuZXdsaW5lLgogICAgIChweXRo b24tc2hlbGwtZm9udC1sb2NrLXdpdGgtZm9udC1sb2NrLWJ1ZmZlcgo= --00000000000092f301057c39e13c--