From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Richard Sharman Newsgroups: gmane.emacs.bugs Subject: bug#45951: 27.1; comint-redirect-results-list-from-process sometimes fails to skip over command if echoed Date: Sun, 17 Jan 2021 18:51:28 -0800 Message-ID: <8CD405D0-5936-4D36-A700-F753B428A5C3@pobox.com> Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.7\)) Content-Type: multipart/alternative; boundary="Apple-Mail=_1A510B1A-C5EB-4900-8A5D-0C8BA8656847" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="27174"; mail-complaints-to="usenet@ciao.gmane.io" To: 45951@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Mon Jan 18 08:04:54 2021 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l1Oas-0006vK-Aq for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 18 Jan 2021 08:04:54 +0100 Original-Received: from localhost ([::1]:42206 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l1Oar-00026Y-3I for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 18 Jan 2021 02:04:53 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:37596) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l1Oa3-0001iQ-Lv for bug-gnu-emacs@gnu.org; Mon, 18 Jan 2021 02:04:03 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:34631) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l1Oa3-0000uJ-Ds for bug-gnu-emacs@gnu.org; Mon, 18 Jan 2021 02:04:03 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l1Oa3-0004rM-9x for bug-gnu-emacs@gnu.org; Mon, 18 Jan 2021 02:04:03 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Richard Sharman Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 18 Jan 2021 07:04:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 45951 X-GNU-PR-Package: emacs X-Debbugs-Original-To: bug-gnu-emacs@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.161095341718617 (code B ref -1); Mon, 18 Jan 2021 07:04:03 +0000 Original-Received: (at submit) by debbugs.gnu.org; 18 Jan 2021 07:03:37 +0000 Original-Received: from localhost ([127.0.0.1]:46173 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1OZd-0004qC-2R for submit@debbugs.gnu.org; Mon, 18 Jan 2021 02:03:37 -0500 Original-Received: from lists.gnu.org ([209.51.188.17]:35922) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Kdk-000732-Dr for submit@debbugs.gnu.org; Sun, 17 Jan 2021 21:51:37 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:47554) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l1Kdk-0006NY-1H for bug-gnu-emacs@gnu.org; Sun, 17 Jan 2021 21:51:36 -0500 Original-Received: from mx4.ncf.ca ([172.83.172.18]:54912) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l1Kdh-00066y-Qi for bug-gnu-emacs@gnu.org; Sun, 17 Jan 2021 21:51:35 -0500 Original-Received: from localhost (localhost [127.0.0.1]) by mx4.ncf.ca (Postfix) with ESMTP id CFD8316B83 for ; Sun, 17 Jan 2021 21:51:32 -0500 (EST) X-Virus-Scanned: Ubuntu amavisd-new at Original-Received: from mx4.ncf.ca ([IPv6:::1]) by localhost (mx4.ncf.ca [IPv6:::1]) (amavisd-new, port 10024) with ESMTP id PDnxZMItpR6F for ; Sun, 17 Jan 2021 21:51:31 -0500 (EST) Original-Received: from mail.ncf.ca (mail.ncf.ca [172.83.172.45]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx4.ncf.ca (Postfix) with ESMTPS id 440FA16B1A for ; Sun, 17 Jan 2021 21:51:31 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.11.0 mx4.ncf.ca 440FA16B1A Original-Received: from [192.168.0.103] (unknown [172.103.218.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: fo019@ncf.ca) by mail.ncf.ca (Postfix) with ESMTPSA id C988125D19 for ; Sun, 17 Jan 2021 21:51:30 -0500 (EST) X-Mailer: Apple Mail (2.3445.9.7) Received-SPF: neutral client-ip=172.83.172.18; envelope-from=rsharman@pobox.com; helo=mx4.ncf.ca X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.779 autolearn=no autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Mon, 18 Jan 2021 02:03:35 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:198074 Archived-At: --Apple-Mail=_1A510B1A-C5EB-4900-8A5D-0C8BA8656847 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 When function comint-redirect-results-list-from-process is examining the output buffer of the command it attempts to skip past the command if it was echoed. However, this test uses looking-at and so will fail if the command contains characters with special meansing to regexps. It can also fail if the command contains a trailing newline character but the buffer may have a control-M character. The latter problem can be fixed by calling comint-carriage-motion before the test is done; the former by looking-at (regexp-quote command) instead of just at command. Note that bug 42662 (27.1; `comint-redirect-results-list-from-process=E2=80= =99=20 treats COMMAND as a regexp) is another symptom of the former problem. Example code to reproduce the problem: (defun example () (interactive) ;; create shell if not already created (save-window-excursion (shell)) (let ((p (get-process "shell")) command reply) (process-send-string p "irb\n") (setq command "puts File.atime(\"example.el\") if = File.exist?(\"example.el\")\n") (setq reply (comint-redirect-results-list-from-process p command = "^.*$" 0)) (process-send-string p "quit\n") ;; cleanup! (message (car reply)) ;; this should be the a time of the file reply )) File example.el creates a shell process then runs =E2=80=9Cirb=E2=80=9D, = the ruby interactive shell, then sends it a command with question-marks in it. The first line returned should be the atime of the file. Patch: --- comint.el.orig 2021-01-17 14:22:17.000000000 -0800 +++ comint.el 2021-01-17 16:49:47.000000000 -0800 @@ -3833,9 +3833,11 @@ (accept-process-output process))) ;; Collect the output (set-buffer output-buffer) + (unless comint-inhibit-carriage-motion + (comint-carriage-motion (point-min)(point-max))) (goto-char (point-min)) ;; Skip past the command, if it was echoed - (and (looking-at command) + (and (looking-at (regexp-quote command)) (forward-line)) (while (and (not (eobp)) (re-search-forward regexp nil t)) Example output in *scratch* buffer before the patch: (example) ("irb " "puts File.atime(\"example.el\") if File.exist?(\"example.el\") " " " "") and after patch applied: (example) ("2021-01-17 18:33:45 -0800" "nil" "") In GNU Emacs 27.1 (build 2, x86_64-apple-darwin17.7.0, NS appkit-1561.61 = Version 10.13.6 (Build 17G14033)) of 2020-12-13 built on henry.local Windowing system distributor 'Apple', version 10.3.1561 System Description: Mac OS X 10.13.6 Recent messages: 2021-01-17 18:33:45 -0800 Result: "2021-01-17 18:33:45 -0800" ("2021-01-17 18:33:45 -0800" "nil" "") Result: ("2021-01-17 18:33:45 -0800" "nil" "") Result: ("2021-01-17 18:33:45 -0800" "nil" "") Quit Type C-x 4 C-o RET to restore the other window. =20 Quit Configured features: NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES THREADS JSON PDUMPER GMP Important settings: value of $LANG: en_CA.UTF-8 locale-coding-system: utf-8-unix Major mode: Shell Minor modes in effect: shell-dirtrack-mode: t tooltip-mode: t global-eldoc-mode: t electric-indent-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t line-number-mode: t transient-mark-mode: t Load-path shadows: None found. Features: (shadow sort mail-extr emacsbug sendmail org-timer org-colview org-clock org-attach org-id org-archive org-agenda ol-eww ol-rmail ol-mhe ol-irc ol-info ol-gnus nnir gnus-sum url url-proxy url-privacy url-expand url-methods url-history mailcap shr url-cookie url-domsuf url-util url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs json map url-vars svg xml dom browse-url gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo parse-time iso8601 gnus-spec gnus-int gnus-range message rmc puny rfc822 mml mml-sec password-cache epa epg epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader gnus-win gnus nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums text-property-search seq byte-opt bytecomp byte-compile cconv mail-utils mm-util mail-prsvr wid-edit ol-docview doc-view jka-compr image-mode exif dired dired-loaddefs ol-bibtex bibtex ol-bbdb ol-w3m reporter org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete org-list org-faces org-entities noutline outline easy-mmode org-version ob-emacs-lisp ob-core ob-eval org-table ol org-keys org-compat advice org-macs org-loaddefs format-spec cal-menu calendar cal-loaddefs apropos edebug gv derived time-date subr-x help-fns radix-tree cl-print debug backtrace help-mode easymenu find-func cl-loaddefs cl-lib shell pcomplete comint ansi-color ring tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/ns-win ns-win ucs-normalize mule-util term/common-win tool-bar dnd fontset image regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock syntax facemenu font-core term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese composite charscript charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs button faces cus-face macroexp files text-properties overlay sha1 md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote threads kqueue cocoa ns multi-tty make-network-process emacs) Memory information: ((conses 16 179913 11829) (symbols 48 19738 1) (strings 32 63869 1510) (string-bytes 1 2209108) (vectors 16 29528) (vector-slots 8 338974 19272) (floats 8 213 69) (intervals 56 717 0) (buffers 1000 19)) --Apple-Mail=_1A510B1A-C5EB-4900-8A5D-0C8BA8656847 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
When function = comint-redirect-results-list-from-process is examining the
output buffer of the command it attempts to skip past the = command if it
was echoed.

However, this test uses looking-at and = so will fail if the command
contains characters = with special meansing to regexps. It can also fail
if= the command contains a trailing newline character but the
buffer may have a control-M character.
The latter problem can be fixed by = calling comint-carriage-motion before
the test is = done; the former by looking-at (regexp-quote command)
instead of just at command.

Note that bug 42662 (27.1; = `comint-redirect-results-list-from-process=E2=80=99 
treats COMMAND as a regexp) is another symptom of the former = problem.


Example code to reproduce the = problem:

(defun example ()
  = (interactive)
  ;; create shell if not already = created
  (save-window-excursion
    (shell))
  (let ((p = (get-process "shell"))
command
= reply)
    (process-send-string p = "irb\n")
    (setq command "puts = File.atime(\"example.el\") if File.exist?(\"example.el\")\n")
    (setq reply = (comint-redirect-results-list-from-process p command "^.*$" = 0))
    (process-send-string p = "quit\n") = ;; cleanup!
    (message (car = reply)) = ;; this should be the a time of the file
    reply
    = ))

File = example.el creates a shell process then runs =E2=80=9Cirb=E2=80=9D, the = ruby
interactive shell, then sends it a command = with question-marks in it.
The first line returned = should be the atime of the file.


Patch:

--- comint.el.orig 2021-01-17 14:22:17.000000000 = -0800
+++ comint.el 2021-01-17 16:49:47.000000000 = -0800
@@ -3833,9 +3833,11 @@
 =  (accept-process-output = process)))
       ;; Collect = the output
       (set-buffer = output-buffer)
+      (unless = comint-inhibit-carriage-motion
+ = (comint-carriage-motion (point-min)(point-max)))
       (goto-char (point-min))
       ;; Skip past the command, if it = was echoed
-      (and (looking-at = command)
+      (and (looking-at = (regexp-quote command))
    = (forward-line))
       (while = (and (not (eobp))
  =  (re-search-forward regexp nil t))



Example output in *scratch* = buffer before the patch:

(example)
("irb
" "puts File.atime(\"example.el\") if = File.exist?(\"example.el\")
" "
" "")

and after patch = applied:

(example)
("2021-01-17 = 18:33:45 -0800" "nil" "")


In GNU Emacs 27.1 (build 2, = x86_64-apple-darwin17.7.0, NS appkit-1561.61 Version 10.13.6 (Build = 17G14033))
of 2020-12-13 built on henry.local
Windowing system distributor 'Apple', version 10.3.1561
System Description:  Mac OS X 10.13.6

Recent messages:
2021-01-17 = 18:33:45 -0800
Result: "2021-01-17 18:33:45 -0800"
("2021-01-17 18:33:45 -0800" "nil" "")

Result: ("2021-01-17 18:33:45 -0800" "nil" "")

Result: ("2021-01-17 18:33:45 -0800" "nil" = "")

Quit
Type C-x 4 C-o RET = to restore the other window. 
Quit
Configured features:
NOTIFY KQUEUE ACL GNUTLS = LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES
THREADS JSON = PDUMPER GMP

Important settings:
  value of = $LANG: en_CA.UTF-8
  locale-coding-system: = utf-8-unix

Major mode: Shell

Minor modes in effect:
  shell-dirtrack-mode: t
  = tooltip-mode: t
  global-eldoc-mode: t
  = electric-indent-mode: t
  mouse-wheel-mode: t
  = tool-bar-mode: t
  menu-bar-mode: t
  = file-name-shadow-mode: t
  global-font-lock-mode: = t
  = font-lock-mode: t
  blink-cursor-mode: t
  = auto-composition-mode: t
  auto-encryption-mode: t
  = auto-compression-mode: t
  line-number-mode: t
  = transient-mark-mode: t

Load-path = shadows:
None found.

Features:
(shadow sort mail-extr emacsbug = sendmail org-timer org-colview org-clock
org-attach org-id = org-archive org-agenda ol-eww ol-rmail ol-mhe ol-irc
ol-info= ol-gnus nnir gnus-sum url url-proxy url-privacy url-expand
url-methods url-history mailcap shr url-cookie url-domsuf = url-util
url-parse auth-source cl-seq eieio eieio-core = cl-macs eieio-loaddefs
json map url-vars svg xml dom = browse-url gnus-group gnus-undo gnus-start
gnus-cloud = nnimap nnmail mail-source utf7 netrc nnoo parse-time iso8601
gnus-spec gnus-int gnus-range message rmc puny rfc822 mml = mml-sec
password-cache epa epg epg-config mm-decode = mm-bodies mm-encode
mail-parse rfc2231 mailabbrev = gmm-utils mailheader gnus-win gnus
nnheader gnus-util = rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
text-property-search seq byte-opt bytecomp byte-compile cconv = mail-utils
mm-util mail-prsvr wid-edit ol-docview doc-view = jka-compr image-mode
exif dired dired-loaddefs ol-bibtex = bibtex ol-bbdb ol-w3m reporter org
ob ob-tangle ob-ref = ob-lob ob-table ob-exp org-macro org-footnote
org-src = ob-comint org-pcomplete org-list org-faces org-entities noutline
outline easy-mmode org-version ob-emacs-lisp ob-core ob-eval = org-table
ol org-keys org-compat advice org-macs = org-loaddefs format-spec cal-menu
calendar cal-loaddefs = apropos edebug gv derived time-date subr-x
help-fns = radix-tree cl-print debug backtrace help-mode easymenu
find-func cl-loaddefs cl-lib shell pcomplete comint = ansi-color ring
tooltip eldoc electric uniquify ediff-hook = vc-hooks lisp-float-type
mwheel term/ns-win ns-win = ucs-normalize mule-util term/common-win
tool-bar dnd = fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register = page
tab-bar menu-bar rfn-eshadow isearch timer select = scroll-bar mouse
jit-lock font-lock syntax facemenu = font-core term/tty-colors frame
minibuffer cl-generic cham = georgian utf-8-lang misc-lang vietnamese
tibetan thai = tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic = chinese
composite charscript charprop case-table epa-hook = jka-cmpr-hook help
simple abbrev obarray cl-preloaded = nadvice loaddefs button faces
cus-face macroexp files = text-properties overlay sha1 md5 base64 format
env = code-pages mule custom widget hashtable-print-readable backquote
threads kqueue cocoa ns multi-tty make-network-process = emacs)

Memory information:
((conses 16 179913 11829)
(symbols 48 19738 = 1)
(strings 32 63869 1510)
(string-bytes 1 = 2209108)
(vectors 16 29528)
(vector-slots = 8 338974 19272)
(floats 8 213 69)
= (intervals 56 717 0)
(buffers 1000 19))
= --Apple-Mail=_1A510B1A-C5EB-4900-8A5D-0C8BA8656847--