From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id QLfaD2mQ/mUB4AAA62LTzQ:P1 (envelope-from ) for ; Sat, 23 Mar 2024 09:18:49 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id QLfaD2mQ/mUB4AAA62LTzQ (envelope-from ) for ; Sat, 23 Mar 2024 09:18:49 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=DUzvCfL6; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Seal: i=2; s=key1; d=yhetil.org; t=1711181929; a=rsa-sha256; cv=pass; b=qQpobIQqDmsFld6fRNPhvOno+L4ILq9O2xWMl4/Ugq+4m5JL4SyWGOOhkrUxqzmvYwcEIo cWbUQQbiUFLNYDGT0oJWncQYuJ3/cgnpIkzyLmcnjlX2Z3hWMbE2SAkUC+kAlqCLCG3UbI cZ6oZ6b8heHP623chVWktflw0/lpLFgz8DIqWp51wBpcOqqcJJSTev+/ayKD5uYo+D8Cnj fi5jJHwiDNX+XgupYprxGSJzznGF4NAeFMVlNyb5ZnaMfO1IInomVIkRJzqZoG5jR9e3fN 61D+b+ScTricxF96+FePClQZEE3pgKLghepzODHSk9mmTbQDsLBLhKq2oT4n5Q== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=DUzvCfL6; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1711181929; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=+DT1qhGRI/uV3KfQieyA3XmAeW8ylZtDkSiqNjdOMX0=; b=V/7iWghzShfSs40bpoXpNkfqigmuZpdXcIYJuNq7sn9KgjxQMwVxpQvO4/XPcQAJci8FK1 uh6hcamlHX8Q+Iio55gGN/Mcg411ul1qOeipF7MwqQFJX0+TG7yeleKbgJA4/E/SiZcIX5 8tMq2W3HXW27JKWQ/4G8oo/EsO2JZby7/TmVNPtOGM2hiSy123APdhpEhA1eS2lPkdt1nI BkF+QQ/TWnL0mYBgcyL3bcmC257WDSDTiHsjSUyk2YAphS9tN9CsVI2au6vmwTXE2sfn3H OjEpc9nRvckp4gVqPvkTv1ilZfW4NcGzJZkG07kvdwXIePTTvuJXFoAXf34+nw== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CAB4514639 for ; Sat, 23 Mar 2024 09:18:48 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rnwZb-0007su-AI; Sat, 23 Mar 2024 04:17:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rnwZT-0007qo-Af for emacs-orgmode@gnu.org; Sat, 23 Mar 2024 04:17:43 -0400 Received: from sender4-pp-f112.zoho.com ([136.143.188.112]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rnwZN-0002CB-13 for emacs-orgmode@gnu.org; Sat, 23 Mar 2024 04:17:43 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1711181853; cv=none; d=zohomail.com; s=zohoarc; b=i1EIUsYkHWJ/CqAOQy4NCPpRvoWuZtgwpI5/ITROw61PKDGmaAVJh/XuSgJyiiUGqkHAytnZF32apveLx4bK4Be4o/r8pHbXwMeRuCTyAPeDle6mzcLhYiPneS93kNhBQuj537Sx1i3IIiGdUGschr/FZEMMM6kGRwQ1xdFY79A= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1711181853; h=Content-Type:Cc:Cc:Date:Date:From:From:MIME-Version:Message-ID:Subject:Subject:To:To:Message-Id:Reply-To; bh=+DT1qhGRI/uV3KfQieyA3XmAeW8ylZtDkSiqNjdOMX0=; b=XI9fsh/ckOHptJAHaaTn0z3hF5C+n4LmYfsTgLuhfCzJJwyLTtfKEGe/ZcBogitmz63XkiwhnDgcruuFypZoy22OkidqxaUJbPsTkggsEjkyyJXU+pnR562iEv4fnAhx6YSbfj+ve9+I1EcOOp8INSnyiejIT3qf7RKJTZFop3s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=excalamus.com; spf=pass smtp.mailfrom=matt@excalamus.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1711181853; s=zmail; d=excalamus.com; i=matt@excalamus.com; h=Date:Date:From:From:To:To:Cc:Cc:Message-Id:Message-Id:Subject:Subject:MIME-Version:Content-Type:Reply-To; bh=+DT1qhGRI/uV3KfQieyA3XmAeW8ylZtDkSiqNjdOMX0=; b=DUzvCfL6WNG2DsK9gfMUnBBOzWKdubfm1wamco3ppP5VfcCuOe4pGKqHn7zn8zuG iwJve+6EQ7way3B9xVH8MojJOTZQP1dcKvnpvYENQUGPubep3LwP/p3rRFdU8xpcI/C uMK0cpqGyKTfWfYlIl4opDX0ryPWI9w1XREU4YaA= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1711181851365379.79114778359417; Sat, 23 Mar 2024 01:17:31 -0700 (PDT) Date: Sat, 23 Mar 2024 09:17:31 +0100 From: Matt To: "Ihor Radchenko" Cc: "emacs-orgmode" Message-Id: <18e6a62eabc.1245ceb87329763.6490883406293378702@excalamus.com> Subject: Re: [BUG] Prompt appears in async shell results MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_1079817_1944981950.1711181851324" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Zoho-Virus-Status: 1 X-Zoho-AV-Stamp: zmail-av-1.1.0/211.117.73 Received-SPF: pass client-ip=136.143.188.112; envelope-from=matt@excalamus.com; helo=sender4-pp-f112.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Spam-Score: -7.78 X-Migadu-Queue-Id: CAB4514639 X-Migadu-Scanner: mx12.migadu.com X-Migadu-Spam-Score: -7.78 X-TUID: iXQle3DaEF/6 ------=_Part_1079817_1944981950.1711181851324 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Friendly bump. I saw no comments on the patch part of my previous message and fear it may = have been missed due to my inclusion of the lengthy and separate topic, "bu= g? org-babel-comint-with-output return value type". The patches fix the prompt appearing in async shell results and are ready t= o go, barring the concern I have about being unable to test how the fix may= affect non-shell languages. > ---- On Mon, 19 Feb 2024 12:07:55 +0100 Ihor Radchenko wrote --- > > Matt matt@excalamus.com> writes: > > > > > - Split prompt filtering and input echoing into two filters > > > + this seems to imply a =3D-hook=3D or =3D-functions=3D type imple= mentation > > > > Not necessarily. You may just split the filter into (1) prompt > > remover; (2) body remover. The filter does not need to interact with > > comint buffer and may simply be provided a regexp/string to be removed= . > > Then, the caller will be responsible to supply prompt regexp/body. >=20 > Attached are patches which split the prompt and echo filtering into separ= ate functions. These are then used to refactor =3Dorg-babel-comint-with-ou= tput=3D and to remove the prompt from async results. I also wrote tests, o= ne for the reported bug and two for the filter functions. Tests for the fi= lter functions are in a new file, 'test-ob-comint.el'. >=20 > Two thoughts: >=20 > 1. Can people please test that the changes to =3Dorg-babel-comint-with-ou= tput=3D haven't broken other languages? >=20 > The filter, which I extracted from =3Dorg-babel-comint-with-output=3D, wo= rks according to the test I wrote. The test is based on ob-shell output. = However, =3Dorg-babel-comint-with-output=3D is used by other languages. I = wasn't sure if echoes showed up for other languages and I had no examples f= rom other languages to work from. The filter is untested for anything but = shell, aside from running 'make test' (which returns no unexpected errors).= If no echo argument is given, the filter simply returns the string it wou= ld otherwise try to remove the echo from. So, I suspect the worst outcome = would be that echoes might start showing up in output. But when have my pr= edictions ever been right? :) -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at=C2=A0https://liberapay.com/org-mode ------=_Part_1079817_1944981950.1711181851324 Content-Type: application/octet-stream; name=0001-testing-lisp-test-ob-comint.el-Make-test-for-prompt-.patch Content-Transfer-Encoding: base64 X-ZM_AttachId: 139299970362500120 Content-Disposition: attachment; filename=0001-testing-lisp-test-ob-comint.el-Make-test-for-prompt-.patch RnJvbSA2NmNjNTE0ZWIyN2E5YzUwZmIwMTI4ZmEzMGU1NWI3NjA5YmI5YjE4IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXR0aGV3IFRyemNpbnNraSA8bWF0dEBleGNhbGFtdXMuY29t PgpEYXRlOiBTdW4sIDE3IE1hciAyMDI0IDE2OjQ3OjIyICswMTAwClN1YmplY3Q6IFtQQVRDSCAx LzddIHRlc3RpbmcvbGlzcC90ZXN0LW9iLWNvbWludC5lbDogTWFrZSB0ZXN0IGZvciBwcm9tcHQK IGZpbHRlcgoKKiB0ZXN0LW9iLWNvbWludC5lbDogIE1ha2UgbmV3IGZpbGUgZm9yIGNvbWludCB0 ZXN0cy4KKHRlc3Qtb3JnLWJhYmVsLWNvbWludC9wcm9tcHQtZmlsdGVyLXJlbW92ZXMtcHJvbXB0 KTogVGVzdCB0aGF0IHRoZQpwcm9tcHQgaXMgcmVtb3ZlZCBmcm9tIHByb2Nlc3MgYnVmZmVyIG91 dHB1dC4KLS0tCiB0ZXN0aW5nL2xpc3AvdGVzdC1vYi1jb21pbnQuZWwgfCA1NSArKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrCiAxIGZpbGUgY2hhbmdlZCwgNTUgaW5zZXJ0aW9ucygr KQogY3JlYXRlIG1vZGUgMTAwNjQ0IHRlc3RpbmcvbGlzcC90ZXN0LW9iLWNvbWludC5lbAoKZGlm ZiAtLWdpdCBhL3Rlc3RpbmcvbGlzcC90ZXN0LW9iLWNvbWludC5lbCBiL3Rlc3RpbmcvbGlzcC90 ZXN0LW9iLWNvbWludC5lbApuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAuLjkx OTNlZjhmMwotLS0gL2Rldi9udWxsCisrKyBiL3Rlc3RpbmcvbGlzcC90ZXN0LW9iLWNvbWludC5l bApAQCAtMCwwICsxLDU1IEBACis7OzsgdGVzdC1vYi1jb21pbnQuZWwgIC0qLSBsZXhpY2FsLWJp bmRpbmc6IHQ7IC0qLQorCis7OyBDb3B5cmlnaHQgKGMpIDIwMjQgTWF0dGhldyBUcnpjaW5za2kK Kzs7IEF1dGhvcnM6IE1hdHRoZXcgVHJ6Y2luc2tpCisKKzs7IFRoaXMgZmlsZSBpcyBub3QgcGFy dCBvZiBHTlUgRW1hY3MuCisKKzs7IFRoaXMgcHJvZ3JhbSBpcyBmcmVlIHNvZnR3YXJlOyB5b3Ug Y2FuIHJlZGlzdHJpYnV0ZSBpdCBhbmQvb3IgbW9kaWZ5Cis7OyBpdCB1bmRlciB0aGUgdGVybXMg b2YgdGhlIEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlIGFzIHB1Ymxpc2hlZCBieQorOzsgdGhl IEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbiwgZWl0aGVyIHZlcnNpb24gMyBvZiB0aGUgTGljZW5z ZSwgb3IKKzs7IChhdCB5b3VyIG9wdGlvbikgYW55IGxhdGVyIHZlcnNpb24uCisKKzs7IFRoaXMg cHJvZ3JhbSBpcyBkaXN0cmlidXRlZCBpbiB0aGUgaG9wZSB0aGF0IGl0IHdpbGwgYmUgdXNlZnVs LAorOzsgYnV0IFdJVEhPVVQgQU5ZIFdBUlJBTlRZOyB3aXRob3V0IGV2ZW4gdGhlIGltcGxpZWQg d2FycmFudHkgb2YKKzs7IE1FUkNIQU5UQUJJTElUWSBvciBGSVRORVNTIEZPUiBBIFBBUlRJQ1VM QVIgUFVSUE9TRS4gIFNlZSB0aGUKKzs7IEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlIGZvciBt b3JlIGRldGFpbHMuCisKKzs7IFlvdSBzaG91bGQgaGF2ZSByZWNlaXZlZCBhIGNvcHkgb2YgdGhl IEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlCis7OyBhbG9uZyB3aXRoIHRoaXMgcHJvZ3JhbS4g IElmIG5vdCwgc2VlIDxodHRwczovL3d3dy5nbnUub3JnL2xpY2Vuc2VzLz4uCisKKwwKKzs7OyBD b21tZW50OgorCis7OyBTZWUgdGVzdGluZy9SRUFETUUgZm9yIGhvdyB0byBydW4gdGVzdHMuCisK KwwKKzs7OyBSZXF1aXJlbWVudHM6CisKKwwKKzs7OyBDb2RlOgorKGVydC1kZWZ0ZXN0IHRlc3Qt b3JnLWJhYmVsLWNvbWludC9wcm9tcHQtZmlsdGVyLXJlbW92ZXMtcHJvbXB0ICgpCisgICJUZXN0 IHRoYXQgcHJvbXB0IGlzIGFjdHVhbGx5IHJlbW92ZWQuIgorICAobGV0KiAoKHByb21wdCAib3Jn X2JhYmVsX3NoX3Byb21wdD4gIikKKyAgICAgICAgIChyZXN1bHRzICJvcmdfYmFiZWxfc2hfcHJv bXB0PiBlY2hvICdvYl9jb21pbnRfYXN5bmNfc2hlbGxfc3RhcnRfZDc4YWM0OWYtZGM4YS00YzM5 LTgyN2MtYzkzMjI1NDg0ZDU5JworIyBwcmludCBtZXNzYWdlCitlY2hvIFwiaGVsbG8gd29ybGRc IgorZWNobyAnb2JfY29taW50X2FzeW5jX3NoZWxsX2VuZF9kNzhhYzQ5Zi1kYzhhLTRjMzktODI3 Yy1jOTMyMjU0ODRkNTknCitvYl9jb21pbnRfYXN5bmNfc2hlbGxfc3RhcnRfZDc4YWM0OWYtZGM4 YS00YzM5LTgyN2MtYzkzMjI1NDg0ZDU5CitvcmdfYmFiZWxfc2hfcHJvbXB0PiBvcmdfYmFiZWxf c2hfcHJvbXB0PiBcImhlbGxvIHdvcmxkXCIKK29yZ19iYWJlbF9zaF9wcm9tcHQ+IG9iX2NvbWlu dF9hc3luY19zaGVsbF9lbmRfZDc4YWM0OWYtZGM4YS00YzM5LTgyN2MtYzkzMjI1NDg0ZDU5Citv cmdfYmFiZWxfc2hfcHJvbXB0PiAiKSkKKyAgICAoc2hvdWxkIChzdHJpbmc9CisgICAgICAgICAg ICAgKG9yZy10cmltIChzdHJpbmctam9pbiAobWFwY2FyICMnb3JnLXRyaW0gKG9yZy1iYWJlbC1j b21pbnQtLXByb21wdC1maWx0ZXIgcmVzdWx0cyBwcm9tcHQpKSAiXG4iKSAiXG4iKQorICAgICAg ICAgICAgICJlY2hvICdvYl9jb21pbnRfYXN5bmNfc2hlbGxfc3RhcnRfZDc4YWM0OWYtZGM4YS00 YzM5LTgyN2MtYzkzMjI1NDg0ZDU5JworIyBwcmludCBtZXNzYWdlCitlY2hvIFwiaGVsbG8gd29y bGRcIgorZWNobyAnb2JfY29taW50X2FzeW5jX3NoZWxsX2VuZF9kNzhhYzQ5Zi1kYzhhLTRjMzkt ODI3Yy1jOTMyMjU0ODRkNTknCitvYl9jb21pbnRfYXN5bmNfc2hlbGxfc3RhcnRfZDc4YWM0OWYt ZGM4YS00YzM5LTgyN2MtYzkzMjI1NDg0ZDU5CitcImhlbGxvIHdvcmxkXCIKK29iX2NvbWludF9h c3luY19zaGVsbF9lbmRfZDc4YWM0OWYtZGM4YS00YzM5LTgyN2MtYzkzMjI1NDg0ZDU5IikpKSkK KworCisocHJvdmlkZSAndGVzdC1vYi1jb21pbnQpCisKKzs7OyB0ZXN0LW9iLWNvbWludC5lbCBl bmRzIGhlcmUKLS0gCjIuNDEuMAoK ------=_Part_1079817_1944981950.1711181851324 Content-Type: application/octet-stream; name=0002-lisp-ob-comint.el-Create-comint-prompt-filter.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970362500140 Content-Disposition: attachment; filename=0002-lisp-ob-comint.el-Create-comint-prompt-filter.patch >From ed5f3e358f2e32c12d3487de9ab17797370eaf81 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 16:54:33 +0100 Subject: [PATCH 2/7] lisp/ob-comint.el: Create comint prompt filter * lisp/ob-comint.el (org-babel-comint--prompt-filter): Extract prompt filtering logic from `org-babel-comint-with-output' into a new function. --- lisp/ob-comint.el | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 1ec84e865..882e19289 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -74,6 +74,19 @@ This is useful when prompt unexpectedly changes." (setq comint-prompt-regexp org-babel-comint-prompt-regexp-old org-babel-comint-prompt-regexp-old tmp)))) +(defun org-babel-comint--prompt-filter (string &optional prompt-regexp) + "Remove PROMPT-REGEXP from STRING. + +PROMPT-REGEXP defaults to `comint-prompt-regexp'." + (let* ((prompt-regexp (or prompt-regexp comint-prompt-regexp)) + (separator "org-babel-comint--prompt-filter-separator\n")) + (while (string-match-p prompt-regexp string) + (setq string + (replace-regexp-in-string + (format "\\(?:%s\\)?\\(?:%s\\)[ \t]*" separator prompt-regexp) + separator string))) + (delete "" (split-string string separator)))) + (defmacro org-babel-comint-with-output (meta &rest body) "Evaluate BODY in BUFFER and return process output. Will wait until EOE-INDICATOR appears in the output, then return -- 2.41.0 ------=_Part_1079817_1944981950.1711181851324 Content-Type: application/octet-stream; name=0003-testing-lisp-test-ob-comint.el-Make-test-for-echo-fi.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970362500160 Content-Disposition: attachment; filename=0003-testing-lisp-test-ob-comint.el-Make-test-for-echo-fi.patch >From 44bccab228b9dee80672b3a3dd9d27b2cc26c4b3 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 16:56:43 +0100 Subject: [PATCH 3/7] testing/lisp/test-ob-comint.el: Make test for echo filter * test-ob-comint.el: (test-org-babel-comint/echo-filter-removes-echo): Test that echoed input is removed from process buffer output. --- testing/lisp/test-ob-comint.el | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/testing/lisp/test-ob-comint.el b/testing/lisp/test-ob-comint.el index 9193ef8f3..8f6def8ca 100644 --- a/testing/lisp/test-ob-comint.el +++ b/testing/lisp/test-ob-comint.el @@ -49,6 +49,26 @@ ob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59 \"hello world\" ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59")))) +(ert-deftest test-org-babel-comint/echo-filter-removes-echo () + "Test that echo is actually removed." + (let* ((echo "echo 'ob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59' +# print message +echo \"hello world\" +echo 'ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59'") + (result "org_babel_sh_prompt> echo 'ob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59' +# print message +echo \"hello world\" +echo 'ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59' +ob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59 +org_babel_sh_prompt> org_babel_sh_prompt> \"hello world\" +org_babel_sh_prompt> ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59 +org_babel_sh_prompt> ")) + (should (string= + (org-babel-comint--echo-filter result echo) + "\nob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59 +org_babel_sh_prompt> org_babel_sh_prompt> \"hello world\" +org_babel_sh_prompt> ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59 +org_babel_sh_prompt> ")))) (provide 'test-ob-comint) -- 2.41.0 ------=_Part_1079817_1944981950.1711181851324 Content-Type: application/octet-stream; name=0004-lisp-ob-comint.el-Create-comint-echo-filter.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970362500180 Content-Disposition: attachment; filename=0004-lisp-ob-comint.el-Create-comint-echo-filter.patch >From c629cb7041d25480cba82812ba5da41329f4e435 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 16:58:52 +0100 Subject: [PATCH 4/7] lisp/ob-comint.el: Create comint echo filter * lisp/ob-comint.el (org-babel-comint--echo-filter): Extract echo filtering logic from `org-babel-comint-with-output' into a new function. --- lisp/ob-comint.el | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 882e19289..e8d8e7609 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -87,6 +87,15 @@ PROMPT-REGEXP defaults to `comint-prompt-regexp'." separator string))) (delete "" (split-string string separator)))) +(defun org-babel-comint--echo-filter (string &optional echo) + "Remove ECHO from STRING." + (and echo string + (string-match + (replace-regexp-in-string "\n" "[\r\n]+" (regexp-quote echo)) + string) + (setq string (substring string (match-end 0)))) + string) + (defmacro org-babel-comint-with-output (meta &rest body) "Evaluate BODY in BUFFER and return process output. Will wait until EOE-INDICATOR appears in the output, then return -- 2.41.0 ------=_Part_1079817_1944981950.1711181851324 Content-Type: application/octet-stream; name=0005-lisp-ob-comint.el-Refactor-org-babel-comint-with-out.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970362500200 Content-Disposition: attachment; filename=0005-lisp-ob-comint.el-Refactor-org-babel-comint-with-out.patch >From f9821b7ff50aeae096be2ebb55f6d920c0afa1e0 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 18:24:43 +0100 Subject: [PATCH 5/7] lisp/ob-comint.el: Refactor `org-babel-comint-with-output' * lisp/ob-comint.el (org-babel-comint-with-output): Replace logic for prompt and echo filtering with `org-babel-comint--prompt-filter' and `org-babel-comint--echo-filter'. Delete `org-babel-comint-prompt-separator' variable and move related comment to `org-babel-comint--prompt-filter'. --- lisp/ob-comint.el | 39 +++++++++------------------------------ 1 file changed, 9 insertions(+), 30 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index e8d8e7609..d13aacccc 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -79,6 +79,9 @@ This is useful when prompt unexpectedly changes." PROMPT-REGEXP defaults to `comint-prompt-regexp'." (let* ((prompt-regexp (or prompt-regexp comint-prompt-regexp)) + ;; We need newline in case if we do progressive replacement + ;; of agglomerated comint prompts with `comint-prompt-regexp' + ;; containing ^. (separator "org-babel-comint--prompt-filter-separator\n")) (while (string-match-p prompt-regexp string) (setq string @@ -112,12 +115,7 @@ or user `keyboard-quit' during execution of body." (let ((buffer (nth 0 meta)) (eoe-indicator (nth 1 meta)) (remove-echo (nth 2 meta)) - (full-body (nth 3 meta)) - (org-babel-comint-prompt-separator - ;; We need newline in case if we do progressive replacement - ;; of agglomerated comint prompts with `comint-prompt-regexp' - ;; containing ^. - "org-babel-comint-prompt-separator\n")) + (full-body (nth 3 meta))) `(org-babel-comint-in-buffer ,buffer (let* ((string-buffer "") (comint-output-filter-functions @@ -161,31 +159,12 @@ or user `keyboard-quit' during execution of body." (goto-char (process-mark (get-buffer-process (current-buffer)))) (insert dangling-text) + ;; remove echo'd FULL-BODY from input + (and ,remove-echo ,full-body + (setq string-buffer (org-babel-comint--echo-filter string-buffer ,full-body))) + ;; Filter out prompts. - (while (string-match-p comint-prompt-regexp string-buffer) - (setq string-buffer - (replace-regexp-in-string - ;; Sometimes, we get multiple agglomerated - ;; prompts together in a single output: - ;; "prompt prompt prompt output" - ;; Or even "promptprompt ...>. - ;; Remove them progressively, so that - ;; possible "^" in the prompt regexp gets to - ;; work as we remove the heading prompt - ;; instance. - (format "\\(?:%s\\)?\\(?:%s\\)[ \t]*" ,org-babel-comint-prompt-separator comint-prompt-regexp) - ,org-babel-comint-prompt-separator - string-buffer))) - ;; remove echo'd FULL-BODY from input - (when (and ,remove-echo ,full-body - (string-match - (replace-regexp-in-string - "\n" "[\r\n]+" (regexp-quote (or ,full-body ""))) - string-buffer)) - (setq string-buffer (substring string-buffer (match-end 0)))) - (delete "" (split-string - string-buffer - ,org-babel-comint-prompt-separator)))))) + (org-babel-comint--prompt-filter string-buffer))))) (defun org-babel-comint-input-command (buffer cmd) "Pass CMD to BUFFER. -- 2.41.0 ------=_Part_1079817_1944981950.1711181851324 Content-Type: application/octet-stream; name=0006-testing-lisp-test-ob-shell.el-Test-async-prompt-remo.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970362500220 Content-Disposition: attachment; filename=0006-testing-lisp-test-ob-shell.el-Test-async-prompt-remo.patch >From 4f943d1e72c1e32f02723a53d4d8f5ee201dfbbb Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 18:27:11 +0100 Subject: [PATCH 6/7] testing/lisp/test-ob-shell.el: Test async prompt removal * testing/lisp/test-ob-shell.el (test-ob-shell/session-async-results): Create test verifying bug report that shell prompt appears in async results. --- testing/lisp/test-ob-shell.el | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el index 879555af0..8cebd8467 100644 --- a/testing/lisp/test-ob-shell.el +++ b/testing/lisp/test-ob-shell.el @@ -129,6 +129,30 @@ echo 2 (if (should (string= ": 1\n: 2\n" (buffer-substring-no-properties (point) (point-max)))) (kill-buffer session-name))))) +(ert-deftest test-ob-shell/session-async-results () + "Test that async evaluation removes prompt from results." + (let* ((session-name "test-ob-shell/session-async-results") + (kill-buffer-query-functions nil) + (start-time (current-time)) + (wait-time (time-add start-time 3)) + uuid-placeholder) + (org-test-with-temp-text + (concat "#+begin_src sh :session " session-name " :async t +# print message +echo \"hello world\" +#+end_src") + (setq uuid-placeholder (org-trim (org-babel-execute-src-block))) + (catch 'too-long + (while (string-match uuid-placeholder (buffer-string)) + (progn + (sleep-for 0.01) + (when (time-less-p wait-time (current-time)) + (throw 'too-long (ert-fail "Took too long to get result from callback")))))) + (search-forward "#+results") + (beginning-of-line 2) + (if (should (string= ": hello world\n" (buffer-substring-no-properties (point) (point-max)))) + (kill-buffer session-name))))) + (ert-deftest test-ob-shell/generic-uses-no-arrays () "Test generic serialization of array into a single string." (org-test-with-temp-text -- 2.41.0 ------=_Part_1079817_1944981950.1711181851324 Content-Type: application/octet-stream; name=0007-lisp-ob-comint.el-Fix-prompt-appearing-in-async-shel.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970362500240 Content-Disposition: attachment; filename=0007-lisp-ob-comint.el-Fix-prompt-appearing-in-async-shel.patch >From abe14703390e9ae56829a118c3d4e488443f6845 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 19:04:39 +0100 Subject: [PATCH 7/7] lisp/ob-comint.el: Fix prompt appearing in async shell results * lisp/ob-comint.el (org-babel-comint-async-filter): Call prompt `org-babel-comint--prompt-filter' Reported-by: "Matthew Trzcinski" Link: https://list.orgmode.org/18d753c1e8a.cfb3e1921191837.5665565128507976741@excalamus.com/ --- lisp/ob-comint.el | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index d13aacccc..f2251892a 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -325,9 +325,10 @@ STRING contains the output originally inserted into the comint buffer." until (and (equal (match-string 1) "start") (equal (match-string 2) uuid)) finally return (+ 1 (match-end 0))))) - ;; Apply callback to clean up the result - (res-str (funcall org-babel-comint-async-chunk-callback - res-str-raw))) + ;; Remove prompt + (res-promptless (org-trim (string-join (mapcar #'org-trim (org-babel-comint--prompt-filter res-str-raw)) "\n") "\n")) + ;; Apply user callback + (res-str (funcall org-babel-comint-async-chunk-callback res-promptless))) ;; Search for uuid in associated org-buffers to insert results (cl-loop for buf in org-buffers until (with-current-buffer buf -- 2.41.0 ------=_Part_1079817_1944981950.1711181851324--