From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id GAM3CIl/lGCcGgAAgWs5BA (envelope-from ) for ; Fri, 07 May 2021 01:45:13 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id iKLlA4l/lGDSGAAA1q6Kng (envelope-from ) for ; Thu, 06 May 2021 23:45:13 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5DC2210A32 for ; Fri, 7 May 2021 01:45:12 +0200 (CEST) Received: from localhost ([::1]:57660 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1leng6-0003ZU-Lw for larch@yhetil.org; Thu, 06 May 2021 19:45:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45560) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lenfZ-0003Y1-M1 for emacs-orgmode@gnu.org; Thu, 06 May 2021 19:44:37 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:41121) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lenfW-0000bi-Qh for emacs-orgmode@gnu.org; Thu, 06 May 2021 19:44:37 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 37F3F5C012C; Thu, 6 May 2021 19:44:33 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Thu, 06 May 2021 19:44:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=37ZkZX HtKcwAtWLNqvr43/zsdP528ivxfUOsR4LpeEs=; b=uF/xdm1MoAF2K0Wwixl0zy TPKLhNcNL0JEPVmLfYEnucGTplsJgibSbo0IjYveAfxGLS72vUGKeVtmPQPPW2Ov JRybyXWczKILRjK2QyRwDfcD+NUzsT7JsOG3QkCW6V2OYVdeaNrv9EnVXue8tDVJ PTd6MaeIQq0R/jhREhP7HWlnyqpERcaDQvYV3q5x2tcK6tE0swO1c5ufeFU1HEnO qf+C/X4PA9ogE+kaeNEQ+YdsmOAutSGYKfINIJh/gUu9vlJSzeu+VeBvUakxYPH2 hDRU9gIsdrpmKIE4Kc8TX3IV5SyDivAzKkSh2ddGV3WkJv5LomWUljmybMuJi1mQ == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeguddgvddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepuffvfhfhkffffgggjggtsehmtderredtfeejnecuhfhrohhmpefpihgtkhcu ufgrvhgrghgvuceonhhitghksehnihgtkhhsrghvrghgvgdrtggrqeenucggtffrrghtth gvrhhnpeetgeekgeeiueetudfhueejgfdtieehffetjeeukeevvedvgeduudehudetgfei tdenucfkphepjeeirdeijedrudekuddrgeehnecuvehluhhsthgvrhfuihiivgeptdenuc frrghrrghmpehmrghilhhfrhhomhepnhhitghksehnihgtkhhsrghvrghgvgdrtggr X-ME-Proxy: Received: from [192.168.0.67] (bras-base-aylmpq0104w-grc-29-76-67-181-45.dsl.bell.ca [76.67.181.45]) by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 6 May 2021 19:44:32 -0400 (EDT) Subject: Re: babel output seems to drop anything before % (in session) To: Ihor Radchenko References: <87h7jhus7k.fsf@spx.local.example> <87wnsc2gp2.fsf@localhost> From: Nick Savage Message-ID: <5780a7fb-17b5-6bde-f3a1-ef741af57734@nicksavage.ca> Date: Thu, 6 May 2021 19:44:32 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: <87wnsc2gp2.fsf@localhost> Content-Type: multipart/mixed; boundary="------------C81FDF6FFE54735480DB29D9" Content-Language: en-US Received-SPF: none client-ip=66.111.4.28; envelope-from=nick@nicksavage.ca; helo=out4-smtp.messagingengine.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Emanuel Berg via General discussions about Org-mode." Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1620344712; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=37ZkZXHtKcwAtWLNqvr43/zsdP528ivxfUOsR4LpeEs=; b=JdbTrA/WLVvyCiwu1MLB2E3Q9FezlGD90GmAnQ1fm9+ak2UR+Lcj3G03PDZAMAlPsRmWlD qSlB5OcM4xOfmqCtszWv/vhHa6i3g43ylUSvWdzK4dn/3wyvajPf2V8btnstxCxGxJ0GuX 4LO/9mMPt70XPdGMSGaWBZW7cQu5iOfJf45ve+3HxWOk7UZ+I0/r6yBXTVBK/lrp2Up6ZD amaz6u+8+hUMzNRus6uHA2kDAZFmCinRGPN4LRcP39dcmM1CJ0XNUoxCrFOJuUfil9PIVa Cv606bvygtYhhL/nbpap0mKIY8HeHLGBRTQF8Xs1rvOHwdjiuZRNhOriqRaBEQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1620344712; a=rsa-sha256; cv=none; b=dZsG0qDamXM//FMTkyuv76IPPT9nNyTY+zMOwd/8jEgBm7BCEs+0aRIpgb9nwsad+6at/V XTX0Ol1Z8RFQV8N0I1P0xgqnPcxPWpy14odc1IvcCQosTEn1/LxPWx5cgZvdljuiSYUhLm 16xNtwFb0npGn8plXRiwHOudly8RyB0CAzu9EZSJGdd4iPbcBRn110CCLgOLlpSvv1DMil nh2du4K2VQHSe/NCzdeHaPR62qalo+BTAAU/GVJE9UltZJXUzmfjYl3v7N96J0xpEelDI8 7BYhR9ypOod5CGPtuL5N8NmW1KU3KcA2JDQFQ0LSHlQvQsSBdVNGOFF6hxOXpA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="uF/xdm1M"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Spam-Score: -1.16 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=messagingengine.com header.s=fm2 header.b="uF/xdm1M"; dmarc=none; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Queue-Id: 5DC2210A32 X-Spam-Score: -1.16 X-Migadu-Scanner: scn0.migadu.com X-TUID: DYwxDUT7JtBU This is a multi-part message in MIME format. --------------C81FDF6FFE54735480DB29D9 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit So I have a patch written that doesn't completely fix the problem, but maybe makes enough progress that someone else can figure it out. The issue is that comint-prompt-regexp is reading the "% " as a prompt, and taking everyone off before it. I've added another parameter to org-babel-comint-with-output in the "meta" to pass our own regexp to replace comint-prompt-regexp, which works except when it doesn't. The regexp I've added is just "\n" now, so the newline characters are removed. The tests that Daniele added as a patch don't quite pass though with this. The issue is that something in the way the output is posted in the output buffer includes the prompt occasionally is included in a line and occasionally not. It seems the first time the block of code is executed, it is included (and therefore needs to be removed) and each other time it is not, so it is only the first time that it is run is not working properly. This is obviously way too fragile to actually merge, but I was hoping the work I've done so far is enough to help someone else make progress. I'll probably take another stab at this tomorrow (since it's bugging me), but thought I'd share what I have for now. Cheers, Nick On 5/6/21 8:24 AM, Ihor Radchenko wrote: > "Nicholas Savage" writes: > >> I can confirm this too on the latest master. >> >> I took a quick peek this morning, and my suspicion is that the problem is somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not positive at this point. > I confirm as well. I also saw an anomaly in the comint buffer. Note that > all the output lines, except "five 0% six" are after the shell prompt. > As I remember, the code expects the result to be exactly at the prompt > line. So, for some reason the first command ("echo five 0% six") of the > second block does not get inserted at the empty line. > > echo one 0% two > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ one 0% two > echo tree 0% four > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ tree 0% four > echo 'org_babel_sh_eoe' > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ echo five 0% six > five 0% six > echo seven 0% eight > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ seven 0% eight > echo 'org_babel_sh_eoe' > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe > yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ > > Best, > Ihor --------------C81FDF6FFE54735480DB29D9 Content-Type: text/x-patch; charset=UTF-8; name="0001-ob-shell.el-Fix-bug-where-shell-output-was-incorrect.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0001-ob-shell.el-Fix-bug-where-shell-output-was-incorrect.pa"; filename*1="tch" >From 6c7d39bfb9be38b54d23fcffbb09f1fcb96751f4 Mon Sep 17 00:00:00 2001 From: Nicholas Savage Date: Thu, 6 May 2021 19:17:33 -0400 Subject: [PATCH] ob-shell.el: Fix bug where shell output was incorrectly truncated on special characters. * lisp/ob-comint.el (org-babel-comint-with-output): Add fifth meta optional argument for providing a custom prompt regexp. lisp/ob-shell.el (org-babel-sh-evaluate): Implements using new argument to prevent incorrect truncation on special characters. If shell output included special characters that also occasionally are included in shell prompts, such as "#" or "%", a regexp was tripping up on them and cutting out part of the line. As ob-shell already cuts out the shell prompts, this was not necessary and instead we can just use \n as the separator. Original functionality was retained for other ob-* files in case this was necessary. --- lisp/ob-comint.el | 20 ++++++++++++-------- lisp/ob-shell.el | 2 +- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 18d4f3c93..27ad6efd7 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -57,13 +57,14 @@ executed inside the protection of `save-excursion' and (defmacro org-babel-comint-with-output (meta &rest body) "Evaluate BODY in BUFFER and return process output. -Will wait until EOE-INDICATOR appears in the output, then return -all process output. If REMOVE-ECHO and FULL-BODY are present and -non-nil, then strip echo'd body from the returned output. META -should be a list containing the following where the last two -elements are optional. +Will wait until EOE-INDICATOR appears in the output, then return all +process output. If REMOVE-ECHO and FULL-BODY are present and non-nil, +then strip echo'd body from the returned output. PROMPT-REGEXP is a +filter that, if provided, overrides the default regexp that tries to +filter out the shell prompt. META should be a list containing the +following where the last three elements are optional. - (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY) + (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY PROMPT-REGEXP) This macro ensures that the filter is removed in case of an error or user `keyboard-quit' during execution of body." @@ -71,7 +72,10 @@ or user `keyboard-quit' during execution of body." (let ((buffer (nth 0 meta)) (eoe-indicator (nth 1 meta)) (remove-echo (nth 2 meta)) - (full-body (nth 3 meta))) + (full-body (nth 3 meta)) + (prompt-regexp (if (nth 4 meta) + (nth 4 meta) + comint-prompt-regexp))) `(org-babel-comint-in-buffer ,buffer (let* ((string-buffer "") (comint-output-filter-functions @@ -111,7 +115,7 @@ or user `keyboard-quit' during execution of body." "\n" "[\r\n]+" (regexp-quote (or ,full-body ""))) string-buffer)) (setq string-buffer (substring string-buffer (match-end 0)))) - (split-string string-buffer comint-prompt-regexp))))) + (split-string string-buffer ,prompt-regexp))))) (def-edebug-spec org-babel-comint-with-output (sexp body)) (defun org-babel-comint-input-command (buffer cmd) diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el index 3eed0c164..9ec0425cb 100644 --- a/lisp/ob-shell.el +++ b/lisp/ob-shell.el @@ -263,7 +263,7 @@ return the value of the last statement in BODY." #'org-trim (butlast (org-babel-comint-with-output - (session org-babel-sh-eoe-output t body) + (session org-babel-sh-eoe-output t body "\n") (dolist (line (append (split-string (org-trim body) "\n") (list org-babel-sh-eoe-indicator))) (insert line) -- 2.20.1 --------------C81FDF6FFE54735480DB29D9--