From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id uLm9L4XkV2X+dgAAG6o9tA:P1 (envelope-from ) for ; Fri, 17 Nov 2023 23:09:10 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id uLm9L4XkV2X+dgAAG6o9tA (envelope-from ) for ; Fri, 17 Nov 2023 23:09:09 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6BBCC5527E for ; Fri, 17 Nov 2023 23:09:09 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=Oel0G6OO; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1"); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=2; s=key1; d=yhetil.org; t=1700258949; a=rsa-sha256; cv=pass; b=urbW/5Jh7If6tS3n+f+9IPKr5gkPn/tJD1D4JxiSVCxBXPagLZWqPya6eCGFVYIH07ArO6 OTO05VwRlqE7sE2/3jqvNUyRhoR3LIEnfQ3ugn/91zWiex5Xo6SngkGr+56lhjOYvXH5Y8 ELE6zyhUIA+TImNetU2woTQ7ST5s3ZMk1c7DI94TaB+mT85V9fqc9JQRpE3hDHdKkXy5fm Ra90QDMpseU8okUjcsNiG3oPcCxzF4uw8Z/c+L0IP9VJA5qZ/AwJf21XGxRJjG/KYmQI73 XhurzmYwcIgYuM14yZecDrCb4cDw12ibEYm/kRt7XoznyOU+oGqXgcBGy9GC+w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1700258949; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=Z0WSVzbEbvTuobpDOVn434LdITykJcXqziqdCKmYG+Y=; b=G8GUIOOoTUnKU4VMhRq4MQP+VLsuklyMpoaeAzuuP8j3bwrdAJcMhrq8FwHZ53AuOP/c6F mhCXONJAuXuKmZiYSGlWanOWJGiGZhTbNbB6RRIM2BDZDaQnKDYfkhugphB7/H83s6cEjW 2wNXvj+eTXlre7bwWqEYUQ9Cq1bofIsT5mMYG7pH7DSCTEzXNR7WLxV2Y42UWXBc2JXJfw 7BrG9+igiG2/zTYvsbgsIRnc4CQQlCVJjvI/nJD8XEUQPCSXo3LzSM7k5Z52mlur71qvRE eD88ZsH6eMQIh6vYJpjf+eTCwK6Z599MeQNUJ4o2MUq7w1e+xheK+u2fSxODHQ== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=Oel0G6OO; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1"); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r470S-0001YK-Q3; Fri, 17 Nov 2023 17:08:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r470Q-0001Xr-Fn for emacs-orgmode@gnu.org; Fri, 17 Nov 2023 17:08:06 -0500 Received: from sender4-op-o14.zoho.com ([136.143.188.14]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r470N-0001Ww-Pm for emacs-orgmode@gnu.org; Fri, 17 Nov 2023 17:08:06 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1700258880; cv=none; d=zohomail.com; s=zohoarc; b=OqrO9DBHvmnamTti07xuPkaqnm4yIx/3s1tA+mPgFG8u0Bbv0JWcEIQ2r+M1K8vLnjeDkWZeEowQRZNFoWLdDWHh42kRHzfJeH0CyZCh+gEwqKhyg9MIOy5rvBsfaTG4ayfKHOdj7FWU6WDuTda+YVmSONG4AbOjZF8Ybtm3vQQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1700258880; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=Z0WSVzbEbvTuobpDOVn434LdITykJcXqziqdCKmYG+Y=; b=kZr7con8iGQ7Auxewytk5QVsNvAxWuEmQpyRS9/43OxAfM5upZMo4raAmzeRcMslid43VCqrgmM/EW83jmAaz+3XeREednBiQhqBNJZaKRHA5J0uJW/2jRWLYP0CNOOrpK52Ok/BvpfMwdCnnZYUlXWxDN/FdwHhOXiB9/NEchc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=excalamus.com; spf=pass smtp.mailfrom=matt@excalamus.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1700258880; s=zmail; d=excalamus.com; i=matt@excalamus.com; h=Date:Date:From:From:To:To:Cc:Cc:Message-ID:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=Z0WSVzbEbvTuobpDOVn434LdITykJcXqziqdCKmYG+Y=; b=Oel0G6OOc8mhf6CJBou4uR/3TIuKwzaXiNEn6qnaJyZjcJyWejU4yr4aRkCBQEoU 7iY/r5PBMe0tKI5PmbmertXL5gmpTjN/E/0rUkTinzgpTJC0qVoqZ7EN+tyUzxlPM/c DY286d6NzfaMVuTUSirL8wcV52/SDXG/QDKJg3W8= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1700258877259569.3958928802425; Fri, 17 Nov 2023 14:07:57 -0800 (PST) Date: Fri, 17 Nov 2023 23:07:57 +0100 From: Matt To: "Ihor Radchenko" Cc: "emacs-orgmode@gnu.org" Message-ID: <18bdf538f2f.126bad4763316098.8581777358227217138@excalamus.com> In-Reply-To: <871qcod9ad.fsf@localhost> References: <25912.63830.726070.953843@gargle.gargle.HOWL> <87bkcmlor9.fsf@t14.reltub.ca> <87o7gldb78.fsf@localhost> <25914.26693.101108.954656@gargle.gargle.HOWL> <87fs1xbis1.fsf@localhost> <25916.238.191509.652552@gargle.gargle.HOWL> <653f8a93.050a0220.f2202.5816@mx.google.com> <87zfzr2ejj.fsf@localhost> <18ba5e231e3.f989cde147196.3154436412643995109@excalamus.com> <875y2e2b9y.fsf@localhost> <18bb07473e2.1007b1565819307.6938164403009000496@excalamus.com> <877cmr2ke4.fsf@localhost> <18bb5337346.dfcd5ebb1139895.801408740607633332@excalamus.com> <18bd3d38ff3.115b786cb2560022.7704006411630399228@excalamus.com> <87fs169h7f.fsf@localhost> <18bd984270d.1048050a22954105.2819987720224062869@excalamus.com> <25942.29051.701153.391260@gargle.gargle.HOWL> <18bd9ea2f7b.b77346232985684.5614027527324280790@excalamus.com> <871qcod9ad.fsf@localhost> Subject: Re: bash source code block: problem after ssh commands MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail Received-SPF: pass client-ip=136.143.188.14; envelope-from=matt@excalamus.com; helo=sender4-op-o14.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx11.migadu.com X-Spam-Score: -8.62 X-Migadu-Queue-Id: 6BBCC5527E X-Migadu-Spam-Score: -8.62 X-TUID: U8R+cmf2eVHz ---- On Fri, 17 Nov 2023 10:20:28 +0100 Ihor Radchenko wrote --- > This has nothing to do with Emacs comint and this is also not a bug in > Emacs Ihor, there were two claims made in the original report. I was referring to Claim 2. That deals with M-x shell and therefore comint-mode. Regarding Claim 1: - Can anyone verify Claim 1? - Is anyone else unable to verify Claim 1 (like me)? - What versions are people using? + M-x org-version + M-x emacs-version I'm running Org mode version 9.7-pre (release_9.6.10-903-g9183e3.dirty @ /home/ahab/.emacs.d/straight/build/org/) on GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0). * The original report has two claims: ** Claim 1. The following block is expected to write a remote file called "foo_file" with contents "foo" as well as give "bar" as the result. #+begin_src bash :results output ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" echo "bar" #+end_src The reported behavior is that "foo_file" is created with "foo" (with "foo" is not stated, but implied) and "bar" is *not* given as the result. ** Claim 2. Copying and pasting the two lines from the first claim into a terminal like xfce4-terminal executes the ssh line as expected and outputs the result of the second line. It was noted that this does not happen with M-x shell. * Comments about the claims: ** Comment 1. tl;dr I can't reproduce the claim that "bar" is *not* the result. The result is "bar" for me. The exact "expected behavior" for a shell block is a little fuzzy. According to my analysis (given below), what Alain reports (remote file and no "bar") is the "expected" behavior. What I see (no remote file and "bar") is actually "unexpected". I used the following to test the claim: #+begin_src bash :results output ssh localhost "echo foo>foo_file" echo "bar" #+end_src I am unable to reproduce the reported behavior (of "bar" not returning). Instead, I get an ssh-askpass permission denied error, foo_file is not created, and "bar" is given as the result. I do not see anywhere in the thread that the original claim was reproduced. The thread preceded something like follows. Leo Butler suggested two work arounds: - add the -f to the ssh command - add a semi-colon and line continuation to the first line. Russell Adams suggested another work around: - add -n to the ssh command Ihor identified that a non-session call does something like the following command: bash -c bash /tmp/temp-file-with-source-block-code.sh where ----- /tmp/temp-file-with-source-block-code ----- ssh localhost "echo foo>foo_file" echo "bar" | tee /tmp/bar.txt ------------------------------------------------- The second line (significantly different from the original report) pipes the echo result to stdout and to a file, bar.txt. Writing to a file allows us to confirm if that line was executed. Ihor suggested that bash -c bash /tmp/temp-file-with-source-block-code.sh does not run the second line because an interactive password prompt is displayed by ssh. The reasoning is that the prompt hangs the process while waiting for input and the second line never runs. Indeed, running the command does not produce /tmp/bar.txt. Ihor is correct about prompts messing with shell blocks (this is not the first time he's seen this). However, the way it's stated does not demonstrate it. This is because Emacs does *not* make a call like bash -c bash /tmp/temp-file-with-source-block-code.sh Alain responded by pointing out that bash -c bash /tmp/temp-file-with-source-block-code.sh does not execute the first line. This is true. Consider calling bash -c bash /tmp/two-lines.sh where ------ /tmp/two-lines.sh ------ echo "first" > /tmp/first.txt echo "second" > /tmm/second.txt ------------------------------- Neither first.txt or second.txt are created. Max Nikulin interjected with a helpful reminder that Bash scripting is a snakepit of footguns. (What Max said is more than that and interesting. I skip it here because it depends on the form of the call.) Before trying to untangle what a given Bash command does, we need to be sure what command is actually called. Unfortunately, there's not a clear Bash command corresponding to how Emacs makes the call. What happens goes something like this: 1. The Lisp function process-file is called with PROGRAM "bash", INFILE a path to a temp file containing the source block code, and ARGS ("-c" "bash") 2. This information is passed to DEFUN ("call-process"), a Lisp object implemented in C 3. DEFUN ("call-process") forwards this information to the C function call_process 4. call_process calls emacs_spawn 5. emacs_spawn creates a subprocess A lot of cleaning and setup happens which is dependent on the system (GNU, Window, Darwin, etc.). There's a call to openp which looks for the executable. An emacs_pipe is set up (I assume for writing to stdout and stderr). I'm unable to give a definitive answer as to what precisely emacs_spawn calls. Do any of the args "get quotes?" I can't say. Bruno Barbier commented that his understanding of process-file is that it gets commands from stdin. Maybe that's what the call to emacs_pipe is doing? He gives the example: #+begin_quote (process-file "bash" "/tmp/test.sh") is more equivalent to: cat /tmp/test.sh | bash #+end_quote He then proposes an experiment to close stdin. To do this, he calls #+begin_src shell :results output exec 0>&- echo OK #+end_src He claims that "exec 0<&-" closes stdin. I believe there is a typo. It's not clear if it has a negative effect, though. According to the [[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting Guide]], #+begin_quote Closing File Descriptors n<&- Close input file descriptor n. 0<&-, <&- Close stdin. n>&- Close output file descriptor n. 1>&-, >&- Close stdout. #+end_quote What Bruno writes corresponds to "closing output file descriptor 0". I honestly don't know what the difference is between an "output file descriptor" and an "input file descriptor". I had no luck finding this information in man bash or info bash. Rerunning the experiment according to the [[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting Guide]], the result is the same: "OK" is *not* printed. #+begin_src shell :results output exec 0<&- echo OK #+end_src Doing the following echoes OK for either direction of the redirection: -- /tmp/exec-OKlt.sh --- exec 0<&- echo OK ---------------------- #+begin_example bash /tmp/exec-OKlt.sh #+end_example -- /tmp/exec-OKgt.sh --- exec 0>&- echo OK ---------------------- #+begin_example bash /tmp/exec-OKgt.sh #+end_example The INFILE passed to process-file looks like, #+begin_src emacs-lisp (process-file "bash" "/tmp/two-lines.sh" '(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH") nil "-c" "bash") #+end_src So, the call Emacs makes is probably more close to: #+begin_example cat /tmp/two-lines.sh | bash -c bash #+end_example What this exactly does is unclear to me. It appears to pass the contents of /tmp/two-lines.sh to a subshell process. That is, it seems to behave like "bash /tmp/two-lines.sh" is run in a subprocess. Running this in xfce4-terminal, I get what I expect: #+begin_example cat /tmp/two-lines.sh | bash -c bash #+end_example Each line is echoed to file so nothing is written to the console. However, both files are created with the expected text. Both lines executed. If I update two-lines to output to std, ------ /tmp/two-lines-tee.sh ------ echo "first" | tee /tmp/first.txt echo "second" | tee /tmp/second.txt ----------------------------------- I see "first" and "second" echoed to the console: ahab@pequod /tmp$ cat two-lines-tee.sh | bash -c bash first second Running the following, neither give an output to the console: #+begin_example ahab@pequod /tmp$ cat exec-OKlt.sh | bash -c bash ahab@pequod /tmp$ cat exec-OKgt.sh | bash -c bash #+end_example This is what we see in Org. I'll be honest, though, I don't really know what to expect with exec 0>&- and exec 0<&-. When I call them in the terminal, it kills the terminal. The surprising bit is that running this in xfce4-terminal ----- /tmp/temp-file-with-source-block-code ----- ssh localhost "echo foo>foo_file" echo "bar" | tee /tmp/bar.txt ------------------------------------------------- #+begin_example cat /tmp/temp-file-with-source-block-code.sh | bash -c bash #+end_example does *not* echo bar (and does not create /tmp/bar.txt) yet it creates foo_file. I get prompted for my password and then the second line doesn't execute. Nothing prints to the console and no bar.txt is created. This is the behavior Alain reports happening in Org (that I am unable to reproduce). That is, the *reported behavior is the expected behavior* (assuming my analysis is correct). However, according to the behavior I see when I run the block (fails to create the remote file and echoes "bar"), Org does the "wrong thing". I can't account for this. Anyway, Ihor's main point stands: a prompt does not work with non-session shell blocks. The following returns exit code 1 (which means fail): #+begin_src bash :results output read -p "What? " #+end_src As far as I can tell, though, that's not what prevents "bar" from being returned. As far as I can reproduce, calling #+begin_src bash :results output ssh localhost "echo foo>foo_file" echo "bar" #+end_src *does* give "bar" for results even though it shouldn't. ** Comment 2. The second claim has nothing to do with Org Babel. I was able to confirm it and provide the steps to reproduce. I think it would make sense to report it upstream and let them decide if it's expected behavior. I'm still happy to do that, but I need to step away from the keyboard :)