Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]

[Matt <matt@excalamus.com>]

 ---- On Tue, 17 Jan 2023 14:53:39 -0500  Osher Jacob  wrote --- 
 > changing shell-command-switch to "/k" or "-k", I get a similar output:

Thanks for checking that.

 > You also mentioned the source code block is being passed through the "-c" flag as a command-line argument.I might be misunderstanding something here, but it seems like it is being passed through the stdin of the shell process when the calls process-file -> call-process are being made.

That the block source is passed through "-c" is an educated guess.  I keep finding myself in C land where it's less clear to me what precisely happens.  You're correct that `call-process' acts on the block source.  It's not clear to me whether that's done through stdin or whether being passed through stdin would even help us.  Here's what I'm seeing.

Evaluating the block goes through 4 lisp commands and then into C:

1. Calls `(org-babel-eval shell-file-name (org-trim body))'

This evaluates as...

(org-babel-eval
  "cmdproxy.exe" ; command, whatever `shell-file-name' is
  (org-trim body) ; query, the block body
  )

The query parameter (the block body) gets inserted into a temp buffer on which `org-babel--shell-command-on-region' is called.

2. Calls `(org-babel--shell-command-on-region command error-buffer)' on a temp buffer containing block body.

This evaluates as...

(org-babel--shell-command-on-region
  command       ; "cmdproxy.exe"
  error-buffer  ;  #<buffer  *Org-Babel Error*>
  )

This in turn calls `process-file' where INPUT-FILE is a temp file containing the contents of the buffer `org-babel--shell-command-on-region' was called on (that is, the temp file contains the block body).

3. Calls `(process-file shell-file-name input-file (if error-file (list t error-file) t) nil shell-command-switch command)'

This evaluates as...

(process-file
  "cmdproxy.exe"                                               ; shell-file-name
  "/path/to/temp/containing/block/source"  ; input-file
  (t "/tmp/babel jTCHe/ob-error-yHOivA")    ; output destination (like call-process's DESTINATION)
  nil                                                                     ; display
 "-c"                                                                    ; args
  "cmdproxy.exe"                                               ; args
)

Of course, I imagine the paths would be Windows paths when run on Windows.

According to the documentation, the args are passed to the process (cmdproxy.exe) "verbatim without filename handling, as `call-process' does."  The `call-process' documentation says, "remaining arguments ARGS are strings passed as command arguments to PROGRAM."

To me, that sounds like cmdproxy.exe gets "passed into" cmdproxy.exe.  This would explain the nested shell calls.  If all this were written out, I imagine it would look something like:

cmdproxy.exe -c cmdproxy.exe

What's not clear to me is how the INPUT-FILE is handled.  

The `process-file' documentation says "normally".  I assume it's how `call-process' handles INPUT-FILE, but the documentation doesn't really address it.

Finally, `process-file' finally calls `call-process'.

4. Calls `(apply 'call-process program (or lc infile) (if stderr-file (list (car buffer) stderr-file) buffer) display args)'

This evaluates as...

(apply 'call-process 
           program                                                                  ; cmdproxy
          (or lc infile)                                                              ; local-copy of the INFILE,  "/path/to/temp/containing/block/source" 
          (if stderr-file (list (car buffer) stderr-file) buffer)  ; (t "/tmp/babel jTCHe/ob-error-yHOivA"  )
          display                                                                     ; nil
          args)                                                                         ; ("-c"  "cmdproxy.exe")

How this actually works is non-trivial (https://git.savannah.gnu.org/cgit/emacs.git/tree/src/callproc.c#n250) and not something I understand at the moment.  We can see here that indeed a call like `cmdproxy.exe -c cmdproxy.exe' is being made.  Still, I'm not sure how the INPUT-FILE gets processed.  For example, is it passed in the second cmdproxy call?  Or, maybe it gets called first and then the second call to cmdproxy happens and hangs?  I don't know.

 > Any ideas on how to proceed from here?

I have two ideas.

1. Another naive work around attempt.  Again, I'm going from memory, documentation, and what I have previously written.  

I have in my init a command to open a terminal when working on Windows that looks like:

    (start-process "cmd" nil "cmd.exe" "/C" "start" "cmd.exe" "/K" "cd" dir)

This starts a cmd process with /c which terminates the prompt after the command that follows is run.  The command  That follows starts another cmd process with /k which changes the directory and leaves the terminal open.

I have another command to open QGIS that does the same thing:

    (start-process "cmd" nil "cmd.exe" "/C" "start" "\"qgis\"" "cmd.exe" "/K" "C:\\Program Files\\QGIS 3.22.3\\bin\\qgis.bat")

It's not clear to me why the extra call to a second cmd with /k is needed.  Maybe it's copy-pasta or Windows being Windows.

Anyway, I mention these because I wonder if we might be able to do something similar.

Try changing `shell-file-name' to "cmd.exe" (ideally, the full path) and `shell-command-switch' to "/k" (or maybe "-k").  My hope would be that the final call would look something like,

cmd.exe /k cmd.exe /k <executes-input-file-as-stdin-or-whatever>

Given my analysis, I'm not sure if there's a way we could trick the call into being `cmd.exe /c cmd.exe /k input-file'.

2. We could write some ob-shell code to explicitly handle Windows cmd and powershell.  For example, call `process-file' similar to the stdin/cmdline branch of `org-babel-sh-evaluate'.

I'm open to this, but, like I've said, I don't have a Windows machine to work on right now.  I'd not be able to verify it.  I, and I'm sure others, would be happy to guide you if that's something you'd like to help implement.

Thoughts?