Re: python-mode: make sure output is not eaten

[Slawomir Nowaczyk <slawomir.nowaczyk.847@student.lu.se>]

On Mon, 28 Aug 2006 17:43:46 -0400
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

#> > I works better -- no longer hangs my Emacs, but after python-send-buffer
#> > the final prompt still doesn't appear.
#> 
#> I believe my last patch fixes this as well.

Yes, it seems to work.

#> > Agreed. But why is removing "_emacs_out ()" any more selective than
#> > removing ">>> _emacs_out ()"?
#> 
#> We're talking about "^_emacs_out " vs "^\\(>>> \\)?_emacs_out": clearly one
#> of the two regexps describes a superset of the other.  If you can guarantee
#> that it'll always match "^>>> _emacs_out ", then it's a different question.

I do not think it is a problem... The following patch seems to work for me:

************************************************************

--- m:/EmacsCVS/EmacsCVS/lisp/progmodes/python.el       2006-08-30 17:05:09.399865600 +0200
+++ c:/Emacs/lisp/progmodes/python.el   2006-08-30 21:26:32.132132800 +0200
@@ -1271,7 +1271,7 @@
           (setq python-preoutput-skip-next-prompt nil)
           (setq line (substring line (match-end 0))))
         ;; Recognize special _emacs_out lines.
-        (if (and (string-match "\\`_emacs_out \\(.*\\)\n\\'" line)
+        (if (and (string-match "\\`>>> _emacs_out \\(.*\\)\n\\'" line)
                  (local-variable-p 'python-preoutput-result))
             (progn
               (setq python-preoutput-result (match-string 1 line))
@@ -1351,7 +1351,8 @@
       ;; seems worth putting in a separate file, and it's probably cleaner
       ;; to put it in a module.
       ;; Ensure we're at a prompt before doing anything else.
-      (python-send-receive "import emacs; print '_emacs_out ()'")))
+      (python-send-string "import emacs")
+      (python-send-receive "print '_emacs_out ()'")))
   (if (derived-mode-p 'python-mode)
       (setq python-buffer (default-value 'python-buffer))) ; buffer-local
   ;; Without this, help output goes into the inferior python buffer if
@@ -1601,7 +1602,7 @@
                  (current-word))))
     ;; First try the symbol we're on.
     (or (and symbol
-            (python-send-receive (format "emacs.eargs(%S, %s)"
+            (python-send-receive (format "\nemacs.eargs(%S, %s)"
                                          symbol python-imports)))
        ;; Try moving to symbol before enclosing parens.
        (let ((s (syntax-ppss)))
@@ -1614,9 +1615,9 @@
                  (skip-chars-backward "a-zA-Z._")
                  (if (< (point) point)
                      (python-send-receive
-                      (format "emacs.eargs(%S, %s)"
-                              (buffer-substring-no-properties (point) point)
-                              python-imports)))))))))))
+                      (format "\nemacs.eargs(%S, %s)"
+                               (buffer-substring-no-properties (point) point)
+                               python-imports)))))))))))
\f
;;;; Info-look functionality.

**********************************************************************
#> > I think I understand your reasoning, but I do not think I agree
#> > with it. If we *want* to wait for prompt after both "import ..."
#> > and "print ...", we can always use python-send-receive twice,
#> > instead of python-send-string followed by python-send-receive. But
#> > I still do not see why one is supposed to be more robust than the
#> > other.
#> 
#> python-send-receive doesn't wait for a prompt but for the _emacs_out
#> cookie, which the `import' does not output.

Right, my mistake.

#> > Also, I have finally found a case in which adding "; print ..."
#> > breaks things: assume the string is "if False: print 1". Observe
#> > that "if False: print 1; print '_emacs_out ()'" does *not* print
#> > anything.
#> 
#> > It may not be a *real* problem, but it does -- IMHO -- show that
#> > sending two separate commands *is* more robust.
#> 
#> It's not the only case, and I agree it's not a purely academic
#> question. But: such cases does not occur in python.el (where we only
#> add the "; print ..." at a few special spots where we have control
#> over the text that precedes it).

Yes, I know. My main objection -- although I admit, it is a purely
aesthetic one -- is that if we cannot make the code work right for the
simple case of sending two commands, the underlying code is too fragile.

#> > #> It's not a command, so it can only called from elisp code. Check the
#> > #> callers and you'll see it is never called with a newline. So it's OK
#> > #> to make such an assumption.
#> 
#> > Well, maybe. But still, either python-send-string works for multiline
#> > strings (in which case python-send-command should work as well)
#> 
#> I see no reason why the two should be so linked.

python-send-command does very little beyond calling python-send-string.
Definitely nothing that would make reasonable for only one of them to
work with multiline strings.

#> > or python-send-string doesn't work for multiline strings, in which case
#> > the assert in python-send-command doesn't help at all (as you wrote,
#> > nothing can call it with multiline command anyway).
#> 
#> In python-send-string, multiline strings work.  In python-send-command
#> multiline strings were very likely to fail because we added "; print ...".
#> Hence the presence of an `assert' (assertions are of no use to the user,
#> their only meaning is for the programmer as a form of documentation that's
#> slightly better sync'd with the code than mere comments).

In other words, the only reason for the assert is that we have chosen to
add "; print" instead of sending two commands?

#> > #> > It should work just fine with multiple lines, the only thing which
#> > #> > might cause problems is indentation, and even that only if there is
#> > #> > an indented block at the very end.
#> > #> 
#> > #> Sending a multiline thingy via python-send-string will look like
#> > #> absolute crap in the resulting output,
#> 
#> > I disagree...
#> 
#> Actually, you seem to be right: the prompt-elimination heuristic
#> seems to work well for this in practice.
#> 
#> Although it points top another problem: the result of the evaluation
#> is printed on the line of the prompt. I.e. if I start with
#> 
#>    >>> 
#> 
#> after python-send-string "print 1" I get
#> 
#>    >>> 1
#>    >>> 
#> 
#> rather than
#> 
#>    >>> 
#>    1
#>    >>> 
#> 
#> or
#> 
#>    >>> print 1
#>    1
#>    >>> 

Yes, I know... that is on the list of bugs I want to fix. I was just
waiting for the current code to stabilise. I believe that the right
thing to do would be to print, in the above case, something like:

>>> # evaluating buffer <test.py>
1
>>> 

or

>>> # evaluating region in buffer <test.py>
1
>>> 

The other mode, python-mode.el, uses something like this, and I remember
it was really helpful, especially when I worked with multiple files. It
really allowed me to see what was happening.

I am not sure what should be printed by python-send-string, but maybe
simple "print 1" would work best.

#> > #> because all the intermediate prompts will get concatenated rather
#> > #> than interleaved with the input.
#> 
#> > Right, but what's wrong with that? If I send a string like "a = 1\n
#> > print 'Test'\n b = 2", what's wrong with getting output like
#> > ">>> Test\n>>> \n >>>"?
#> 
#> In a session log, the prompt is expected to be followed by user
#> input, not program output.

Sure... some day we might want to do that. But it's pretty far down on
my list of priorities.

#> > #> So I see no point in trying to support this in a case such as
#> > #> `python-send-command' where we don't use this flexibility anyway (and
#> > #> where we need to analyze the output, so the more control over it the
#> > #> better).
#> 
#> > You may be right... but if you think it makes sense, the assert should
#> > be in python-send-string, not in python-send-command.
#> 
#> I see no point in restricting the use of python-send-string where the only
#> potential problem is ugly output, but not incorrect execution.

I agree.

#> Whereas in python-send-command, the result could have been incorrect
#> execution.

Yes, but -- IMHO -- this is only due to the bad design decision of
adding "; print"

#> Also adding the assert to python-send-command did not restrict the
#> user in any way, only the python.el hacker.

Right. Fine, this is probably not important enough to warrant further
discussion.

#> > I have hard time imagining how the current code could work at all
#> > if it send anything to Python interpreter while it was busy, but I
#> > will take a closer look at this.
#> 
#> Indeed, if the python process is busy when we send the command, the
#> result may be wrong. But I expect that in most cases it'll be right.

It seems to work to some degree in my very limited testing... but there
are a lot of ways to break it. There is some kind of queueing
implemented in python-mode.el, which detects busy interpreter and waits
with sending commands... but I haven't checked how is it implemented.

#> Still, I think checking busy/idle-ness is a good idea in itself and
#> your above suggestion as well.

I agree.

#> Note that checking idle/busy-ness can be tricky. Even assuming the
#> program's output does not look like a prompt, you may have to count
#> the number of newlines sent to the python's process to know how many
#> prompts to expect in return. Or alternatively, you'd have to send
#> each input line one at a time, waiting for the prompt before sending
#> the next one.

I was only thinking about checking busyness of things like
python-send-region -- when we can wait for _emacs_out. I believe it is
not worth the trouble to worry about python-send-string.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Hardware: The parts of a computer system that can be kicked.