python-mode: make sure output is not eaten

python-mode: make sure output is not eaten

[Slawomir Nowaczyk]

Hello,

the following path makes sure python-mode doesn't eat output coming
from the Python process. Without it, calling python-send-buffer on
a Python script which produces output, such as simple

   print 'OK'

wouldn't show anything in Inferior Python buffer -- the output would
get eaten by the python-preoutput-filter together with the
"_emacs_out" sentinel (at least on my machine... there seems to be a
race condition, because output was showing just fine when I run
python-send-command under edebug).

Anyway, this patch makes sure that output is printed when necessary.

**********************************************************************

--- EmacsCVS/lisp/progmodes/python.el       2006-08-21 10:03:44.950083200 +0200
+++ Emacs/lisp/progmodes/python.el   2006-08-21 21:58:30.049228800 +0200
@@ -1282,7 +1282,7 @@
          "")
         ((string-match "_emacs_out \\(.*\\)\n" s)
          (setq python-preoutput-result (match-string 1 s))
-         "")
+         (substring s 0 (match-beginning 0)))
        ((string-match ".*\n" s)
         s)
        ((or (eq t (compare-strings s nil nil "_emacs_ok\n" nil (length s)))
@@ -1409,7 +1409,7 @@
   "Evaluate STRING in inferior Python process."
   (interactive "sPython command: ")
   (comint-send-string (python-proc) string)
-  (comint-send-string (python-proc) "\n\n"))
+  (comint-send-string (python-proc) "\n"))

 (defun python-send-buffer ()
   "Send the current buffer to the inferior Python process."

**********************************************************************

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

If the code and the comments disagree, then both are probably wrong.

Re: python-mode: make sure output is not eaten

[Stefan Monnier]

> the following path makes sure python-mode doesn't eat output coming
> from the Python process. Without it, calling python-send-buffer on
> a Python script which produces output, such as simple

Does the patch below give good results as well?


        Stefan


--- python.el	20 aoû 2006 14:13:46 -0400	1.41
+++ python.el	22 aoû 2006 02:08:00 -0400	
@@ -1264,9 +1264,25 @@
   (when python-preoutput-leftover
     (setq s (concat python-preoutput-leftover s))
     (setq python-preoutput-leftover nil))
+  (let ((start 0)
+        (res ""))
+    ;; First process whole lines.
+    (while (string-match "\n" s start)
+      (let ((line (substring s start (setq start (match-end 0)))))
+        (cond
+         ((string= line "_emacs_ok\n")
+          (when python-preoutput-continuation
+            (funcall python-preoutput-continuation)
+            (setq python-preoutput-continuation nil)))
+         ((string-match "\\`_emacs_out \\(.*\\)\n" line)
+          (setq python-preoutput-result (match-string 1 line)))
+         (t (setq res (concat res line))))))
+    ;; Then process the remaining partial line.
+    (unless (zerop start) (setq s (substring s start)))
   (cond ((and (string-match (rx string-start (repeat 3 (any ".>"))
 				" " string-end)
-                            s)
+                              s start)
+                ;; Isn't that just (not (bolp)) ?  --Stef
               (/= (let ((inhibit-field-text-motion t))
                     (line-beginning-position))
                   (point)))
@@ -1274,23 +1290,14 @@
          ;; What is this all about, exactly?  --Stef
 	 ;; (if (and (eq ?. (aref s 0)))
 	 ;;     (accept-process-output (get-buffer-process (current-buffer)) 1))
-         "")
-        ((string= s "_emacs_ok\n")
-         (when python-preoutput-continuation
-           (funcall python-preoutput-continuation)
-           (setq python-preoutput-continuation nil))
-         "")
-        ((string-match "_emacs_out \\(.*\\)\n" s)
-         (setq python-preoutput-result (match-string 1 s))
-         "")
-	((string-match ".*\n" s)
-	 s)
+           ;; Throw away this prompt-like thingy.
+           res)
 	((or (eq t (compare-strings s nil nil "_emacs_ok\n" nil (length s)))
 	     (let ((end (min (length "_emacs_out ") (length s))))
 	       (eq t (compare-strings s nil end "_emacs_out " nil end))))
 	 (setq python-preoutput-leftover s)
-	 "")
-        (t s)))
+	 res)
+        (t (concat res s)))))
 
 (autoload 'comint-check-proc "comint")
 

Re: python-mode: make sure output is not eaten

[Slawomir Nowaczyk]

On Tue, 22 Aug 2006 02:10:02 -0400
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

#> > the following path makes sure python-mode doesn't eat output coming
#> > from the Python process. Without it, calling python-send-buffer on
#> > a Python script which produces output, such as simple
#> 
#> Does the patch below give good results as well?

Not quite... I get the following error:
   error in process filter: Args out of range: ">>> >>> ", 22

It seems like this change is unnecessary:

#>    (cond ((and (string-match (rx string-start (repeat 3 (any ".>"))
#>  				" " string-end)
#> -                            s)
#> +                              s start)
as you setq `s' to the appropriate substring one line earlier, in 
  (unless (zerop start) (setq s (substring s start)))

Also, "\n\n" in python-send-string still needs to be changed into a
single "\n" in order to avoid spurious prompts.

Finally, when you test this patch, make sure you also try enabling
eldoc-mode (it doesn't work too well, a patch to improve it a bit is
forthcoming, but even now it works for built-in functions like "apply"
or "map"). It is important to make sure no extra ">>>"s get added to
the inferior buffer when eldoc is talking to Python process.

This is -- partially -- the reason why it is important to save
whatever text exists in front of "_emacs_out"... For example, when I
python-send-buffer I can get output like "\nTest!\n>>> _emacs_out
()\n>>> "... and the ">>>" after "Test!" must show in the buffer. On
the other hand, after eldoc asks Python about argument list, the
output may look like this: "_emacs_out apply(object[, args[,
kwargs]])\n>>> " and it is important that *nothing* is shown in the
inferior buffer.

The solution that seems easiest is to always eat the final prompt, but
let the one in front of _emacs_out be printed.

Anyway, the following patch, to be applied after yours, makes thing
work OK for me:

**********************************************************************

--- EmacsCVS/lisp/progmodes/python.el       2006-08-22 19:49:58.809940800 +0200
+++ Emacs/lisp/progmodes/python.el   2006-08-22 20:11:40.010976000 +0200
@@ -1274,14 +1274,15 @@
           (when python-preoutput-continuation
             (funcall python-preoutput-continuation)
             (setq python-preoutput-continuation nil)))
-         ((string-match "\\`_emacs_out \\(.*\\)\n" line)
-          (setq python-preoutput-result (match-string 1 line)))
+         ((string-match "_emacs_out \\(.*\\)\n" line)
+          (setq python-preoutput-result (match-string 1 line))
+          (setq res (concat res (substring line 0 (match-beginning 0)))))
          (t (setq res (concat res line))))))
     ;; Then process the remaining partial line.
     (unless (zerop start) (setq s (substring s start)))
   (cond ((and (string-match (rx string-start (repeat 3 (any ".>"))
                                " " string-end)
-                              s start)
+                              s); start)
                 ;; Isn't that just (not (bolp)) ?  --Stef
               (/= (let ((inhibit-field-text-motion t))
                     (line-beginning-position))
@@ -1416,7 +1417,7 @@
   "Evaluate STRING in inferior Python process."
   (interactive "sPython command: ")
   (comint-send-string (python-proc) string)
-  (comint-send-string (python-proc) "\n\n"))
+  (comint-send-string (python-proc) "\n"))

 (defun python-send-buffer ()
   "Send the current buffer to the inferior Python process."

**********************************************************************

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

The Main Library at Indiana University sinks over an inch every
year because when it was built, engineers failed to take into
account the weight of all the books that would occupy the building.

Re: python-mode: make sure output is not eaten

[Stefan Monnier]

> #>    (cond ((and (string-match (rx string-start (repeat 3 (any ".>"))
> #>  				" " string-end)
> #> -                            s)
> #> +                              s start)
> as you setq `s' to the appropriate substring one line earlier, in 
>   (unless (zerop start) (setq s (substring s start)))

Oh, right.  I missed this one spot, sorry.

> Also, "\n\n" in python-send-string still needs to be changed into a
> single "\n" in order to avoid spurious prompts.

Yes, of course.  BTW do you have any idea why the current code uses "\n\n"?

> Finally, when you test this patch, make sure you also try enabling
> eldoc-mode (it doesn't work too well, a patch to improve it a bit is
> forthcoming, but even now it works for built-in functions like "apply"
> or "map"). It is important to make sure no extra ">>>"s get added to
> the inferior buffer when eldoc is talking to Python process.

Would it be because eldoc ends up sending commands to the underlying process
while some other command is still being processed?  Maybe we'd need
a "python-process-busy" variable to prevent eldoc from interfering.

> This is -- partially -- the reason why it is important to save
> whatever text exists in front of "_emacs_out"... For example, when I
> python-send-buffer I can get output like "\nTest!\n>>> _emacs_out
> ()\n>>> "... and the ">>>" after "Test!" must show in the buffer. On
> the other hand, after eldoc asks Python about argument list, the
> output may look like this: "_emacs_out apply(object[, args[,
> kwargs]])\n>>> " and it is important that *nothing* is shown in the
> inferior buffer.

The handling of the prompt looks pretty messy, indeed.  I think we should
never send a command followed by "\n" followed by "print _emacs_out ()", but
instead we should send the command augmented with "; print _emacs_out ()\n".
This should avoid those nasty intermediate prompts.

How 'bout the patch below which also removes the _emacs_ok and preoutput
continuation stuff which doesn't seem to be used any more.


        Stefan


--- python.el	20 aoû 2006 14:13:46 -0400	1.41
+++ python.el	23 aoû 2006 00:00:28 -0400	
@@ -1250,47 +1250,45 @@
 (defvar python-preoutput-result nil
   "Data from last `_emacs_out' line seen by the preoutput filter.")
 
-(defvar python-preoutput-continuation nil
-  "If non-nil, funcall this when `python-preoutput-filter' sees `_emacs_ok'.")
-
 (defvar python-preoutput-leftover nil)
 
 ;; Using this stops us getting lines in the buffer like
 ;; >>> ... ... >>>
-;; Also look for (and delete) an `_emacs_ok' string and call
-;; `python-preoutput-continuation' if we get it.
 (defun python-preoutput-filter (s)
   "`comint-preoutput-filter-functions' function: ignore prompts not at bol."
   (when python-preoutput-leftover
     (setq s (concat python-preoutput-leftover s))
     (setq python-preoutput-leftover nil))
+  (let ((start 0)
+        (res ""))
+    ;; First process whole lines.
+    (while (string-match "\n" s start)
+      (let ((line (substring s start (setq start (match-end 0)))))
+        (if (string-match "\\`_emacs_out \\(.*\\)\n\\'" line)
+            (setq python-preoutput-result (match-string 1 line))
+          (setq res (concat res line)))))
+    ;; Then process the remaining partial line.
+    (unless (zerop start) (setq s (substring s start)))
   (cond ((and (string-match (rx string-start (repeat 3 (any ".>"))
 				" " string-end)
                             s)
-              (/= (let ((inhibit-field-text-motion t))
-                    (line-beginning-position))
-                  (point)))
+                ;; Drop this prompt only if it's not gonna be inserted at BOL.
+                (if (zerop (length res))
+                    (not (bolp))
+                  (string-match res ".\\'")))
 	 ;; The need for this seems to be system-dependent:
          ;; What is this all about, exactly?  --Stef
 	 ;; (if (and (eq ?. (aref s 0)))
 	 ;;     (accept-process-output (get-buffer-process (current-buffer)) 1))
-         "")
-        ((string= s "_emacs_ok\n")
-         (when python-preoutput-continuation
-           (funcall python-preoutput-continuation)
-           (setq python-preoutput-continuation nil))
-         "")
-        ((string-match "_emacs_out \\(.*\\)\n" s)
-         (setq python-preoutput-result (match-string 1 s))
-         "")
-	((string-match ".*\n" s)
-	 s)
-	((or (eq t (compare-strings s nil nil "_emacs_ok\n" nil (length s)))
-	     (let ((end (min (length "_emacs_out ") (length s))))
-	       (eq t (compare-strings s nil end "_emacs_out " nil end))))
+           res)
+          ((let ((end (min (length "_emacs_out ") (length s))))
+             (eq t (compare-strings s nil end "_emacs_out " nil end)))
+           ;; The leftover string is a prefix of _emacs_out so we don't know
+           ;; yet whether it's an _emacs_out or something else: wait until we
+           ;; get more output so we can resolve this ambiguity.
 	 (setq python-preoutput-leftover s)
-	 "")
-        (t s)))
+           res)
+          (t (concat res s)))))
 
 (autoload 'comint-check-proc "comint")
 
@@ -1342,9 +1340,8 @@
   ;; file.  The code might be inline here, but there's enough that it
   ;; seems worth putting in a separate file, and it's probably cleaner
   ;; to put it in a module.
-  (python-send-string "import emacs")
   ;; Ensure we're at a prompt before doing anything else.
-  (python-send-receive "print '_emacs_out ()'")
+  (python-send-receive "import emacs; print '_emacs_out ()'")
   ;; Without this, help output goes into the inferior python buffer if
   ;; the process isn't already running.
   (sit-for 1 t)        ;Should we use accept-process-output instead?  --Stef
@@ -1362,9 +1359,8 @@
   (let ((end (marker-position (process-mark (python-proc)))))
     (with-current-buffer python-buffer (goto-char (point-max)))
     (compilation-forget-errors)
-    (python-send-string command)
     ;; Must wait until this has completed before re-setting variables below.
-    (python-send-receive "print '_emacs_out ()'")
+    (python-send-receive (concat comand "; print '_emacs_out ()'"))
     (with-current-buffer python-buffer
       (set-marker compilation-parsing-end end)
       (setq compilation-last-buffer (current-buffer)))))
@@ -1409,7 +1405,7 @@
   "Evaluate STRING in inferior Python process."
   (interactive "sPython command: ")
   (comint-send-string (python-proc) string)
-  (comint-send-string (python-proc) "\n\n"))
+  (comint-send-string (python-proc) "\n"))
 
 (defun python-send-buffer ()
   "Send the current buffer to the inferior Python process."

Re: python-mode: make sure output is not eaten

[Steven Huwig]

On Aug 23, 2006, at 12:04 AM, Stefan Monnier wrote:
>
> Yes, of course.  BTW do you have any idea why the current code uses  
> "\n\n"?

Might it be because the Python interactive interpreter needs the extra
newline due to significant whitespace?  E.g.

 >>> for i in range(1,5):
...     print i,
...
1 2 3 4

a DEDENT token is created by the unindented second newline, closing
the block.  A single newline cannot do that as it is also the
statement terminator.

Probably it compensates for indented text without a trailing newline.
It doesn't look like python-mode itself has any situations where this
matters, but it is possible -- but unlikely -- for user input to be
affected.

If you do the following in python-mode with both versions:

C-c C-s for i in range(1, 5): C-q C-j
<tab> print i, <return>

you will see the difference in behavior.


-- Steven Huwig

Re: python-mode: make sure output is not eaten

[Steven Huwig]

On Aug 24, 2006, at 8:18 PM, Steven Huwig wrote:

>
> Probably it compensates for indented text without a trailing newline.
> It doesn't look like python-mode itself has any situations where this
> matters, but it is possible -- but unlikely -- for user input to be
> affected.
>
> If you do the following in python-mode with both versions:
>
> C-c C-s for i in range(1, 5): C-q C-j
> <tab> print i, <return>
>
> you will see the difference in behavior.
>

To explain my post a little more, I think the new code is the right
thing.  If someone is astute enough to type in multi-line Python
commands at an Emacs interactive prompt, then they will likely be
astute enough to realize that they need to add an extra newline.

Plus I don't think the usage pattern is typical in the first place;
there is a comint buffer for such interaction.

-- Steven Huwig

Re: python-mode: make sure output is not eaten

[Steven Huwig]

On Aug 24, 2006, at 8:18 PM, Steven Huwig wrote:

>
> Probably it compensates for indented text without a trailing newline.
> It doesn't look like python-mode itself has any situations where this
> matters, but it is possible -- but unlikely -- for user input to be
> affected.
>
> If you do the following in python-mode with both versions:
>
> C-c C-s for i in range(1, 5): C-q C-j
> <tab> print i, <return>
>
> you will see the difference in behavior.
>

To explain my post a little more, I think the new code is the right
thing.  If someone is astute enough to type in multi-line Python
commands at an Emacs interactive prompt, then they will likely be
astute enough to realize that they need to add an extra newline.

Plus I don't think the usage pattern is typical in the first place;
there is a comint buffer for such interaction.

-- Steven Huwig

Re: python-mode: make sure output is not eaten

[Slawomir Nowaczyk]

On Thu, 24 Aug 2006 20:18:07 -0400
Steven Huwig <steven.huwig@gmail.com> wrote:

#> On Aug 23, 2006, at 12:04 AM, Stefan Monnier wrote:
#> >
#> > Yes, of course.  BTW do you have any idea why the current code uses  
#> > "\n\n"?
#> 
#> Might it be because the Python interactive interpreter needs the extra
#> newline due to significant whitespace?  E.g.
#> 
#>  >>> for i in range(1,5):
#> ...     print i,
#> ...
#> 1 2 3 4
#> 
#> a DEDENT token is created by the unindented second newline, closing
#> the block.  A single newline cannot do that as it is also the
#> statement terminator.
#> 
#> Probably it compensates for indented text without a trailing newline.
#> It doesn't look like python-mode itself has any situations where this
#> matters, but it is possible -- but unlikely -- for user input to be
#> affected.
#> 
#> If you do the following in python-mode with both versions:
#> 
#> C-c C-s for i in range(1, 5): C-q C-j
#> <tab> print i, <return>
#> 
#> you will see the difference in behavior.

OK, you are right... In my testing I only made sure that ending code
with indented block works fine with commands like python-send-buffer and
python-send-region, I haven't tested with bare python-send-string. Yes,
double "\n" might be an attempt to get that working.

#> To explain my post a little more, I think the new code is the right
#> thing. If someone is astute enough to type in multi-line Python
#> commands at an Emacs interactive prompt, then they will likely be
#> astute enough to realize that they need to add an extra newline.
#> 
#> Plus I don't think the usage pattern is typical in the first place;
#> there is a comint buffer for such interaction.

I agree.

Besides, if anything, TRT would be to allow people start an indented
block with C-c C-s and then *continue* it in comint buffer... but it is
neither easy nor important enough to worry about it right now, IMHO.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Write your questions down on the back of $20 dollar bill and send them to me.

Re: python-mode: make sure output is not eaten

[Slawomir Nowaczyk]

On Wed, 23 Aug 2006 00:04:40 -0400
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

#> > Also, "\n\n" in python-send-string still needs to be changed into a
#> > single "\n" in order to avoid spurious prompts.
#> 
#> Yes, of course.  BTW do you have any idea why the current code uses "\n\n"?

Not, not really -- it looks to me like an attempt to fix some prompt
problems which didn't quite work. Or it may be an attempt to have 
C-c C-s work for indented blocks, as Steven suggested.

#> > Finally, when you test this patch, make sure you also try enabling
#> > eldoc-mode (it doesn't work too well, a patch to improve it a bit is
#> > forthcoming, but even now it works for built-in functions like "apply"
#> > or "map"). It is important to make sure no extra ">>>"s get added to
#> > the inferior buffer when eldoc is talking to Python process.
#> 
#> Would it be because eldoc ends up sending commands to the
#> underlying process while some other command is still being
#> processed? Maybe we'd need a "python-process-busy" variable to
#> prevent eldoc from interfering.

Not really, at least not in my testing... the problems I am seeing
come from the fact that when eldoc sends commands, Python responds
with line that looks like "_emacs_out reload(module)\n>>> " and the
comint filter needs to remove the trailing prompt as well.

OTOH, it seems possible that we might need a "python-process-busy"
variable, as I do not know what will happen if I eldoc tries sending
commands to busy Python interpreter... but maybe there is something in
comint mode which we can test to check if subprocess is busy?

It is not related to current problems, though.

#> > This is -- partially -- the reason why it is important to save
#> > whatever text exists in front of "_emacs_out"... For example,
#> > when I python-send-buffer I can get output like "\nTest!\n>>>
#> > _emacs_out ()\n>>> "... and the ">>>" after "Test!" must show in
#> > the buffer. On the other hand, after eldoc asks Python about
#> > argument list, the output may look like this: "_emacs_out
#> > apply(object[, args[,kwargs]])\n>>> " and it is important that
#> > *nothing* is shown in the inferior buffer.
#> 
#> The handling of the prompt looks pretty messy, indeed. I think we
#> should never send a command followed by "\n" followed by "print
#> _emacs_out ()", but instead we should send the command augmented
#> with "; print _emacs_out ()\n". This should avoid those nasty
#> intermediate prompts.

I am not sure about that. It's not the intermediate prompts that are a
problem, it's rather the trailing one: which is always present, but
shouldn't really be shown.

An alternative, which I think might be a good one, would be to have
the caller specify what to do with the prompt... after all, it is not
the filter function, but the one which sends something to Python, that
really knows if prompt should be shown or not.

We could have variable python-inhibit-print which could be let-bound
by functions such as python-eldoc-function which *know* they should
never print anything...

OTOH, I like your solution of printing prompt only if there is some
other output -- seems clean to me.

#> How 'bout the patch below which also removes the _emacs_ok and
#> preoutput continuation stuff which doesn't seem to be used any
#> more.

You have misspelled command as "comand" in 
    (defun python-send-command (command)

Other than that, your patch looks OK to me. Please install, it's
definitely an improvement.

However, if you change the
   (string-match "\\`_emacs_out \\(.*\\)\n\\'" line)
to
   (string-match "\\`\\(?:>>> \\)?_emacs_out \\(.*\\)\n\\'" line)

then you do not need to do the trick of augmenting commands with ";
print" statement: things work just fine if you do python-send-string
followed by python-send-receive. 

I think this is cleaner, because adding ";" causes syntax error if the
command ends with an indented block (try C-c C-s if 1: C-q C-j    print 1
RET). As Steven pointed out, this is probably not an important use case,
but still... syntax error is a bit too much :)

If you agree, here is a patch, to be applied over yours:

--- m:/EmacsCVS/EmacsCVS/lisp/progmodes/python.el       2006-08-25 10:41:46.370473600 +0200
+++ c:/Emacs/lisp/progmodes/python.el   2006-08-25 10:47:42.112004800 +0200
@@ -1264,7 +1264,7 @@
     ;; First process whole lines.
     (while (string-match "\n" s start)
       (let ((line (substring s start (setq start (match-end 0)))))
-        (if (string-match "\\`_emacs_out \\(.*\\)\n\\'" line)
+        (if (string-match "\\`\\(?:>>> \\)?_emacs_out \\(.*\\)\n\\'" line)
             (setq python-preoutput-result (match-string 1 line))
           (setq res (concat res line)))))
     ;; Then process the remaining partial line.
@@ -1341,7 +1341,8 @@
   ;; seems worth putting in a separate file, and it's probably cleaner
   ;; to put it in a module.
   ;; Ensure we're at a prompt before doing anything else.
-  (python-send-receive "import emacs; print '_emacs_out ()'")
+  (python-send-string "import emacs")
+  (python-send-receive "print '_emacs_out ()'")
   ;; Without this, help output goes into the inferior python buffer if
   ;; the process isn't already running.
   (sit-for 1 t)        ;Should we use accept-process-output instead?  --Stef
@@ -1360,7 +1361,8 @@
     (with-current-buffer python-buffer (goto-char (point-max)))
     (compilation-forget-errors)
     ;; Must wait until this has completed before re-setting variables below.
-    (python-send-receive (concat comand "; print '_emacs_out ()'"))
+    (python-send-string command)
+    (python-send-receive "print '_emacs_out ()'")
     (with-current-buffer python-buffer
       (set-marker compilation-parsing-end end)
       (setq compilation-last-buffer (current-buffer)))))


-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

If at first you dont succeed, destroy all evidence that you tried.

Re: python-mode: make sure output is not eaten

[Ken Manheimer]

On 8/24/06, Steven Huwig <steven.huwig@gmail.com> wrote:
>
> On Aug 23, 2006, at 12:04 AM, Stefan Monnier wrote:
> >
> > Yes, of course.  BTW do you have any idea why the current code uses
> > "\n\n"?
>
> Might it be because the Python interactive interpreter needs the extra
> newline due to significant whitespace?  E.g.
>
>  >>> for i in range(1,5):
> ...     print i,
> ...
> 1 2 3 4

i haven't looked at the code in question, but have a salient point to
offer.  the handling of interactively entered python code is different
than the processing of source code from a file, specifically in that
the end of top-level blocks is detected by an empty line of input.

this is necessary because, during interaction, you don't want to
require people to enter the first line of the subsequent top-level
construct to determine that the previous top-level construct has
concluded.  instead, it's assumed they've concluded when an empty line
is encountered.

i suspect that the "\n\n" you all are referring to is specifically to
provide that newline.

> a DEDENT token is created by the unindented second newline, closing
> the block.  A single newline cannot do that as it is also the
> statement terminator.

... and a double newline doesn't have this effect for file input,
where the interpreter/compiler *can* wait for the first line of the
next top-level construct, or end-of-file.

 > [...]

-- 
ken
ken.manheimer@gmail.com
http://myriadicity.net

Re: python-mode: make sure output is not eaten

[Stefan Monnier]

> i suspect that the "\n\n" you all are referring to is specifically to
> provide that newline.

Indeed.  But I think it was a mistake for python.el to do that.  I suspect
that in most cases, if we want to send multi-line statements to the python
process, we should do it via a temp file, but even if we don't it's
preferable to have a separate command.  Or at least, only add "\n\n" if the
string already contains a \n.

The main problem with sending multiple lines directly is that you get
multiple prompts in return and it's difficult for the process filter to drop
all the unneeded ones without risking dropping some needed ones also.


        Stefan

Re: python-mode: make sure output is not eaten

[Stefan Monnier]

> #> Yes, of course.  BTW do you have any idea why the current code uses "\n\n"?

> Not, not really -- it looks to me like an attempt to fix some prompt
> problems which didn't quite work. Or it may be an attempt to have 
> C-c C-s work for indented blocks, as Steven suggested.

I've installed a half-way solution which I believe keeps the old behavior
when needed.

> Not really, at least not in my testing... the problems I am seeing
> come from the fact that when eldoc sends commands, Python responds
> with line that looks like "_emacs_out reload(module)\n>>> " and the
> comint filter needs to remove the trailing prompt as well.

I've added python-preoutput-skip-next-prompt for that purpose.  This way
such prompts should get removed unambiguously.

> OTOH, I like your solution of printing prompt only if there is some
> other output -- seems clean to me.

That's just what was there before (modulo the fact that it didn't catch all
situations).

> You have misspelled command as "comand" in 
>     (defun python-send-command (command)

Thanks.  Fixed.

> Other than that, your patch looks OK to me. Please install, it's
> definitely an improvement.

I've installed a variation of it.  Please fix it as you see fit.

> However, if you change the
>    (string-match "\\`_emacs_out \\(.*\\)\n\\'" line)
> to
>    (string-match "\\`\\(?:>>> \\)?_emacs_out \\(.*\\)\n\\'" line)

> then you do not need to do the trick of augmenting commands with ";
> print" statement: things work just fine if you do python-send-string
> followed by python-send-receive.

Silently interpreting/removing output from the process is always risky,
since the output may appear for other reasons than the ones we expected.
So I'd rather be extra careful to only recognize _emacs_out when I'm sure
it's really the internal thingy.


        Stefan

Re: python-mode: make sure output is not eaten

[Slawomir Nowaczyk]

On Fri, 25 Aug 2006 18:53:41 -0400
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

#> > Not, not really -- it looks to me like an attempt to fix some
#> > prompt problems which didn't quite work. Or it may be an attempt to
#> > have C-c C-s work for indented blocks, as Steven suggested.
#> 
#> I've installed a half-way solution which I believe keeps the old
#> behavior when needed.

Have you installed the complete patch? Because current version of the
CVS code doesn't quite work for me... it hangs emacs whenever I try
python-send-buffer (I need C-g to get it make it responsive again),
which seems to be related to the "while" loop in python-send-receive
never finishing due to the (local-variable-p 'python-preoutput-result)
in python-preoutput-filter never being true...

I do not understand the purpose of most of your code, so I do not know
how should it be fixed.

In general, I do not particularly like your patch (I am talking about
prompt handling, there are a couple of other things you have changed
which look OK).

I still think 
  (python-send-string "import emacs")
  (python-send-receive "print '_emacs_out ()'")
is cleaner than
  (python-send-receive "import emacs; print '_emacs_out ()'")

I do not understand why "command" python-send-command shouldn't contain
newline... It should work just fine with multiple lines, the only thing
which might cause problems is indentation, and even that only if there
is an indented block at the very end.

If you are thinking about conditionally adding one or two newlines after
python-send-string, than the correct condition should be "add double
newline if and only if the last line is indented", IMHO.

I do not understand the purpose of python-preoutput-result variable. It
is used in python-send-receive and in python-preoutput-filter, but those
uses seem unrelated... if I read the code correctly, python-send-receive
will call (make-local-variable 'python-preoutput-filter), but it will
also always call (kill-local-variable 'python-preoutput-filter)... what
is the point of it? And when could testing (local-variable-p
'python-preoutput-result) in python-preoutput-filter make sense?

Also, python-mode-running doesn't seem to be used anywhere.

#> > Not really, at least not in my testing... the problems I am seeing
#> > come from the fact that when eldoc sends commands, Python responds
#> > with line that looks like "_emacs_out reload(module)\n>>> " and the
#> > comint filter needs to remove the trailing prompt as well.
#> 
#> I've added python-preoutput-skip-next-prompt for that purpose.  This way
#> such prompts should get removed unambiguously.

I do not think this is the right way to solve the problem. IMHO, the
variable should be called python-inhibit-output and should prevent
*anything* from being printed in Inferior Python buffer -- thus,
functions like eldoc could bind it to 't and we would not need to worry
about them at all.

After all, it doesn't matter how the output of python-eldoc-function
looks like, it should never appear in the comint buffer. And it would
let us simplify the common case of *user* wanting to run Python code.

#> > However, if you change the
#> >    (string-match "\\`_emacs_out \\(.*\\)\n\\'" line)
#> > to
#> >    (string-match "\\`\\(?:>>> \\)?_emacs_out \\(.*\\)\n\\'" line)
#> 
#> > then you do not need to do the trick of augmenting commands with ";
#> > print" statement: things work just fine if you do python-send-string
#> > followed by python-send-receive.
#> 
#> Silently interpreting/removing output from the process is always risky,
#> since the output may appear for other reasons than the ones we expected.
#> So I'd rather be extra careful to only recognize _emacs_out when I'm sure
#> it's really the internal thingy.

I do not quite understand... You never *know* that it is the internal
thingy anyway (try typing "print '_emacs_out'" in the Inferior Python
buffer)...

I do not see the danger of removing the prompt in front of _emacs_out.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

If vegetarians love animals so much, why do they eat all their food???

Re: python-mode: make sure output is not eaten

[Stefan Monnier]

> #> > Not, not really -- it looks to me like an attempt to fix some
> #> > prompt problems which didn't quite work. Or it may be an attempt to
> #> > have C-c C-s work for indented blocks, as Steven suggested.
> #> 
> #> I've installed a half-way solution which I believe keeps the old
> #> behavior when needed.

> Have you installed the complete patch? Because current version of the
> CVS code doesn't quite work for me... it hangs emacs whenever I try
> python-send-buffer (I need C-g to get it make it responsive again),
> which seems to be related to the "while" loop in python-send-receive
> never finishing due to the (local-variable-p 'python-preoutput-result)
> in python-preoutput-filter never being true...

Hmm... looks like the two pieces of code don't run in the same buffer maybe?
I've just installed a patch which may fix it.

> I still think 
>   (python-send-string "import emacs")
>   (python-send-receive "print '_emacs_out ()'")
> is cleaner than
>   (python-send-receive "import emacs; print '_emacs_out ()'")

Again, here's the issue: the process's output can be absolutely arbitrary.
If we see "\n>>> " somewhere in the process's output we have in general no
guarantee that it's actually a prompt.  Similarly if we see "_emacs_out " we
have in general no guarantee that it's coming from python.el or from
emacs.py.  So we should be careful in the process filter to be as selective
as possible in what we remove.

In interactive use, it's perfectly OK to do:

   >>> import emacs
   >>> print '_emacs_out ()'
   _emacs_out ()
   >>> 

but from python.el this basically turns into:

   >>> import emacs
   print '_emacs_out ()'
   >>> _emacs_out ()
   >>> 

because we don't wait for the prompt in between the two.  That's not
something you'd normally do interactively.  Instead if you didn't want to
wait for the prompt between the two, you'd do something like:

   >>> import emacs; print '_emacs_out ()'
   _emacs_out ()
   >>> 

which has the advantage that the _emacs_out () thingy appears at its normal
place so you can be more selective and risk misidentifying it in
fewer cases.

> I do not understand why "command" python-send-command shouldn't contain
> newline...

It's not a command, so it can only called from elisp code.  Check the
callers and you'll see it is never called with a newline.  So it's OK to
make such an assumption.

> It should work just fine with multiple lines, the only thing which might
> cause problems is indentation, and even that only if there is an indented
> block at the very end.

Sending a multiline thingy via python-send-string will look like absolute
crap in the resulting output, because all the intermediate prompts will get
concatenated rather than interleaved with the input.  So I see no point in
trying to support this in a case such as `python-send-command' where we
don't use this flexibility anyway (and where we need to analyze the output,
so the more control over it the better).

> If you are thinking about conditionally adding one or two newlines after
> python-send-string, than the correct condition should be "add double
> newline if and only if the last line is indented", IMHO.

Sure, go ahead.  I just tried a simple heuristic to preserve the previous
"\n\n" thingy without imposing it in the case where I know it's unneeded and
even annoying.

> I do not understand the purpose of python-preoutput-result variable. It
> is used in python-send-receive and in python-preoutput-filter, but those
> uses seem unrelated...

They're very much related: the accept-process-output call in
python-send-receive will let process filters run, so it's basically an
indirect call to python-preoutput-filter.

> if I read the code correctly, python-send-receive
> will call (make-local-variable 'python-preoutput-filter), but it will
> also always call (kill-local-variable 'python-preoutput-filter)... what
> is the point of it? And when could testing (local-variable-p
> 'python-preoutput-result) in python-preoutput-filter make sense?

Yes, it checks that the filter is being run from within the
accept-process-output of python-send-receive.

> Also, python-mode-running doesn't seem to be used anywhere.

It's used right where it's used ;-)

  (unless (boundp 'python-mode-running)	; kill the recursion from jython-mode
    (let ((python-mode-running t))
      (python-maybe-jython))))

i.e. it's used to prevent python-maybe-jython -> jython-mode ->
python-mode -> python-maybe-jython -> jython-mode -> python-mode ->
python-maybe-jython -> ...

> #> > Not really, at least not in my testing... the problems I am seeing
> #> > come from the fact that when eldoc sends commands, Python responds
> #> > with line that looks like "_emacs_out reload(module)\n>>> " and the
> #> > comint filter needs to remove the trailing prompt as well.
> #> 
> #> I've added python-preoutput-skip-next-prompt for that purpose.  This way
> #> such prompts should get removed unambiguously.

> I do not think this is the right way to solve the problem. IMHO, the
> variable should be called python-inhibit-output and should prevent
> *anything* from being printed in Inferior Python buffer -- thus,
> functions like eldoc could bind it to 't and we would not need to worry
> about them at all.

> After all, it doesn't matter how the output of python-eldoc-function
> looks like, it should never appear in the comint buffer. And it would
> let us simplify the common case of *user* wanting to run Python code.

Sounds good.  As long as you can be sure that there can be no other output
at the same time (i.e. the python process is idle, waiting at the prompt
when you send the eargs command).  Currently, we do not check that python is
idle, so we have no guarantee that we can just discard all the output
between after we send the `eargs' (ou modpath, or complete, ...) until after
we see the _emacs_out.

> #> > However, if you change the
> #> >    (string-match "\\`_emacs_out \\(.*\\)\n\\'" line)
> #> > to
> #> >    (string-match "\\`\\(?:>>> \\)?_emacs_out \\(.*\\)\n\\'" line)
> #> 
> #> > then you do not need to do the trick of augmenting commands with ";
> #> > print" statement: things work just fine if you do python-send-string
> #> > followed by python-send-receive.
> #> 
> #> Silently interpreting/removing output from the process is always risky,
> #> since the output may appear for other reasons than the ones we expected.
> #> So I'd rather be extra careful to only recognize _emacs_out when I'm sure
> #> it's really the internal thingy.

> I do not quite understand... You never *know* that it is the internal
> thingy anyway (try typing "print '_emacs_out'" in the Inferior Python
> buffer)...

Thanks to the local-variable-p check, the filter should now correctly let it
go through, except for unusual cases.


        Stefan

Re: python-mode: make sure output is not eaten

[Slawomir Nowaczyk]

On Sat, 26 Aug 2006 10:48:25 -0400
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

#> > Have you installed the complete patch? Because current version of the
#> > CVS code doesn't quite work for me... it hangs emacs whenever I try
#> > python-send-buffer (I need C-g to get it make it responsive again),
#> > which seems to be related to the "while" loop in python-send-receive
#> > never finishing due to the (local-variable-p 'python-preoutput-result)
#> > in python-preoutput-filter never being true...
#> 
#> Hmm... looks like the two pieces of code don't run in the same buffer maybe?
#> I've just installed a patch which may fix it.

I works better -- no longer hangs my Emacs, but after python-send-buffer
the final prompt still doesn't appear.

Commenting out (set (make-local-variable 'python-preoutput-skip-next-prompt) t))
in python-preoutput-filter makes it work for me.

#> > I still think 
#> >   (python-send-string "import emacs")
#> >   (python-send-receive "print '_emacs_out ()'")
#> > is cleaner than
#> >   (python-send-receive "import emacs; print '_emacs_out ()'")
#> 
#> Again, here's the issue: the process's output can be absolutely
#> arbitrary. If we see "\n>>> " somewhere in the process's output we
#> have in general no guarantee that it's actually a prompt. Similarly
#> if we see "_emacs_out " we have in general no guarantee that it's
#> coming from python.el or from emacs.py. So we should be careful in
#> the process filter to be as selective as possible in what we remove.

Agreed. But why is removing "_emacs_out ()" any more selective than
removing ">>> _emacs_out ()"?

#> In interactive use, it's perfectly OK to do:
#> 
#>    >>> import emacs
#>    >>> print '_emacs_out ()'
#>    _emacs_out ()
#>    >>> 
#> 
#> but from python.el this basically turns into:
#> 
#>    >>> import emacs
#>    print '_emacs_out ()'
#>    >>> _emacs_out ()
#>    >>> 
#> 
#> because we don't wait for the prompt in between the two.  That's not
#> something you'd normally do interactively.  Instead if you didn't want to
#> wait for the prompt between the two, you'd do something like:
#> 
#>    >>> import emacs; print '_emacs_out ()'
#>    _emacs_out ()
#>    >>> 
#> 
#> which has the advantage that the _emacs_out () thingy appears at its normal
#> place so you can be more selective and risk misidentifying it in
#> fewer cases.

I think I understand your reasoning, but I do not think I agree with it.
If we *want* to wait for prompt after both "import ..." and "print ...",
we can always use python-send-receive twice, instead of
python-send-string followed by python-send-receive. But I still do not
see why one is supposed to be more robust than the other.

Also, I have finally found a case in which adding "; print ..." breaks
things: assume the string is "if False: print 1". Observe that "if
False: print 1; print '_emacs_out ()'" does *not* print anything.

It may not be a *real* problem, but it does -- IMHO -- show that sending
two separate commands *is* more robust.

#> > I do not understand why "command" python-send-command shouldn't
#> > contain newline...
#> 
#> It's not a command, so it can only called from elisp code. Check the
#> callers and you'll see it is never called with a newline. So it's OK
#> to make such an assumption.

Well, maybe. But still, either python-send-string works for multiline
strings (in which case python-send-command should work as well) or
python-send-string doesn't work for multiline strings, in which case the
assert in python-send-command doesn't help at all (as you wrote, nothing
can call it with multiline command anyway).

#> > It should work just fine with multiple lines, the only thing which
#> > might cause problems is indentation, and even that only if there is
#> > an indented block at the very end.
#> 
#> Sending a multiline thingy via python-send-string will look like
#> absolute crap in the resulting output,

I disagree...

#> because all the intermediate prompts will get concatenated rather
#> than interleaved with the input.

Right, but what's wrong with that? If I send a string like "a = 1\n
print 'Test'\n b = 2", what's wrong with getting output like
">>> Test\n>>> \n >>>"?

Sure, doing that may not be the smartest thing to do, comint buffers
work much better for this kind of stuff, but still... the output
actually makes sense.

#> So I see no point in trying to support this in a case such as
#> `python-send-command' where we don't use this flexibility anyway (and
#> where we need to analyze the output, so the more control over it the
#> better).

You may be right... but if you think it makes sense, the assert should
be in python-send-string, not in python-send-command.

#> > If you are thinking about conditionally adding one or two newlines after
#> > python-send-string, than the correct condition should be "add double
#> > newline if and only if the last line is indented", IMHO.
#> 
#> Sure, go ahead.  I just tried a simple heuristic to preserve the previous
#> "\n\n" thingy without imposing it in the case where I know it's unneeded and
#> even annoying.
#> 
#> > I do not understand the purpose of python-preoutput-result variable. It
#> > is used in python-send-receive and in python-preoutput-filter, but those
#> > uses seem unrelated...
#> 
#> They're very much related: the accept-process-output call in
#> python-send-receive will let process filters run, so it's basically an
#> indirect call to python-preoutput-filter.

OK, I see... I forgot the filter function can be called from within
accept-process-output.

#> > Also, python-mode-running doesn't seem to be used anywhere.
#> 
#> It's used right where it's used ;-)
#> 
#>   (unless (boundp 'python-mode-running)	; kill the recursion from jython-mode
#>     (let ((python-mode-running t))
#>       (python-maybe-jython))))
#> 
#> i.e. it's used to prevent python-maybe-jython -> jython-mode ->
#> python-mode -> python-maybe-jython -> jython-mode -> python-mode ->
#> python-maybe-jython -> ...

OK, I was mislead by the name... I thought it was supposed to be an
implementation of a feature you mentioned earlier, a check whether
Python interpreter is busy so that things like eldoc won't be called
when they shouldn't.

#> > I do not think this is the right way to solve the problem. IMHO, the
#> > variable should be called python-inhibit-output and should prevent
#> > *anything* from being printed in Inferior Python buffer -- thus,
#> > functions like eldoc could bind it to 't and we would not need to worry
#> > about them at all.
#> 
#> > After all, it doesn't matter how the output of python-eldoc-function
#> > looks like, it should never appear in the comint buffer. And it would
#> > let us simplify the common case of *user* wanting to run Python code.
#> 
#> Sounds good. As long as you can be sure that there can be no other
#> output at the same time (i.e. the python process is idle, waiting at
#> the prompt when you send the eargs command). Currently, we do not
#> check that python is idle, so we have no guarantee that we can just
#> discard all the output between after we send the `eargs' (ou modpath,
#> or complete, ...) until after we see the _emacs_out.

I have hard time imagining how the current code could work at all if it
send anything to Python interpreter while it was busy, but I will take a
closer look at this.

#> > I do not quite understand... You never *know* that it is the internal
#> > thingy anyway (try typing "print '_emacs_out'" in the Inferior Python
#> > buffer)...
#> 
#> Thanks to the local-variable-p check, the filter should now correctly let it
#> go through, except for unusual cases.

Yes, I see, it's working now.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Hard Disk:  A device that allows users to delete vast quantities of
data with simple mnemonic commands.

Re: python-mode: make sure output is not eaten

[Stefan Monnier]

> #> > Have you installed the complete patch? Because current version of the
> #> > CVS code doesn't quite work for me... it hangs emacs whenever I try
> #> > python-send-buffer (I need C-g to get it make it responsive again),
> #> > which seems to be related to the "while" loop in python-send-receive
> #> > never finishing due to the (local-variable-p 'python-preoutput-result)
> #> > in python-preoutput-filter never being true...
> #> 
> #> Hmm... looks like the two pieces of code don't run in the same buffer
> #> maybe?  I've just installed a patch which may fix it.

> I works better -- no longer hangs my Emacs, but after python-send-buffer
> the final prompt still doesn't appear.

I believe my last patch fixes this as well.

> Agreed. But why is removing "_emacs_out ()" any more selective than
> removing ">>> _emacs_out ()"?

We're talking about "^_emacs_out " vs "^\\(>>> \\)?_emacs_out": clearly one
of the two regexps describes a superset of the other.  If you can guarantee
that it'll always match "^>>> _emacs_out ", then it's a different question.

> I think I understand your reasoning, but I do not think I agree with it.
> If we *want* to wait for prompt after both "import ..." and "print ...",
> we can always use python-send-receive twice, instead of
> python-send-string followed by python-send-receive.  But I still do not
> see why one is supposed to be more robust than the other.

python-send-receive doesn't wait for a prompt but for the _emacs_out cookie,
which the `import' does not output.

> Also, I have finally found a case in which adding "; print ..." breaks
> things: assume the string is "if False: print 1". Observe that "if
> False: print 1; print '_emacs_out ()'" does *not* print anything.

> It may not be a *real* problem, but it does -- IMHO -- show that sending
> two separate commands *is* more robust.

It's not the only case, and I agree it's not a purely academic question.
But: such cases does not occur in python.el (where we only add the
"; print ..." at a few special spots where we have control over the text
that precedes it).

> #> > I do not understand why "command" python-send-command shouldn't
> #> > contain newline...
> #> 
> #> It's not a command, so it can only called from elisp code. Check the
> #> callers and you'll see it is never called with a newline. So it's OK
> #> to make such an assumption.

> Well, maybe. But still, either python-send-string works for multiline
> strings (in which case python-send-command should work as well)

I see no reason why the two should be so linked.

> or python-send-string doesn't work for multiline strings, in which case
> the assert in python-send-command doesn't help at all (as you wrote,
> nothing can call it with multiline command anyway).

In python-send-string, multiline strings work.  In python-send-command
multiline strings were very likely to fail because we added "; print ...".
Hence the presence of an `assert' (assertions are of no use to the user,
their only meaning is for the programmer as a form of documentation that's
slightly better sync'd with the code than mere comments).

> #> > It should work just fine with multiple lines, the only thing which
> #> > might cause problems is indentation, and even that only if there is
> #> > an indented block at the very end.
> #> 
> #> Sending a multiline thingy via python-send-string will look like
> #> absolute crap in the resulting output,

> I disagree...

Actually, you seem to be right: the prompt-elimination heuristic seems to
work well for this in practice.

Although it points top another problem: the result of the evaluation is
printed on the line of the prompt.  I.e. if I start with

   >>> 

after python-send-string "print 1" I get

   >>> 1
   >>> 

rather than

   >>> 
   1
   >>> 

or

   >>> print 1
   1
   >>> 

> #> because all the intermediate prompts will get concatenated rather
> #> than interleaved with the input.

> Right, but what's wrong with that? If I send a string like "a = 1\n
> print 'Test'\n b = 2", what's wrong with getting output like
> ">>> Test\n>>> \n >>>"?

In a session log, the prompt is expected to be followed by user input, not
program output.

> #> So I see no point in trying to support this in a case such as
> #> `python-send-command' where we don't use this flexibility anyway (and
> #> where we need to analyze the output, so the more control over it the
> #> better).

> You may be right... but if you think it makes sense, the assert should
> be in python-send-string, not in python-send-command.

I see no point in restricting the use of python-send-string where the only
potential problem is ugly output, but not incorrect execution.  Whereas in
python-send-command, the result could have been incorrect execution.
Also adding the assert to python-send-command did not restrict the user in
any way, only the python.el hacker.

> #> > I do not think this is the right way to solve the problem. IMHO, the
> #> > variable should be called python-inhibit-output and should prevent
> #> > *anything* from being printed in Inferior Python buffer -- thus,
> #> > functions like eldoc could bind it to 't and we would not need to worry
> #> > about them at all.
> #> 
> #> > After all, it doesn't matter how the output of python-eldoc-function
> #> > looks like, it should never appear in the comint buffer. And it would
> #> > let us simplify the common case of *user* wanting to run Python code.
> #> 
> #> Sounds good. As long as you can be sure that there can be no other
> #> output at the same time (i.e. the python process is idle, waiting at
> #> the prompt when you send the eargs command). Currently, we do not
> #> check that python is idle, so we have no guarantee that we can just
> #> discard all the output between after we send the `eargs' (ou modpath,
> #> or complete, ...) until after we see the _emacs_out.

> I have hard time imagining how the current code could work at all if it
> send anything to Python interpreter while it was busy, but I will take a
> closer look at this.

Indeed, if the python process is busy when we send the command, the result
may be wrong.  But I expect that in most cases it'll be right.  Still,
I think checking busy/idle-ness is a good idea in itself and your above
suggestion as well.

Note that checking idle/busy-ness can be tricky.  Even assuming the
program's output does not look like a prompt, you may have to count the
number of newlines sent to the python's process to know how many prompts to
expect in return.  Or alternatively, you'd have to send each input line one
at a time, waiting for the prompt before sending the next one.


        Stefan

Re: python-mode: make sure output is not eaten

[Slawomir Nowaczyk]

On Mon, 28 Aug 2006 17:43:46 -0400
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

#> > I works better -- no longer hangs my Emacs, but after python-send-buffer
#> > the final prompt still doesn't appear.
#> 
#> I believe my last patch fixes this as well.

Yes, it seems to work.

#> > Agreed. But why is removing "_emacs_out ()" any more selective than
#> > removing ">>> _emacs_out ()"?
#> 
#> We're talking about "^_emacs_out " vs "^\\(>>> \\)?_emacs_out": clearly one
#> of the two regexps describes a superset of the other.  If you can guarantee
#> that it'll always match "^>>> _emacs_out ", then it's a different question.

I do not think it is a problem... The following patch seems to work for me:

************************************************************

--- m:/EmacsCVS/EmacsCVS/lisp/progmodes/python.el       2006-08-30 17:05:09.399865600 +0200
+++ c:/Emacs/lisp/progmodes/python.el   2006-08-30 21:26:32.132132800 +0200
@@ -1271,7 +1271,7 @@
           (setq python-preoutput-skip-next-prompt nil)
           (setq line (substring line (match-end 0))))
         ;; Recognize special _emacs_out lines.
-        (if (and (string-match "\\`_emacs_out \\(.*\\)\n\\'" line)
+        (if (and (string-match "\\`>>> _emacs_out \\(.*\\)\n\\'" line)
                  (local-variable-p 'python-preoutput-result))
             (progn
               (setq python-preoutput-result (match-string 1 line))
@@ -1351,7 +1351,8 @@
       ;; seems worth putting in a separate file, and it's probably cleaner
       ;; to put it in a module.
       ;; Ensure we're at a prompt before doing anything else.
-      (python-send-receive "import emacs; print '_emacs_out ()'")))
+      (python-send-string "import emacs")
+      (python-send-receive "print '_emacs_out ()'")))
   (if (derived-mode-p 'python-mode)
       (setq python-buffer (default-value 'python-buffer))) ; buffer-local
   ;; Without this, help output goes into the inferior python buffer if
@@ -1601,7 +1602,7 @@
                  (current-word))))
     ;; First try the symbol we're on.
     (or (and symbol
-            (python-send-receive (format "emacs.eargs(%S, %s)"
+            (python-send-receive (format "\nemacs.eargs(%S, %s)"
                                          symbol python-imports)))
        ;; Try moving to symbol before enclosing parens.
        (let ((s (syntax-ppss)))
@@ -1614,9 +1615,9 @@
                  (skip-chars-backward "a-zA-Z._")
                  (if (< (point) point)
                      (python-send-receive
-                      (format "emacs.eargs(%S, %s)"
-                              (buffer-substring-no-properties (point) point)
-                              python-imports)))))))))))
+                      (format "\nemacs.eargs(%S, %s)"
+                               (buffer-substring-no-properties (point) point)
+                               python-imports)))))))))))
\f
;;;; Info-look functionality.

**********************************************************************
#> > I think I understand your reasoning, but I do not think I agree
#> > with it. If we *want* to wait for prompt after both "import ..."
#> > and "print ...", we can always use python-send-receive twice,
#> > instead of python-send-string followed by python-send-receive. But
#> > I still do not see why one is supposed to be more robust than the
#> > other.
#> 
#> python-send-receive doesn't wait for a prompt but for the _emacs_out
#> cookie, which the `import' does not output.

Right, my mistake.

#> > Also, I have finally found a case in which adding "; print ..."
#> > breaks things: assume the string is "if False: print 1". Observe
#> > that "if False: print 1; print '_emacs_out ()'" does *not* print
#> > anything.
#> 
#> > It may not be a *real* problem, but it does -- IMHO -- show that
#> > sending two separate commands *is* more robust.
#> 
#> It's not the only case, and I agree it's not a purely academic
#> question. But: such cases does not occur in python.el (where we only
#> add the "; print ..." at a few special spots where we have control
#> over the text that precedes it).

Yes, I know. My main objection -- although I admit, it is a purely
aesthetic one -- is that if we cannot make the code work right for the
simple case of sending two commands, the underlying code is too fragile.

#> > #> It's not a command, so it can only called from elisp code. Check the
#> > #> callers and you'll see it is never called with a newline. So it's OK
#> > #> to make such an assumption.
#> 
#> > Well, maybe. But still, either python-send-string works for multiline
#> > strings (in which case python-send-command should work as well)
#> 
#> I see no reason why the two should be so linked.

python-send-command does very little beyond calling python-send-string.
Definitely nothing that would make reasonable for only one of them to
work with multiline strings.

#> > or python-send-string doesn't work for multiline strings, in which case
#> > the assert in python-send-command doesn't help at all (as you wrote,
#> > nothing can call it with multiline command anyway).
#> 
#> In python-send-string, multiline strings work.  In python-send-command
#> multiline strings were very likely to fail because we added "; print ...".
#> Hence the presence of an `assert' (assertions are of no use to the user,
#> their only meaning is for the programmer as a form of documentation that's
#> slightly better sync'd with the code than mere comments).

In other words, the only reason for the assert is that we have chosen to
add "; print" instead of sending two commands?

#> > #> > It should work just fine with multiple lines, the only thing which
#> > #> > might cause problems is indentation, and even that only if there is
#> > #> > an indented block at the very end.
#> > #> 
#> > #> Sending a multiline thingy via python-send-string will look like
#> > #> absolute crap in the resulting output,
#> 
#> > I disagree...
#> 
#> Actually, you seem to be right: the prompt-elimination heuristic
#> seems to work well for this in practice.
#> 
#> Although it points top another problem: the result of the evaluation
#> is printed on the line of the prompt. I.e. if I start with
#> 
#>    >>> 
#> 
#> after python-send-string "print 1" I get
#> 
#>    >>> 1
#>    >>> 
#> 
#> rather than
#> 
#>    >>> 
#>    1
#>    >>> 
#> 
#> or
#> 
#>    >>> print 1
#>    1
#>    >>> 

Yes, I know... that is on the list of bugs I want to fix. I was just
waiting for the current code to stabilise. I believe that the right
thing to do would be to print, in the above case, something like:

>>> # evaluating buffer <test.py>
1
>>> 

or

>>> # evaluating region in buffer <test.py>
1
>>> 

The other mode, python-mode.el, uses something like this, and I remember
it was really helpful, especially when I worked with multiple files. It
really allowed me to see what was happening.

I am not sure what should be printed by python-send-string, but maybe
simple "print 1" would work best.

#> > #> because all the intermediate prompts will get concatenated rather
#> > #> than interleaved with the input.
#> 
#> > Right, but what's wrong with that? If I send a string like "a = 1\n
#> > print 'Test'\n b = 2", what's wrong with getting output like
#> > ">>> Test\n>>> \n >>>"?
#> 
#> In a session log, the prompt is expected to be followed by user
#> input, not program output.

Sure... some day we might want to do that. But it's pretty far down on
my list of priorities.

#> > #> So I see no point in trying to support this in a case such as
#> > #> `python-send-command' where we don't use this flexibility anyway (and
#> > #> where we need to analyze the output, so the more control over it the
#> > #> better).
#> 
#> > You may be right... but if you think it makes sense, the assert should
#> > be in python-send-string, not in python-send-command.
#> 
#> I see no point in restricting the use of python-send-string where the only
#> potential problem is ugly output, but not incorrect execution.

I agree.

#> Whereas in python-send-command, the result could have been incorrect
#> execution.

Yes, but -- IMHO -- this is only due to the bad design decision of
adding "; print"

#> Also adding the assert to python-send-command did not restrict the
#> user in any way, only the python.el hacker.

Right. Fine, this is probably not important enough to warrant further
discussion.

#> > I have hard time imagining how the current code could work at all
#> > if it send anything to Python interpreter while it was busy, but I
#> > will take a closer look at this.
#> 
#> Indeed, if the python process is busy when we send the command, the
#> result may be wrong. But I expect that in most cases it'll be right.

It seems to work to some degree in my very limited testing... but there
are a lot of ways to break it. There is some kind of queueing
implemented in python-mode.el, which detects busy interpreter and waits
with sending commands... but I haven't checked how is it implemented.

#> Still, I think checking busy/idle-ness is a good idea in itself and
#> your above suggestion as well.

I agree.

#> Note that checking idle/busy-ness can be tricky. Even assuming the
#> program's output does not look like a prompt, you may have to count
#> the number of newlines sent to the python's process to know how many
#> prompts to expect in return. Or alternatively, you'd have to send
#> each input line one at a time, waiting for the prompt before sending
#> the next one.

I was only thinking about checking busyness of things like
python-send-region -- when we can wait for _emacs_out. I believe it is
not worth the trouble to worry about python-send-string.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Hardware: The parts of a computer system that can be kicked.

Re: python-mode: make sure output is not eaten

[Stefan Monnier]

> #> > Agreed. But why is removing "_emacs_out ()" any more selective than
> #> > removing ">>> _emacs_out ()"?
> #> 
> #> We're talking about "^_emacs_out " vs "^\\(>>> \\)?_emacs_out": clearly
> #> one of the two regexps describes a superset of the other.  If you can
> #> guarantee that it'll always match "^>>> _emacs_out ", then it's
> #> a different question.

> I do not think it is a problem... The following patch seems to work for me:

But in what way is that better?

> #> I see no reason why the two should be so linked.
> python-send-command does very little beyond calling python-send-string.

It did a lot more: it waited for the command to terminate, which involves
monitoring the output, trying to recognize some special pattern in there.

> #> > or python-send-string doesn't work for multiline strings, in which case
> #> > the assert in python-send-command doesn't help at all (as you wrote,
> #> > nothing can call it with multiline command anyway).
> #> 
> #> In python-send-string, multiline strings work.  In python-send-command
> #> multiline strings were very likely to fail because we added "; print ...".
> #> Hence the presence of an `assert' (assertions are of no use to the user,
> #> their only meaning is for the programmer as a form of documentation that's
> #> slightly better sync'd with the code than mere comments).

> In other words, the only reason for the assert is that we have chosen to
> add "; print" instead of sending two commands?

Yup.

> I am not sure what should be printed by python-send-string, but maybe
> simple "print 1" would work best.

If the string is simple/short, then it should be printed, as if the user had
typed it in.

> Yes, but -- IMHO -- this is only due to the bad design decision of
> adding "; print"

;-)

> #> Also adding the assert to python-send-command did not restrict the
> #> user in any way, only the python.el hacker.

> Right. Fine, this is probably not important enough to warrant further
> discussion.

Especially since python-send-command doesn't wait for the command to
terminate any more, so it doesn't add any `print' any more, whether on the
same line or on a separate one.


        Stefan

Re: python-mode: make sure output is not eaten

[Slawomir Nowaczyk]

On Wed, 30 Aug 2006 18:10:33 -0400
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

#>>>> Agreed. But why is removing "_emacs_out ()" any more selective
#>>>> than removing ">>> _emacs_out ()"?
#>>> 
#>>> We're talking about "^_emacs_out " vs "^\\(>>> \\)?_emacs_out":
#>>> clearly one of the two regexps describes a superset of the other.
#>>> If you can guarantee that it'll always match "^>>> _emacs_out ",
#>>> then it's a different question.
#>
#>> I do not think it is a problem... The following patch seems to work
#>> for me:
#> 
#> But in what way is that better?

It probably isn't... but it does guarantee I can always match "^>>> _emacs_out" :)

#> > #> Also adding the assert to python-send-command did not restrict
#> > #> the user in any way, only the python.el hacker.
#> 
#> > Right. Fine, this is probably not important enough to warrant
#> > further discussion.
#> 
#> Especially since python-send-command doesn't wait for the command to
#> terminate any more, so it doesn't add any `print' any more, whether
#> on the same line or on a separate one.

Exactly.

-- 
 Best wishes,
   Slawomir Nowaczyk
     ( slawomir.nowaczyk.847@student.lu.se )

Be nice to other people--they outnumber you six billion to one.