bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

[Eli Zaretskii]

4 of the Python tests fail on MS-Windows due to some issue related to
the prompts displayed by the Python interpreter.  The 4 failed tests
are:

   FAILED  python-shell-make-comint-4
   FAILED  python-shell-prompt-detect-1
   FAILED  python-shell-prompt-detect-2
   FAILED  python-shell-prompt-set-calculated-regexps-6

I show below the relevant portions of python-tests.log.

I need help in figuring out what causes these.  One possible suspect
is some buffering issue, due to which Emacs does not receive the
prompt that Python prints.  Another possibility might be the version
of the Python (I have 2.6.6 installed).  Or maybe something else.

Let me know if I can collect more information about the failures, or
you'd like me to try something in order to find out the reasons.

Thanks.

Here are the relevant portions of the test log for the 4 failed tests:

  Test python-shell-make-comint-4 condition:
      (ert-test-failed
       ((should
	 (string= python-shell--prompt-calculated-input-regexp
		  (concat "^\\(extralargeinputprompt\\|\\.\\.> \\|" "block\\|py> \\|pdf\\|sml\\|in\\)")))
	:form
	(string= "^\\(extralargeinputprompt\\|block\\|pdf\\|sml\\|in\\)" "^\\(extralargeinputprompt\\|\\.\\.> \\|block\\|py> \\|pdf\\|sml\\|in\\)")
	:value nil))
     FAILED  192/223  python-shell-make-comint-4
  Test python-shell-prompt-detect-1 condition:
      (ert-test-failed
       ((should
	 (equal
	  (python-shell-prompt-detect)
	  '...))
	:form
	(equal nil
	       (">>> " "... " ""))
	:value nil :explanation
	(different-types nil
			 (">>> " "... " ""))))
     FAILED  193/223  python-shell-prompt-detect-1
  Test python-shell-prompt-detect-2 condition:
      (ert-test-failed
       ((should
	 (equal
	  (python-shell-prompt-detect)
	  '...))
	:form
	(equal nil
	       ("py> " "..> " "out "))
	:value nil :explanation
	(different-types nil
			 ("py> " "..> " "out "))))
     FAILED  194/223  python-shell-prompt-detect-2
  Test python-shell-prompt-set-calculated-regexps-6 condition:
      (ert-test-failed
       ((should
	 (string= python-shell--prompt-calculated-input-regexp "^\\(\\.\\.> \\|p\\.> \\|\\)"))
	:form
	(string= "^\\(\\)" "^\\(\\.\\.> \\|p\\.> \\|\\)")
	:value nil))
     FAILED  204/223  python-shell-prompt-set-calculated-regexps-6


In GNU Emacs 25.0.50.384 (i686-pc-mingw32)
 of 2015-08-29
Repository revision: 5e63c842007b0f85e91735a7c4e00be0b7fe9ba5
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
 `configure --prefix=/d/usr --enable-checking=yes,glyphs --with-wide-int
 'CFLAGS=-gdwarf-4 -g3 -O0''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message dired format-spec
rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util help-fns help-mode easymenu cl-loaddefs pcase cl-lib mail-prsvr
mail-utils time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table
w32-win w32-vars term/common-win tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 84975 6948)
 (symbols 56 19924 0)
 (miscs 48 36 107)
 (strings 16 15967 5012)
 (string-bytes 1 393017)
 (vectors 16 11212)
 (vector-slots 8 402827 5209)
 (floats 8 125 127)
 (intervals 40 260 63)
 (buffers 856 11))

bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

[Eli Zaretskii]

Ping!  I really need help in debugging these failures.

TIA

> Date: Sat, 29 Aug 2015 15:21:01 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> 4 of the Python tests fail on MS-Windows due to some issue related to
> the prompts displayed by the Python interpreter.  The 4 failed tests
> are:
> 
>    FAILED  python-shell-make-comint-4
>    FAILED  python-shell-prompt-detect-1
>    FAILED  python-shell-prompt-detect-2
>    FAILED  python-shell-prompt-set-calculated-regexps-6
> 
> I show below the relevant portions of python-tests.log.
> 
> I need help in figuring out what causes these.  One possible suspect
> is some buffering issue, due to which Emacs does not receive the
> prompt that Python prints.  Another possibility might be the version
> of the Python (I have 2.6.6 installed).  Or maybe something else.
> 
> Let me know if I can collect more information about the failures, or
> you'd like me to try something in order to find out the reasons.
> 
> Thanks.
> 
> Here are the relevant portions of the test log for the 4 failed tests:
> 
>   Test python-shell-make-comint-4 condition:
>       (ert-test-failed
>        ((should
> 	 (string= python-shell--prompt-calculated-input-regexp
> 		  (concat "^\\(extralargeinputprompt\\|\\.\\.> \\|" "block\\|py> \\|pdf\\|sml\\|in\\)")))
> 	:form
> 	(string= "^\\(extralargeinputprompt\\|block\\|pdf\\|sml\\|in\\)" "^\\(extralargeinputprompt\\|\\.\\.> \\|block\\|py> \\|pdf\\|sml\\|in\\)")
> 	:value nil))
>      FAILED  192/223  python-shell-make-comint-4
>   Test python-shell-prompt-detect-1 condition:
>       (ert-test-failed
>        ((should
> 	 (equal
> 	  (python-shell-prompt-detect)
> 	  '...))
> 	:form
> 	(equal nil
> 	       (">>> " "... " ""))
> 	:value nil :explanation
> 	(different-types nil
> 			 (">>> " "... " ""))))
>      FAILED  193/223  python-shell-prompt-detect-1
>   Test python-shell-prompt-detect-2 condition:
>       (ert-test-failed
>        ((should
> 	 (equal
> 	  (python-shell-prompt-detect)
> 	  '...))
> 	:form
> 	(equal nil
> 	       ("py> " "..> " "out "))
> 	:value nil :explanation
> 	(different-types nil
> 			 ("py> " "..> " "out "))))
>      FAILED  194/223  python-shell-prompt-detect-2
>   Test python-shell-prompt-set-calculated-regexps-6 condition:
>       (ert-test-failed
>        ((should
> 	 (string= python-shell--prompt-calculated-input-regexp "^\\(\\.\\.> \\|p\\.> \\|\\)"))
> 	:form
> 	(string= "^\\(\\)" "^\\(\\.\\.> \\|p\\.> \\|\\)")
> 	:value nil))
>      FAILED  204/223  python-shell-prompt-set-calculated-regexps-6
> 
> 
> In GNU Emacs 25.0.50.384 (i686-pc-mingw32)
>  of 2015-08-29
> Repository revision: 5e63c842007b0f85e91735a7c4e00be0b7fe9ba5
> Windowing system distributor `Microsoft Corp.', version 5.1.2600
> Configured using:
>  `configure --prefix=/d/usr --enable-checking=yes,glyphs --with-wide-int
>  'CFLAGS=-gdwarf-4 -g3 -O0''
> 
> Configured features:
> XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
> TOOLKIT_SCROLL_BARS
> 
> Important settings:
>   value of $LANG: ENU
>   locale-coding-system: cp1255
> 
> Major mode: Lisp Interaction
> 
> Minor modes in effect:
>   tooltip-mode: t
>   global-eldoc-mode: t
>   electric-indent-mode: t
>   mouse-wheel-mode: t
>   tool-bar-mode: t
>   menu-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   blink-cursor-mode: t
>   auto-composition-mode: t
>   auto-encryption-mode: t
>   auto-compression-mode: t
>   line-number-mode: t
>   transient-mark-mode: t
> 
> Recent messages:
> For information about GNU Emacs and the GNU system, type C-h C-a.
> 
> Load-path shadows:
> None found.
> 
> Features:
> (shadow sort gnus-util mail-extr emacsbug message dired format-spec
> rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
> mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
> mm-util help-fns help-mode easymenu cl-loaddefs pcase cl-lib mail-prsvr
> mail-utils time-date mule-util tooltip eldoc electric uniquify
> ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table
> w32-win w32-vars term/common-win tool-bar dnd fontset image regexp-opt
> fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
> page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
> font-lock syntax facemenu font-core frame cl-generic cham georgian
> utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
> japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
> ethiopic indian cyrillic chinese charscript case-table epa-hook
> jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
> loaddefs button faces cus-face macroexp files text-properties overlay
> sha1 md5 base64 format env code-pages mule custom widget
> hashtable-print-readable backquote w32notify w32 multi-tty
> make-network-process emacs)
> 
> Memory information:
> ((conses 16 84975 6948)
>  (symbols 56 19924 0)
>  (miscs 48 36 107)
>  (strings 16 15967 5012)
>  (string-bytes 1 393017)
>  (vectors 16 11212)
>  (vector-slots 8 402827 5209)
>  (floats 8 125 127)
>  (intervals 40 260 63)
>  (buffers 856 11))
> 
> 
> 
> 

bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]

tags 21376 + patch
quit

Eli Zaretskii <eliz@gnu.org> writes:

> 4 of the Python tests fail on MS-Windows due to some issue related to
> the prompts displayed by the Python interpreter.  The 4 failed tests
> are:
>
>    FAILED  python-shell-make-comint-4
>    FAILED  python-shell-prompt-detect-1
>    FAILED  python-shell-prompt-detect-2
>    FAILED  python-shell-prompt-set-calculated-regexps-6
>
> I show below the relevant portions of python-tests.log.
>
> I need help in figuring out what causes these.  One possible suspect
> is some buffering issue, due to which Emacs does not receive the
> prompt that Python prints.  Another possibility might be the version
> of the Python (I have 2.6.6 installed).  Or maybe something else.

It seems to be a bug in Python 2.x on Windows (Python 3.6.x doesn't show
the problem).  In unbuffered mode it rejects carriage returns.  Compare
results of

    printf 'print("hello world")\r\n' > hello-world-with-crnl.py
    python -i    < hello-world-with-crnl.py
    python -i -u < hello-world-with-crnl.py

    printf 'print("hello world")\n' > hello-world-with-nl.py
    python -i    < hello-world-with-nl.py
    python -i -u < hello-world-with-nl.py

(Setting the PYTHONUNBUFFERED environment variable, as python.el does,
is equivalent to passing '-u'.)

The bug can be worked around with the following patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 1440 bytes --]

From a8b2295d38f50044e0396c29aefd5f465e848c84 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Tue, 15 Aug 2017 17:49:10 -0400
Subject: [PATCH v1] Work around w32-python-2.7 bug to fix prompt detection
 (Bug#21376)

* lisp/progmodes/python.el (python-shell-prompt-detect): Don't put
carriage returns into the temporary file when running in unbuffered
mode, the w32 build of python 2.7 chokes on them.
---
 lisp/progmodes/python.el | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index d774874a43..b7385f870f 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -2251,7 +2251,11 @@ python-shell-prompt-detect
                 ;; `condition-case' and displaying the error message to
                 ;; the user in the no-prompts warning.
                 (ignore-errors
-                  (let ((code-file (python-shell--save-temp-file code)))
+                  (let ((code-file
+                         ;; Python 2.7 on Windows does not handle
+                         ;; carriage returns in unbuffered mode.
+                         (let ((inhibit-eol-conversion (getenv "PYTHONUNBUFFERED")))
+                           (python-shell--save-temp-file code))))
                     ;; Use `process-file' as it is remote-host friendly.
                     (process-file
                      interpreter
-- 
2.14.1

bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 21376@debbugs.gnu.org,  fgallina@gnu.org
> Date: Tue, 15 Aug 2017 21:41:18 -0400
> 
> >    FAILED  python-shell-make-comint-4
> >    FAILED  python-shell-prompt-detect-1
> >    FAILED  python-shell-prompt-detect-2
> >    FAILED  python-shell-prompt-set-calculated-regexps-6
> >
> > I show below the relevant portions of python-tests.log.
> >
> > I need help in figuring out what causes these.  One possible suspect
> > is some buffering issue, due to which Emacs does not receive the
> > prompt that Python prints.  Another possibility might be the version
> > of the Python (I have 2.6.6 installed).  Or maybe something else.
> 
> It seems to be a bug in Python 2.x on Windows (Python 3.6.x doesn't show
> the problem).  In unbuffered mode it rejects carriage returns.  Compare
> results of
> 
>     printf 'print("hello world")\r\n' > hello-world-with-crnl.py
>     python -i    < hello-world-with-crnl.py
>     python -i -u < hello-world-with-crnl.py
> 
>     printf 'print("hello world")\n' > hello-world-with-nl.py
>     python -i    < hello-world-with-nl.py
>     python -i -u < hello-world-with-nl.py
> 
> (Setting the PYTHONUNBUFFERED environment variable, as python.el does,
> is equivalent to passing '-u'.)
> 
> The bug can be worked around with the following patch:

Thanks, but after applying the patch I still see the same 4 tests
mentioned above failing.  Do I need to do something in addition to
just running the test, as in

  cd test && make lisp/progmodes/python-tests.log

?

bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

[Noam Postavsky]

On Wed, Aug 16, 2017 at 10:26 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> Thanks, but after applying the patch I still see the same 4 tests
> mentioned above failing.  Do I need to do something in addition to
> just running the test, as in
>
>   cd test && make lisp/progmodes/python-tests.log

Hmm, that's odd, it works for me. Can you check what the following
expressions insert:

(let ((pyfile (make-temp-file "test" nil ".py")))
  (unwind-protect
      (progn (let ((inhibit-eol-conversion nil))
               (with-temp-file pyfile
                 (insert "print('hello world')\n")))
             (call-process "python2" pyfile '(t t) nil
                           "-i" "-u"))
    (delete-file pyfile)))

(let ((pyfile (make-temp-file "test" nil ".py")))
  (unwind-protect
      (progn (let ((inhibit-eol-conversion t))
               (with-temp-file pyfile
                 (insert "print('hello world')\n")))
             (call-process "python2" pyfile '(t t) nil
                           "-i" "-u"))
    (delete-file pyfile)))

I get this for the first (where ^M is really a carriage return character)

    Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40)
      [MSC v.1500 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>>   File "<stdin>", line 1
        print('hello world')^M
                            ^
    SyntaxError: invalid syntax
    >>>

and this for the second:

    Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40)
      [MSC v.1500 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> hello world
    >>>

bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Wed, 16 Aug 2017 12:14:56 -0400
> Cc: 21376@debbugs.gnu.org, Fabián E. Gallina <fgallina@gnu.org>
> 
> On Wed, Aug 16, 2017 at 10:26 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Thanks, but after applying the patch I still see the same 4 tests
> > mentioned above failing.  Do I need to do something in addition to
> > just running the test, as in
> >
> >   cd test && make lisp/progmodes/python-tests.log
> 
> Hmm, that's odd, it works for me. Can you check what the following
> expressions insert:
> 
> (let ((pyfile (make-temp-file "test" nil ".py")))
>   (unwind-protect
>       (progn (let ((inhibit-eol-conversion nil))
>                (with-temp-file pyfile
>                  (insert "print('hello world')\n")))
>              (call-process "python2" pyfile '(t t) nil
>                            "-i" "-u"))
>     (delete-file pyfile)))
> 
> (let ((pyfile (make-temp-file "test" nil ".py")))
>   (unwind-protect
>       (progn (let ((inhibit-eol-conversion t))
>                (with-temp-file pyfile
>                  (insert "print('hello world')\n")))
>              (call-process "python2" pyfile '(t t) nil
>                            "-i" "-u"))
>     (delete-file pyfile)))
> 
> I get this for the first (where ^M is really a carriage return character)
> 
>     Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40)
>       [MSC v.1500 64 bit (AMD64)] on win32
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>>   File "<stdin>", line 1
>         print('hello world')^M
>                             ^
>     SyntaxError: invalid syntax
>     >>>
> 
> and this for the second:
> 
>     Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40)
>       [MSC v.1500 64 bit (AMD64)] on win32
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>> hello world
>     >>>

Same here, except that Python version is 2.6.6.  Also, I needed to
replace "python2" with an explicit absolute file name of python.exe,
as I don't have "python2" here and Python is not on PATH.  But I don't
believe these differences are important, are they?

bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

[Noam Postavsky]

On Wed, Aug 16, 2017 at 12:38 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>     Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40)
>>       [MSC v.1500 64 bit (AMD64)] on win32
>>     Type "help", "copyright", "credits" or "license" for more information.
>>     >>>   File "<stdin>", line 1
>>         print('hello world')^M
>>                             ^
>>     SyntaxError: invalid syntax
>>     >>>
>>
>
> Same here, except that Python version is 2.6.6.  Also, I needed to
> replace "python2" with an explicit absolute file name of python.exe,
> as I don't have "python2" here and Python is not on PATH.  But I don't
> believe these differences are important, are they?

Yes, I've been messing around with PATH and bouncing between Python 2
and 3. Anyway, it sounds like your version of Python has the exact
same issue. Two questions:

Is it possible you didn't recompile python.elc after applying the patch?

How does the test run at all if you don't have Python on PATH?

python-tests.el has:

    (defvar python-tests-shell-interpreter "python")
    ...
    (ert-deftest python-shell-calculate-command-1 ()
      "Check the command to execute is calculated correctly.
    Using `python-shell-interpreter' and
    `python-shell-interpreter-args'."
      (skip-unless (executable-find python-tests-shell-interpreter))

bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Wed, 16 Aug 2017 12:59:45 -0400
> Cc: 21376@debbugs.gnu.org, Fabián E. Gallina <fgallina@gnu.org>
> 
> Is it possible you didn't recompile python.elc after applying the patch?

Sorry, I was blind: I thought your patch changed python-tests.el, not
python.el.  Now that I compiled python.el, these tests pass.

Sorry for my confusion, and thanks for fixing this.

bug#21376: 25.0.50; Python tests fail on MS-Windows -- issues with the prompt

[npostavs]

tags 21376 fixed
close 21376 26.1
quit

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Noam Postavsky <npostavs@users.sourceforge.net>
>> Date: Wed, 16 Aug 2017 12:59:45 -0400
>> Cc: 21376@debbugs.gnu.org, Fabián E. Gallina <fgallina@gnu.org>
>> 
>> Is it possible you didn't recompile python.elc after applying the patch?
>
> Sorry, I was blind: I thought your patch changed python-tests.el, not
> python.el.  Now that I compiled python.el, these tests pass.
>
> Sorry for my confusion, and thanks for fixing this.

No problem, glad we cleared it up.  I've pushed to master.

[1: 9714545675]: 2017-08-20 16:43:37 -0400
  Work around w32-python-2.x bug to fix prompt detection (Bug#21376)
  http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=9714545675af046d730352eb03dc00e93b6f7d3c