On Sep 4, 2021, at 2:49 AM, Augusto Stoffel <arstoffel@gmail.com> wrote:

On Fri,  3 Sep 2021 at 19:04, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

   Change Python eval to send directly instead of using temporary files

FWIW, sending large amounts of text via pty can be troublesome (some
OSes have been known to insert additional control chars every 256
bytes, or to do weird things after 4096 bytes leading the send to never
complete, ...), which is why it's very common for comint modes to send
regions of text via temp files.


       Stefan

Okay, the 4096 bytes limit indeed exists in IPython (but not in the
standard Python interpreters, apparently).

I've attached a patch that reverts to using temporary files for
sufficiently long strings (would this magic 4096 ever require
customization?).  The patch also solves bug#32042, which is mostly
unrelated.

I would like pass text inline as much as possible because the
back-and-forth of temp files is pretty slow over Tramp, which makes the
Python shell rather annoying to use.

As to some OSes inserting additional control characters every 256 bytes,
I've never encountered this, but it seems a rather tricky problem to
work around.  Before this commit, the "plumbing code" sent to the
interpreter could already be above 256 bytes (e.g., if the generated
temp file names are modestly long).  Inserting newline characters at
strategic places would print random prompt strings, and therefore also
introduce complications.

<0001-Fixes-for-python-shell-send-string-and-python-shell-.patch>

Today I was trying to export my monthly org document that uses org babel python blocks to produce tables. I can get it to export fine if I revert python.el to the version before e32c7d2a8d - Change Python eval to send directly instead of using temporary files.

Below I show the python session buffer, first where it is working and then where it is broken for me. I’m running Emacs on macOS 11.5.1 and compile from the master branch.

In the “Broken session buffer” below, I recognize the code from that commit, but I really don’t understand the problem. Any ideas on what other information I can gather that would be useful?

Working session buffer when using previous python.el to export document

Python 3.9.6 (default, Jun 29 2021, 05:25:02) 
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> org_babel_python_eoe
>>> python.el: native completion setup loaded
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> 2021-08-24org_babel_python_eoe
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> org_babel_python_eoe
>>> 2021-08-24org_babel_python_eoe
>>> 2021-08-24org_babel_python_eoe
>>> 2021-08-24org_babel_python_eoe
>>> 2021-08-24org_babel_python_eoe
>>> 2021-08-27org_babel_python_eoe
>>> 2021-08-27org_babel_python_eoe
>>> 2021-08-27org_babel_python_eoe
>>> 2021-08-27org_babel_python_eoe
>>> 2021-08-27org_babel_python_eoe
>>> 2021-08-27org_babel_python_eoe
>>>

Broken session buffer with current python.el in e32c7d2a8d

Python 3.9.6 (default, Jun 29 2021, 05:25:02) 
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
org_babel_python_eoe
>>>   File "<stdin>", line 1
    exec("def __PYTHON_EL_eval(source, filename):\n    import ast, sys\n    if sys.version_info[0] == 2:\n        from __builtin__ import compile, eval, globals\n    else:\n        from builtins import compile, eval, globals\n    sys.stdout.write('\\n')\n    try:\n        p, e = ast.parse(source, filename), None\n    except SyntaxError:\n        t, v, tb = sys.exc_info()\n        sys.excepthook(t, v, tb.tb_next)\n        return\n    if p.body and isinstance(p.body[-1], ast.Expr):\n        e = p.body.pop()\n    try:\n        g = globals()\n        exec(compile(p, filename, 'exec'), g, g)\n        if e:\n            return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g)\n    except Exception:\n        t, v, tb = sys.exc_info()\n        sys.excepthook(t, v, tb.tb_next)");__PYTHON_EL_eval("\ndef __PYTHON_EL_native_completion_setup():\n    try:\n        import readline\n\n        try:\n            import __builtin__\n        except ImportError:\n            # Python 3\n            import builtins as __bexec("def __PYTHON_EL_eval(source, filename):\n    import ast, sys\n    if sys.version_info[0] == 2:\n        from __builtin__ import compile, eval, globals\n    else:\n        from builtins import compile, eval, globals\n    sys.stdout.write('\\n')\n    try:\n        p, e = ast.parse(source, filename), None\n    except SyntaxError:\n        t, v, tb = sys.exc_info()\n        sys.excepthook(t, v, tb.tb_next)\n        return\n    if p.body and isinstance(p.body[-1], ast.Expr):\n        e = p.body.pop()\n    try:\n        g = globals()\n        exec(compile(p, filename, 'exec'), g, g)\n        if e:\n            return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g)\n    except Exception:\n        t, v, tb = sys.exc_info()\n        sys.excepthook(t, v, tb.tb_next)");__PYTHON_EL_eval("try:\n    import ast\n    with open('/var/folders/kf/zdpzgs9d30b3jj4bdkdjf1vw0000gn/T/babel-VMz5YY/python-Xmy5oT') as __org_babel_python_tmpfile:\n        __org_babel_python_ast = ast.parse(__org_babel_python_tmpfile.read())\n    __org_babel_python_final = __org_babel_python_ast.body[-1]\n    if isinstance(__org_babel_python_final, ast.Expr):\n        __org_babel_python_ast.body = __org_babel_python_ast.body[:-1]\n        exec(compile(__org_babel_python_ast, '<string>', 'exec'))\n        __org_babel_python_final = eval(compile(ast.Expression(\n            __org_babel_python_final.value), '<string>', 'eval'))\n        with open('/var/folders/kf/zdpzgs9d30b3jj4bdkdjf1vw0000gn/T/babel-VMz5YY/python-hzPBKk', 'w') as __org_babel_python_tmpfile:\n            if False:\n                import pprint\n                __org_babel_python_tmpfile.write(pprint.pformat(__org_babel_python_final))\n            else:\n                __org_babel_python_tmpfile.write(str(__org_babel_python_final))\n    else:\n        exec(compile(__org_babel_python_ast, '<string>', 'exec'))\n        __org_babel_python_final = None\nexcept:\n    raise\nfinally:\n    print('org_babel_python_eoe')", "<string>")
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          ^
SyntaxError: invalid syntax
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 3, in <module>
NameError: name 'dfMembership' is not defined
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 2, in <module>
NameError: name 'dfMembership' is not defined
>>> 
2021-08-24org_babel_python_eoe
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 2, in <module>
NameError: name 'dfYPTdata' is not defined
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 2, in <module>
NameError: name 'dfTrainingRpt' is not defined
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 2, in <module>
NameError: name 'dfScoutbookScoutsRB' is not defined
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 3, in <module>
NameError: name 'dfCompareSBwithTM' is not defined
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 1, in <module>
NameError: name 'dfAdvData' is not defined
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 1, in <module>
NameError: name 'dfScoutbookScoutsRB' is not defined
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 1, in <module>
NameError: name 'dfCompareSBwithTM' is not defined
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 3, in <module>
NameError: name 'dfScoutbookScoutsRB' is not defined
>>> 
org_babel_python_eoe
Traceback (most recent call last):
  File "<string>", line 18, in __PYTHON_EL_eval
  File "<string>", line 8, in <module>
  File "<string>", line 22, in <module>
NameError: name 'dfAdvData' is not defined
>>>