On Sep 4, 2021, at 2:49 AM, Augusto Stoffel > wrote: On Fri, 3 Sep 2021 at 19:04, Stefan Monnier > wrote: Change Python eval to send directly instead of using temporary files FWIW, sending large amounts of text via pty can be troublesome (some OSes have been known to insert additional control chars every 256 bytes, or to do weird things after 4096 bytes leading the send to never complete, ...), which is why it's very common for comint modes to send regions of text via temp files. Stefan Okay, the 4096 bytes limit indeed exists in IPython (but not in the standard Python interpreters, apparently). I've attached a patch that reverts to using temporary files for sufficiently long strings (would this magic 4096 ever require customization?). The patch also solves bug#32042, which is mostly unrelated. I would like pass text inline as much as possible because the back-and-forth of temp files is pretty slow over Tramp, which makes the Python shell rather annoying to use. As to some OSes inserting additional control characters every 256 bytes, I've never encountered this, but it seems a rather tricky problem to work around. Before this commit, the "plumbing code" sent to the interpreter could already be above 256 bytes (e.g., if the generated temp file names are modestly long). Inserting newline characters at strategic places would print random prompt strings, and therefore also introduce complications. <0001-Fixes-for-python-shell-send-string-and-python-shell-.patch> Today I was trying to export my monthly org document that uses org babel python blocks to produce tables. I can get it to export fine if I revert python.el to the version before e32c7d2a8d - Change Python eval to send directly instead of using temporary files. Below I show the python session buffer, first where it is working and then where it is broken for me. I’m running Emacs on macOS 11.5.1 and compile from the master branch. In the “Broken session buffer” below, I recognize the code from that commit, but I really don’t understand the problem. Any ideas on what other information I can gather that would be useful? Working session buffer when using previous python.el to export document Python 3.9.6 (default, Jun 29 2021, 05:25:02) [Clang 12.0.5 (clang-1205.0.22.9)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> org_babel_python_eoe >>> python.el: native completion setup loaded >>> org_babel_python_eoe >>> org_babel_python_eoe >>> org_babel_python_eoe >>> 2021-08-24org_babel_python_eoe >>> org_babel_python_eoe >>> org_babel_python_eoe >>> org_babel_python_eoe >>> org_babel_python_eoe >>> org_babel_python_eoe >>> org_babel_python_eoe >>> org_babel_python_eoe >>> org_babel_python_eoe >>> org_babel_python_eoe >>> 2021-08-24org_babel_python_eoe >>> 2021-08-24org_babel_python_eoe >>> 2021-08-24org_babel_python_eoe >>> 2021-08-24org_babel_python_eoe >>> 2021-08-27org_babel_python_eoe >>> 2021-08-27org_babel_python_eoe >>> 2021-08-27org_babel_python_eoe >>> 2021-08-27org_babel_python_eoe >>> 2021-08-27org_babel_python_eoe >>> 2021-08-27org_babel_python_eoe >>> Broken session buffer with current python.el in e32c7d2a8d Python 3.9.6 (default, Jun 29 2021, 05:25:02) [Clang 12.0.5 (clang-1205.0.22.9)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> org_babel_python_eoe >>> File "", line 1 exec("def __PYTHON_EL_eval(source, filename):\n import ast, sys\n if sys.version_info[0] == 2:\n from __builtin__ import compile, eval, globals\n else:\n from builtins import compile, eval, globals\n sys.stdout.write('\\n')\n try:\n p, e = ast.parse(source, filename), None\n except SyntaxError:\n t, v, tb = sys.exc_info()\n sys.excepthook(t, v, tb.tb_next)\n return\n if p.body and isinstance(p.body[-1], ast.Expr):\n e = p.body.pop()\n try:\n g = globals()\n exec(compile(p, filename, 'exec'), g, g)\n if e:\n return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g)\n except Exception:\n t, v, tb = sys.exc_info()\n sys.excepthook(t, v, tb.tb_next)");__PYTHON_EL_eval("\ndef __PYTHON_EL_native_completion_setup():\n try:\n import readline\n\n try:\n import __builtin__\n except ImportError:\n # Python 3\n import builtins as __bexec("def __PYTHON_EL_eval(source, filename):\n import ast, sys\n if sys.version_info[0] == 2:\n from __builtin__ import compile, eval, globals\n else:\n from builtins import compile, eval, globals\n sys.stdout.write('\\n')\n try:\n p, e = ast.parse(source, filename), None\n except SyntaxError:\n t, v, tb = sys.exc_info()\n sys.excepthook(t, v, tb.tb_next)\n return\n if p.body and isinstance(p.body[-1], ast.Expr):\n e = p.body.pop()\n try:\n g = globals()\n exec(compile(p, filename, 'exec'), g, g)\n if e:\n return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g)\n except Exception:\n t, v, tb = sys.exc_info()\n sys.excepthook(t, v, tb.tb_next)");__PYTHON_EL_eval("try:\n import ast\n with open('/var/folders/kf/zdpzgs9d30b3jj4bdkdjf1vw0000gn/T/babel-VMz5YY/python-Xmy5oT') as __org_babel_python_tmpfile:\n __org_babel_python_ast = ast.parse(__org_babel_python_tmpfile.read())\n __org_babel_python_final = __org_babel_python_ast.body[-1]\n if isinstance(__org_babel_python_final, ast.Expr):\n __org_babel_python_ast.body = __org_babel_python_ast.body[:-1]\n exec(compile(__org_babel_python_ast, '', 'exec'))\n __org_babel_python_final = eval(compile(ast.Expression(\n __org_babel_python_final.value), '', 'eval'))\n with open('/var/folders/kf/zdpzgs9d30b3jj4bdkdjf1vw0000gn/T/babel-VMz5YY/python-hzPBKk', 'w') as __org_babel_python_tmpfile:\n if False:\n import pprint\n __org_babel_python_tmpfile.write(pprint.pformat(__org_babel_python_final))\n else:\n __org_babel_python_tmpfile.write(str(__org_babel_python_final))\n else:\n exec(compile(__org_babel_python_ast, '', 'exec'))\n __org_babel_python_final = None\nexcept:\n raise\nfinally:\n print('org_babel_python_eoe')", "") ^ SyntaxError: invalid syntax >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 3, in NameError: name 'dfMembership' is not defined >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 2, in NameError: name 'dfMembership' is not defined >>> 2021-08-24org_babel_python_eoe >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 2, in NameError: name 'dfYPTdata' is not defined >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 2, in NameError: name 'dfTrainingRpt' is not defined >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 2, in NameError: name 'dfScoutbookScoutsRB' is not defined >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 3, in NameError: name 'dfCompareSBwithTM' is not defined >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 1, in NameError: name 'dfAdvData' is not defined >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 1, in NameError: name 'dfScoutbookScoutsRB' is not defined >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 1, in NameError: name 'dfCompareSBwithTM' is not defined >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 3, in NameError: name 'dfScoutbookScoutsRB' is not defined >>> org_babel_python_eoe Traceback (most recent call last): File "", line 18, in __PYTHON_EL_eval File "", line 8, in File "", line 22, in NameError: name 'dfAdvData' is not defined >>>