bug#45348: 27.1; python-eldoc-setup-code uses obsolete python function

bug#45348: 27.1; python-eldoc-setup-code uses obsolete python function

[Steven De Herdt]

[-- Attachment #1: Type: text/plain, Size: 5696 bytes --]

Hello folks

This report concerns eldoc for Python.  I'm using Debian testing, with
Python 3.9, in my daily config I use ipython3 as
python-shell-interpreter.  Every time I place point in the argument list
of a Python function without documentation, the minibuffer expands to
show a warning ("DeprecationWarning: `formatargspec` is deprecated since
Python 3.5. Use `signature` and the `Signature` object directly"), an
unrelated snippet of my code, as well as the intended function signature.

This behaviour can be partly reproduced in a minimal setting with the
attached func.py:
* emacs -Q func.py
* C-u C-c C-p , choose "python3 -i"
* in buffer for func.py: C-c C-c
* place point at argument (13) of the call to func
The DeprecationWarning only shows the first time with python3.  My
ipython3 shows it every time however, that's a bit annoying.  I'm not
sure the repeated warning is an issue with ipython, but the eldoc setup
code would need to upgrade from deprecated functionality anyway.

Maybe I'll try to make python-eldoc-setup-code use inspect.signature if
it annoys me too much. :)

Thanks for all the emacs!
-Steven



In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.23,
cairo version 1.16.0)
 of 2020-11-08, modified by Debian built on x86-ubc-01
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Debian GNU/Linux bullseye/sid

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark set
Make dedicated process? (y or n) y
Shell native completion is enabled.
Can’t guess python-indent-offset, using defaults: 4
Sent: #! /usr/bin/python3...
Mark set
user-error: Beginning of history; no preceding item
user-error: End of history; no default available

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --build
 x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
 --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --with-cairo
 --with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars
 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs-6jKC2B/emacs-27.1+1=.
-fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD
JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LC_TIME: nl_BE.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils thingatpt compile
cl-extra help-mode python easymenu tramp-sh tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat shell pcomplete
parse-time iso8601 time-date ls-lisp format-spec auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
seq byte-opt gv bytecomp byte-compile cconv comint ring cl-loaddefs
cl-lib ansi-color tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 75346 9259)
 (symbols 48 8919 1)
 (strings 32 26350 2263)
 (string-bytes 1 947097)
 (vectors 16 15081)
 (vector-slots 8 187151 10066)
 (floats 8 55 29)
 (intervals 56 274 0)
 (buffers 1000 15))

[-- Attachment #2: func.py --]
[-- Type: text/x-python, Size: 58 bytes --]

#! /usr/bin/python3

def func(a):
  return 15*a

func(13)

bug#45348: 27.1; python-eldoc-setup-code uses obsolete python function

[Tomas Nordin]

[-- Attachment #1: Type: text/plain, Size: 3243 bytes --]

Steven De Herdt <steven.deherdt@gmail.com> writes:

> Hello folks
>
> This report concerns eldoc for Python.  I'm using Debian testing, with
> Python 3.9, in my daily config I use ipython3 as
> python-shell-interpreter.  Every time I place point in the argument list
> of a Python function without documentation, the minibuffer expands to
> show a warning ("DeprecationWarning: `formatargspec` is deprecated since
> Python 3.5. Use `signature` and the `Signature` object directly"), an
> unrelated snippet of my code, as well as the intended function signature.
>
> This behaviour can be partly reproduced in a minimal setting with the
> attached func.py:
> * emacs -Q func.py
> * C-u C-c C-p , choose "python3 -i"
> * in buffer for func.py: C-c C-c
> * place point at argument (13) of the call to func
> The DeprecationWarning only shows the first time with python3.  My
> ipython3 shows it every time however, that's a bit annoying.  I'm not
> sure the repeated warning is an issue with ipython, but the eldoc setup
> code would need to upgrade from deprecated functionality anyway.

I can reproduce this with Python 3.7.3 on Debian stable. And yes,
strange behaviour with Ipython, which is maybe not about the eldoc setup
code.

> Maybe I'll try to make python-eldoc-setup-code use inspect.signature
if
> it annoys me too much. :)

Yes, did you figure out something safe and sound? Otherwise can you
please try the attached patch. It fixes the problem here.

The patch touch the string of python code `python-eldoc-setup-code' and
do a little more than simply not call formatargspec with python 3, but
does not change the intended effect.

I think the patch improves readability somewhat. But also I identified a
few bugs with the python code in the string. Those bugs doesn't surface
due to the (needed) wrapping try block. Here are the observed bugs,

- if obj is not a string, if isinstance(obj, str_type) fails and doc
  does not get defined. After this, doc is referred to in the next
  test, (if not doc and callable(obj):) which will then error.

- if obj is a class instance and has an empty docstring and is not
  callable (often not), then (if not doc and callable(obj)) fails and
  doc.splitlines()[0] will Error.

- again, if obj is a class instance, not callable but has no
  docstring, doc will be None and doc.splitlines() will error.

- if obj is provided as the object itself, the function always fail
  because doc does not get defined.

Those bugs are fixed with the patch, as well as using inspect.signature
with python 3 as recommended by the deprecation warning.

Besides the patch, I wonder what other users thinks about the resulting
behaviour of this python function. Here is a a suitable docstring for
`python-eldoc-setup-code', would it be provided:

"""Provide a docstring or signature on obj.

Prefer a docstring, and fall back on the signature if no docstring
is found."""

Meaning, if there is a docstring, we will not get the call signature,
we get that only when there is no docstring. Personally I would prefer
the call signature always, like in elisp. Then perhaps when doing C-c
C-f (python-eldoc-at-point) show the (full) docstring. But that would be
another bug-report I guess.

Best regards
--
Tomas


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: pydoc-45348.patch --]
[-- Type: text/x-diff, Size: 1898 bytes --]

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index d58b32f3c3..6acfafdb68 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -4533,33 +4533,35 @@ python-eldoc-setup-code
   "def __PYDOC_get_help(obj):
     try:
         import inspect
+        from sys import version_info as version_info
+
         try:
-            str_type = basestring
-            argspec_function = inspect.getargspec
-        except NameError:
-            str_type = str
-            argspec_function = inspect.getfullargspec
-        if isinstance(obj, str_type):
             obj = eval(obj, globals())
-        doc = inspect.getdoc(obj)
+            doc = inspect.getdoc(obj)
+        except TypeError:
+            doc = inspect.getdoc(obj)
+
         if not doc and callable(obj):
-            target = None
             if inspect.isclass(obj) and hasattr(obj, '__init__'):
                 target = obj.__init__
                 objtype = 'class'
             else:
                 target = obj
                 objtype = 'def'
-            if target:
-                args = inspect.formatargspec(*argspec_function(target))
-                name = obj.__name__
-                doc = '{objtype} {name}{args}'.format(
-                    objtype=objtype, name=name, args=args
-                )
-        else:
+            if version_info.major == 3:
+                args = str(inspect.signature(target))
+            else:
+                args = inspect.formatargspec(*inspect.getargspec(target))
+
+            name = obj.__name__
+            doc = '{objtype} {name}{args}'.format(
+                objtype=objtype, name=name, args=args)
+        elif doc:
             doc = doc.splitlines()[0]
+        else:
+            doc = ''
     except:
-        doc = ''
+        return ''
     return doc"
   "Python code to setup documentation retrieval."
   :type 'string

bug#45348: 27.1; python-eldoc-setup-code uses obsolete python function

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

On Wed, Dec 23, 2020, 23:54 Tomas Nordin <tomasn@posteo.net> wrote:

>
> Meaning, if there is a docstring, we will not get the call signature,
> we get that only when there is no docstring. Personally I would prefer
> the call signature always, like in elisp. Then perhaps when doing C-c
> C-f (python-eldoc-at-point) show the (full) docstring. But that would be
> another bug-report I guess.
>

If you think you can program this, the latest ElDoc allows you to do just
that. See the docstrings of eldoc-documentation-functions,
eldoc-documentation-strategy, eldoc-display-functions and other variables
prefixed eldoc-echo-area. Also see the new interactive command M-x eldoc.

All of those things have been touched recently by me, and I'd appreciate
the feedback.

João

João

>

[-- Attachment #2: Type: text/html, Size: 1420 bytes --]

bug#45348: 27.1; python-eldoc-setup-code uses obsolete python function

[Tomas Nordin]

João Távora <joaotavora@gmail.com> writes:

> On Wed, Dec 23, 2020, 23:54 Tomas Nordin <tomasn@posteo.net> wrote:
>
>>
>> Meaning, if there is a docstring, we will not get the call signature,
>> we get that only when there is no docstring. Personally I would prefer
>> the call signature always, like in elisp. Then perhaps when doing C-c
>> C-f (python-eldoc-at-point) show the (full) docstring. But that would be
>> another bug-report I guess.
>>
>
> If you think you can program this, the latest ElDoc allows you to do just
> that. See the docstrings of eldoc-documentation-functions,
> eldoc-documentation-strategy, eldoc-display-functions and other variables
> prefixed eldoc-echo-area. Also see the new interactive command M-x eldoc.
>
> All of those things have been touched recently by me, and I'd appreciate
> the feedback.

OK, thanks. Then we know the API to read up on when it comes to that.

I would be happy to try and help improving the doc support in python.el.
The strategy I would suggest to do that would be

-- C-c C-d         python-describe-at-point

Make this display full documentation akin to C-h f in elisp. Document
this function to do so, currently it is un-documented. And current
effect is to leave the help text in the python shell, which works
sometimes.

-- C-c C-f         python-eldoc-at-point

Make this output the first line of docstring, if there is no docstring
show nothing. This is almost what happens now, only if there is no
docstring the signature is given instead.

-- Automatic help display of objects

Make this show signatures always. (akin to elisp mode).

But I don't know if anybody care to see this happen.

Best regards
--
Tomas

bug#45348: 27.1; python-eldoc-setup-code uses obsolete python function

[João Távora]

Tomas Nordin <tomasn@posteo.net> writes:

>> All of those things have been touched recently by me, and I'd appreciate
>> the feedback.
>
> OK, thanks. Then we know the API to read up on when it comes to that.
>
> I would be happy to try and help improving the doc support in python.el.
> The strategy I would suggest to do that would be
>
> -- C-c C-d         python-describe-at-point
>
> Make this display full documentation akin to C-h f in elisp. Document

> -- Automatic help display of objects
>
> Make this show signatures always. (akin to elisp mode).
>
> But I don't know if anybody care to see this happen.

I would.  Or at least I think these ideas are welcome input for solving
the broader problem of inconsistency of documentation systems in Emacs.
In fact, I'm going to take this opportunity to lay out a plan that I've
been thinking of, to collect some early feedback, and to seize on any
motivation that you or someone else may have to work on this.

I quite agree with you when you say there is something that should
display full documentation akin to C-h f in Elisp.  But this is not only
for Python, it is for all languages -- including Elisp.

In my opinion, that something should be 'M-x eldoc' (and we can chose a
prominent shortcut for it later).  In the latest Emacs, when invoked
interactively, it pops up a new buffer (like C-h f does) which contains
detailed information about how to use a particular element of the
language (a function, a procedure, a variable, etc).  When invoked
automatically by a background process, it behaves more discreetly, but
and normally displays a shorter string in the echo area.

Under the hood, the ElDoc framework also now possesses the main tools to
help segregate succint short summaries that can be shown in a
constrained setting -- such as the echo area -- from longer descriptions
that are more suitable for larger outlets, such as buffers, separate
frame and long-winded popup widgets.  These tools are the aforementioned
eldoc-documentation-functions and eldoc-display-functions.

In my view we're only missing a protocol for the former to communicate
to the latter how small or large the produces pieces of documentation
are.  How?  Here's the current situation in Elisp mode: M-x eldoc will
call on one or two backends (sometimes both, depending on
eldoc-documentation-strategy) to document the thing at point: the
variable documentation one and the function documentation one.  For
historical reasons, they return simple one liners for displaying in the
echo area.  But each could _additionally_ return much richer
descriptions like the ones that C-h v and C-h f do.  They could return
this in new a :FULL-DOC property:

* eldoc-display-in-echo-area would ignore it, since it is content
  that doesn't fit its target:

* eldoc-display-in-buffer function would utilize to format a buffer
  that would look much like the *Help* buffer does nowadays.

Whether that new property should be a string, a function, a buffer, or
any of the above is an implementation detail we'd have to think up.
Regardless, I'm sure something like this could be thought up for some
future Python backend(s) as well: an Eldoc backend that returns a short
summary (the signature, presumably) and a longer description of the
thing at point.

Other pieces of the puzzle to consider here are the somewhat
underused/defunct help-at-pt.el which happens to have a good shortcut in
'C-h .'.  It could be merged with ElDoc and lend that shortcut to M-x
eldoc.

João