Python interactive navigation around nested functions

Python interactive navigation around nested functions

[Dima Kogan]

Hi. The current way python-mode handles nested functions seems wrong to
me. I had thought that it was a bug, but the test suite explicitly
checks for the current behavior, so I want to ask about it here before
proposing a patch.

This is the first check in the (python-nav-beginning-of-defun-1) test in
test/automated/python-tests.el. Let's say you have a python buffer:

    def decoratorFunctionWithArguments(arg1, arg2, arg3):
        '''print decorated function call data to stdout.

        Usage:

        @decoratorFunctionWithArguments('arg1', 'arg2')
        def func(a, b, c=True):
            pass
        '''

        def wwrap(f):
            print 'Inside wwrap()'
            def wrapped_f(*args):
                print 'Inside wrapped_f()'
                print 'Decorator arguments:', arg1, arg2, arg3
                f(*args)
                print 'After f(*args)'
            return wrapped_f
        return wwrap

The point is on "return wrapped_f". The user then hits C-M-a to navigate
to the beginning of the current function. The point is unambiguously
inside wwrap() and not inside wrapped_f(), so I claim it should end up
at the "def wwrap(f)" line. However the current behavior (and that test
suite check) say it should end up on "def "wrapped_f()".

This seems wrong. In fact, there are two different "go-to-previous-def"
functions: one that supposedly handles nested defuns
(python-nav-beginning-of-defun) and one that doesn't
(python-nav-backward-defun), and it doesn't make sense that they both do
the same thing here. 

I propose to make C-M-a to go to "def wwrap" here. I already have the
patches ready, but the test case is giving me pause.

dima

Re: Python interactive navigation around nested functions

[Stefan Monnier]

>         def wwrap(f):
>             print 'Inside wwrap()'
>             def wrapped_f(*args):
>                 print 'Inside wrapped_f()'
>                 print 'Decorator arguments:', arg1, arg2, arg3
>                 f(*args)
>                 print 'After f(*args)'
>             return wrapped_f
>         return wwrap

> The point is on "return wrapped_f".  The user then hits C-M-a to navigate
> to the beginning of the current function.  The point is unambiguously
> inside wwrap() and not inside wrapped_f(), so I claim it should end up
> at the "def wwrap(f)" line.  However the current behavior (and that test
> suite check) say it should end up on "def "wrapped_f()".

[ Note: the below is the view from the generic side of Emacs, because
  I don't know much about Python in general and python-mode
  in particular.  ]

Traditionally (IMO), C-M-a goes to the nearest beginning of defun at the
same AST level or higher.  So from "return wrapped_f", it seems reasonable
to jump to "def wrapped_f(*args):".

If you want to jump to the beginning of the enclosing defun, I think we
need another function, which currently doesn't exist in the "generic"
part of Emacs.  If C-M-a always jumped to the beginning of the enclosing
defun, then it would always jump to BOB when called from outside
a function, and that's clearly not how it behaves usually.


        Stefan

Re: Python interactive navigation around nested functions

[Andreas Röhler]

On 20.06.2016 04:43, Stefan Monnier wrote:
>>          def wwrap(f):
>>              print 'Inside wwrap()'
>>              def wrapped_f(*args):
>>                  print 'Inside wrapped_f()'
>>                  print 'Decorator arguments:', arg1, arg2, arg3
>>                  f(*args)
>>                  print 'After f(*args)'
>>              return wrapped_f
>>          return wwrap
>> The point is on "return wrapped_f".  The user then hits C-M-a to navigate
>> to the beginning of the current function.  The point is unambiguously
>> inside wwrap() and not inside wrapped_f(), so I claim it should end up
>> at the "def wwrap(f)" line.  However the current behavior (and that test
>> suite check) say it should end up on "def "wrapped_f()".
> [ Note: the below is the view from the generic side of Emacs, because
>    I don't know much about Python in general and python-mode
>    in particular.  ]
>
> Traditionally (IMO), C-M-a goes to the nearest beginning of defun at the
> same AST level or higher.  So from "return wrapped_f", it seems reasonable
> to jump to "def wrapped_f(*args):".
>
> If you want to jump to the beginning of the enclosing defun, I think we
> need another function, which currently doesn't exist in the "generic"
> part of Emacs.  If C-M-a always jumped to the beginning of the enclosing
> defun, then it would always jump to BOB when called from outside
> a function, and that's clearly not how it behaves usually.
>
>
>          Stefan
>
>


It's not about jumping to enclosing function --which would mean upwards--
but reaching the AST level entry node.

This is a bug, noted for python-mode.el too:
https://bugs.launchpad.net/python-mode/+bug/1594263

Re: Python interactive navigation around nested functions

[Stefan Monnier]

> It's not about jumping to enclosing function --which would mean upwards--
> but reaching the AST level entry node.

I don't know what you mean by "the AST level entry node".

> This is a bug, noted for python-mode.el too:
> https://bugs.launchpad.net/python-mode/+bug/1594263

So both python-modes behave identically in this respect.  That would not
be surprising if the behavior you don't like is the one usually
considered as right.


        Stefan

Re: Python interactive navigation around nested functions

[Andreas Röhler]

On 20.06.2016 09:34, Stefan Monnier wrote:
>> It's not about jumping to enclosing function --which would mean upwards--
>> but reaching the AST level entry node.
> I don't know what you mean by "the AST level entry node".
>
>> This is a bug, noted for python-mode.el too:
>> https://bugs.launchpad.net/python-mode/+bug/1594263
> So both python-modes behave identically in this respect.  That would not
> be surprising if the behavior you don't like is the one usually
> considered as right.
>
>
>

Start and end of a functions definition is not about like or don't like.
Also not about "usually considered".

Re: Python interactive navigation around nested functions

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1013 bytes --]

On 2016-06-20 04:28, Andreas Röhler wrote:
> On 20.06.2016 09:34, Stefan Monnier wrote: 
>
>> So both python-modes behave identically in this respect.  That would not
>> be surprising if the behavior you don't like is the one usually
>> considered as right.
> 
> Start and end of a functions definition is not about like or don't like.
> Also not about "usually considered".

I don't understand your aggressiveness, nor your point.  Stefan is just pointing out that C-M-a doesn't usually go to the beginning of the enclosing function definition (and the docs don't claim that it does, either):

  (beginning-of-defun &optional ARG)
  
  Move backward to the beginning of a defun.
  (...)
  With ARG, do it that many times.  Negative ARG means move forward
  to the ARGth following beginning of defun.

I agree with the OP that it would be very nice to have a way to go back to the beginning of the "current" defun (some sort of super C-M-u).  But I don't understand the vitriol.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Python interactive navigation around nested functions

[Andreas Röhler]

On 20.06.2016 15:57, Clément Pit--Claudel wrote:
> On 2016-06-20 04:28, Andreas Röhler wrote:
>> On 20.06.2016 09:34, Stefan Monnier wrote:
>>
>>> So both python-modes behave identically in this respect.  That would not
>>> be surprising if the behavior you don't like is the one usually
>>> considered as right.
>> Start and end of a functions definition is not about like or don't like.
>> Also not about "usually considered".
> I don't understand your aggressiveness,

Wherefrom do you derive any? The "you don't like" was introduced by 
Stefan, not me.


>   nor your point.  Stefan is just pointing out that C-M-a doesn't usually go to the beginning of the enclosing
>   function definition

In any case it's not about enclosing, which would be reached by up-list 
related stuff, it's about start of current.

> (and the docs don't claim that it does, either):
>
>    (beginning-of-defun &optional ARG)
>    
>    Move backward to the beginning of a defun.


What means "a" defun? User may expect the beginning of definition of 
code at point. Which is not the case in current python.el, but fixed in 
python-mode.el meanwhile.

>    (...)
>    With ARG, do it that many times.  Negative ARG means move forward
>    to the ARGth following beginning of defun.
>
> I agree with the OP that it would be very nice to have a way to go back to the beginning of the "current" defun (some sort of super C-M-u).

C-M-u would match the enclosing - python-mode.el delivers py-up. But 
that is not at stake here.


>   But I don't understand the vitriol.
>

No idea what you mean.

Re: Python interactive navigation around nested functions

[Dima Kogan]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>         def wwrap(f):
>>             print 'Inside wwrap()'
>>             def wrapped_f(*args):
>>                 print 'Inside wrapped_f()'
>>                 print 'Decorator arguments:', arg1, arg2, arg3
>>                 f(*args)
>>                 print 'After f(*args)'
>>             return wrapped_f
>>         return wwrap
>
>
> Traditionally (IMO), C-M-a goes to the nearest beginning of defun at the
> same AST level or higher.  So from "return wrapped_f", it seems reasonable
> to jump to "def wrapped_f(*args):".

Hmmm. That's not at all how I always thought this was supposed to work,
but I actually see this behavior in several modes across several
languages, so it looks like you're right. I don't really LIKE it,
though, so let me ponder.

Thanks for setting me straight

Re: Python interactive navigation around nested functions

[Dima Kogan]

Dima Kogan <lists@dima.secretsauce.net> writes:

> I don't really LIKE it, though, so let me ponder.

OK. I pondered for 10 minutes. I think the current behavior is strange,
and I think the following would be much more natural:

1. If we're on a function-definition line, C-M-a should go up to the
previous definition, at the SAME OR HIGHER AST level

2. If we're NOT on a function-definition line, C-M-a should go up to the
previous definition, at a HIGHER AST level.

This would do what I would want, and I think is far more reasonable. The
current behavior has some side-effects, which can probably be called
"bugs" without controversy. Say you have this:

    def bbb():
        print 11
        print 22

        def ccc():
            print 33
            print 44

        print 55
        print 55
        print 55
        print 55

    def ddd():
        print 123

From any of the "print 55" lines, C-M-h selects the whole "def ddd"
block, which I would not expect at all. If the ddd block isn't there,
then it selects all of the "ccc" block, which is wrong too: neither of
these contain the statement we started on. One could call this a
separate bug from the original complaint, but changing the C-M-a
definition makes this work naturally.

dima

Re: Python interactive navigation around nested functions

[Stefan Monnier]

> 1. If we're on a function-definition line, C-M-a should go up to the
> previous definition, at the SAME OR HIGHER AST level

OK, that's what it's supposed to be already.

> 2. If we're NOT on a function-definition line, C-M-a should go up to the
> previous definition, at a HIGHER AST level.

That's a significant change.  It means that if we're between two
functions (e.g. on a static var declaration in a C file), C-M-a would
jump to BOB.

> From any of the "print 55" lines, C-M-h selects the whole "def ddd"
> block, which I would not expect at all. If the ddd block isn't there,
> then it selects all of the "ccc" block, which is wrong too: neither of
> these contain the statement we started on. One could call this a
> separate bug from the original complaint, but changing the C-M-a
> definition makes this work naturally.

These behaviors are indeed problematic (AFAICT they're plain bugs w.r.t
the behavior described in the docstring), but they affect C-M-h, so
hopefully we can fix them in C-M-h without having to change C-M-a.

Of course by C-M-a we can mean different things:
- the backend navigation code provided by the major mode via
  *-of-defun-function.
- the command bound to C-M-a.
- the beginning-of-defun command.

They don't do exactly the same thing either, and the constraints w.r.t
changing them are slightly different (e.g. some allow the introduction
of a whole new backend API, or the rebinding of C-M-a to another command).


        Stefan

Re: Python interactive navigation around nested functions

[Dima Kogan]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> 2. If we're NOT on a function-definition line, C-M-a should go up to the
>> previous definition, at a HIGHER AST level.
>
> That's a significant change.  It means that if we're between two
> functions (e.g. on a static var declaration in a C file), C-M-a would
> jump to BOB.

Hi. So to be clear, my thinking here is about getting do-what-I-mean
behavior out of C-M-a.

The current implementation in emacs already has special-case behavior if
no leading block-start is found: instead of going to BOB, the point is
left where it is. The proposal could be modified slightly to
special-case it differently, and try to go to the prev-defun-start at
the same-or-higher level, which would produce the current behavior in
this case.

Here's a case where the current behavior is not what the user wants (at
least never what I want). Say you have this:

  defun f1():

    # stuff

    defun f2():
      # stuff

    # stuff
    # stuff
    # lots and lots of stuff
    # So much stuff that when I'm here I know I'm in f1(), but f2 isn't
    #   something i'm thinking about at all

If I C-M-a, at the end of f1(), I, the human editing this source am
thinking about f1(), but not thinking about f1() at all, and I want to
go to the start of f1(). I could hit C-M-u instead, but that will stop
at any if/while/etc blocks.

The behavior the feels "right" is implemented here:

  https://github.com/dkogan/emacs-snapshot/blob/python_indent/lisp/progmodes/python.el

There're a few more fixes in that tree that I'll submit separately in a
bit.


> Of course by C-M-a we can mean different things:
> - the backend navigation code provided by the major mode via
>   *-of-defun-function.
> - the command bound to C-M-a.
> - the beginning-of-defun command.
>
> They don't do exactly the same thing either, and the constraints w.r.t
> changing them are slightly different (e.g. some allow the introduction
> of a whole new backend API, or the rebinding of C-M-a to another command).

Yeah, I'd rather not proliferate multiple similar-yet-slightly-different
behaviors, so I'll simply use my own python nav functions in my work if
this gets rejected.

Thanks

dima

Re: Python interactive navigation around nested functions

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 421 bytes --]

On 2016-06-24 18:10, Dima Kogan wrote:
> Yeah, I'd rather not proliferate multiple similar-yet-slightly-different
> behaviors, so I'll simply use my own python nav functions in my work if
> this gets rejected.

That would be too bad;.I think you're making a good point; and still the frustrating part with a python-specific fix is that C-M-a will have a different meaning depending on the programming language :/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Python interactive navigation around nested functions

[Andreas Röhler]

On 25.06.2016 01:23, Clément Pit--Claudel wrote:
> On 2016-06-24 18:10, Dima Kogan wrote:
>> Yeah, I'd rather not proliferate multiple similar-yet-slightly-different
>> behaviors, so I'll simply use my own python nav functions in my work if
>> this gets rejected.
> That would be too bad

Not that much, as it displays another basic strength of Emacs.
For me a major reason for preferring it over other tools, which might be 
smarter in detail.

> ;.I think you're making a good point; and still the frustrating part with a python-specific fix is that C-M-a will have a different meaning depending on the programming language :/
>

Nope. A function definition as displayed here is pretty comparable with 
its role in Emacs Lisp.
A case would be def/class distinction - which is not at stake.

Re: Python interactive navigation around nested functions

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 547 bytes --]

On 2016-06-26 15:29, Andreas Röhler wrote:
>> I think you're making a good point; and still the frustrating part
>> with a python-specific fix is that C-M-a will have a different
>> meaning depending on the programming language :/
> 
> Nope. A function definition as displayed here is pretty comparable
> with its role in Emacs Lisp. A case would be def/class distinction -
> which is not at stake.

Sorry, I don't understand what you mean. My point was that a python-specific fix for the issue described by the OP would be confusing.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Python interactive navigation around nested functions

[Andreas Röhler]

On 27.06.2016 02:39, Clément Pit--Claudel wrote:
> On 2016-06-26 15:29, Andreas Röhler wrote:
>>> I think you're making a good point; and still the frustrating part
>>> with a python-specific fix is that C-M-a will have a different
>>> meaning depending on the programming language :/
>> Nope. A function definition as displayed here is pretty comparable
>> with its role in Emacs Lisp. A case would be def/class distinction -
>> which is not at stake.
> Sorry, I don't understand what you mean.

So the confusion seems to be complete :)
What do you expact M-x beginning-of-defun RET to do in a Python source 
buffer?

BTW there is a cherished bug in Emacs Lisp,  linked to 
open-paren-in-column-0-is-defun-start proceeding. C-M-a doesn't jump  
reliably to beginning of a functions definition there.
Do you suggest to expand this bug to all modes?

That would not just spread bad habits, getting familiar of inconsistent 
movements, but make the code useless for employment in templates and 
other functions.

Re: Python interactive navigation around nested functions

[Stefan Monnier]

>   defun f1():

>     # stuff

>     defun f2():
>       # stuff

>     # stuff
>     # stuff
>     # lots and lots of stuff
>     # So much stuff that when I'm here I know I'm in f1(), but f2 isn't
>     #   something i'm thinking about at all

I guess the question is how to distinguish this case.  IOW, how much is
"lots and lots of stuff".  The current behavior is meant for cases like:

   defun f1():
     # Stuff
     defun f2():
       # More stuff
     defun f3():
       # Yet more stuff
     defun f4():
       # You get the idea
     defun f5():
       # Aha
     # Here we go

Also for interactive use, we generally prefer to move less than more,
since it's fairly easy for the user to repeat the command until she gets
where she wants to, whereas if the command moves too far, there's not
much she can do, other than try and find some other command.


        Stefan

Re: Python interactive navigation around nested functions

[Andreas Röhler]

On 25.06.2016 02:18, Stefan Monnier wrote:
>>    defun f1():
>>      # stuff
>>      defun f2():
>>        # stuff
>>      # stuff
>>      # stuff
>>      # lots and lots of stuff
>>      # So much stuff that when I'm here I know I'm in f1(), but f2 isn't
>>      #   something i'm thinking about at all
> I guess the question is how to distinguish this case.  IOW, how much is
> "lots and lots of stuff".  The current behavior is meant for cases like:
>
>     defun f1():
>       # Stuff
>       defun f2():
>         # More stuff
>       defun f3():
>         # Yet more stuff
>       defun f4():
>         # You get the idea
>       defun f5():
>         # Aha
>       # Here we go
>
> Also for interactive use, we generally prefer to move less than more,

Sounds like a bet.
Why not jump from the end to start precisely?

Re: Python interactive navigation around nested functions

[Andreas Röhler]

On 21.06.2016 08:05, Dima Kogan wrote:
> Dima Kogan <lists@dima.secretsauce.net> writes:
>
>> I don't really LIKE it, though, so let me ponder.
> OK. I pondered for 10 minutes. I think the current behavior is strange,
> and I think the following would be much more natural:
>
> 1. If we're on a function-definition line, C-M-a should go up to the
> previous definition, at the SAME OR HIGHER AST level

A common proceeding is saying "at start". i.e. at first character.
Than go upward.

Re: Python interactive navigation around nested functions

[Andreas Röhler]

On 21.06.2016 07:45, Dima Kogan wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>          def wwrap(f):
>>>              print 'Inside wwrap()'
>>>              def wrapped_f(*args):
>>>                  print 'Inside wrapped_f()'
>>>                  print 'Decorator arguments:', arg1, arg2, arg3
>>>                  f(*args)
>>>                  print 'After f(*args)'
>>>              return wrapped_f
>>>          return wwrap
>>
>> Traditionally (IMO), C-M-a goes to the nearest beginning of defun at the
>> same AST level or higher.  So from "return wrapped_f", it seems reasonable
>> to jump to "def wrapped_f(*args):".
> Hmmm. That's not at all how I always thought this was supposed to work,
> but I actually see this behavior in several modes across several
> languages,

Thats because of bugs in lisp mode WRT beginning-of-defun --cherished 
for decades-- it can't be done right that way. From there the bad manner 
might have expanded.

Tried to fix that in lisp-mode here:

https://github.com/andreas-roehler/werkstatt/blob/master/subroutines/gnu-emacs-fixes.el



>   so it looks like you're right. I don't really LIKE it,
> though, so let me ponder.
>
> Thanks for setting me straight
>