bug#11974: 24.1.50; python-nav-forward-sexp-function doesn't skip over strings

bug#11974: 24.1.50; python-nav-forward-sexp-function doesn't skip over strings

[Ivan Andrus]

Actually, the new function `python-nav-forward-sexp-function' doesn't do
what I expect at all.  I use forward-sexp, up-list and friends a lot,
so perhaps I'm not typical.  I do think sexp-like movement for python
would be AWESOME!  Let me show some use cases where the current
functionality is broken for me, and then attempt to articulate what I
would like/expect.  I shall always assume that forward-sexp has a
positive argument, and the obvious statements hold for negative
arguments.

In sage-mode we narrow to a docstring by doing the equivalent of

(let* ((bos (python-info-ppss-context 'string)))
  (goto-char bos)
  (forward-sexp 1)
  (narrow-to-region bos (point)))

This is broken now since forward-sexp goes to the end of the block, far
past the end of the string (or not moving at all).  If you have any
undindented code at the end of your buffer, then forward-sexp (and up-list)
will refuse to move past it e.g.

def main():
    2+3 # I get stuck here

print main()


What I would expect form a "perfect" python forward-sexp is to skip over
a block if and only if you are at the very beginning of the block.
Otherwise is should skip over a statement if and only if you are at the
beginning of a statement.  Otherwise is should act exactly as an
unmodified forward-sexp would, skipping over words, parenthesized
expressions, and strings.

I guess this is because I think of python as having a set of parens
around a block as well as parens around each statement in addition to
all parens and strings that are currently there.

I don't guarantee that this would be a good experience, but I think it
would.  Hopefully up-list would then move to the end of the block etc.
I'm not sure how to deal with down-list.  Perhaps going to the first
statement if it's at the beginning of a block and into a parenthesized
expression otherwise?

-Ivan

bug#11974: 24.1.50; python-nav-forward-sexp-function doesn't skip over strings

[Stefan Monnier]

> (let* ((bos (python-info-ppss-context 'string)))
>   (goto-char bos)
>   (forward-sexp 1)
>   (narrow-to-region bos (point)))

> This is broken now since forward-sexp goes to the end of the block, far
> past the end of the string (or not moving at all).  If you have any
> undindented code at the end of your buffer, then forward-sexp (and up-list)
> will refuse to move past it e.g.

There's a similar problem in SMIE's implementation of forward-sexp.
Basically, the issue is that there are several "sexp"s that start at
point and forward-sexp has to choose which one to skip over.
Every choice is valid, but since there's no way for the user to say
which choice she wants, any choice is bound to disappoint and/or
surprise the user sometimes.

That doesn't mean that it's a hopeless problem and that we shouldn't try
to improve the behavior, tho: in practice (maybe just for historical
reasons), I find that I generally prefer "the smallest sexp", especially
since I can repeat forward-sexp if it didn't skip enough.

I.e. Python's forward-sexp should only try to skip over
indentation-delimited blocks when it has a clear indication that only
skipping the current line/statement would be wrong, or that there's an
easy/intuitive way for the user to get the other behavior (e.g. by
starting the forward-sexp from elsewhere).

For example, with SMIE if I have
   a + b * c
and I'm before "b", forward-sexp will only skip over "b", but if I move
to just before the "+", then forward-sexp will skip to after "c".
I'm not sure if Python's indentation-delimited blocks can find a similar
"convention" for the user to express her intention.

> What I would expect form a "perfect" python forward-sexp is to skip over
> a block if and only if you are at the very beginning of the block.

You mean that forward-sexp would generally skip over just a statement,
except if it's the first statement of a block in which case it skips the
whole block?  That sounds a bit arbitrary, making it difficult for the
user to skip over just the first statement.

[ Bear in mind, I never write Python.  ]
How 'bout

  def foo():
      toto
      bar

if I'm before `toto' on the same line, forward-sexp should skip over
"toto", but if I'm right after the ":" (and before the \n) then
forward-sexp should skip to after "bar".


        Stefan

bug#11974: 24.1.50; python-nav-forward-sexp-function doesn't skip over strings

[Ivan Andrus]

Oops, forgot to cc the buglist.

On Jul 19, 2012, at 9:52 AM, Stefan Monnier wrote:

>> (let* ((bos (python-info-ppss-context 'string)))
>> (goto-char bos)
>> (forward-sexp 1)
>> (narrow-to-region bos (point)))
> 
>> This is broken now since forward-sexp goes to the end of the block, far
>> past the end of the string (or not moving at all).  If you have any
>> undindented code at the end of your buffer, then forward-sexp (and up-list)
>> will refuse to move past it e.g.
> 
> There's a similar problem in SMIE's implementation of forward-sexp.
> Basically, the issue is that there are several "sexp"s that start at
> point and forward-sexp has to choose which one to skip over.
> Every choice is valid, but since there's no way for the user to say
> which choice she wants, any choice is bound to disappoint and/or
> surprise the user sometimes.

I didn't realize SMIE implemented forward-sexp.  Yet another reason for me to use it.

> That doesn't mean that it's a hopeless problem and that we shouldn't try
> to improve the behavior, tho: in practice (maybe just for historical
> reasons), I find that I generally prefer "the smallest sexp", especially
> since I can repeat forward-sexp if it didn't skip enough.

I agree.  The smallest is usually best.

> I.e. Python's forward-sexp should only try to skip over
> indentation-delimited blocks when it has a clear indication that only
> skipping the current line/statement would be wrong, or that there's an
> easy/intuitive way for the user to get the other behavior (e.g. by
> starting the forward-sexp from elsewhere).
> 
> For example, with SMIE if I have
>  a + b * c
> and I'm before "b", forward-sexp will only skip over "b", but if I move
> to just before the "+", then forward-sexp will skip to after "c".
> I'm not sure if Python's indentation-delimited blocks can find a similar
> "convention" for the user to express her intention.
> 
>> What I would expect form a "perfect" python forward-sexp is to skip over
>> a block if and only if you are at the very beginning of the block.
> 
> You mean that forward-sexp would generally skip over just a statement,
> except if it's the first statement of a block in which case it skips the
> whole block?  That sounds a bit arbitrary, making it difficult for the
> user to skip over just the first statement.

No, it would only skip over a block if it's at the block opening.  In some sense the entire block is one statement hence my thinking.  See below.

> [ Bear in mind, I never write Python.  ]
> How 'bout
> 
> def foo():
>     toto
>     bar
> 
> if I'm before `toto' on the same line, forward-sexp should skip over
> "toto", but if I'm right after the ":" (and before the \n) then
> forward-sexp should skip to after "bar".

In my mind you should skip to after bar only if the cursor is before the def (or possibly anywhere in the def keyword).  If I'm anywhere after that I consider myself to be "inside" the block.  But that's only the way I think and may not be representative of other (more serious) python programmers etc. etc.

I guess I would treat both if and else as separate "block openers" rather than if opening a single block.  Maybe an easy rule would be if you are at the beginning of the statement (well between the end of the last and the beginning of the current) go to the next statement that has the same indentation?  Anywhere else do as a "normal" forward-sexp.

-Ivan

bug#11974: 24.1.50; python-nav-forward-sexp-function doesn't skip over strings

[Stefan Monnier]

>> def foo():
>>    toto
>>    bar
> In my mind you should skip to after bar only if the cursor is before the def
> (or possibly anywhere in the def keyword).

This is the obvious non-controversial case, indeed.  If that's
sufficient, then I think there is no remaining problem.

I just assumed that Fabián (or others) also wanted to be able to skip
just the "block" of statements without the introductory element (like
def/if/...).


        Stefan

bug#11974: 24.1.50; python-nav-forward-sexp-function doesn't skip over strings

[Ivan Andrus]

On Jul 20, 2012, at 12:54 AM, Stefan Monnier wrote:

>>> def foo():
>>>   toto
>>>   bar
>> In my mind you should skip to after bar only if the cursor is before the def
>> (or possibly anywhere in the def keyword).
> 
> This is the obvious non-controversial case, indeed.  If that's
> sufficient, then I think there is no remaining problem.
> 
> I just assumed that Fabián (or others) also wanted to be able to skip
> just the "block" of statements without the introductory element (like
> def/if/...).


I'm okay if people like the current definition of "sexp" for python-mode, but I still think the implementatin is slightly broken.  In the simple example below I cannot get past the end of the main function using `forward-sexp'.  Is this by design?


def main():
    return 2+3 # I get stuck here

print main()

If this is by design then I'll work around it in my code.

-Ivan