A new major-mode for Python

A new major-mode for Python

[Fabian Ezequiel Gallina]

Hello Emacs devs,

I know the status of GNU/Emacs python.el has been discussed several
times on this list and I know there were some hopes for a merge
between python-mode.el[0] and python.el that never happened (mainly
because of Copyright issues AFAIK).

The thing is that I work a lot with Python and Emacs but I
wasn't happy with any of the available Python major modes.

I liked some stuff from the default python.el and some other from
python-mode.el and because of this I have come up with a *new*
major-mode[1] for Python which is based on the things I like and need
on a daily basis.

Most of the code is written from scratch. There are some parts that
are inspired by Emacs' python.el but I *never* used any of code you'll
find on python-mode.el in order to avoid Copyright issues in the case
there is some interest into merging this to the trunk.

The mode itself it's pretty solid, It's what I've been using for
almost a year now. After fixing a couple of things here and there, I'm
quite happy with the results.

I know the code is *not* perfect. I tried to be as clean as possible
and to name things the best way I could, but I'm not an Elisp guru,
and probably some parts of it are ugly at best (I'd like to receive
some pointers when that kind of stuff happens, since the idea is to
learn a little bit more about best practices).

Here is the list of what it currently implements:

    * Syntax highlighting

    * Solid (auto)indentation support

    * auto-detection of indentation levels for current file

    * Triple quoted strings support (stolen without guilt from
      GNU/Emacs' original python.el)

    * Fancy variable assignment colorization

    * Movement commands you’ll expect from a major-mode.

    * Python shell integration (not only for Python 2 but also Python 3!)

    * Python shell completion (Same as above!)

    * Nice generic shell integration that could support virtually any
      text based python shell

    * PDB Tracking (it even supports ipdb!)

    * Symbol completion that sucks because a running inferior shell
      process and valid code in the current buffer are needed (Don’t
      blame me, it’s like that in every python-mode I know). I don’t
      use this thing a lot, I use ropemacs instead

    * Eldoc support (this suffers the same drawbacks as the symbol
      completion, but it’s the only sane way to do it from Elisp)

    * add-log-current-defun support

    * hideshow support

    * outline support

    * fill paragraph

Things that (perhaps) are good to add at some point:

    * python-check
    * ffap support (I never used it though)
    * some skeletons (I never used them since I use yasnippet)

Things I don't think are necessary to be added:

    * Bicycle Repair Man integration: since it is discontinued and
      there are other really nice refactoring tools out there (like
      rope which can be integrated via ropemacs).

Since I'd love to contribute this to Emacs I don't have any issues in
signing papers for Copyright assignment.

In the meanwhile you can find this major mode at github[1].


[0] http://launchpadlibrarian.net/61970301/python-mode.el
[1] http://github.com/fgallina/python.el


Any ideas or suggestions are appreciated,
-- 
Fabián E. Gallina
http://www.from-the-cloud.com

Re: A new major-mode for Python

[Stefan Monnier]

> I have come up with a *new* major-mode[1] for Python

Help!!!

More seriously.  Rather than a new python mode I'm interested in
a maintainer for our python mode (i.e. someone who will fix bugs in it,
add new features, and will accept to work with us when we want to make
changes that he does not like, typically to make it behave in a more
standard way).

So rather than replace our python.el with yours, I offer you to take
over maintainership of our python.el.  This may sound like a crappy
offer (why would you maintain some old code you don't like when you
already have your own that works better, right?), but note that, as
a maintainer of our python.el, you'll have the opportunity to replace
pretty much any of "our" code with yours, thus morphing our python.el
into yours.

Another way to look at it, is that I'm asking you to take a "diff
old-python.el new-python.el" and cut it into manageable and meaningful
patches, so it is easier to see that it is moving forward.

Or maybe, what I'm asking really is to merge our python.el with yours.


        Stefan

Re: A new major-mode for Python

[Chong Yidong]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> So rather than replace our python.el with yours, I offer you to take
> over maintainership of our python.el.  This may sound like a crappy
> offer (why would you maintain some old code you don't like when you
> already have your own that works better, right?), but note that, as
> a maintainer of our python.el, you'll have the opportunity to replace
> pretty much any of "our" code with yours, thus morphing our python.el
> into yours.
>
> Another way to look at it, is that I'm asking you to take a "diff
> old-python.el new-python.el" and cut it into manageable and meaningful
> patches, so it is easier to see that it is moving forward.

100% agreement.

Scanning through your version of python.el, the code looks to be of good
quality.  As Stefan said, merging it into our existing python.el is much
better than setting up a wholesale replacement.  Among other things,
features won't get lost.  So I hope you will agree to take this on.


Should you be willing, we'll need a copyright assignment for your
contributions to Emacs.  This can be done by emailing the following
information to fsf-records@gnu.org; they will send you the assignment
form for your past and future changes.

Please use your full legal name (in ASCII characters) as the subject
line of the message.  Thanks.
----------------------------------------------------------------------
REQUEST: SEND FORM FOR PAST AND FUTURE CHANGES

[What is the name of the program or package you're contributing to?]

GNU Emacs and python.el

[Did you copy any files or text written by someone else in these changes?
Even if that material is free software, we need to know about it.]


[Do you have an employer who might have a basis to claim to own
your changes?  Do you attend a school which might make such a claim?]


[For the copyright registration, what country are you a citizen of?]


[What year were you born?]


[Please write your email address here.]


[Please write your postal address here.]





[Which files have you changed so far, and which new files have you written
so far?]

Re: A new major-mode for Python

[Fabian Ezequiel Gallina]

2011/2/15 Stefan Monnier <monnier@iro.umontreal.ca>:
>> I have come up with a *new* major-mode[1] for Python
>
> Help!!!
>
> More seriously.  Rather than a new python mode I'm interested in
> a maintainer for our python mode (i.e. someone who will fix bugs in it,
> add new features, and will accept to work with us when we want to make
> changes that he does not like, typically to make it behave in a more
> standard way).
>

At first, I started to hack on python.el, I even integrated it
with ipython. But then I realized lot of things were implemented
in a really complicated manner for a newbie like me and I was
struggling to make it work as I wished.

For instance I never liked the shell integration, and when I tried to
simplify it into something more like python-mode.el, but with the
possibility to launch dedicated shells for files, I found myself
fighting with the code instead of feeling comfortable to build upon
it.

I also felt lot of stuff were not necessary. Mainly the need of an
external python file to interact with the shell (coupling the shell
integration to an specific python version), all jython specific code
and the Bicycle Repairman integration.

It was at that time I started thinking in writing a python mode from
scratch in order to have a *clean* working version of the things I
expected from it. The result is this thing I uploaded to github :)

> So rather than replace our python.el with yours, I offer you to take
> over maintainership of our python.el.  This may sound like a crappy
> offer (why would you maintain some old code you don't like when you
> already have your own that works better, right?), but note that, as
> a maintainer of our python.el, you'll have the opportunity to replace
> pretty much any of "our" code with yours, thus morphing our python.el
> into yours.
>
> Another way to look at it, is that I'm asking you to take a "diff
> old-python.el new-python.el" and cut it into manageable and meaningful
> patches, so it is easier to see that it is moving forward.
>
> Or maybe, what I'm asking really is to merge our python.el with yours.
>

The possibility of being the maintainer is never a crappy offer, but
I'm really scared about the effort we would put into doing the kind of
merge you propose, I can't really see that approach as
productive. (The mode I wrote shares *very little* code with trunk's
python.el)

However since this mode implements the most important features from
python.el I think we can provide some define-obsolete-function-alias,
define-obsolete-variable-alias in order to keep it the most compatible
as possible.

I'm willing to add/re-implement ffap, python-check, skeletons. But I
can't think of a good reason to add Bicycle Repairman again.

When that stuff is done the new python.el would contain all the
features of the previous one while adding an important one:
compatibility with python 3 (even when using the default shell setup)

So in shorter terms what I'm willing to do is:

   1) Re-implement ffap, python-check, add some skeletons

   2) Set several define-obsolete-{variable,function}-alias(es) in
      order to add a some nice compatibility to trunk's python.el

   3) Maintain this stuff :)


What do you think? It still sounds like a bad idea?


Thanks,
-- 
Fabián E. Gallina
http://www.from-the-cloud.com

Re: A new major-mode for Python

[Neal Becker]

I get an error trying to byte-compile it (grabbed from github):

Compiling file /home/nbecker/.emacs.d/python.el at Wed Feb 16 09:21:37 2011
python.el:265:16:Error: Symbol's value as variable is void: python-rx-
constituents

Re: A new major-mode for Python

[Stefan Monnier]

> The possibility of being the maintainer is never a crappy offer, but
> I'm really scared about the effort we would put into doing the kind of
> merge you propose, I can't really see that approach as
> productive. (The mode I wrote shares *very little* code with trunk's
> python.el)

> However since this mode implements the most important features from
> python.el I think we can provide some define-obsolete-function-alias,
> define-obsolete-variable-alias in order to keep it the most compatible
> as possible.

I'm not so much concerned about backward compatibility, really.  I just
would rather have a smoother transition.

> I'm willing to add/re-implement ffap, python-check, skeletons. But I
> can't think of a good reason to add Bicycle Repairman again.

As a maintainer you are free to decide that some feature should
be dropped.  So I don't mind if the repairman goes.  As for skeletons
and ffap, I can't think of any reason why it would be much more work
than just copying the current code: it is pretty self-contained.

> When that stuff is done the new python.el would contain all the
> features of the previous one while adding an important one.

I'm not too concerned about features.  I just want the transition to be
done in meaningful chunks.  E.g. one commit can rip out the old
indentation and install your new one.  Another one could be "drop
bycicle because noone wants to repair the repairman".

> What do you think? It still sounds like a bad idea?

I think we can work something out.  And if Christoph is OK with helping
along, this could go very smoothly.


        Stefan

Re: A new major-mode for Python

[Chong Yidong]

Fabian Ezequiel Gallina <galli.87@gmail.com> writes:

> The possibility of being the maintainer is never a crappy offer, but
> I'm really scared about the effort we would put into doing the kind of
> merge you propose, I can't really see that approach as
> productive. (The mode I wrote shares *very little* code with trunk's
> python.el)
>
> It was at that time I started thinking in writing a python mode from
> scratch in order to have a *clean* working version of the things I
> expected from it. The result is this thing I uploaded to github :)

It's good to have a clean working version as the base from which to
work, and I don't object in principle to removing functionality that's
truly not needed.  But it's preferable to make changes in the trunk code
systematically.

Even though the mode you wrote does not closely match what's in the
trunk, it should still be possible to break it up into separate pieces.

For example, the first step could involve replacing
python-beginning-of-defun and python-end-of-defun and the functions they
use (i.e. the the indentation engine).  This would allow other hackers
to study the diff, and figure out (i) in what ways the new code really
is an improvement, (ii) whether any features are lost, and (iii) whether
any commands and customizable options, on which users might depend, are
lost or changed in a backward-incompatible way.

And then repeat the procedure for the other parts: font lock, python
shell support, pdbtrack support, and so forth.

It is a bit more work than just dumping a brand new file into place.
But I don't think it's an unreasonable amount of work, especially if
other hackers are willing to help, and the advantages would be great.

Re: A new major-mode for Python

[Christoph]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

 > I'm not so much concerned about backward compatibility, really.  I just
 > would rather have a smoother transition.

I think the move can be done step by step and shouldn't take too long 
overall.

 > I'm not too concerned about features.  I just want the transition to be
 > done in meaningful chunks.  E.g. one commit can rip out the old
 > indentation and install your new one.  Another one could be "drop
 > bycicle because noone wants to repair the repairman".

[...]

 > I think we can work something out.  And if Christoph is OK with helping
 > along, this could go very smoothly.

I am OK with that.

Here is what I would propose for the next steps:

If Fabian is OK with the general approach described by you and Chong and
OK with signing the FSF papers, he should go ahead and do that. In the
meantime, we could polish Fabians implementation as hosted on github and
start thinking about the integration in steps.

By polish, I mean a) resolve issues the current implementation has (I 
have been using it for a couple of days and found a couple, especially 
on Windows) and b) implement the features that are in the Emacs' 
python.el but not in Fabians implementation and that we don't want to loose.

I am available in whatever capacity Fabian is comfortable, i.e. as a
tester/bug reporter or I can provide patches for improvements.

As for the integration with the current python.el, I think that some
cleanup is needed first. For example the pdbtrack implementation that
was done a couple of years ago pulled in all this unecessary code that
was never cleaned up. I would clean up this part right away. Then maybe
remove bicycle repair main support and other stuff that we decide is not
really required. I think, this will make integrating Fabians changes
easier.

Fabian, what do you think?

Christoph

Re: A new major-mode for Python

[m h]

On Tue, Feb 15, 2011 at 1:05 AM, Fabian Ezequiel Gallina
<galli.87@gmail.com> wrote:
> Hello Emacs devs,
>
> I know the status of GNU/Emacs python.el has been discussed several
> times on this list and I know there were some hopes for a merge
> between python-mode.el[0] and python.el that never happened (mainly
> because of Copyright issues AFAIK).

Fabian-

As a python-mode.el user, I'm curious how the feature set compares to
python-mode.el?  What is missing, what is new/extra?  (Not trying to
start flame war or anything, just curious)

cheers,
matt

Re: A new major-mode for Python

[Fabian Ezequiel Gallina]

2011/2/16 Neal Becker <ndbecker2@gmail.com>:
> I get an error trying to byte-compile it (grabbed from github):
>
> Compiling file /home/nbecker/.emacs.d/python.el at Wed Feb 16 09:21:37 2011
> python.el:265:16:Error: Symbol's value as variable is void: python-rx-
> constituents
>

The trunk compiles without warnings with Emacs 23.2. If you are
interested in filling a bug, let's use github's issue tracker[0] for
now.


[0] https://github.com/fgallina/python.el/issues


Regards,
-- 
Fabián E. Gallina
http://www.from-the-cloud.com

Re: A new major-mode for Python

[Fabian Ezequiel Gallina]

2011/2/17 Christoph <cschol2112@googlemail.com>:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> I'm not so much concerned about backward compatibility, really.  I just
>> would rather have a smoother transition.
>
> I think the move can be done step by step and shouldn't take too long
> overall.
>
>> I'm not too concerned about features.  I just want the transition to be
>> done in meaningful chunks.  E.g. one commit can rip out the old
>> indentation and install your new one.  Another one could be "drop
>> bycicle because noone wants to repair the repairman".

Sounds reasonable to me. Let's do it.

>
> [...]
>
>> I think we can work something out.  And if Christoph is OK with helping
>> along, this could go very smoothly.
>
> I am OK with that.
>
> Here is what I would propose for the next steps:
>
> If Fabian is OK with the general approach described by you and Chong and
> OK with signing the FSF papers, he should go ahead and do that. In the
> meantime, we could polish Fabians implementation as hosted on github and
> start thinking about the integration in steps.

I do agree, and I sent the form request already, they should get back
to me soon.

>
> By polish, I mean a) resolve issues the current implementation has (I have
> been using it for a couple of days and found a couple, especially on
> Windows) and b) implement the features that are in the Emacs' python.el but
> not in Fabians implementation and that we don't want to loose.
>
> I am available in whatever capacity Fabian is comfortable, i.e. as a
> tester/bug reporter or I can provide patches for improvements.
>

Any help is appreciated, I assume you have signed the papers already
so any contribution will be OK to inclusion.

> As for the integration with the current python.el, I think that some
> cleanup is needed first. For example the pdbtrack implementation that
> was done a couple of years ago pulled in all this unecessary code that
> was never cleaned up. I would clean up this part right away. Then maybe
> remove bicycle repair main support and other stuff that we decide is not
> really required. I think, this will make integrating Fabians changes
> easier.
>
> Fabian, what do you think?
>

That sounds nice. If you could start those cleanups in the current
python.el part the merge should get even simpler.


Regards,
-- 
Fabián E. Gallina
http://www.from-the-cloud.com

Re: A new major-mode for Python

[Fabian Ezequiel Gallina]

2011/2/17 m h <sesquile@gmail.com>:
> On Tue, Feb 15, 2011 at 1:05 AM, Fabian Ezequiel Gallina
> <galli.87@gmail.com> wrote:
>> Hello Emacs devs,
>>
>> I know the status of GNU/Emacs python.el has been discussed several
>> times on this list and I know there were some hopes for a merge
>> between python-mode.el[0] and python.el that never happened (mainly
>> because of Copyright issues AFAIK).
>
> Fabian-
>
> As a python-mode.el user, I'm curious how the feature set compares to
> python-mode.el?  What is missing, what is new/extra?  (Not trying to
> start flame war or anything, just curious)
>

I'm really not sure, the time I used python-mode.el was just because
of its shell integration. I don't think you'll miss much things tough.


Regards,
-- 
Fabián E. Gallina
http://www.from-the-cloud.com

Re: A new major-mode for Python

[Christoph]

Fabian Ezequiel Gallina <galli.87@gmail.com> writes:

> I do agree, and I sent the form request already, they should get back
> to me soon.

Well, they send you a letter that you have to sign and send back. This
can take a while (from my own experience). 

But in the meantime we can go forward with your implementation on
github. I will sign up and start putting in some bug reports.

> Any help is appreciated, I assume you have signed the papers already
> so any contribution will be OK to inclusion.

Yes, I have.

> That sounds nice. If you could start those cleanups in the current
> python.el part the merge should get even simpler.

OK, I will get started on this.

Christoph

Re: A new major-mode for Python

[Fabian Ezequiel Gallina]

2011/2/17 Christoph <cschol2112@googlemail.com>:
> Fabian Ezequiel Gallina <galli.87@gmail.com> writes:
>
>> I do agree, and I sent the form request already, they should get back
>> to me soon.
>
> Well, they send you a letter that you have to sign and send back. This
> can take a while (from my own experience).
>

Hi all,

After a couple weeks, I received the confirmation about my papers.

Now I'm kinda lost how we go from here, does everybody agree if I
start creating patches against the trunk version?


Regards,
-- 
Fabián E. Gallina
http://www.from-the-cloud.com

Re: A new major-mode for Python

[Christoph Scholtes]

On 3/28/2011 4:55 PM, Fabian Ezequiel Gallina wrote:

> Now I'm kinda lost how we go from here, does everybody agree if I
> start creating patches against the trunk version?

I have one cleanup patch that I would like to apply to the trunk first. 
It removes some obsolete stuff that was introduced when the pdbtrack 
support was added, but never cleaned up. Your patches might apply 
cleaner after this is removed.

I will try to send it later tonight for review.

Christoph