bug#21783: 25.0.50; python.el does not support new Python 3.5 keywords

bug#21783: 25.0.50; python.el does not support new Python 3.5 keywords

[Lele Gaifax]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

Python 3.5, released in mid September 2015, introduced a few new keywords
to better support asynchronous code, "async" and "await" in particular. See
https://www.python.org/dev/peps/pep-0492/ for details.

To avoid breaking already written users code, they are technically not
"reserved keywords", but accordingly to the plan they will become that in
Python 3.7 (https://www.python.org/dev/peps/pep-0492/#id75).

Currently python.el doesn't know anything about them, so it obviously cannot
properly indent the following code:

  async def coroutine(foo, bar):
  |

where "|" represent the cursor position, nor highlight them in any way.

I'm attaching a patch that implements such support, with some test cases. See
also the thread http://thread.gmane.org/gmane.emacs.devel/190950 where I
initially sent the same patch and where I've been solicited to create a bug
report.

Hope this helps,

thanks in advance,
ciao, lele.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Add PEP492 support to python.el --]
[-- Type: text/x-diff, Size: 4450 bytes --]

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 6ff12b5..9f5c087 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -384,7 +384,10 @@ python-mode-map
   (defconst python-rx-constituents
     `((block-start          . ,(rx symbol-start
                                    (or "def" "class" "if" "elif" "else" "try"
-                                       "except" "finally" "for" "while" "with")
+                                       "except" "finally" "for" "while" "with"
+                                       ;; Python 3.5+ PEP492
+                                       (and "async" (+ space)
+                                            (or "def" "for" "with")))
                                    symbol-end))
       (dedenter            . ,(rx symbol-start
                                    (or "elif" "else" "except" "finally")
@@ -395,7 +398,11 @@ python-mode-map
                                   symbol-end))
       (decorator            . ,(rx line-start (* space) ?@ (any letter ?_)
                                    (* (any word ?_))))
-      (defun                . ,(rx symbol-start (or "def" "class") symbol-end))
+      (defun                . ,(rx symbol-start
+                                   (or "def" "class"
+                                       ;; Python 3.5+ PEP492
+                                       (and "async" (+ space) "def"))
+                                   symbol-end))
       (if-name-main         . ,(rx line-start "if" (+ space) "__name__"
                                    (+ space) "==" (+ space)
                                    (any ?' ?\") "__main__" (any ?' ?\")
@@ -527,6 +534,8 @@ python-font-lock-keywords
           ;; fontified like that in order to keep font-lock consistent between
           ;; Python versions.
           "nonlocal"
+          ;; Python 3.5+ PEP492
+          (and "async" (+ space) (or "def" "for" "with"))
           ;; Extra:
           "self")
          symbol-end)
diff --git a/test/automated/python-tests.el b/test/automated/python-tests.el
index 44b05e2..23d799a 100644
--- a/test/automated/python-tests.el
+++ b/test/automated/python-tests.el
@@ -604,6 +604,42 @@ python-tests-visible-string
    (should (eq (car (python-indent-context)) :after-line))
    (should (= (python-indent-calculate-indentation) 0))))
 
+(ert-deftest python-indent-after-async-block-1 ()
+  "Test PEP492 async def."
+  (python-tests-with-temp-buffer
+   "
+async def foo(a, b, c=True):
+"
+   (should (eq (car (python-indent-context)) :no-indent))
+   (should (= (python-indent-calculate-indentation) 0))
+   (goto-char (point-max))
+   (should (eq (car (python-indent-context)) :after-block-start))
+   (should (= (python-indent-calculate-indentation) 4))))
+
+(ert-deftest python-indent-after-async-block-2 ()
+  "Test PEP492 async with."
+  (python-tests-with-temp-buffer
+   "
+async with foo(a) as mgr:
+"
+   (should (eq (car (python-indent-context)) :no-indent))
+   (should (= (python-indent-calculate-indentation) 0))
+   (goto-char (point-max))
+   (should (eq (car (python-indent-context)) :after-block-start))
+   (should (= (python-indent-calculate-indentation) 4))))
+
+(ert-deftest python-indent-after-async-block-3 ()
+  "Test PEP492 async for."
+  (python-tests-with-temp-buffer
+   "
+async for a in sequencer():
+"
+   (should (eq (car (python-indent-context)) :no-indent))
+   (should (= (python-indent-calculate-indentation) 0))
+   (goto-char (point-max))
+   (should (eq (car (python-indent-context)) :after-block-start))
+   (should (= (python-indent-calculate-indentation) 4))))
+
 (ert-deftest python-indent-after-backslash-1 ()
   "The most common case."
   (python-tests-with-temp-buffer
@@ -1483,6 +1519,26 @@ python-tests-visible-string
                 (beginning-of-line)
                 (point))))))
 
+(ert-deftest python-nav-beginning-of-defun-3 ()
+  (python-tests-with-temp-buffer
+   "
+class C(object):
+
+    async def m(self):
+        return await self.c()
+
+    async def c(self):
+        pass
+"
+   (python-tests-look-at "self.c()")
+   (should (= (save-excursion
+                (python-nav-beginning-of-defun)
+                (point))
+              (save-excursion
+                (python-tests-look-at "async def m" -1)
+                (beginning-of-line)
+                (point))))))
+
 (ert-deftest python-nav-end-of-defun-1 ()
   (python-tests-with-temp-buffer
    "

[-- Attachment #3: Type: text/plain, Size: 211 bytes --]


-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it  |                 -- Fortunato Depero, 1929.

bug#21783: Copyright assignment

[Lele Gaifax]

Maybe I could have stated that I fulfilled the assignment process (# 988143),
should that smooth the acceptance of the patch.
-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it  |                 -- Fortunato Depero, 1929.

bug#21783: Copyright assignment

[Eli Zaretskii]

> From: Lele Gaifax <lele@metapensiero.it>
> Date: Fri, 18 Dec 2015 11:29:51 +0100
> 
> Maybe I could have stated that I fulfilled the assignment process (# 988143),
> should that smooth the acceptance of the patch.

Thanks.  This patch should be reviewed and approved first.  Fabián,
can you look into this patch, please?

bug#21783: Copyright assignment

[Lele Gaifax]

Is there anything else I could do to help getting this into Emacs 25?

thank you&ciao, lele.
-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it  |                 -- Fortunato Depero, 1929.

bug#21783: Copyright assignment

[Lars Ingebrigtsen]

Lele Gaifax <lele@metapensiero.it> writes:

> Is there anything else I could do to help getting this into Emacs 25?

I think we're basically waiting for somebody with Python mode experience
to review the patch.  Fabián?  Mark?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21783: Patch applied

[Jorgen Schäfer]

[-- Attachment #1: Type: text/plain, Size: 185 bytes --]

Hello!
I just pushed this patch to git master as 1f6b0bc1. Thank you so much for
your contribution and I am very sorry that it took so long to get into the
repository.

Regards,
Jorgen

[-- Attachment #2: Type: text/html, Size: 256 bytes --]

bug#21783: closed (Patch applied)

[Lele Gaifax]

GNU bug Tracking System <help-debbugs@gnu.org> writes:

> I just pushed this patch to git master as 1f6b0bc1. Thank you so much for
> your contribution and I am very sorry that it took so long to get into the
> repository.

Thank you Jorgen, no need to apologize, really!

But I wonder: is there any chance to get this "backported" to the emacs-25
branch as well? Is there anything I should do to make that happen?

thanks again,
ciao, lele.
-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele@metapensiero.it  |                 -- Fortunato Depero, 1929.