bug#23387: 25.0.93; Unicode quote inserted in Python mode outside of strings

bug#23387: 25.0.93; Unicode quote inserted in Python mode outside of strings

[Philipp Stephani]

emacs -Q
M-x electric-quote-mode
M-x customize-variable RET electric-quote-string, set it to t, save
M-x python-mode
Hit '

Expected: ' (ASCII apostrophe) is inserted because point is outside a string
Actual: ’ (Unicode quote) is inserted

This seems to happen only in Python mode, not in e.g. emacs-lisp-mode.



In GNU Emacs 25.0.93.5 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8)
 of 2016-04-24 built on localhost
Repository revision: 0cd2e923dba8d8c7128b0c084ce6af22069e8db5
Windowing system distributor 'The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04 LTS

Configured using:
 'configure --with-modules
 LDFLAGS=-Wl,-fuse-ld=gold,--export-dynamic-symbol=__google_auxv'

Configured features:
XPM JPEG TIFF GIF PNG SOUND GSETTINGS NOTIFY FREETYPE XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-quote-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Electric-Quote mode enabled
Quit
Creating customization items...
Creating customization items ...done
Resetting customization items...done
Creating customization setup...done
To install your edits, invoke [State] and choose the Set operation
Source file ‘/usr/local/google/home/phst/ThirdParty/Emacs/lisp/progmodes/python.el’ newer than byte-compiled file
Can’t guess python-indent-offset, using defaults: 4
Source file ‘/usr/local/google/home/phst/ThirdParty/Emacs/lisp/dired.el’ newer than byte-compiled file

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired rfc822 mml mml-sec epg
epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mail-utils
python tramp-sh tramp tramp-compat auth-source cl-seq eieio eieio-core
cl-macs gnus-util mm-util mail-prsvr password-cache tramp-loaddefs
trampver ucs-normalize shell pcomplete format-spec advice json map seq
byte-opt gv bytecomp byte-compile cl-extra cconv comint ring ansi-color
cus-edit wid-edit thingatpt help-fns help-mode easymenu cl-loaddefs
pcase cl-lib cus-start cus-load time-date mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list newcomment elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer cl-preloaded nadvice loaddefs button faces
cus-face macroexp files text-properties overlay sha1 md5 base64 format
env code-pages mule custom widget hashtable-print-readable backquote
inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 239874 9745)
 (symbols 48 24466 0)
 (miscs 40 400 230)
 (strings 32 29068 7264)
 (string-bytes 1 922223)
 (vectors 16 39237)
 (vector-slots 8 730263 4252)
 (floats 8 240 60)
 (intervals 56 261 0)
 (buffers 976 14)
 (heap 1024 32672 1237))

-- 
Google Germany GmbH
Erika-Mann-Straße 33
80636 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle

Diese E-Mail ist vertraulich.  Wenn Sie nicht der richtige Adressat sind,
leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen
Sie die E-Mail und alle Anhänge.  Vielen Dank.

This e-mail is confidential.  If you are not the right addressee please do not
forward it, please inform the sender, and please erase this e-mail including
any attachments.  Thanks.

bug#23387: 25.0.93; Unicode quote inserted in Python mode outside of strings

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 489 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am Mi., 27. Apr. 2016 um
16:16 Uhr:

>
> emacs -Q
> M-x electric-quote-mode
> M-x customize-variable RET electric-quote-string, set it to t, save
> M-x python-mode
> Hit '
>
> Expected: ' (ASCII apostrophe) is inserted because point is outside a
> string
> Actual: ’ (Unicode quote) is inserted
>
> This seems to happen only in Python mode, not in e.g. emacs-lisp-mode.
>
> Also happens in C++ mode, and probably others.

[-- Attachment #2: Type: text/html, Size: 778 bytes --]

bug#23387: 25.0.93; Unicode quote inserted in Python mode outside of strings

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am Do., 28. Apr. 2016 um
19:11 Uhr:

> Philipp Stephani <p.stephani2@gmail.com> schrieb am Mi., 27. Apr. 2016 um
> 16:16 Uhr:
>
>>
>> emacs -Q
>> M-x electric-quote-mode
>> M-x customize-variable RET electric-quote-string, set it to t, save
>> M-x python-mode
>> Hit '
>>
>> Expected: ' (ASCII apostrophe) is inserted because point is outside a
>> string
>> Actual: ’ (Unicode quote) is inserted
>>
>> This seems to happen only in Python mode, not in e.g. emacs-lisp-mode.
>>
>> Also happens in C++ mode, and probably others.
>

The root cause is that electric-quote-post-self-insert-function uses
syntax-ppss to check whether point is in a string or comment. Before it
makes any replacement, the ASCII apostrophe is still in the buffer, so
syntax-ppss considers point to be inside a string. It looks like calling
syntax-ppss with (forward-point -1) as argument instead of (point) should
fix it, but I'm not sure whether that's the right way.

[-- Attachment #2: Type: text/html, Size: 1619 bytes --]

bug#23387: 25.0.93; Unicode quote inserted in Python mode outside of strings

[Philipp Stephani]

[-- Attachment #1.1: Type: text/plain, Size: 1176 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am Do., 28. Apr. 2016 um
19:22 Uhr:

> Philipp Stephani <p.stephani2@gmail.com> schrieb am Do., 28. Apr. 2016 um
> 19:11 Uhr:
>
>> Philipp Stephani <p.stephani2@gmail.com> schrieb am Mi., 27. Apr. 2016
>> um 16:16 Uhr:
>>
>>>
>>> emacs -Q
>>> M-x electric-quote-mode
>>> M-x customize-variable RET electric-quote-string, set it to t, save
>>> M-x python-mode
>>> Hit '
>>>
>>> Expected: ' (ASCII apostrophe) is inserted because point is outside a
>>> string
>>> Actual: ’ (Unicode quote) is inserted
>>>
>>> This seems to happen only in Python mode, not in e.g. emacs-lisp-mode.
>>>
>>> Also happens in C++ mode, and probably others.
>>
>
> The root cause is that electric-quote-post-self-insert-function uses
> syntax-ppss to check whether point is in a string or comment. Before it
> makes any replacement, the ASCII apostrophe is still in the buffer, so
> syntax-ppss considers point to be inside a string. It looks like calling
> syntax-ppss with (forward-point -1) as argument instead of (point) should
> fix it, but I'm not sure whether that's the right way.
>

I've attached a patch.

[-- Attachment #1.2: Type: text/html, Size: 2067 bytes --]

[-- Attachment #2: 0001-Fix-insertion-of-Unicode-quotes-in-strings.patch --]
[-- Type: application/octet-stream, Size: 2052 bytes --]

From cd7c17079c7efdd00fadc01c6622db3baab9d48a Mon Sep 17 00:00:00 2001
From: Philipp Stephani <phst@google.com>
Date: Sat, 30 Apr 2016 12:45:22 +0200
Subject: [PATCH] Fix insertion of Unicode quotes in strings.

* lisp/electric.el (electric-quote-post-self-insert-function): Parse
until before point to fix Bug#23387.
* test/automated/electric-tests.el (electric-quote-string): Add unit
test for electric-quote-string.
---
 lisp/electric.el                 | 5 ++++-
 test/automated/electric-tests.el | 8 ++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/lisp/electric.el b/lisp/electric.el
index ab79943..74b4e91 100644
--- a/lisp/electric.el
+++ b/lisp/electric.el
@@ -444,7 +444,10 @@ electric-quote-post-self-insert-function
     (let ((start
            (if (and comment-start comment-use-syntax)
                (when (or electric-quote-comment electric-quote-string)
-                 (let ((syntax (syntax-ppss)))
+                 ;; Here the quote has already been inserted, so we
+                 ;; might be inside a string.  Therefore use the
+                 ;; position before point for syntactic parsing.
+                 (let ((syntax (save-excursion (syntax-ppss (1- (point))))))
                    (and (or (and electric-quote-comment (nth 4 syntax))
                             (and electric-quote-string (nth 3 syntax)))
                         (nth 8 syntax))))
diff --git a/test/automated/electric-tests.el b/test/automated/electric-tests.el
index 107b2e7..afd7076 100644
--- a/test/automated/electric-tests.el
+++ b/test/automated/electric-tests.el
@@ -584,5 +584,13 @@ autowrapping-7
                   (skip-chars-backward "\"")
                   (mark-sexp -1)))
 
+\f
+;;; Electric quotes
+(define-electric-pair-test electric-quote-string
+  "" "'" :expected-string "'" :expected-point 2
+  :fixture-fn #'electric-quote-local-mode
+  :bindings '((electric-quote-string . t))
+  :test-in-comments nil :test-in-strings nil)
+
 (provide 'electric-tests)
 ;;; electric-tests.el ends here
-- 
2.8.1

bug#23387: 25.0.93; Unicode quote inserted in Python mode outside of strings

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

Thanks for the bug report and patch, but I ran into a problem with the 
patch. Suppose I do this:

emacs -Q
M-x electric-quote-mode
M-x customize-variable RET electric-quote-string, set it to t, save
M-x python-mode
'abc'

That is, I type apostrophe-a-b-c-apostrophe in Python mode. With the 
patch, the buffer will contain:

'abc’

The opening apostrophe is OK, but the closing apostrophe is considered 
to be inside a string so it is transmogrified into a right single 
quotation mark. A Python programmer would want both to be apostrophes.

I installed the attached patches into emacs-25 instead, as they should 
address this problem too. Please give it a try.

[-- Attachment #2: 0001-Don-t-electrically-quote-in-Python.patch --]
[-- Type: application/x-patch, Size: 1773 bytes --]

[-- Attachment #3: 0002-Add-electric-quote-string-unit-test.patch --]
[-- Type: application/x-patch, Size: 1001 bytes --]

bug#23387: 25.0.93; Unicode quote inserted in Python mode outside of strings

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

Paul Eggert <eggert@cs.ucla.edu> schrieb am Mo., 2. Mai 2016 um 18:05 Uhr:

> Thanks for the bug report and patch, but I ran into a problem with the
> patch. Suppose I do this:
>
> emacs -Q
> M-x electric-quote-mode
> M-x customize-variable RET electric-quote-string, set it to t, save
> M-x python-mode
> 'abc'
>
> That is, I type apostrophe-a-b-c-apostrophe in Python mode. With the
> patch, the buffer will contain:
>
> 'abc’
>
> The opening apostrophe is OK, but the closing apostrophe is considered
> to be inside a string so it is transmogrified into a right single
> quotation mark. A Python programmer would want both to be apostrophes.
>

I'm not sure, I think if a user has electric-quote-string enabled, that's
the behavior they want, because it's inside a string. However, I can see
that both approaches are valid. There is probably no good way that would
work in all circumstances.

[-- Attachment #2: Type: text/html, Size: 1243 bytes --]