bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Eli Zaretskii]

To reproduce:

  emacs -Q
  M-x electric-quote-mode RET
  M-x set-variable RET electric-quote-replace-double RET t RET

Type:

  "foo \"foo\""

You get this in the buffer:

  "foo \”foo\”"

I expected "foo \“foo\”" instead.


In GNU Emacs 27.0.50 (build 56, i686-pc-mingw32)
 of 2017-12-22 built on HOME-C4E4A596F7
Repository revision: b003171d27dfa4f0a5e6f8d9eb632b1930748e95
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...
Electric-Quote mode enabled
You can run the command ‘electric-quote-mode’ with M-x e-q-m RET
Electric-Quote mode enabled
Scanning for dabbrevs...done
user-error: No dynamic expansion for ‘Electric-quote-rep’ found
Scanning for dabbrevs...done
user-error: No dynamic expansion for ‘Electric-quote-replace-d’ found

Configured using:
 'configure --prefix=/d/usr --enable-checking=yes,glyphs --with-wide-int
 --with-modules --enable-check-lisp-object-type 'CFLAGS=-O0 -gdwarf-4
 -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS MODULES JSON LCMS2

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-quote-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr dabbrev emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv dired dired-loaddefs format-spec rfc822 mml
mml-sec password-cache epa derived epg epg-config gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils cus-edit wid-edit thingatpt help-fns
radix-tree help-mode easymenu cl-loaddefs cl-lib cus-start cus-load
elec-pair time-date mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table term/w32-win
w32-win w32-vars term/common-win tool-bar dnd fontset image regexp-opt
fringe tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote w32notify w32 lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 122305 12056)
 (symbols 56 22916 1)
 (miscs 48 43 120)
 (strings 16 35784 2142)
 (string-bytes 1 868306)
 (vectors 16 15963)
 (vector-slots 8 742591 12998)
 (floats 8 66 193)
 (intervals 40 317 128)
 (buffers 880 12))

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Fr., 22. Dez. 2017 um 14:41 Uhr:

> To reproduce:
>
>   emacs -Q
>   M-x electric-quote-mode RET
>   M-x set-variable RET electric-quote-replace-double RET t RET
>
> Type:
>
>   "foo \"foo\""
>
> You get this in the buffer:
>
>   "foo \”foo\”"
>
> I expected "foo \“foo\”" instead.
>
>
I think it's not completely clear what to expect here. After all, electric
quote is for human-language text, which normally doesn't contain
backslashes.
At least in the context of Emacs Lisp strings, I'd expect "foo \"foo\""
here, i.e., ASCII quotes. The non-ASCII quotes don't need to be escaped, so
presumably escaping means that the user intended to type an ASCII quote.

[-- Attachment #2: Type: text/html, Size: 1118 bytes --]

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sat, 30 Dec 2017 22:07:16 +0000
> Cc: 29812@debbugs.gnu.org
> 
>    emacs -Q
>    M-x electric-quote-mode RET
>    M-x set-variable RET electric-quote-replace-double RET t RET
> 
>  Type:
> 
>    "foo \"foo\""
> 
>  You get this in the buffer:
> 
>    "foo \”foo\”"
> 
>  I expected "foo \“foo\”" instead.
> 
> I think it's not completely clear what to expect here. After all, electric quote is for human-language text, which
> normally doesn't contain backslashes.

AFAIK, electric-quote-replace-double is supposed to work in comments
and strings in buffers under programming language modes, not only in
text modes.  And it works correctly for me in C modes and also in Lisp
comments, so why not in Lisp strings?

> At least in the context of Emacs Lisp strings, I'd expect "foo \"foo\"" here, i.e., ASCII quotes. The non-ASCII
> quotes don't need to be escaped, so presumably escaping means that the user intended to type an ASCII
> quote.

Typing just a quote in a Lisp string terminates the string, so I
wouldn't expect that to produce curved quotes.  And a backslash just
quotes the next character, so there's nothing wrong with having it
before curved quotes.

Anyway, if this feature is not supposed to work reliably in
programming language strings, perhaps we shouldn't try?  Having it
sometimes work and sometimes not is IMO confusing.

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2143 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 31. Dez. 2017 um 16:51 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sat, 30 Dec 2017 22:07:16 +0000
> > Cc: 29812@debbugs.gnu.org
> >
> >    emacs -Q
> >    M-x electric-quote-mode RET
> >    M-x set-variable RET electric-quote-replace-double RET t RET
> >
> >  Type:
> >
> >    "foo \"foo\""
> >
> >  You get this in the buffer:
> >
> >    "foo \”foo\”"
> >
> >  I expected "foo \“foo\”" instead.
> >
> > I think it's not completely clear what to expect here. After all,
> electric quote is for human-language text, which
> > normally doesn't contain backslashes.
>
> AFAIK, electric-quote-replace-double is supposed to work in comments
> and strings in buffers under programming language modes, not only in
> text modes.  And it works correctly for me in C modes and also in Lisp
> comments, so why not in Lisp strings?
>

Does it work as expected for you in C strings? I see the same behavior in C
strings as in Lisp.


>
> > At least in the context of Emacs Lisp strings, I'd expect "foo \"foo\""
> here, i.e., ASCII quotes. The non-ASCII
> > quotes don't need to be escaped, so presumably escaping means that the
> user intended to type an ASCII
> > quote.
>
> Typing just a quote in a Lisp string terminates the string, so I
> wouldn't expect that to produce curved quotes.  And a backslash just
> quotes the next character, so there's nothing wrong with having it
> before curved quotes.
>

True, but why would you quote a character that doesn't need quoting? Doing
so could be taken as a hint that the character doesn't need quoting.


>
> Anyway, if this feature is not supposed to work reliably in
> programming language strings, perhaps we shouldn't try?  Having it
> sometimes work and sometimes not is IMO confusing.
>

It should work in comments and strings, yes. However, given that the
behavior is heuristic in all cases it's hard to define what the correct
behavior should be.
It seems that the behavior you expect should be relatively easy to
implement, though. I'll try to send a patch.

[-- Attachment #2: Type: text/html, Size: 3075 bytes --]

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 31 Dec 2017 16:16:02 +0000
> Cc: 29812@debbugs.gnu.org
> 
>  AFAIK, electric-quote-replace-double is supposed to work in comments
>  and strings in buffers under programming language modes, not only in
>  text modes.  And it works correctly for me in C modes and also in Lisp
>  comments, so why not in Lisp strings?
> 
> Does it work as expected for you in C strings? I see the same behavior in C strings as in Lisp.

In C, "\"foo\"" produces ASCII quotes.

>  Anyway, if this feature is not supposed to work reliably in
>  programming language strings, perhaps we shouldn't try?  Having it
>  sometimes work and sometimes not is IMO confusing.
> 
> It should work in comments and strings, yes. However, given that the behavior is heuristic in all cases it's hard
> to define what the correct behavior should be.

Well, can you give an example where it does work in strings?  Maybe
I'm missing something, because it looked to me as if it never works in
that case.

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 31. Dez. 2017 um 17:38 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 31 Dec 2017 16:16:02 +0000
> > Cc: 29812@debbugs.gnu.org
> >
> >  AFAIK, electric-quote-replace-double is supposed to work in comments
> >  and strings in buffers under programming language modes, not only in
> >  text modes.  And it works correctly for me in C modes and also in Lisp
> >  comments, so why not in Lisp strings?
> >
> > Does it work as expected for you in C strings? I see the same behavior
> in C strings as in Lisp.
>
> In C, "\"foo\"" produces ASCII quotes.
>

Did you enable electric-quote-string?


>
> >  Anyway, if this feature is not supposed to work reliably in
> >  programming language strings, perhaps we shouldn't try?  Having it
> >  sometimes work and sometimes not is IMO confusing.
> >
> > It should work in comments and strings, yes. However, given that the
> behavior is heuristic in all cases it's hard
> > to define what the correct behavior should be.
>
> Well, can you give an example where it does work in strings?  Maybe
> I'm missing something, because it looked to me as if it never works in
> that case.
>

Depends on what you mean with "work". A bare " should always close the
string; after a \ it currently inserts an opening quote because it only
looks back one character.

[-- Attachment #2: Type: text/html, Size: 2068 bytes --]

bug#29812: [PATCH] Ignore escape characters for context-sensitive quotes (Bug#29812)

[Philipp Stephani]

* lisp/electric.el (electric-quote-post-self-insert-function): Skip
over escape characters when determining whether a context-sensitive
quote should be opening or closing.

* test/lisp/electric-tests.el
(electric-quote-replace-double-escaped-open)
(electric-quote-replace-double-escaped-close): New unit tests.
---
 lisp/electric.el            |  1 +
 test/lisp/electric-tests.el | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+)

diff --git a/lisp/electric.el b/lisp/electric.el
index cee3562139..9473ef374d 100644
--- a/lisp/electric.el
+++ b/lisp/electric.el
@@ -504,6 +504,7 @@ electric-quote-post-self-insert-function
                             electric-quote-replace-double)
                         (save-excursion
                           (backward-char)
+                          (skip-syntax-backward "\\")
                           (or (bobp) (bolp)
                               (memq (char-before) (list q< q<<))
                               (memq (char-syntax (char-before))
diff --git a/test/lisp/electric-tests.el b/test/lisp/electric-tests.el
index 7df2449b9e..793cd7c5d3 100644
--- a/test/lisp/electric-tests.el
+++ b/test/lisp/electric-tests.el
@@ -735,6 +735,24 @@ electric-quote-replace-double-after-paren
   :bindings '((electric-quote-replace-double . t))
   :test-in-comments nil :test-in-strings nil)
 
+(define-electric-pair-test electric-quote-replace-double-escaped-open
+  "foo \\" "-----\"" :expected-string "foo \\“"
+  :expected-point 7 :modes '(emacs-lisp-mode c-mode)
+  :fixture-fn #'electric-quote-local-mode
+  :bindings '((electric-quote-replace-double . t)
+              (electric-quote-comment . t)
+              (electric-quote-string . t))
+  :test-in-comments t :test-in-strings t :test-in-code nil)
+
+(define-electric-pair-test electric-quote-replace-double-escaped-close
+  "foo \\“foo\\" "----------\"" :expected-string "foo \\“foo\\”"
+  :expected-point 12 :modes '(emacs-lisp-mode c-mode)
+  :fixture-fn #'electric-quote-local-mode
+  :bindings '((electric-quote-replace-double . t)
+              (electric-quote-comment . t)
+              (electric-quote-string . t))
+  :test-in-comments t :test-in-strings t :test-in-code nil)
+
 ;; Simulate ‘markdown-mode’: it sets both ‘comment-start’ and
 ;; ‘comment-use-syntax’, but derives from ‘text-mode’.
 (define-electric-pair-test electric-quote-markdown-in-text
-- 
2.15.1

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 31 Dec 2017 16:49:31 +0000
> Cc: 29812@debbugs.gnu.org
> 
>  In C, "\"foo\"" produces ASCII quotes.
> 
> Did you enable electric-quote-string?

Should I?

>  Well, can you give an example where it does work in strings?  Maybe
>  I'm missing something, because it looked to me as if it never works in
>  that case.
> 
> Depends on what you mean with "work".

I mean some way of inserting “foo” inside a string.  Is that possible
somehow?

> A bare " should always close the string; after a \ it currently inserts
> an opening quote because it only looks back one character. 

Which is a bug, isn't it?

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 31. Dez. 2017 um 18:00 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 31 Dec 2017 16:49:31 +0000
> > Cc: 29812@debbugs.gnu.org
> >
> >  In C, "\"foo\"" produces ASCII quotes.
> >
> > Did you enable electric-quote-string?
>
> Should I?
>

Yes, it's nil by default.


>
> >  Well, can you give an example where it does work in strings?  Maybe
> >  I'm missing something, because it looked to me as if it never works in
> >  that case.
> >
> > Depends on what you mean with "work".
>
> I mean some way of inserting “foo” inside a string.  Is that possible
> somehow?
>

Sure, either by inserting the characters in some other way, or by using ``
and '' (double apostrophe).


>
> > A bare " should always close the string; after a \ it currently inserts
> > an opening quote because it only looks back one character.
>
> Which is a bug, isn't it?
>

Maybe. As said, it's a heuristic, and there's no unambiguous "correct"
behavior. But the patch I've sent modifies the behavior so that it ignores
the escape character.

[-- Attachment #2: Type: text/html, Size: 1970 bytes --]

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 31 Dec 2017 17:03:14 +0000
> Cc: 29812@debbugs.gnu.org
> 
>  I mean some way of inserting “foo” inside a string.  Is that possible
>  somehow?
> 
> Sure, either by inserting the characters in some other way, or by using `` and '' (double apostrophe).

Then maybe we should just give up on electric-quote-replace-double
inside strings, and use double apostrophes instead?

bug#29812: [PATCH] Ignore escape characters for context-sensitive quotes (Bug#29812)

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 471 bytes --]

Philipp Stephani <p.stephani2@gmail.com> schrieb am So., 31. Dez. 2017 um
17:51 Uhr:

> * lisp/electric.el (electric-quote-post-self-insert-function): Skip
> over escape characters when determining whether a context-sensitive
> quote should be opening or closing.
>
> * test/lisp/electric-tests.el
> (electric-quote-replace-double-escaped-open)
> (electric-quote-replace-double-escaped-close): New unit tests.
>

No comments, so I've pushed this to master as 6735df4443.

[-- Attachment #2: Type: text/html, Size: 783 bytes --]

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 31. Dez. 2017 um 18:30 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 31 Dec 2017 17:03:14 +0000
> > Cc: 29812@debbugs.gnu.org
> >
> >  I mean some way of inserting “foo” inside a string.  Is that possible
> >  somehow?
> >
> > Sure, either by inserting the characters in some other way, or by using
> `` and '' (double apostrophe).
>
> Then maybe we should just give up on electric-quote-replace-double
> inside strings, and use double apostrophes instead?
>

Why? It works as designed and expected – that is, a double quote will
terminate the string. That is what users want most of the time. Also, there
are many languages where strings aren't double-quoted, such as Python.

[-- Attachment #2: Type: text/html, Size: 1212 bytes --]

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sun, 07 Jan 2018 12:56:31 +0000
> Cc: 29812@debbugs.gnu.org
> 
>  >  I mean some way of inserting “foo” inside a string.  Is that possible
>  >  somehow?
>  >
>  > Sure, either by inserting the characters in some other way, or by using `` and '' (double apostrophe).
> 
>  Then maybe we should just give up on electric-quote-replace-double
>  inside strings, and use double apostrophes instead?
> 
> Why? It works as designed and expected – that is, a double quote will terminate the string. That is what users
> want most of the time. Also, there are many languages where strings aren't double-quoted, such as Python. 

Once again, my problem is that one cannot insert “foo” inside strings
(unless in languages where strings are quoted 'like this', I guess).
So I'm saying that we probably shouldn't advertise this method for
text in strings in programming modes, because it doesn't really work
there.

bug#29812: 27.0.50; electric-quote-replace-double misbehaves in Lisp strings

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1235 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am So., 7. Jan. 2018 um 17:50 Uhr:

> > From: Philipp Stephani <p.stephani2@gmail.com>
> > Date: Sun, 07 Jan 2018 12:56:31 +0000
> > Cc: 29812@debbugs.gnu.org
> >
> >  >  I mean some way of inserting “foo” inside a string.  Is that possible
> >  >  somehow?
> >  >
> >  > Sure, either by inserting the characters in some other way, or by
> using `` and '' (double apostrophe).
> >
> >  Then maybe we should just give up on electric-quote-replace-double
> >  inside strings, and use double apostrophes instead?
> >
> > Why? It works as designed and expected – that is, a double quote will
> terminate the string. That is what users
> > want most of the time. Also, there are many languages where strings
> aren't double-quoted, such as Python.
>
> Once again, my problem is that one cannot insert “foo” inside strings
> (unless in languages where strings are quoted 'like this', I guess).
> So I'm saying that we probably shouldn't advertise this method for
> text in strings in programming modes, because it doesn't really work
> there.
>

Feel free to clarify the NEWS entry. (You mentioned strings there in commit
e92f5537a8222187525ef5066dba051211db5290.)

[-- Attachment #2: Type: text/html, Size: 1743 bytes --]