bug#13131: 24.1; Allow curly quotes to be found by searching for straight quotes?

bug#13131: 24.1; Allow curly quotes to be found by searching for straight quotes?

[Reuben Thomas]

isearch kindly matches multiple spaces when you insert a space;
similarly, it'd be good if searching for a straight quote also matched
curly quotes (arguably only in modes where that really makes sense, i.e.
text modes).


In GNU Emacs 24.1.1 (x86_64-pc-linux-gnu, GTK+ Version 2.24.12)
 of 2012-09-22 on batsu, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.11300000
Configured using:
 `configure '--build' 'x86_64-linux-gnu' '--build' 'x86_64-linux-gnu'
 '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib'
 '--localstatedir=/var/lib' '--infodir=/usr/share/info'
 '--mandir=/usr/share/man' '--with-pop=yes'
 '--enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/24.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.1/site-lisp:/usr/share/emacs/site-lisp'
 '--with-crt-dir=/usr/lib/x86_64-linux-gnu' '--with-x=yes'
 '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars'
 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector
 --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -O2'
 'CPPFLAGS=-D_FORTIFY_SOURCE=2''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: en_GB.UTF-8
  value of $LC_NUMERIC: en_GB.UTF-8
  value of $LC_TIME: en_GB.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: LaTeX/P

Minor modes in effect:
  TeX-PDF-mode: t
  buffer-face-mode: t
  flyspell-mode: t
  smart-quotes-mode: t
  recentf-mode: t
  show-paren-mode: t
  server-mode: t
  savehist-mode: t
  minibuffer-electric-default-mode: t
  iswitchb-mode: t
  icomplete-mode: t
  global-auto-revert-mode: t
  desktop-save-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  visual-line-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent input:
C-s C-s C-s C-s C-s C-s C-s C-s C-s C-a C-c C-c <return> 
C-s s t i l l SPC o f C-a <down> <down> <down> <down> 
<left> <left> <left> <left> <right> C-s H e r e SPC 
a r e SPC <right> <left> <M-backspace> i s SPC <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <left> <left> <backspace> C-s 
w i C-a C-x C-s C-s w i t h SPC o n e SPC C-s <down> 
<left> <left> <left> <left> <left> <left> <left> <left> 
<left> <left> <left> <left> <left> <left> <left> <left> 
<left> <left> <left> <left> <left> <left> <left> <left> 
<left> <left> <left> <left> <left> <left> <left> <left> 
<left> <left> <left> <left> <left> <left> <left> <left> 
<left> <left> " o n e SPC c c o <backspace> <backspace> 
<backspace> a c c o r d " M-b M-b M-c <right> <right> 
<right> <right> <right> <right> <right> <right> SPC 
h a s SPC m a n y SPC c o n c o r d s M-d M-d M-d M-d 
C-_ <backspace> <backspace> d s : C-d C-d C-x C-s C-d 
C-x C-s C-a <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <tab> C-_ <left> <left> <left> <left> <left> 
<left> <left> <left> <left> <left> <left> <left> <left> 
<right> <right> <M-backspace> <backspace> , SPC a y 
e , SPC o n C-x C-s <down> <down> <down> <down> <down> 
<down> <down> <down> <down> <down> <down> C-s T h e 
r e ' <backspace> ' s <backspace> <backspace> M-x r 
e p o r t - e m c a s - <backspace> <backspace> <backspace> 
<backspace> a c s - b u g <return>

Recent messages:
Saving file /home/rrt/Writing/Poetry/youth/youth.tex...
Wrote /home/rrt/Writing/Poetry/youth/youth.tex
Saving file /home/rrt/Writing/Poetry/youth/youth.tex...
Wrote /home/rrt/Writing/Poetry/youth/youth.tex
Undo!
Auto-saving...done
Auto-saving...done
Saving file /home/rrt/Writing/Poetry/youth/youth.tex...
Wrote /home/rrt/Writing/Poetry/youth/youth.tex
Mark saved where search started

Load-path shadows:
/home/rrt/local/share/emacs/site-lisp/dict hides /usr/share/emacs24/site-lisp/emacs-goodies-el/dict
/home/rrt/local/share/emacs/site-lisp/browse-kill-ring hides /usr/share/emacs24/site-lisp/emacs-goodies-el/browse-kill-ring
/home/rrt/.emacs.d/elpa/dictionary-1.8.7/link hides /usr/share/emacs24/site-lisp/dictionary-el/link
/home/rrt/.emacs.d/elpa/dictionary-1.8.7/dictionary hides /usr/share/emacs24/site-lisp/dictionary-el/dictionary
/home/rrt/.emacs.d/elpa/dictionary-1.8.7/dictionary-init hides /usr/share/emacs24/site-lisp/dictionary-el/dictionary-init
/home/rrt/.emacs.d/elpa/dictionary-1.8.7/connection hides /usr/share/emacs24/site-lisp/dictionary-el/connection
/usr/share/emacs24/site-lisp/auctex/multi-prompt hides /usr/share/emacs/site-lisp/auctex/multi-prompt
/usr/share/emacs24/site-lisp/auctex/latex hides /usr/share/emacs/site-lisp/auctex/latex
/usr/share/emacs24/site-lisp/auctex/texmathp hides /usr/share/emacs/site-lisp/auctex/texmathp
/usr/share/emacs24/site-lisp/auctex/tex-fold hides /usr/share/emacs/site-lisp/auctex/tex-fold
/usr/share/emacs24/site-lisp/auctex/tex-jp hides /usr/share/emacs/site-lisp/auctex/tex-jp
/usr/share/emacs24/site-lisp/auctex/preview hides /usr/share/emacs/site-lisp/auctex/preview
/usr/share/emacs24/site-lisp/auctex/tex-bar hides /usr/share/emacs/site-lisp/auctex/tex-bar
/usr/share/emacs24/site-lisp/auctex/tex-mik hides /usr/share/emacs/site-lisp/auctex/tex-mik
/usr/share/emacs24/site-lisp/auctex/toolbar-x hides /usr/share/emacs/site-lisp/auctex/toolbar-x
/usr/share/emacs24/site-lisp/auctex/tex hides /usr/share/emacs/site-lisp/auctex/tex
/usr/share/emacs24/site-lisp/auctex/prv-emacs hides /usr/share/emacs/site-lisp/auctex/prv-emacs
/usr/share/emacs24/site-lisp/auctex/context hides /usr/share/emacs/site-lisp/auctex/context
/usr/share/emacs24/site-lisp/auctex/tex-info hides /usr/share/emacs/site-lisp/auctex/tex-info
/usr/share/emacs24/site-lisp/auctex/tex-font hides /usr/share/emacs/site-lisp/auctex/tex-font
/usr/share/emacs24/site-lisp/auctex/tex-style hides /usr/share/emacs/site-lisp/auctex/tex-style
/usr/share/emacs24/site-lisp/auctex/bib-cite hides /usr/share/emacs/site-lisp/auctex/bib-cite
/usr/share/emacs24/site-lisp/auctex/font-latex hides /usr/share/emacs/site-lisp/auctex/font-latex
/usr/share/emacs24/site-lisp/auctex/context-nl hides /usr/share/emacs/site-lisp/auctex/context-nl
/usr/share/emacs24/site-lisp/auctex/plain-tex hides /usr/share/emacs/site-lisp/auctex/plain-tex
/usr/share/emacs24/site-lisp/auctex/context-en hides /usr/share/emacs/site-lisp/auctex/context-en
/usr/share/emacs24/site-lisp/auctex/tex-buf hides /usr/share/emacs/site-lisp/auctex/tex-buf
/usr/share/emacs/site-lisp/golang-mode/go-mode-load hides /usr/share/emacs/24.1/site-lisp/golang-mode/go-mode-load
/usr/share/emacs/site-lisp/golang-mode/go-mode hides /usr/share/emacs/24.1/site-lisp/golang-mode/go-mode
/usr/share/emacs/24.1/site-lisp/cmake-data/cmake-mode hides /usr/share/emacs/site-lisp/cmake-mode
/usr/share/emacs24/site-lisp/dictionaries-common/ispell hides /usr/share/emacs/24.1/lisp/textmodes/ispell
/usr/share/emacs24/site-lisp/dictionaries-common/flyspell hides /usr/share/emacs/24.1/lisp/textmodes/flyspell
/usr/share/emacs/site-lisp/rst hides /usr/share/emacs/24.1/lisp/textmodes/rst

Features:
(plain-tex qp newcomment shadow sort gnus-util mail-extr emacsbug
message format-spec rfc822 mml mml-sec mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils jka-compr find-func pp
help-mode view multi-isearch etags texmathp latexenc make-mode autoconf
autoconf-mode vc-git lua-mode cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs flymake compile comint
ansi-color ring preview prv-emacs warnings tex-buf noutline outline
font-latex byte-opt bytecomp byte-compile cconv macroexp latex tex-style
tex dbus xml face-remap regexp-opt flyspell ispell smart-quotes
auto-dictionary-autoloads c-eldoc-autoloads dictionary-autoloads
diff-git-autoloads dired-isearch-autoloads full-ack-autoloads
guess-style-autoloads kill-ring-search-autoloads magit-autoloads
mv-shell-autoloads tumble-autoloads http-post-simple-autoloads package
tabulated-list completing-help recentf tree-widget wid-edit uniquify
paren server savehist minibuf-eldef iswitchb icomplete autorevert
desktop cus-start cus-load ropemacs pymacs user-site-loaddefs advice
advice-preload yasnippet help-fns derived edmacro kmacro easymenu assoc
cl go-mode-load emacs-goodies-el emacs-goodies-custom
emacs-goodies-loaddefs easy-mmode preview-latex tex-site auto-loads
time-date tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd
tool-bar dnd fontset image fringe lisp-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer loaddefs button faces
cus-face files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
make-network-process dbusbind dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

-- 
http://rrt.sc3d.org/

bug#13131: 24.1; Allow curly quotes to be found by searching for straight quotes?

[Juri Linkov]

> isearch kindly matches multiple spaces when you insert a space;
> similarly, it'd be good if searching for a straight quote also matched
> curly quotes (arguably only in modes where that really makes sense, i.e.
> text modes).

I suppose under "straight quote" you mean Unicode char named APOSTROPHE, and
"curly quotes" are LEFT SINGLE QUOTATION MARK and RIGHT SINGLE QUOTATION MARK.

There is a mapping of LEFT DOUBLE QUOTATION MARK and RIGHT DOUBLE QUOTATION MARK
to "``" and "''" in `IT-unicode-translations' in lisp/term/internal.el, but
it's not suitable for this.  I can't find more mappings in the source code,
ucs-normalize.el says that Punctuation is excluded from Normalization:

;; HFS Normalization excludes following area for decomposition.
;;  U+02000 .. U+02FFF :: Punctuation, symbols, dingbats, arrows, etc.

But it seems Normalization can't help anyway because there is no
decomposition information for such quotes in UnicodeData.txt.

Maybe the Unicode standard has a separate document for equivalence mappings
for punctuation.  I can found only the following table.  Is this what is needed?

http://unicode.org/cldr/charts/supplemental/character_fallback_substitutions.html

bug#13131: 24.1; Allow curly quotes to be found by searching for straight quotes?

[Eli Zaretskii]

> From: Juri Linkov <juri@jurta.org>
> Date: Mon, 10 Dec 2012 09:59:05 +0200
> Cc: 13131@debbugs.gnu.org
> 
> There is a mapping of LEFT DOUBLE QUOTATION MARK and RIGHT DOUBLE QUOTATION MARK
> to "``" and "''" in `IT-unicode-translations' in lisp/term/internal.el

Irrelevant to the issue at hand.  That mapping (and internal.el in
general) is used by the MS-DOS build of Emacs to allow a legible
display of characters that the DOS codepage doesn't support.

> I can't find more mappings in the source code, ucs-normalize.el says
> that Punctuation is excluded from Normalization:
> 
> ;; HFS Normalization excludes following area for decomposition.
> ;;  U+02000 .. U+02FFF :: Punctuation, symbols, dingbats, arrows, etc.
> 
> But it seems Normalization can't help anyway because there is no
> decomposition information for such quotes in UnicodeData.txt.

This is not a normalization issue.  This is about an Emacs specific
feature, and should be treated as such.

We should begin by discussing whether we really want this feature, by
default or otherwise.  One of the aspects we need to consider, IMO, is
what do spell-checkers do with these characters: if they don't treat
them as word-constituents, that would be one argument against the
change; if they do, it's an argument for the change.

> Maybe the Unicode standard has a separate document for equivalence mappings
> for punctuation.  I can found only the following table.  Is this what is needed?
> 
> http://unicode.org/cldr/charts/supplemental/character_fallback_substitutions.html

This seems to be relevant to display, not to the issue at hand.  If
anything, it should tell us to treat ' as we treat ’, not the other
way around.

bug#13131: 24.1; Allow curly quotes to be found by searching for straight quotes?

[Juri Linkov]

>> http://unicode.org/cldr/charts/supplemental/character_fallback_substitutions.html
>
> This seems to be relevant to display, not to the issue at hand.  If
> anything, it should tell us to treat ' as we treat ’, not the other
> way around.

Most of the equivalent sequences in character_fallback_substitutions.html
are NFC and NFKC that means normalization using the decomposition property
whose equivalence we already argeed to implement in the search.

A few of the equivalent sequences are marked "Explicit" and
all of them are defined in a separate very small file

http://unicode.org/repos/cldr/trunk/common/supplemental/characters.xml

whose mappings we could also add to our equivalence tables explicitly.

bug#13131: 24.1; Allow curly quotes to be found by searching for straight quotes?

[Reuben Thomas]

[-- Attachment #1: Type: text/plain, Size: 1558 bytes --]

On 10 December 2012 08:28, Eli Zaretskii <eliz@gnu.org> wrote:

>
> We should begin by discussing whether we really want this feature, by
> default or otherwise.  One of the aspects we need to consider, IMO, is
> what do spell-checkers do with these characters: if they don't treat
> them as word-constituents, that would be one argument against the
> change; if they do, it's an argument for the change.
>

At least aspell now seems to accept curly quotes by default in some locales.

However,  I think the rationale for the feature is independent of
spell-checkers. Emacs has had support for displaying curly quotes for some
time. There are various ways to enter them conveniently (the Emacs Wiki
lists smart-quotes.el, which I use, and typopunct.el. The sole author of
smart-quotes.el would be quite happy to assign copyright to GNU and have it
added to Emacs; I suggested he contact the maintainers about this in August
2011. (Incidentally, I'd be happy to facilitate the process if Emacs
maintainers were interested.)

Since curly quotes make perfect sense as part of words, in exactly the same
way as straight quotes, it seems reasonable to treat them the same for
movement, even if spelling support is not yet perfect; indeed, any
additional incentive to spelling program authors to improve Unicode
punctuation support is welcome.

For writing text, the introduction of Unicode has been a boon, and it would
be great if Emacs's "semantic" support would catch up with its excellent
display support. This is one aspect of that.

-- 
http://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 1981 bytes --]

bug#13131: 24.1; Allow curly quotes to be found by searching for straight quotes?

[Juri Linkov]

> isearch kindly matches multiple spaces when you insert a space;
> similarly, it'd be good if searching for a straight quote also matched
> curly quotes (arguably only in modes where that really makes sense, i.e.
> text modes).

This is implemented now by char-fold feature.

bug#13131: 24.1; Allow curly quotes to be found by searching for straight quotes?

[Reuben Thomas]

[-- Attachment #1: Type: text/plain, Size: 420 bytes --]

On Sun, 28 Jul 2019 at 23:59, Juri Linkov <juri@linkov.net> wrote:

> > isearch kindly matches multiple spaces when you insert a space;
> > similarly, it'd be good if searching for a straight quote also matched
> > curly quotes (arguably only in modes where that really makes sense, i.e.
> > text modes).
>
> This is implemented now by char-fold feature.
>

Thanks, that's a great improvement!

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 1004 bytes --]