bug#67857: 30.0.50; ispell fails on apostrophes when the dictionary includes contractions

bug#67857: 30.0.50; ispell fails on apostrophes when the dictionary includes contractions

[Bob Rogers]

   1.  "emacs -Q" (in an environment that uses hunspell by default).

   2.  In the *scratch* buffer, type "this wiouldn't work".

   3.  Type M-b three times to move point before the "w" in the second
word.

   4.  Type M-$.  The sole offered replacement is "wouldn't" but the
text it offers to replace is "wiouldn"; typing "0" to take the
replacement results in "this wouldn't't work" in the buffer.

   On my system, the value of ispell-really-hunspell is "1.6.2"; further
configuration data appear below.  The problem seems to be that the
dictionary hunspell is using contains contractions, but ispell is
expecting just [:alpha:] words.  I tried tweaking the regexps in
ispell-dictionary-alist to accept apostrophes, but got the error:

	ispell--run-on-word: Ispell and its process have different
	character maps: (* & wiouldn 1 1: wouldn't)

It seems like a job tailor-made for OTHERCHARS, but I get the same error
whether I add the apostrophe to OTHERCHARS via

	(setf (nth 3 (assoc nil ispell-dictionary-alist)) "['0-9]")

or to CASECHARS and NOT-CASECHARS in tandem, via (e.g.):

	(setf (nth 1 (assoc nil ispell-dictionary-alist)) "['[:alpha:]]")

   I do not understand how to get past this point, so either the code is
failing somehow (and I appreciate that it might be hard to glean the
right config bits for the dictionar(y|ies) in use), I am missing or
misunderstanding some part of the documentation, or the documentation is
lacking.  TIA,

					-- Bob Rogers
					   http://www.rgrjr.com/

------------------------------------------------------------------------
In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.34, cairo version 1.16.0) of 2023-12-16 built on orion
Repository revision: 059ae68087bf4498a0529d07319f5c5c2671cad7
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101004
System Description: openSUSE Leap 15.5

Configured using:
 'configure --with-dbus=no --with-gsettings=no --with-gif=ifavailable
 --with-tiff=no --with-gnutls=yes --with-gconf=no'

Configured features:
ACL CAIRO FREETYPE GIF GLIB GMP GNUTLS HARFBUZZ JPEG LIBSELINUX LIBXML2
MODULES NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS
TOOLKIT_SCROLL_BARS X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

bug#67857: 30.0.50; ispell fails on apostrophes when the dictionary includes contractions

[Eli Zaretskii]

tags 67857 notabug
thanks

> Date: Sat, 16 Dec 2023 13:03:46 -0800
> From: Bob Rogers <rogers@rgrjr.com>
> 
>    1.  "emacs -Q" (in an environment that uses hunspell by default).
> 
>    2.  In the *scratch* buffer, type "this wiouldn't work".
> 
>    3.  Type M-b three times to move point before the "w" in the second
> word.
> 
>    4.  Type M-$.  The sole offered replacement is "wouldn't" but the
> text it offers to replace is "wiouldn"; typing "0" to take the
> replacement results in "this wouldn't't work" in the buffer.

Doesn't happen for me.

>    On my system, the value of ispell-really-hunspell is "1.6.2";

I use Hunspell 1.3.2, FWIW.

> further
> configuration data appear below.  The problem seems to be that the
> dictionary hunspell is using contains contractions, but ispell is
> expecting just [:alpha:] words.  I tried tweaking the regexps in
> ispell-dictionary-alist to accept apostrophes, but got the error:
> 
> 	ispell--run-on-word: Ispell and its process have different
> 	character maps: (* & wiouldn 1 1: wouldn't)
> 
> It seems like a job tailor-made for OTHERCHARS, but I get the same error
> whether I add the apostrophe to OTHERCHARS via
> 
> 	(setf (nth 3 (assoc nil ispell-dictionary-alist)) "['0-9]")
> 
> or to CASECHARS and NOT-CASECHARS in tandem, via (e.g.):
> 
> 	(setf (nth 1 (assoc nil ispell-dictionary-alist)) "['[:alpha:]]")
> 
>    I do not understand how to get past this point, so either the code is
> failing somehow (and I appreciate that it might be hard to glean the
> right config bits for the dictionar(y|ies) in use), I am missing or
> misunderstanding some part of the documentation, or the documentation is
> lacking.  TIA,

You are looking at the wrong places to try fixing this.  First, the
default for OTHERCHARS in English and American dictionaries is "[']"
(see ispell-dictionary-base-alist), and likewise for the "default"
dictionary, so this already includes the apostrophe.  However, when
ispell.el uses Hunspell, it doesn't use this database for OTHERCHARS;
instead, it looks in the dictionary's affix file, see
ispell-parse-hunspell-affix-file.  So you should look in your affix
file and see if the apostrophe is mentioned there.  In my affix file
for English, I see this line:

  WORDCHARS 0123456789'

which includes the apostrophe.  See hunspell(4) man page for more
details about the affix file and its settings.

My affix file also has this line, which also includes the apostrophe:

  TRY esianrtolcdugmphbyfvkwzESIANRTOLCDUGMPHBYFVKWZ'-

Not sure if it's relevant, but I though I'd mention it for
completeness.

I see no bug in Emacs here.  ispell.el works as intended.

bug#67857: 30.0.50; ispell fails on apostrophes when the dictionary includes contractions

[Bob Rogers]

   From: Eli Zaretskii <eliz@gnu.org>
   Date: Sun, 17 Dec 2023 08:59:42 +0200

   > Date: Sat, 16 Dec 2023 13:03:46 -0800
   > From: Bob Rogers <rogers@rgrjr.com>
   > 
   > . . .
   > 
   >    4.  Type M-$.  The sole offered replacement is "wouldn't" but the
   > text it offers to replace is "wiouldn"; typing "0" to take the
   > replacement results in "this wouldn't't work" in the buffer.

   Doesn't happen for me . . .

   You are looking at the wrong places to try fixing this.  First, the
   default for OTHERCHARS in English and American dictionaries is "[']"
   (see ispell-dictionary-base-alist), and likewise for the "default"
   dictionary, so this already includes the apostrophe.

Yes, I see this.

   However, when ispell.el uses Hunspell, it doesn't use this database
   for OTHERCHARS; instead, it looks in the dictionary's affix file, see
   ispell-parse-hunspell-affix-file.  So you should look in your affix
   file and see if the apostrophe is mentioned there.  In my affix file
   for English, I see this line:

     WORDCHARS 0123456789'

   which includes the apostrophe.  See hunspell(4) man page for more
   details about the affix file and its settings.

Yes, this is my problem; the apostrophe was missing, and ispell DTRT
when I add it.  (I think I understand now what "different character
maps" means.)  Thanks for pointing me in the right direction.

   My affix file also has this line, which also includes the apostrophe:

     TRY esianrtolcdugmphbyfvkwzESIANRTOLCDUGMPHBYFVKWZ'-

   Not sure if it's relevant, but I though I'd mention it for
   completeness.

Mine has the apostrophe (though not the hyphen), so I left it alone.

   I see no bug in Emacs here.  ispell.el works as intended.

Agreed; this is probably an openSUSE config problem.  Unfortunately, I
can't file a bug with them because they are still shipping 27.2 (at
least in their openSUSE Leap 15.5 release) and I can't reproduce this
issue in that version.

					-- Bob

bug#67857: 30.0.50; ispell fails on apostrophes when the dictionary includes contractions

[Eli Zaretskii]

> From: Bob Rogers <rogers@rgrjr.com>
> Date: Sun, 17 Dec 2023 13:22:07 -0800
> CC: 67857@debbugs.gnu.org
> 
>    I see no bug in Emacs here.  ispell.el works as intended.
> 
> Agreed; this is probably an openSUSE config problem.  Unfortunately, I
> can't file a bug with them because they are still shipping 27.2 (at
> least in their openSUSE Leap 15.5 release) and I can't reproduce this
> issue in that version.

OK, thanks for getting back to us.  I'm therefore closing this bug.